You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Residual risks the PR #395 review marked informational — filed so they are tracked rather than growing that diff. All in scripts/skill_evolution.py.
Non-string host patches crash after full fleet spend. A host error_analyst_fn returning a truthy non-string (dict, response object) crashes collect_patches at the quality gate (patch.strip() → AttributeError) outside the per-future exception handler — after every analyst has already run. Suggested fix: type-check each future result at collection (warn + drop non-strings), defending the documented "patch text or None" contract.
No timeout on analyst futures. A blocking host error_analyst_fn hangs collect_patches indefinitely (as_completed has no timeout; the built-in path only implicitly relies on client network timeouts). Suggested fix: an optional analyst_timeout_s applied per future, timed-out analysts degraded to warnings like other host failures.
Patch provenance is not auditable. Run artifacts never record whether a patch came from the host analyst or the built-in prompt — *_patches.json/selection.txt can't show error_analyst_fn was even configured. Suggested fix: a source field per patch record (builtin | host), written by _write_evolution_artifacts.
Correction-as-hypothesis counterweight is builtin-only. The prompt rule added in skill_evolution: host integration hooks (analyst fn, incumbent score, richer trajectories) #395 protects the built-in analyst; a host analyst receives raw sessions where correction_boundaries labels user assertions as correct_fact with no such rule. Suggested fix: document the hazard in the error_analyst_fn docstring (host analysts should treat user corrections as hypotheses) — the engine cannot enforce it.
No stability signal on the growing host contract.evolve_skill is now 17 keyword params over an implicit session-dict schema in a scripts/ module with no __all__/version marker, while positioned as an adopt-don't-fork engine. Suggested fix: a short "host contract" section in the module docstring + __all__, and a CHANGELOG note when the session schema changes.
Refs: PR #395 (host integration hooks), the 8-lens review comment there, and #384/#361 for the surrounding U6 work.
Residual risks the PR #395 review marked informational — filed so they are tracked rather than growing that diff. All in
scripts/skill_evolution.py.error_analyst_fnreturning a truthy non-string (dict, response object) crashescollect_patchesat the quality gate (patch.strip()→AttributeError) outside the per-future exception handler — after every analyst has already run. Suggested fix: type-check each future result at collection (warn + drop non-strings), defending the documented "patch text or None" contract.error_analyst_fnhangscollect_patchesindefinitely (as_completedhas no timeout; the built-in path only implicitly relies on client network timeouts). Suggested fix: an optionalanalyst_timeout_sapplied per future, timed-out analysts degraded to warnings like other host failures.*_patches.json/selection.txtcan't showerror_analyst_fnwas even configured. Suggested fix: asourcefield per patch record (builtin|host), written by_write_evolution_artifacts.correction_boundarieslabels user assertions ascorrect_factwith no such rule. Suggested fix: document the hazard in theerror_analyst_fndocstring (host analysts should treat user corrections as hypotheses) — the engine cannot enforce it.evolve_skillis now 17 keyword params over an implicit session-dict schema in ascripts/module with no__all__/version marker, while positioned as an adopt-don't-fork engine. Suggested fix: a short "host contract" section in the module docstring +__all__, and a CHANGELOG note when the session schema changes.Refs: PR #395 (host integration hooks), the 8-lens review comment there, and #384/#361 for the surrounding U6 work.