feat(bin): count task attempts against a durable retry budget (CFVC-13) - #63
Conversation
2fe01dd to
af60d09
Compare
Nothing counted attempts before this: no state field, no metadata field, and brief rule 5's "if you hit the same obstacle twice, stop" made the worker the arbiter of its own retry budget, evaluated from a context that resets on every relaunch. "Should this be retried?" is now arithmetic over a number on disk. bin/fm-attempt.sh owns state/<id>.attempt (attempt=, attempt_budget=, terminal=). Every ship or scout spawn checks the budget before it creates anything and commits the increment when it publishes task metadata, which also carries the count as attempt=/attempt_budget=. An absent field reads as attempt 1, so a task dispatched before this retries at 2 rather than restarting its budget. Secondmates are exempt: their relaunch is unattended liveness recovery, not a retry. Exhaustion is a named stop, not a silent one. It records the terminal state and declares the failure on the task's own status log. Teardown retires the count on an ordinary release, which is only reachable once the work landed, and keeps it under --force, so discarding between attempts cannot make the budget unbounded. Compatibility with the two delivered dependencies, neither of which is on the trunk yet, and neither of whose producers this duplicates: - CFVC-11 (fm/cfvc-11-terminal-vocabulary, loopspecs/terminal-states.json): exhaustion terminates in that unified vocabulary's budget_exhausted rather than a third name. The regression asserts membership whenever that file is present and says plainly when it is not. - CFVC-12 (fm/cfvc-12-task-outcome-failure): the status declaration uses the failed: verb that its terminal-outcome derivation reads, so exhaustion books outcome=failed through that owner instead of a second ledger producer. The three-member v1 outcome enum is untouched. Brief rule 5's prose is deleted in the same change and the rule list renumbered, with its numbered cross-reference updated.
… document fm-attempt.sh
…TS.md state inventory
…ta parity check The projection parity case spawns ONE task id twice - opted out, then projected - and byte-compares the two metadata files after normalizing the fields that legitimately differ per run. The durable attempt counter adds another such field: the second spawn of that id is attempt 2 by construction, so the comparison failed on a real per-run fact rather than on a projection difference. Only attempt= is normalized. attempt_budget= stays compared, because the budget is a property of the task rather than of the run, and projection must not change it - normalizing both would have retired real coverage to silence one line. Caught by fork CI, not locally: this suite is real-herdr-gated and the task brief carries no Herdr lab guard, so CI is its verification path here.
CI proved the increment semantics wrong. The Herdr suite drives a same-identity reclaim: a task whose session dies leaves an agent-free husk, and fm-spawn is re-run with the same id to replace the dead endpoint while the worktree, branch and work are preserved. Counting every launch charged that recovery against the retry budget, so a task could not be brought back after two session restarts - "fm-hibit-resume-r1 has spent its retry budget: 2 of 2 attempts used". That is the same harm already avoided for secondmates and not applied to crewmates. The contract now reads the spec literally: an attempt increments ONLY when the prior attempt has a recorded FAILED terminal outcome. A spawn following no recorded failure - dead runtime, husk, freeze, any recovery reclaim - is a CONTINUATION: the count persists, does not move, and is never refused. The outcome record draws the line, not the spawn event, which is exactly why this increment depends on the terminal outcome that can say the prior attempt failed. A wedged worker is no exception; relaunching one continues unless the wedge was recorded as a failure first. The failure signal is the `failed:` verb on the task's own status log - the same declaration the ledger derives outcome=failed from, so this reads CFVC-12's evidence rather than a private second signal. failures= records how many were seen when the count last moved, so only a failure newer than the last counted attempt spends the next one. This script's own budget-exhaustion declaration is excluded from that tally, or one refusal would manufacture its own successor. A refusal preserves the tally it found: it opens nothing, so it must not consume the pending failure. A forced teardown needs its own record. The discard ends the attempt AND deletes the status log carrying the declaration, so teardown --force now calls `fm-attempt.sh end`, which marks the attempt ended (ended=1). Without it a re-dispatch after a discard would read as a continuation and discarding between attempts would make the budget unbounded - the hole keeping the record closed. Witnessed red first, as instructed: tests/fm-backend-herdr-presentation-e2e.sh reproduces the exact CI refusal against the pre-correction tree and passes after, 23 cases, with the default-session tripwire intact. The suite drives Herdr only through the guarded lab helper, which refuses the default session by construction.
Trunk began requiring --reason-code on every ship and scout spawn after this branch was cut, so the attempt suite's six ship spawns were refused before they could publish a count. Each now passes NL_RULE_CLASSIFICATION, matching the other spawn-driving suites. The secondmate case is deliberately left alone: the reason code is refused on --secondmate, which provisions a standing home rather than dispatching a task.
af60d09 to
3594339
Compare
Rebased onto the current fork trunkRebased onto Conflicts and how they were resolvedSix conflicts across two of the five commits. Every one was two independent additions landing at the same point, and every one resolved as a union:
One integration fix the rebase exposedTrunk began requiring Test results, disclosed in full
The reporting was not trusted on absence: the runner produced real |
…3) (#63) * feat(bin): count task attempts against a durable retry budget Nothing counted attempts before this: no state field, no metadata field, and brief rule 5's "if you hit the same obstacle twice, stop" made the worker the arbiter of its own retry budget, evaluated from a context that resets on every relaunch. "Should this be retried?" is now arithmetic over a number on disk. bin/fm-attempt.sh owns state/<id>.attempt (attempt=, attempt_budget=, terminal=). Every ship or scout spawn checks the budget before it creates anything and commits the increment when it publishes task metadata, which also carries the count as attempt=/attempt_budget=. An absent field reads as attempt 1, so a task dispatched before this retries at 2 rather than restarting its budget. Secondmates are exempt: their relaunch is unattended liveness recovery, not a retry. Exhaustion is a named stop, not a silent one. It records the terminal state and declares the failure on the task's own status log. Teardown retires the count on an ordinary release, which is only reachable once the work landed, and keeps it under --force, so discarding between attempts cannot make the budget unbounded. Compatibility with the two delivered dependencies, neither of which is on the trunk yet, and neither of whose producers this duplicates: - CFVC-11 (fm/cfvc-11-terminal-vocabulary, loopspecs/terminal-states.json): exhaustion terminates in that unified vocabulary's budget_exhausted rather than a third name. The regression asserts membership whenever that file is present and says plainly when it is not. - CFVC-12 (fm/cfvc-12-task-outcome-failure): the status declaration uses the failed: verb that its terminal-outcome derivation reads, so exhaustion books outcome=failed through that owner instead of a second ledger producer. The three-member v1 outcome enum is untouched. Brief rule 5's prose is deleted in the same change and the rule list renumbered, with its numbered cross-reference updated. * no-mistakes(review): validate default budget env, fix retire premise, document fm-attempt.sh * no-mistakes(document): document tooling status-line producers in AGENTS.md state inventory * test(herdr): normalize the per-run attempt count in the projection meta parity check The projection parity case spawns ONE task id twice - opted out, then projected - and byte-compares the two metadata files after normalizing the fields that legitimately differ per run. The durable attempt counter adds another such field: the second spawn of that id is attempt 2 by construction, so the comparison failed on a real per-run fact rather than on a projection difference. Only attempt= is normalized. attempt_budget= stays compared, because the budget is a property of the task rather than of the run, and projection must not change it - normalizing both would have retired real coverage to silence one line. Caught by fork CI, not locally: this suite is real-herdr-gated and the task brief carries no Herdr lab guard, so CI is its verification path here. * fix(bin): spend an attempt on a recorded failure, not on a launch CI proved the increment semantics wrong. The Herdr suite drives a same-identity reclaim: a task whose session dies leaves an agent-free husk, and fm-spawn is re-run with the same id to replace the dead endpoint while the worktree, branch and work are preserved. Counting every launch charged that recovery against the retry budget, so a task could not be brought back after two session restarts - "fm-hibit-resume-r1 has spent its retry budget: 2 of 2 attempts used". That is the same harm already avoided for secondmates and not applied to crewmates. The contract now reads the spec literally: an attempt increments ONLY when the prior attempt has a recorded FAILED terminal outcome. A spawn following no recorded failure - dead runtime, husk, freeze, any recovery reclaim - is a CONTINUATION: the count persists, does not move, and is never refused. The outcome record draws the line, not the spawn event, which is exactly why this increment depends on the terminal outcome that can say the prior attempt failed. A wedged worker is no exception; relaunching one continues unless the wedge was recorded as a failure first. The failure signal is the `failed:` verb on the task's own status log - the same declaration the ledger derives outcome=failed from, so this reads CFVC-12's evidence rather than a private second signal. failures= records how many were seen when the count last moved, so only a failure newer than the last counted attempt spends the next one. This script's own budget-exhaustion declaration is excluded from that tally, or one refusal would manufacture its own successor. A refusal preserves the tally it found: it opens nothing, so it must not consume the pending failure. A forced teardown needs its own record. The discard ends the attempt AND deletes the status log carrying the declaration, so teardown --force now calls `fm-attempt.sh end`, which marks the attempt ended (ended=1). Without it a re-dispatch after a discard would read as a continuation and discarding between attempts would make the budget unbounded - the hole keeping the record closed. Witnessed red first, as instructed: tests/fm-backend-herdr-presentation-e2e.sh reproduces the exact CI refusal against the pre-correction tree and passes after, 23 cases, with the default-session tripwire intact. The suite drives Herdr only through the guarded lab helper, which refuses the default session by construction. * test(attempt): give the spawn cases the reason code trunk now requires Trunk began requiring --reason-code on every ship and scout spawn after this branch was cut, so the attempt suite's six ship spawns were refused before they could publish a count. Each now passes NL_RULE_CLASSIFICATION, matching the other spawn-driving suites. The secondmate case is deliberately left alone: the reason code is refused on --secondmate, which provisions a standing home rather than dispatching a task.
…3) (#63) * feat(bin): count task attempts against a durable retry budget Nothing counted attempts before this: no state field, no metadata field, and brief rule 5's "if you hit the same obstacle twice, stop" made the worker the arbiter of its own retry budget, evaluated from a context that resets on every relaunch. "Should this be retried?" is now arithmetic over a number on disk. bin/fm-attempt.sh owns state/<id>.attempt (attempt=, attempt_budget=, terminal=). Every ship or scout spawn checks the budget before it creates anything and commits the increment when it publishes task metadata, which also carries the count as attempt=/attempt_budget=. An absent field reads as attempt 1, so a task dispatched before this retries at 2 rather than restarting its budget. Secondmates are exempt: their relaunch is unattended liveness recovery, not a retry. Exhaustion is a named stop, not a silent one. It records the terminal state and declares the failure on the task's own status log. Teardown retires the count on an ordinary release, which is only reachable once the work landed, and keeps it under --force, so discarding between attempts cannot make the budget unbounded. Compatibility with the two delivered dependencies, neither of which is on the trunk yet, and neither of whose producers this duplicates: - CFVC-11 (fm/cfvc-11-terminal-vocabulary, loopspecs/terminal-states.json): exhaustion terminates in that unified vocabulary's budget_exhausted rather than a third name. The regression asserts membership whenever that file is present and says plainly when it is not. - CFVC-12 (fm/cfvc-12-task-outcome-failure): the status declaration uses the failed: verb that its terminal-outcome derivation reads, so exhaustion books outcome=failed through that owner instead of a second ledger producer. The three-member v1 outcome enum is untouched. Brief rule 5's prose is deleted in the same change and the rule list renumbered, with its numbered cross-reference updated. * no-mistakes(review): validate default budget env, fix retire premise, document fm-attempt.sh * no-mistakes(document): document tooling status-line producers in AGENTS.md state inventory * test(herdr): normalize the per-run attempt count in the projection meta parity check The projection parity case spawns ONE task id twice - opted out, then projected - and byte-compares the two metadata files after normalizing the fields that legitimately differ per run. The durable attempt counter adds another such field: the second spawn of that id is attempt 2 by construction, so the comparison failed on a real per-run fact rather than on a projection difference. Only attempt= is normalized. attempt_budget= stays compared, because the budget is a property of the task rather than of the run, and projection must not change it - normalizing both would have retired real coverage to silence one line. Caught by fork CI, not locally: this suite is real-herdr-gated and the task brief carries no Herdr lab guard, so CI is its verification path here. * fix(bin): spend an attempt on a recorded failure, not on a launch CI proved the increment semantics wrong. The Herdr suite drives a same-identity reclaim: a task whose session dies leaves an agent-free husk, and fm-spawn is re-run with the same id to replace the dead endpoint while the worktree, branch and work are preserved. Counting every launch charged that recovery against the retry budget, so a task could not be brought back after two session restarts - "fm-hibit-resume-r1 has spent its retry budget: 2 of 2 attempts used". That is the same harm already avoided for secondmates and not applied to crewmates. The contract now reads the spec literally: an attempt increments ONLY when the prior attempt has a recorded FAILED terminal outcome. A spawn following no recorded failure - dead runtime, husk, freeze, any recovery reclaim - is a CONTINUATION: the count persists, does not move, and is never refused. The outcome record draws the line, not the spawn event, which is exactly why this increment depends on the terminal outcome that can say the prior attempt failed. A wedged worker is no exception; relaunching one continues unless the wedge was recorded as a failure first. The failure signal is the `failed:` verb on the task's own status log - the same declaration the ledger derives outcome=failed from, so this reads CFVC-12's evidence rather than a private second signal. failures= records how many were seen when the count last moved, so only a failure newer than the last counted attempt spends the next one. This script's own budget-exhaustion declaration is excluded from that tally, or one refusal would manufacture its own successor. A refusal preserves the tally it found: it opens nothing, so it must not consume the pending failure. A forced teardown needs its own record. The discard ends the attempt AND deletes the status log carrying the declaration, so teardown --force now calls `fm-attempt.sh end`, which marks the attempt ended (ended=1). Without it a re-dispatch after a discard would read as a continuation and discarding between attempts would make the budget unbounded - the hole keeping the record closed. Witnessed red first, as instructed: tests/fm-backend-herdr-presentation-e2e.sh reproduces the exact CI refusal against the pre-correction tree and passes after, 23 cases, with the default-session tripwire intact. The suite drives Herdr only through the guarded lab helper, which refuses the default session by construction. * test(attempt): give the spawn cases the reason code trunk now requires Trunk began requiring --reason-code on every ship and scout spawn after this branch was cut, so the attempt suite's six ship spawns were refused before they could publish a count. Each now passes NL_RULE_CLASSIFICATION, matching the other spawn-driving suites. The secondmate case is deliberately left alone: the reason code is refused on --secondmate, which provisions a standing home rather than dispatching a task.
What this does
CFVC-13: "should this be retried?" becomes arithmetic over a durable count instead of a model judgment.
Nothing counted attempts before this - no state field, no meta field - and generated brief rule 5 ("if you hit the same obstacle twice... stop") made the worker the arbiter of its own retry budget, evaluated from a context that resets on every relaunch.
bin/fm-attempt.sh(new) ownsstate/<id>.attempt(attempt=,attempt_budget=,terminal=) and the whole contract: record format, migration rule, refusal, exhaustion. Verbs:show,check,open,retire.bin/fm-spawn.shchecks the budget before it creates anything (a refused retry allocates no worktree and no endpoint) and commits the increment when it publishes task metadata (a spawn that never launched costs nothing). The count is published asattempt=/attempt_budget=;--attempt-budget <n>raises it deliberately and the raised value is recorded.bin/fm-teardown.shretires the count on an ordinary release and keeps it under--force, so discarding between attempts cannot make the budget unbounded.bin/fm-brief.shdeletes rule 5's prose, renumbers the rules, and updates the one numbered cross-reference.An absent
attempt=reads as attempt 1, so a task dispatched before this retries at 2 rather than restarting its budget. Secondmates are exempt: their relaunch is unattended liveness recovery, not a retry. There is deliberately no reset verb - raising the budget is the only way past an exhausted count, and it is recorded.Dependency compatibility
Both dependencies are delivered on branches and are not on the trunk yet. Neither producer is duplicated.
fm/cfvc-11-terminal-vocabulary,loopspecs/terminal-states.json): exhaustion terminates in that unified vocabulary'sbudget_exhaustedrather than a third name. The regression asserts membership whenever that file is present and says plainly when it is not - a silent skip would read as a pass - with a discriminating negative control.fm/cfvc-12-task-outcome-failure): exhaustion appends thefailed:verb its terminal-outcome derivation reads, so it booksoutcome=failedthrough that owner instead of a second ledger producer. The three-member v1 outcome enum is untouched.Check state - please read before merging
The pipeline reported
checks-passedon an empty check rollup. Its CI step loggedno CI checks reported - still monitoring until merged or closed. No checks examined this head, so it is unverified, not green. Treat it accordingly.Why this PR exists alongside upstream kunchenguid#1979-series PR 2010
The pipeline pushed the branch to this fork but opened its pull request at the upstream venue: kunchenguid#2010.
That upstream PR is incomplete. Rebasing onto the upstream trunk dropped part of the fix content the pipeline itself had produced for the accepted review finding: the exact-premise correction in
bin/fm-attempt.sh's header,bin/fm-teardown.sh's retire rationale, anddocs/architecture.md. Those hunks are present here and absent there.This PR is the complete change, based on the fork trunk, which is the deliberate base for this lane.
Evidence
tests/fm-attempt.test.shsuite against a tree withbin/fm-attempt.shremoved; the spawn case against the unmodifiedbin/fm-spawn.sh; the teardown retire case against the unmodifiedbin/fm-teardown.sh; the brief case against the unmodifiedbin/fm-brief.sh.tests/fm-attempt.test.sh,tests/fm-brief.test.sh,tests/fm-teardown.test.sh,bin/fm-lint.sh,bin/fm-doc-audience-check.sh,bin/fm-test-run.sh --check-coverage.tests/fm-session-start.test.shandtests/fm-launch.test.shfail identically on the untouched base in this environment, so they are pre-existing and unrelated.