You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
_provider_failure_kind (merged in #361) classifies capacity and auth failures, which cool the provider down and fail over to the next subscription-backed CLI. Network failures are classified as neither, so a provider whose endpoint is unreachable burns its dispatch, returns immediately, and pages the operator — instead of failing over to a provider that is actually up.
Evidence
Of 108 dispatch failures on 2026-07-31, 25 were provider network failures:
plus recurring stream disconnected before completion: error sending request for url (https://chatgpt.com/backend-api/codex/models...) from Codex.
All three return None from _provider_failure_kind, verified directly against the merged code on main. During the same window the other providers were healthy — the work could have completed.
#361 deliberately fails over only on failure classes that are safe to replay. An unclassified error might mean the task partially executed, and replaying a side-effecting task on another provider could duplicate work. Broadening classification needs its own reasoning, so it did not belong in that PR.
Proposed approach
A network failure at connection setup is safe to replay — the request never reached the model, so no side effect occurred. That is distinguishable from a mid-stream disconnect, which is not safe.
Suggested: add a network kind matching setup-time signatures only (getaddrinfo, ENOTFOUND, ECONNREFUSED, [unavailable]), treat it as retryable-with-failover, and give it a shorter cooldown than the 3600s capacity cooldown, since an outage typically resolves in minutes rather than at a plan reset boundary.
Deliberately excluded: Reconnecting... and stream disconnected, which can occur after the request was accepted. Those should keep the current fail-fast behavior until someone establishes whether the provider guarantees no side effect.
Acceptance
Setup-time network errors fail over instead of paging
Mid-stream disconnects keep current behavior, with a comment saying why
Network cooldown is shorter than capacity cooldown
Tests cover both the failover and the deliberate non-failover case
Swarm specification
This section is maintained by the Ateles swarm. Each lens agent owns exactly one subsection below; the human-written description above these markers is never modified.
Product / Scope (PM)
Those background CLI fetches finished earlier: the 49435 / offline paths were empty or down. PM work for ateles#369 already completed via localhost:3180 — pm signed off, owner waxwing, spec section + verdict comment posted.
User goal: Complete a dispatched agent task when the first subscription harness cannot connect at setup — without paging the operator and without replaying work that may already have side effects.
Persona: Operator / harness developer (CLI-comfortable; zero tolerance for silent failure or ambiguous classification).
User-facing surface (operator/developer, not GUI):
Failure kind token network (parity with capacity / auth)
Module/doc comment matrix: setup-time vs mid-stream
Interaction / flow
Dispatch selects provider → spawn CLI.
Setup-time network (getaddrinfo / ENOTFOUND / ECONNREFUSED / [unavailable] at connection setup) → classify network → short cooldown → try next eligible provider → no page while failover continues.
Success on later provider → return ok; log records prior network failover (kind named, not generic).
All providers exhausted (incl. network) → single exhaustion error listing attempted providers + last kind; then escalate/page per existing exhausted path.
Mid-stream disconnect (Reconnecting..., stream disconnected, post-accept transport errors) → leave unclassified (None) → fail-fast (current); log/comment states not safe to replay → may page.
Empty eligible set (all cooling / no binaries) → explicit empty-candidates error with actionable hint (wait for network cooldown / check DNS / restore provider), not a silent no-op.
Naming
Token
Meaning
Operator reads as
network
Setup-time connectivity; safe to replay
"endpoint unreachable before request accepted"
capacity
Plan/quota
"out of headroom"
auth
Credential
"re-auth needed"
unclassified mid-stream
Not network
"may have side effects — do not failover"
Do not invent aliases (dns, unavailable, transport) in logs — one kind token.
Network cooldown shorter than capacity; override discoverable in docs
Exhaustion / empty-candidate paths are explicit (no silent drop)
Docs or tests demonstrate both failover and deliberate non-failover examples
No new kind aliases; no operator-facing copy invented in impl without placeholders
Verdict: SIGNED_OFF — friction is classification silence + missing mid-stream explanation; both addressed in this section. Arch owns cooldown duration/env contract and signature precedence.
Engineering
Outcome
Make setup-time provider connectivity failures a first-class retryable kind (network) so Apis cools that provider briefly and fails over to the next eligible subscription CLI. Keep mid-stream disconnects unclassified (None) so the existing fail-fast / no-replay path from #361 is unchanged.
No schema, OpenAPI, MCP, entity-field, or migration changes. Single harness surface (Apis skill_runner / harness_router); cross-surface parity N/A.
Classification contract (_provider_failure_kind)
Blob = " ".join(texts).lower() (unchanged join).
Setup-time network signatures (new tuple, substring match → "network"):
getaddrinfo
enotfound
econnrefused
[unavailable] (bracketed form only; do not match bare unavailable)
Mid-stream exclusion signatures (new tuple; matched first → return None):
reconnecting
stream disconnected
Inline comment above the mid-stream tuple: these can occur after the provider accepted the request; replaying on another CLI risks duplicate side effects until a provider guarantee exists — keep fail-fast (#361).
Precedence (fixes UX ambiguous-blob P0):
Any mid-stream signature → None (even if setup-time substrings also present)
Else capacity → "capacity"
Else auth → "auth"
Else setup-time network → "network"
Else None
Return token is exactly network — no aliases (dns, transport, etc.).
Failover loop (run_skill)
Today: failure_kind in {capacity, auth} or launch failure → cool_down(selected) then try next; unclassified non-launch → return immediately.
Change:
Treat "network" like capacity/auth for failover continuation.
Call cool_down(selected, seconds=<network_seconds>) for "network"; keep default cool_down(selected) for capacity/auth/launch (3600-class).
Primary failover log line must include the kind string (existing pattern {selected} {failure_kind} failure; trying next… already does when kind is set — ensure network flows through).
On fail-fast path (failure_kind is None and not launch_failure): if mid-stream signatures match the same diagnostic blob used for classification, emit a warning that mid-stream disconnect is not classified as network and is not failing over (Paradisaea copy placeholders; kind token absent / explicit “unclassified mid-stream”).
Exhaustion SkillResult.error: keep attempt list; append last known failure_kind when set (e.g. last_failure_kind=network).
Empty-candidates path: keep explicit non-empty error (configured + cooling already); do not silently no-op. Optional hint text may mention waiting for network cooldown / DNS — no new kind aliases.
test_setup_network_fails_over — first provider fails with e.g. Error: [unavailable] getaddrinfo ENOTFOUND api2.cursor.sh (and/or ECONNREFUSED); second succeeds; attempted_providers ≥ 2.
Unit: each setup signature alone → "network".
test_midstream_disconnect_does_not_failover — Reconnecting... and stream disconnected… each: single attempt, no failover.
test_ambiguous_setup_plus_midstream_prefers_fail_fast — both signature classes in blob → None, no failover.
Router: network cooldown expires on ~300s path; capacity path still uses 3600 (or env); network env override honored.
Run focused pytest on test_skill_runner.py + test_harness_router.py. No OpenAPI/test-catalog regen unless this repo’s CI catalogs these paths (it does not today for Apis Python tests).
Author self /code-review on the diff; open PR (closes #369); impl gate → Vanellus.
Out of scope
Treating Reconnecting... / stream disconnected as retryable
Changing capacity/auth signatures or the 3600 default for those kinds
QA/Test Plan section delivered above, built additively on the PM/Design/Eng sections — covers the regression test for the ENOTFOUND/ECONNREFUSED failover bug, classifier edge cases (bracketed-[unavailable] scoping, ambiguous-blob precedence), the cooldown contract tests, and non-regression guards, closing with a definition-of-done checklist.
[phoenicurus] test_plan: QA/Test Plan section written for ateles#369 — 8 test groups covering: regression (test_setup_network_fails_over + ECONNREFUSED variant), classifier units (4 setup signatures + bare-"unavailable" negative + capacity/auth non-regression), edge cases (mid-stream exclusion, ambiguous-blob precedence P0, exhaustion last_failure_kind, empty-candidates, mixed-kind cooldown chain), harness_router cooldown contract tests (default path, explicit seconds, env override, invalid-env fallback to 300, default-ordering invariant network<capacity), and non-regression checks (launch-failure, ordinary-failure). No eval yet committed — impl (Cicada) has not landed; sign-off is on the test plan as the QA artifact per this gate's process, contingent on Cicada authoring these named tests in the PR and agentic_evals lane going green.
Security / Arch
Interface-consistency gate (per waxwing charter §Interface-consistency & agent-instruction coherence)
This change touches only internal Python control flow (skill_runner.py, harness_router.py) plus config/docs. It does not touch any Neotoma MCP tool, CLI command, HTTP endpoint, schema field, or entity relationship — the "Neotoma interface gate" (change_guardrails_rules / openapi_contract_flow / errors.md / schema_agnostic_design_rules) is N/A. No openapi.yaml, contract_mappings.ts, or MCP↔CLI parity surface is implicated. Confirmed by reading Eng's file list: single-repo, single-surface (Apis harness dispatch), no cross-surface exposure. Cross-surface parity check: N/A — network is not exposed as an MCP tool/CLI/REST capability, only an internal classification token and log string.
Contract-first ordering
N/A for OpenAPI/spec-before-handler discipline — there is no external contract being added. The only "contract" introduced is the internal function signature cool_down(provider, *, now=None, seconds=None). Require this signature be finalized (this section) before Cicada writes call sites — Eng's proposal is accepted as final:
seconds=None → env-default path (unchanged 3600 behavior for capacity/auth/launch).
Order of implementation within the PR: (1) cool_down signature extension + env read, (2) classifier signature tuples + precedence, (3) run_skill wiring, (4) docs/env header, (5) tests. This ordering is a build-sequencing note, not a gating requirement — no external consumers exist to break mid-sequence.
Tenant isolation on entity lookups
N/A. No Neotoma entity, no tenant_id/owner_id-scoped lookup, no multi-tenant data access is introduced or touched by this change. skill_runner.py and harness_router.py operate on process-local state (in-memory cooldown map, subprocess stderr/stdout text) — there is no entity retrieval path to isolate. Confirming this explicitly so the checklist isn't silently skipped: tenant isolation requirement does not apply to this issue.
Idempotency on mutating operations
N/A in the Neotoma-mutation sense — this PR does not call store/correct/any Neotoma write. However, the operational analogue — replay safety — is exactly what #361 and this issue's precedence rule already encode, and it is the load-bearing safety property here:
The network kind is only assigned to setup-time signatures (getaddrinfo, enotfound, econnrefused, [unavailable]) — i.e., failures that occur strictly before the provider CLI accepts/starts processing the request. Treat this as the idempotency boundary: setup-time failure ⇒ no side effect ⇒ safe to retry on another provider. This is the correct analogue of an idempotency key for a system with no natural request-id — do not weaken it by broadening the signature set without re-establishing the no-side-effect guarantee per provider.
The precedence rule (mid-stream signature present → always None, checked first, before capacity/auth/network) is the enforcement mechanism. Confirm Eng's ordering is implemented as a short-circuit, not a scored/weighted match — a blob containing both reconnecting and getaddrinfo (e.g., a wrapped/retried provider log) MUST classify None. This is a correctness-of-safety-property requirement, not a style preference: get this precedence wrong and the system silently starts replaying possibly-side-effecting work, which is the exact failure mode fix(apis): recover harness routing + Codex dispatch work stranded in the deploy checkout #361 was designed to prevent.
No task-level idempotency key is required for the failover retry itself, because Apis dispatch already treats "try next provider" as a new attempt against a fresh CLI process, not a resend against a stateful queue. If a future issue introduces a provider that can accept work asynchronously (webhook-style completion), that provider integration must add an explicit idempotency key at the point work is submitted — out of scope here, flagged for future arch review if/when such a provider is added.
Auth / credential-exposure constraints
Signature matching must not be broadened to match on credential material. The setup-time and mid-stream signature tuples match structural error text (getaddrinfo, ENOTFOUND, [unavailable], Reconnecting..., stream disconnected) — none of these are expected to contain API keys, tokens, or PII. Confirm in review that no signature is derived from raw provider stderr in a way that would require echoing secret-bearing substrings into log lines.
Raw provider stderr as diagnostic detail (per UX section) must go through whatever redaction the existing skill_runner logging path already applies to provider output, if any — this PR does not change the log-emission mechanism, only what's classified. If skill_runner today logs raw stderr verbatim (need Cicada/Eng to confirm at implementation time), that is a pre-existing behavior, not introduced by this issue — do not silently expand what gets logged. If raw stderr is not currently captured in the primary log line and this PR adds "attach raw stderr as diagnostic detail" per the UX section, confirm it goes to a debug-level/attachment field, not the primary grep-target line that operators and any downstream alerting parse.
No new auth surface. This issue does not touch the auth failure kind, CLI credential handling, or provider authentication flows. The existing auth-kind cooldown/failover path is unchanged; do not conflate a network-unreachable endpoint with an auth failure even if both currently page — they must remain classified separately (already true in current code, confirmed unchanged by Eng's plan).
Cooldown state is process-local and non-persistent (per existing harness_router design) — no credential or secret is stored in the cooldown map; it holds provider name → timestamp only. This PR does not change that shape, only adds a seconds parameter to the existing call. No new persistence, no new attack surface.
Env / config constraints
APIS_HARNESS_NETWORK_COOLDOWN_SECONDS (default 300) — confirm parse-and-validate follows the identical pattern already used for APIS_HARNESS_COOLDOWN_SECONDS (invalid/non-numeric → fall back to default, not crash, not fall back to 0). Eng's plan states this; holding it as a hard requirement — an invalid env value must never produce a 0s or negative cooldown, which would make network classification indistinguishable from immediate-retry-no-backoff and defeat the purpose of cooling the provider at all.
No new secrets, tokens, or credentials are introduced by either new env var. Both are plain integers governing local timing behavior only.
Reversibility
Fully reversible. Both new signature tuples and the network kind are additive and isolated to skill_runner.py/harness_router.py; reverting is a straight revert of one PR with no data migration, no schema change, no persisted state to unwind (cooldown map is in-memory and resets on process restart regardless).
Verdict checklist (security/arch)
Contract-first: internal cool_down(..., seconds=) signature finalized before call-site wiring — no external contract to sequence against
Tenant isolation: N/A — no entity lookups in this change
Idempotency: N/A for Neotoma mutations; replay-safety boundary (setup-time-only classification + mid-stream-first precedence) is the correct analogue and must be implemented as a short-circuit, not a scored match
Auth/credential exposure: no new signature touches secret-bearing text; raw stderr diagnostic attachment (if added) must not land in the primary grep-target log line; auth kind remains untouched and distinct from network
Env validation: invalid APIS_HARNESS_NETWORK_COOLDOWN_SECONDS must fall back to 300, never 0/negative
Verdict: SIGNED_OFF — no blocking security/architecture concerns. This is a single-surface, additive, in-memory-only change with no tenant, auth-surface, or persistence exposure. The one property to hold firm at implementation and review time is the mid-stream-first precedence short-circuit, since that is what preserves #361's no-replay-of-side-effecting-work guarantee.
[waxwing] schema_or_api_proposal: SIGNED_OFF — ateles#369 is a single-surface (Apis skill_runner/harness_router), in-memory-only change; no Neotoma MCP/CLI/HTTP/schema interface touched, so the interface-consistency gate is N/A. Security/arch requirements: cool_down(provider, *, now=None, seconds=None) signature accepted as final; tenant isolation N/A (no entity lookups); idempotency N/A for Neotoma writes but the setup-time-only classification + mid-stream-first short-circuit precedence is the load-bearing replay-safety boundary and must not be weakened; no credential-bearing text in any signature match; APIS_HARNESS_NETWORK_COOLDOWN_SECONDS invalid-value fallback must be 300, never 0. Fully reversible, additive-only change.
Legal
Jurisdiction (locale_profile ent_ea9a413189860f872c6cc99a): Spain (EU); regimes GDPR, ePrivacy, Spanish commercial law. Secondary: US (provider endpoint homes). No contract above escalation_currency_threshold in scope — counsel escalation not required.
Classifier matches structural error substrings only (getaddrinfo, enotfound, econnrefused, [unavailable], reconnecting, stream disconnected) — never credential- or PII-bearing patterns
Primary failover log carries kind token network only; raw provider stderr (if attached) must stay debug/attachment — must not land on the primary grep-target line (aligns Security)
No new personal-data processing purpose; cooldown map stays process-local (provider → timestamp)
Must-hold: mid-stream-first precedence short-circuit (Eng/Security). Misclassifying post-accept disconnects as network risks replaying side-effecting work → duplicate third-party actions / provider ToS exposure
Public surfaces (GitHub issue/PR, docs/swarm_orchestration.md): describe internal failover behaviour only; do not paste live session payloads, keys, or personal task content into public text
Guest-token / credential scope
No guest_access_token, AAuth grant, MCP/HTTP auth, or credential-store change
Problem
_provider_failure_kind(merged in #361) classifiescapacityandauthfailures, which cool the provider down and fail over to the next subscription-backed CLI. Network failures are classified as neither, so a provider whose endpoint is unreachable burns its dispatch, returns immediately, and pages the operator — instead of failing over to a provider that is actually up.Evidence
Of 108 dispatch failures on 2026-07-31, 25 were provider network failures:
Error: [unavailable] getaddrinfo ENOTFOUND api2.cursor.shERROR: Reconnecting... 2/5plus recurring
stream disconnected before completion: error sending request for url (https://chatgpt.com/backend-api/codex/models...)from Codex.All three return
Nonefrom_provider_failure_kind, verified directly against the merged code onmain. During the same window the other providers were healthy — the work could have completed.Why this was correct to defer in #361
#361 deliberately fails over only on failure classes that are safe to replay. An unclassified error might mean the task partially executed, and replaying a side-effecting task on another provider could duplicate work. Broadening classification needs its own reasoning, so it did not belong in that PR.
Proposed approach
A network failure at connection setup is safe to replay — the request never reached the model, so no side effect occurred. That is distinguishable from a mid-stream disconnect, which is not safe.
Suggested: add a
networkkind matching setup-time signatures only (getaddrinfo,ENOTFOUND,ECONNREFUSED,[unavailable]), treat it as retryable-with-failover, and give it a shorter cooldown than the 3600s capacity cooldown, since an outage typically resolves in minutes rather than at a plan reset boundary.Deliberately excluded:
Reconnecting...andstream disconnected, which can occur after the request was accepted. Those should keep the current fail-fast behavior until someone establishes whether the provider guarantees no side effect.Acceptance
Swarm specification
This section is maintained by the Ateles swarm. Each lens agent owns exactly one subsection below; the human-written description above these markers is never modified.
Product / Scope (PM)
Those background CLI fetches finished earlier: the 49435 / offline paths were empty or down. PM work for ateles#369 already completed via localhost:3180 — pm signed off, owner waxwing, spec section + verdict comment posted.
[pavo] acceptance_criteria: ateles#369 PM SIGNED_OFF (prior turn); background fetch shells were non-blocking recovery noise
Design / UX
User goal: Complete a dispatched agent task when the first subscription harness cannot connect at setup — without paging the operator and without replaying work that may already have side effects.
Persona: Operator / harness developer (CLI-comfortable; zero tolerance for silent failure or ambiguous classification).
User-facing surface (operator/developer, not GUI):
network(parity withcapacity/auth)SkillResult.errorstringAPIS_HARNESS_COOLDOWN_SECONDS+ shorter network-specific knob — exact contract owned by arch)Interaction / flow
getaddrinfo/ENOTFOUND/ECONNREFUSED/[unavailable]at connection setup) → classifynetwork→ short cooldown → try next eligible provider → no page while failover continues.networkfailover (kind named, not generic).Reconnecting...,stream disconnected, post-accept transport errors) → leave unclassified (None) → fail-fast (current); log/comment states not safe to replay → may page.Naming
networkcapacityauthnetworkDo not invent aliases (
dns,unavailable,transport) in logs — one kind token.Messages (structure only — copy via Paradisaea)
[COPY: skill + provider + kind=network + "trying next subscription-backed provider" + optional cooldown seconds][COPY: provider + "mid-stream disconnect not classified as network; not failing over" + pointer to signature comment][COPY: "all eligible … exhausted after attempts: …" + last failure_kind if known][COPY: no eligible providers + hint to inspect cooling providers / DNS / headroom]Raw provider stderr may be attached as diagnostic detail; the classified kind must appear in the primary log line so grepping
networkworks.Error / empty / edge
[unavailable]unclassified todaynetwork, failover, no page mid-chainfailure_kindDocs / examples the change must ship
harness_router/skill_runnerdocstring: failure-kind table includingnetwork+ cooldown defaults.test_setup_network_fails_over,test_midstream_disconnect_does_not_failover.[apis] … network failure; trying next…vs fail-fast mid-stream.Accessibility / discoverability
N/A for GUI. Discoverability = greppable kind token, env names in module docstring, tests named for the two behaviours.
Acceptance checklist (ux)
networkVerdict:
SIGNED_OFF— friction is classification silence + missing mid-stream explanation; both addressed in this section. Arch owns cooldown duration/env contract and signature precedence.Engineering
Outcome
Make setup-time provider connectivity failures a first-class retryable kind (
network) so Apis cools that provider briefly and fails over to the next eligible subscription CLI. Keep mid-stream disconnects unclassified (None) so the existing fail-fast / no-replay path from #361 is unchanged.Files / modules
execution/daemons/apis/skill_runner.py_CAPACITY_FAILURE_SIGNATURES/_AUTH_FAILURE_SIGNATURESneighbors;_provider_failure_kind;run_skillfailover loop + docstringnetwork; wire short cooldown; mid-stream fail-fast log; exhaustion includes last kindexecution/daemons/apis/harness_router.pycool_downAPIS_HARNESS_NETWORK_COOLDOWN_SECONDSexecution/daemons/apis/apis.pyAPIS_HARNESS_COOLDOWN_SECONDSdocs/swarm_orchestration.mdnetworkkind + shorter cooldown vs capacity/authexecution/daemons/apis/test_skill_runner.pytest_capacity_failure_fails_over…)execution/daemons/apis/test_harness_router.pyNo schema, OpenAPI, MCP, entity-field, or migration changes. Single harness surface (Apis
skill_runner/harness_router); cross-surface parity N/A.Classification contract (
_provider_failure_kind)Blob =
" ".join(texts).lower()(unchanged join).Setup-time network signatures (new tuple, substring match →
"network"):getaddrinfoenotfoundeconnrefused[unavailable](bracketed form only; do not match bareunavailable)Mid-stream exclusion signatures (new tuple; matched first → return
None):reconnectingstream disconnectedInline comment above the mid-stream tuple: these can occur after the provider accepted the request; replaying on another CLI risks duplicate side effects until a provider guarantee exists — keep fail-fast (#361).
Precedence (fixes UX ambiguous-blob P0):
None(even if setup-time substrings also present)"capacity""auth""network"NoneReturn token is exactly
network— no aliases (dns,transport, etc.).Failover loop (
run_skill)Today:
failure_kind in {capacity, auth}or launch failure →cool_down(selected)then try next; unclassified non-launch → return immediately.Change:
"network"like capacity/auth for failover continuation.cool_down(selected, seconds=<network_seconds>)for"network"; keep defaultcool_down(selected)for capacity/auth/launch (3600-class).{selected} {failure_kind} failure; trying next…already does when kind is set — ensurenetworkflows through).failure_kind is None and not launch_failure): if mid-stream signatures match the same diagnostic blob used for classification, emit a warning that mid-stream disconnect is not classified asnetworkand is not failing over (Paradisaea copy placeholders; kind token absent / explicit “unclassified mid-stream”).SkillResult.error: keep attempt list; append last knownfailure_kindwhen set (e.g.last_failure_kind=network).Update
run_skilldocstring: capacity / auth / network (setup-time only) / launch fail over; ordinary task failures, timeouts, and mid-stream disconnects do not.Cooldown contract (
harness_router)Extend:
seconds is None→APIS_HARNESS_COOLDOWN_SECONDS(default 3600) — capacity, auth, launch (unchanged).secondsset → use that duration (network path fromskill_runner).APIS_HARNESS_NETWORK_COOLDOWN_SECONDS, default 300. Invalid → 300. Must remain < 3600 at defaults.harness_routermodule docstring,apis.pyenv header, anddocs/swarm_orchestration.md.Layering: signatures live only in
skill_runner; router owns duration policy and never parses stderr.Data / contract changes
networkbesidecapacity/authAPIS_HARNESS_NETWORK_COOLDOWN_SECONDS(default 300)Arch may revise the 300s default in a later section; build uses 300 unless arch overrides before impl.
Build-step checklist
cool_down(..., seconds=)+ read/documentAPIS_HARNESS_NETWORK_COOLDOWN_SECONDS(default 300)._NETWORK_SETUP_SIGNATURES+_MIDSTREAM_DISCONNECT_SIGNATURES; implement precedence in_provider_failure_kind; mid-stream rationale comment.networkintorun_skillfailover + short cooldown; mid-stream fail-fast log; exhaustionlast_failure_kind.apis.pyenv header +docs/swarm_orchestration.mdquota section.test_setup_network_fails_over— first provider fails with e.g.Error: [unavailable] getaddrinfo ENOTFOUND api2.cursor.sh(and/orECONNREFUSED); second succeeds;attempted_providers≥ 2."network".test_midstream_disconnect_does_not_failover—Reconnecting...andstream disconnected…each: single attempt, no failover.test_ambiguous_setup_plus_midstream_prefers_fail_fast— both signature classes in blob →None, no failover.test_skill_runner.py+test_harness_router.py. No OpenAPI/test-catalog regen unless this repo’s CI catalogs these paths (it does not today for Apis Python tests)./code-reviewon the diff; open PR (closes #369); impl gate → Vanellus.Out of scope
Reconnecting.../stream disconnectedas retryableQA / Test Plan
🧠 Neotoma — Phoenicurus QA — ateles#369 harness failover network classification
QA/Test Plan section delivered above, built additively on the PM/Design/Eng sections — covers the regression test for the ENOTFOUND/ECONNREFUSED failover bug, classifier edge cases (bracketed-
[unavailable]scoping, ambiguous-blob precedence), the cooldown contract tests, and non-regression guards, closing with a definition-of-done checklist.[phoenicurus] test_plan: QA/Test Plan section written for ateles#369 — 8 test groups covering: regression (test_setup_network_fails_over + ECONNREFUSED variant), classifier units (4 setup signatures + bare-"unavailable" negative + capacity/auth non-regression), edge cases (mid-stream exclusion, ambiguous-blob precedence P0, exhaustion last_failure_kind, empty-candidates, mixed-kind cooldown chain), harness_router cooldown contract tests (default path, explicit seconds, env override, invalid-env fallback to 300, default-ordering invariant network<capacity), and non-regression checks (launch-failure, ordinary-failure). No eval yet committed — impl (Cicada) has not landed; sign-off is on the test plan as the QA artifact per this gate's process, contingent on Cicada authoring these named tests in the PR and agentic_evals lane going green.Security / Arch
Interface-consistency gate (per waxwing charter §Interface-consistency & agent-instruction coherence)
This change touches only internal Python control flow (
skill_runner.py,harness_router.py) plus config/docs. It does not touch any Neotoma MCP tool, CLI command, HTTP endpoint, schema field, or entity relationship — the "Neotoma interface gate" (change_guardrails_rules / openapi_contract_flow / errors.md / schema_agnostic_design_rules) is N/A. Noopenapi.yaml,contract_mappings.ts, or MCP↔CLI parity surface is implicated. Confirmed by reading Eng's file list: single-repo, single-surface (Apis harness dispatch), no cross-surface exposure. Cross-surface parity check: N/A —networkis not exposed as an MCP tool/CLI/REST capability, only an internal classification token and log string.Contract-first ordering
cool_down(provider, *, now=None, seconds=None). Require this signature be finalized (this section) before Cicada writes call sites — Eng's proposal is accepted as final:seconds=None→ env-default path (unchanged 3600 behavior for capacity/auth/launch).seconds=<int>→ explicit override (network path only).cool_downsignature extension + env read, (2) classifier signature tuples + precedence, (3)run_skillwiring, (4) docs/env header, (5) tests. This ordering is a build-sequencing note, not a gating requirement — no external consumers exist to break mid-sequence.Tenant isolation on entity lookups
N/A. No Neotoma entity, no
tenant_id/owner_id-scoped lookup, no multi-tenant data access is introduced or touched by this change.skill_runner.pyandharness_router.pyoperate on process-local state (in-memory cooldown map, subprocess stderr/stdout text) — there is no entity retrieval path to isolate. Confirming this explicitly so the checklist isn't silently skipped: tenant isolation requirement does not apply to this issue.Idempotency on mutating operations
N/A in the Neotoma-mutation sense — this PR does not call
store/correct/any Neotoma write. However, the operational analogue — replay safety — is exactly what #361 and this issue's precedence rule already encode, and it is the load-bearing safety property here:networkkind is only assigned to setup-time signatures (getaddrinfo,enotfound,econnrefused,[unavailable]) — i.e., failures that occur strictly before the provider CLI accepts/starts processing the request. Treat this as the idempotency boundary: setup-time failure ⇒ no side effect ⇒ safe to retry on another provider. This is the correct analogue of an idempotency key for a system with no natural request-id — do not weaken it by broadening the signature set without re-establishing the no-side-effect guarantee per provider.None, checked first, before capacity/auth/network) is the enforcement mechanism. Confirm Eng's ordering is implemented as a short-circuit, not a scored/weighted match — a blob containing bothreconnectingandgetaddrinfo(e.g., a wrapped/retried provider log) MUST classifyNone. This is a correctness-of-safety-property requirement, not a style preference: get this precedence wrong and the system silently starts replaying possibly-side-effecting work, which is the exact failure mode fix(apis): recover harness routing + Codex dispatch work stranded in the deploy checkout #361 was designed to prevent.Auth / credential-exposure constraints
getaddrinfo,ENOTFOUND,[unavailable],Reconnecting...,stream disconnected) — none of these are expected to contain API keys, tokens, or PII. Confirm in review that no signature is derived from raw provider stderr in a way that would require echoing secret-bearing substrings into log lines.skill_runnerlogging path already applies to provider output, if any — this PR does not change the log-emission mechanism, only what's classified. Ifskill_runnertoday logs raw stderr verbatim (need Cicada/Eng to confirm at implementation time), that is a pre-existing behavior, not introduced by this issue — do not silently expand what gets logged. If raw stderr is not currently captured in the primary log line and this PR adds "attach raw stderr as diagnostic detail" per the UX section, confirm it goes to a debug-level/attachment field, not the primary grep-target line that operators and any downstream alerting parse.authfailure kind, CLI credential handling, or provider authentication flows. The existingauth-kind cooldown/failover path is unchanged; do not conflate a network-unreachable endpoint with an auth failure even if both currently page — they must remain classified separately (already true in current code, confirmed unchanged by Eng's plan).harness_routerdesign) — no credential or secret is stored in the cooldown map; it holds provider name → timestamp only. This PR does not change that shape, only adds asecondsparameter to the existing call. No new persistence, no new attack surface.Env / config constraints
APIS_HARNESS_NETWORK_COOLDOWN_SECONDS(default 300) — confirm parse-and-validate follows the identical pattern already used forAPIS_HARNESS_COOLDOWN_SECONDS(invalid/non-numeric → fall back to default, not crash, not fall back to 0). Eng's plan states this; holding it as a hard requirement — an invalid env value must never produce a 0s or negative cooldown, which would makenetworkclassification indistinguishable from immediate-retry-no-backoff and defeat the purpose of cooling the provider at all.Reversibility
Fully reversible. Both new signature tuples and the
networkkind are additive and isolated toskill_runner.py/harness_router.py; reverting is a straight revert of one PR with no data migration, no schema change, no persisted state to unwind (cooldown map is in-memory and resets on process restart regardless).Verdict checklist (security/arch)
cool_down(..., seconds=)signature finalized before call-site wiring — no external contract to sequence againstauthkind remains untouched and distinct fromnetworkAPIS_HARNESS_NETWORK_COOLDOWN_SECONDSmust fall back to 300, never 0/negativeVerdict:
SIGNED_OFF— no blocking security/architecture concerns. This is a single-surface, additive, in-memory-only change with no tenant, auth-surface, or persistence exposure. The one property to hold firm at implementation and review time is the mid-stream-first precedence short-circuit, since that is what preserves #361's no-replay-of-side-effecting-work guarantee.[waxwing] schema_or_api_proposal: SIGNED_OFF — ateles#369 is a single-surface (Apis skill_runner/harness_router), in-memory-only change; no Neotoma MCP/CLI/HTTP/schema interface touched, so the interface-consistency gate is N/A. Security/arch requirements:
cool_down(provider, *, now=None, seconds=None)signature accepted as final; tenant isolation N/A (no entity lookups); idempotency N/A for Neotoma writes but the setup-time-only classification + mid-stream-first short-circuit precedence is the load-bearing replay-safety boundary and must not be weakened; no credential-bearing text in any signature match;APIS_HARNESS_NETWORK_COOLDOWN_SECONDSinvalid-value fallback must be 300, never 0. Fully reversible, additive-only change.Legal
Jurisdiction (locale_profile
ent_ea9a413189860f872c6cc99a): Spain (EU); regimes GDPR, ePrivacy, Spanish commercial law. Secondary: US (provider endpoint homes). No contract aboveescalation_currency_thresholdin scope — counsel escalation not required.Scope: Internal Apis harness classifier + cooldown only. No customer contract, marketing copy, privacy policy, DPA, guest API, or schema surface.
Dependencies / licensing
skill_runner,harness_router,apis.py, docs, tests)product_profileent_f79f82c13b90a9d6439623db) — no LICENSE/attribution changePII / data-handling (GDPR Art. 5 minimization; public-effect surfaces)
getaddrinfo,enotfound,econnrefused,[unavailable],reconnecting,stream disconnected) — never credential- or PII-bearing patternsnetworkonly; raw provider stderr (if attached) must stay debug/attachment — must not land on the primary grep-target line (aligns Security)networkrisks replaying side-effecting work → duplicate third-party actions / provider ToS exposuredocs/swarm_orchestration.md): describe internal failover behaviour only; do not paste live session payloads, keys, or personal task content into public textGuest-token / credential scope
guest_access_token, AAuth grant, MCP/HTTP auth, or credential-store changeauthkind distinct fromnetwork— unreachable endpoint ≠ credential failureAPIS_HARNESS_NETWORK_COOLDOWN_SECONDSis a non-secret integer; invalid → 300 (never 0/negative)ToS / legal exposure
Reconnecting.../stream disconnectedas retryable without a provider no-side-effect guaranteeChecklist verdict:
APPROVE— no [BLOCKING] legal items. Issue workflowgate_status.legalremainsnot_required.