Skip to content

Harness failover: classify provider network failures as retryable #369

Description

@markmhendrickson

Problem

_provider_failure_kind (merged in #361) classifies capacity and auth failures, which cool the provider down and fail over to the next subscription-backed CLI. Network failures are classified as neither, so a provider whose endpoint is unreachable burns its dispatch, returns immediately, and pages the operator — instead of failing over to a provider that is actually up.

Evidence

Of 108 dispatch failures on 2026-07-31, 25 were provider network failures:

  • 13× Error: [unavailable] getaddrinfo ENOTFOUND api2.cursor.sh
  • 12× ERROR: Reconnecting... 2/5

plus recurring stream disconnected before completion: error sending request for url (https://chatgpt.com/backend-api/codex/models...) from Codex.

All three return None from _provider_failure_kind, verified directly against the merged code on main. During the same window the other providers were healthy — the work could have completed.

Why this was correct to defer in #361

#361 deliberately fails over only on failure classes that are safe to replay. An unclassified error might mean the task partially executed, and replaying a side-effecting task on another provider could duplicate work. Broadening classification needs its own reasoning, so it did not belong in that PR.

Proposed approach

A network failure at connection setup is safe to replay — the request never reached the model, so no side effect occurred. That is distinguishable from a mid-stream disconnect, which is not safe.

Suggested: add a network kind matching setup-time signatures only (getaddrinfo, ENOTFOUND, ECONNREFUSED, [unavailable]), treat it as retryable-with-failover, and give it a shorter cooldown than the 3600s capacity cooldown, since an outage typically resolves in minutes rather than at a plan reset boundary.

Deliberately excluded: Reconnecting... and stream disconnected, which can occur after the request was accepted. Those should keep the current fail-fast behavior until someone establishes whether the provider guarantees no side effect.

Acceptance

  • Setup-time network errors fail over instead of paging
  • Mid-stream disconnects keep current behavior, with a comment saying why
  • Network cooldown is shorter than capacity cooldown
  • Tests cover both the failover and the deliberate non-failover case

Swarm specification

This section is maintained by the Ateles swarm. Each lens agent owns exactly one subsection below; the human-written description above these markers is never modified.

Product / Scope (PM)

Those background CLI fetches finished earlier: the 49435 / offline paths were empty or down. PM work for ateles#369 already completed via localhost:3180 — pm signed off, owner waxwing, spec section + verdict comment posted.

[pavo] acceptance_criteria: ateles#369 PM SIGNED_OFF (prior turn); background fetch shells were non-blocking recovery noise

Design / UX

User goal: Complete a dispatched agent task when the first subscription harness cannot connect at setup — without paging the operator and without replaying work that may already have side effects.

Persona: Operator / harness developer (CLI-comfortable; zero tolerance for silent failure or ambiguous classification).

User-facing surface (operator/developer, not GUI):

  • Failure kind token network (parity with capacity / auth)
  • Apis dispatch logs on failover / fail-fast
  • Exhaustion SkillResult.error string
  • Cooldown config discoverability (APIS_HARNESS_COOLDOWN_SECONDS + shorter network-specific knob — exact contract owned by arch)
  • Module/doc comment matrix: setup-time vs mid-stream

Interaction / flow

  1. Dispatch selects provider → spawn CLI.
  2. Setup-time network (getaddrinfo / ENOTFOUND / ECONNREFUSED / [unavailable] at connection setup) → classify network → short cooldown → try next eligible provider → no page while failover continues.
  3. Success on later provider → return ok; log records prior network failover (kind named, not generic).
  4. All providers exhausted (incl. network) → single exhaustion error listing attempted providers + last kind; then escalate/page per existing exhausted path.
  5. Mid-stream disconnect (Reconnecting..., stream disconnected, post-accept transport errors) → leave unclassified (None) → fail-fast (current); log/comment states not safe to replay → may page.
  6. Empty eligible set (all cooling / no binaries) → explicit empty-candidates error with actionable hint (wait for network cooldown / check DNS / restore provider), not a silent no-op.

Naming

Token Meaning Operator reads as
network Setup-time connectivity; safe to replay "endpoint unreachable before request accepted"
capacity Plan/quota "out of headroom"
auth Credential "re-auth needed"
unclassified mid-stream Not network "may have side effects — do not failover"

Do not invent aliases (dns, unavailable, transport) in logs — one kind token.

Messages (structure only — copy via Paradisaea)

  • Failover warning: [COPY: skill + provider + kind=network + "trying next subscription-backed provider" + optional cooldown seconds]
  • Fail-fast mid-stream: [COPY: provider + "mid-stream disconnect not classified as network; not failing over" + pointer to signature comment]
  • Exhaustion: [COPY: "all eligible … exhausted after attempts: …" + last failure_kind if known]
  • Empty candidates: [COPY: no eligible providers + hint to inspect cooling providers / DNS / headroom]

Raw provider stderr may be attached as diagnostic detail; the classified kind must appear in the primary log line so grepping network works.

Error / empty / edge

State Severity Required behaviour
Setup ENOTFOUND / [unavailable] unclassified today P0 (fixed by this issue) Classify network, failover, no page mid-chain
Mid-stream still pages P1 (accepted) Explicit non-classification + comment why
Log omits failure_kind P1 Always include kind string on failover path
Network cooldown == capacity 3600s P1 Shorter network cooldown; document override env
Docs missing setup vs mid-stream P1 Ship matrix + 2 examples (failover + deliberate non-failover)
Ambiguous blob matching both setup + mid-stream P0 if misclassified Prefer fail-fast when mid-stream signature present (Waxwing to confirm precedence)

Docs / examples the change must ship

  1. harness_router / skill_runner docstring: failure-kind table including network + cooldown defaults.
  2. Inline comment above mid-stream exclusions: why not retryable until provider guarantees no side effect.
  3. Test names as living examples: test_setup_network_fails_over, test_midstream_disconnect_does_not_failover.
  4. One operator-facing note (runbook or module doc): how to read [apis] … network failure; trying next… vs fail-fast mid-stream.

Accessibility / discoverability

N/A for GUI. Discoverability = greppable kind token, env names in module docstring, tests named for the two behaviours.

Acceptance checklist (ux)

  • Setup-time network errors fail over; primary log line includes network
  • Mid-stream signatures remain unclassified; comment explains why
  • Network cooldown shorter than capacity; override discoverable in docs
  • Exhaustion / empty-candidate paths are explicit (no silent drop)
  • Docs or tests demonstrate both failover and deliberate non-failover examples
  • No new kind aliases; no operator-facing copy invented in impl without placeholders

Verdict: SIGNED_OFF — friction is classification silence + missing mid-stream explanation; both addressed in this section. Arch owns cooldown duration/env contract and signature precedence.

Engineering

Outcome

Make setup-time provider connectivity failures a first-class retryable kind (network) so Apis cools that provider briefly and fails over to the next eligible subscription CLI. Keep mid-stream disconnects unclassified (None) so the existing fail-fast / no-replay path from #361 is unchanged.

Files / modules

Path Region Change
execution/daemons/apis/skill_runner.py _CAPACITY_FAILURE_SIGNATURES / _AUTH_FAILURE_SIGNATURES neighbors; _provider_failure_kind; run_skill failover loop + docstring Add setup-time network + mid-stream exclusion signatures; classify network; wire short cooldown; mid-stream fail-fast log; exhaustion includes last kind
execution/daemons/apis/harness_router.py module docstring; cool_down Accept explicit duration; document APIS_HARNESS_NETWORK_COOLDOWN_SECONDS
execution/daemons/apis/apis.py Environment variables header (~L42–43) Document network cooldown env next to APIS_HARNESS_COOLDOWN_SECONDS
docs/swarm_orchestration.md “Quota-aware execution in Apis” State network kind + shorter cooldown vs capacity/auth
execution/daemons/apis/test_skill_runner.py harness failover tests (~test_capacity_failure_fails_over…) Effect tests for failover + deliberate non-failover + precedence
execution/daemons/apis/test_harness_router.py cooldown tests Assert network duration < capacity default; env override

No schema, OpenAPI, MCP, entity-field, or migration changes. Single harness surface (Apis skill_runner / harness_router); cross-surface parity N/A.

Classification contract (_provider_failure_kind)

Blob = " ".join(texts).lower() (unchanged join).

Setup-time network signatures (new tuple, substring match → "network"):

  • getaddrinfo
  • enotfound
  • econnrefused
  • [unavailable] (bracketed form only; do not match bare unavailable)

Mid-stream exclusion signatures (new tuple; matched first → return None):

  • reconnecting
  • stream disconnected

Inline comment above the mid-stream tuple: these can occur after the provider accepted the request; replaying on another CLI risks duplicate side effects until a provider guarantee exists — keep fail-fast (#361).

Precedence (fixes UX ambiguous-blob P0):

  1. Any mid-stream signature → None (even if setup-time substrings also present)
  2. Else capacity → "capacity"
  3. Else auth → "auth"
  4. Else setup-time network → "network"
  5. Else None

Return token is exactly network — no aliases (dns, transport, etc.).

Failover loop (run_skill)

Today: failure_kind in {capacity, auth} or launch failure → cool_down(selected) then try next; unclassified non-launch → return immediately.

Change:

  1. Treat "network" like capacity/auth for failover continuation.
  2. Call cool_down(selected, seconds=<network_seconds>) for "network"; keep default cool_down(selected) for capacity/auth/launch (3600-class).
  3. Primary failover log line must include the kind string (existing pattern {selected} {failure_kind} failure; trying next… already does when kind is set — ensure network flows through).
  4. On fail-fast path (failure_kind is None and not launch_failure): if mid-stream signatures match the same diagnostic blob used for classification, emit a warning that mid-stream disconnect is not classified as network and is not failing over (Paradisaea copy placeholders; kind token absent / explicit “unclassified mid-stream”).
  5. Exhaustion SkillResult.error: keep attempt list; append last known failure_kind when set (e.g. last_failure_kind=network).
  6. Empty-candidates path: keep explicit non-empty error (configured + cooling already); do not silently no-op. Optional hint text may mention waiting for network cooldown / DNS — no new kind aliases.

Update run_skill docstring: capacity / auth / network (setup-time only) / launch fail over; ordinary task failures, timeouts, and mid-stream disconnects do not.

Cooldown contract (harness_router)

Extend:

cool_down(provider, *, now=None, seconds=None) -> None
  • seconds is NoneAPIS_HARNESS_COOLDOWN_SECONDS (default 3600) — capacity, auth, launch (unchanged).
  • seconds set → use that duration (network path from skill_runner).
  • Network duration source: APIS_HARNESS_NETWORK_COOLDOWN_SECONDS, default 300. Invalid → 300. Must remain < 3600 at defaults.
  • Document both envs in harness_router module docstring, apis.py env header, and docs/swarm_orchestration.md.

Layering: signatures live only in skill_runner; router owns duration policy and never parses stderr.

Data / contract changes

Surface Change
Failure kind token Add network beside capacity / auth
Env Add APIS_HARNESS_NETWORK_COOLDOWN_SECONDS (default 300)
Logs / errors Kind on failover line; last kind on exhaustion; mid-stream fail-fast warning
Persistence / APIs None

Arch may revise the 300s default in a later section; build uses 300 unless arch overrides before impl.

Build-step checklist

  1. Extend cool_down(..., seconds=) + read/document APIS_HARNESS_NETWORK_COOLDOWN_SECONDS (default 300).
  2. Add _NETWORK_SETUP_SIGNATURES + _MIDSTREAM_DISCONNECT_SIGNATURES; implement precedence in _provider_failure_kind; mid-stream rationale comment.
  3. Wire network into run_skill failover + short cooldown; mid-stream fail-fast log; exhaustion last_failure_kind.
  4. Update apis.py env header + docs/swarm_orchestration.md quota section.
  5. Tests (effect-level, not classifier-only):
    • test_setup_network_fails_over — first provider fails with e.g. Error: [unavailable] getaddrinfo ENOTFOUND api2.cursor.sh (and/or ECONNREFUSED); second succeeds; attempted_providers ≥ 2.
    • Unit: each setup signature alone → "network".
    • test_midstream_disconnect_does_not_failoverReconnecting... and stream disconnected… each: single attempt, no failover.
    • test_ambiguous_setup_plus_midstream_prefers_fail_fast — both signature classes in blob → None, no failover.
    • Router: network cooldown expires on ~300s path; capacity path still uses 3600 (or env); network env override honored.
  6. Run focused pytest on test_skill_runner.py + test_harness_router.py. No OpenAPI/test-catalog regen unless this repo’s CI catalogs these paths (it does not today for Apis Python tests).
  7. Author self /code-review on the diff; open PR (closes #369); impl gate → Vanellus.

Out of scope

  • Treating Reconnecting... / stream disconnected as retryable
  • Changing capacity/auth signatures or the 3600 default for those kinds
  • Metered fallback, paging redesign, Neotoma/schema/MCP surfaces

QA / Test Plan

🧠 Neotoma — Phoenicurus QA — ateles#369 harness failover network classification

QA/Test Plan section delivered above, built additively on the PM/Design/Eng sections — covers the regression test for the ENOTFOUND/ECONNREFUSED failover bug, classifier edge cases (bracketed-[unavailable] scoping, ambiguous-blob precedence), the cooldown contract tests, and non-regression guards, closing with a definition-of-done checklist.

[phoenicurus] test_plan: QA/Test Plan section written for ateles#369 — 8 test groups covering: regression (test_setup_network_fails_over + ECONNREFUSED variant), classifier units (4 setup signatures + bare-"unavailable" negative + capacity/auth non-regression), edge cases (mid-stream exclusion, ambiguous-blob precedence P0, exhaustion last_failure_kind, empty-candidates, mixed-kind cooldown chain), harness_router cooldown contract tests (default path, explicit seconds, env override, invalid-env fallback to 300, default-ordering invariant network<capacity), and non-regression checks (launch-failure, ordinary-failure). No eval yet committed — impl (Cicada) has not landed; sign-off is on the test plan as the QA artifact per this gate's process, contingent on Cicada authoring these named tests in the PR and agentic_evals lane going green.

Security / Arch

Interface-consistency gate (per waxwing charter §Interface-consistency & agent-instruction coherence)

This change touches only internal Python control flow (skill_runner.py, harness_router.py) plus config/docs. It does not touch any Neotoma MCP tool, CLI command, HTTP endpoint, schema field, or entity relationship — the "Neotoma interface gate" (change_guardrails_rules / openapi_contract_flow / errors.md / schema_agnostic_design_rules) is N/A. No openapi.yaml, contract_mappings.ts, or MCP↔CLI parity surface is implicated. Confirmed by reading Eng's file list: single-repo, single-surface (Apis harness dispatch), no cross-surface exposure. Cross-surface parity check: N/Anetwork is not exposed as an MCP tool/CLI/REST capability, only an internal classification token and log string.

Contract-first ordering

  • N/A for OpenAPI/spec-before-handler discipline — there is no external contract being added. The only "contract" introduced is the internal function signature cool_down(provider, *, now=None, seconds=None). Require this signature be finalized (this section) before Cicada writes call sites — Eng's proposal is accepted as final:
    • seconds=None → env-default path (unchanged 3600 behavior for capacity/auth/launch).
    • seconds=<int> → explicit override (network path only).
  • Order of implementation within the PR: (1) cool_down signature extension + env read, (2) classifier signature tuples + precedence, (3) run_skill wiring, (4) docs/env header, (5) tests. This ordering is a build-sequencing note, not a gating requirement — no external consumers exist to break mid-sequence.

Tenant isolation on entity lookups

N/A. No Neotoma entity, no tenant_id/owner_id-scoped lookup, no multi-tenant data access is introduced or touched by this change. skill_runner.py and harness_router.py operate on process-local state (in-memory cooldown map, subprocess stderr/stdout text) — there is no entity retrieval path to isolate. Confirming this explicitly so the checklist isn't silently skipped: tenant isolation requirement does not apply to this issue.

Idempotency on mutating operations

N/A in the Neotoma-mutation sense — this PR does not call store/correct/any Neotoma write. However, the operational analogue — replay safety — is exactly what #361 and this issue's precedence rule already encode, and it is the load-bearing safety property here:

  • The network kind is only assigned to setup-time signatures (getaddrinfo, enotfound, econnrefused, [unavailable]) — i.e., failures that occur strictly before the provider CLI accepts/starts processing the request. Treat this as the idempotency boundary: setup-time failure ⇒ no side effect ⇒ safe to retry on another provider. This is the correct analogue of an idempotency key for a system with no natural request-id — do not weaken it by broadening the signature set without re-establishing the no-side-effect guarantee per provider.
  • The precedence rule (mid-stream signature present → always None, checked first, before capacity/auth/network) is the enforcement mechanism. Confirm Eng's ordering is implemented as a short-circuit, not a scored/weighted match — a blob containing both reconnecting and getaddrinfo (e.g., a wrapped/retried provider log) MUST classify None. This is a correctness-of-safety-property requirement, not a style preference: get this precedence wrong and the system silently starts replaying possibly-side-effecting work, which is the exact failure mode fix(apis): recover harness routing + Codex dispatch work stranded in the deploy checkout #361 was designed to prevent.
  • No task-level idempotency key is required for the failover retry itself, because Apis dispatch already treats "try next provider" as a new attempt against a fresh CLI process, not a resend against a stateful queue. If a future issue introduces a provider that can accept work asynchronously (webhook-style completion), that provider integration must add an explicit idempotency key at the point work is submitted — out of scope here, flagged for future arch review if/when such a provider is added.

Auth / credential-exposure constraints

  • Signature matching must not be broadened to match on credential material. The setup-time and mid-stream signature tuples match structural error text (getaddrinfo, ENOTFOUND, [unavailable], Reconnecting..., stream disconnected) — none of these are expected to contain API keys, tokens, or PII. Confirm in review that no signature is derived from raw provider stderr in a way that would require echoing secret-bearing substrings into log lines.
  • Raw provider stderr as diagnostic detail (per UX section) must go through whatever redaction the existing skill_runner logging path already applies to provider output, if any — this PR does not change the log-emission mechanism, only what's classified. If skill_runner today logs raw stderr verbatim (need Cicada/Eng to confirm at implementation time), that is a pre-existing behavior, not introduced by this issue — do not silently expand what gets logged. If raw stderr is not currently captured in the primary log line and this PR adds "attach raw stderr as diagnostic detail" per the UX section, confirm it goes to a debug-level/attachment field, not the primary grep-target line that operators and any downstream alerting parse.
  • No new auth surface. This issue does not touch the auth failure kind, CLI credential handling, or provider authentication flows. The existing auth-kind cooldown/failover path is unchanged; do not conflate a network-unreachable endpoint with an auth failure even if both currently page — they must remain classified separately (already true in current code, confirmed unchanged by Eng's plan).
  • Cooldown state is process-local and non-persistent (per existing harness_router design) — no credential or secret is stored in the cooldown map; it holds provider name → timestamp only. This PR does not change that shape, only adds a seconds parameter to the existing call. No new persistence, no new attack surface.

Env / config constraints

  • APIS_HARNESS_NETWORK_COOLDOWN_SECONDS (default 300) — confirm parse-and-validate follows the identical pattern already used for APIS_HARNESS_COOLDOWN_SECONDS (invalid/non-numeric → fall back to default, not crash, not fall back to 0). Eng's plan states this; holding it as a hard requirement — an invalid env value must never produce a 0s or negative cooldown, which would make network classification indistinguishable from immediate-retry-no-backoff and defeat the purpose of cooling the provider at all.
  • No new secrets, tokens, or credentials are introduced by either new env var. Both are plain integers governing local timing behavior only.

Reversibility

Fully reversible. Both new signature tuples and the network kind are additive and isolated to skill_runner.py/harness_router.py; reverting is a straight revert of one PR with no data migration, no schema change, no persisted state to unwind (cooldown map is in-memory and resets on process restart regardless).

Verdict checklist (security/arch)

  • Contract-first: internal cool_down(..., seconds=) signature finalized before call-site wiring — no external contract to sequence against
  • Tenant isolation: N/A — no entity lookups in this change
  • Idempotency: N/A for Neotoma mutations; replay-safety boundary (setup-time-only classification + mid-stream-first precedence) is the correct analogue and must be implemented as a short-circuit, not a scored match
  • Auth/credential exposure: no new signature touches secret-bearing text; raw stderr diagnostic attachment (if added) must not land in the primary grep-target log line; auth kind remains untouched and distinct from network
  • Env validation: invalid APIS_HARNESS_NETWORK_COOLDOWN_SECONDS must fall back to 300, never 0/negative
  • Interface gate: N/A — no MCP/CLI/HTTP/schema surface touched; cross-surface parity check N/A

Verdict: SIGNED_OFF — no blocking security/architecture concerns. This is a single-surface, additive, in-memory-only change with no tenant, auth-surface, or persistence exposure. The one property to hold firm at implementation and review time is the mid-stream-first precedence short-circuit, since that is what preserves #361's no-replay-of-side-effecting-work guarantee.

[waxwing] schema_or_api_proposal: SIGNED_OFF — ateles#369 is a single-surface (Apis skill_runner/harness_router), in-memory-only change; no Neotoma MCP/CLI/HTTP/schema interface touched, so the interface-consistency gate is N/A. Security/arch requirements: cool_down(provider, *, now=None, seconds=None) signature accepted as final; tenant isolation N/A (no entity lookups); idempotency N/A for Neotoma writes but the setup-time-only classification + mid-stream-first short-circuit precedence is the load-bearing replay-safety boundary and must not be weakened; no credential-bearing text in any signature match; APIS_HARNESS_NETWORK_COOLDOWN_SECONDS invalid-value fallback must be 300, never 0. Fully reversible, additive-only change.

Legal

Jurisdiction (locale_profile ent_ea9a413189860f872c6cc99a): Spain (EU); regimes GDPR, ePrivacy, Spanish commercial law. Secondary: US (provider endpoint homes). No contract above escalation_currency_threshold in scope — counsel escalation not required.

Scope: Internal Apis harness classifier + cooldown only. No customer contract, marketing copy, privacy policy, DPA, guest API, or schema surface.

Dependencies / licensing

  • PR adds no new npm/pip/OS packages (Eng file list only: skill_runner, harness_router, apis.py, docs, tests)
  • Ateles licence remains MIT (product_profile ent_f79f82c13b90a9d6439623db) — no LICENSE/attribution change
  • N/A full dependency-licence matrix for this PR (nothing added); reject any late dependency without re-check

PII / data-handling (GDPR Art. 5 minimization; public-effect surfaces)

  • Classifier matches structural error substrings only (getaddrinfo, enotfound, econnrefused, [unavailable], reconnecting, stream disconnected) — never credential- or PII-bearing patterns
  • Primary failover log carries kind token network only; raw provider stderr (if attached) must stay debug/attachment — must not land on the primary grep-target line (aligns Security)
  • No new personal-data processing purpose; cooldown map stays process-local (provider → timestamp)
  • Must-hold: mid-stream-first precedence short-circuit (Eng/Security). Misclassifying post-accept disconnects as network risks replaying side-effecting work → duplicate third-party actions / provider ToS exposure
  • Public surfaces (GitHub issue/PR, docs/swarm_orchestration.md): describe internal failover behaviour only; do not paste live session payloads, keys, or personal task content into public text

Guest-token / credential scope

  • No guest_access_token, AAuth grant, MCP/HTTP auth, or credential-store change
  • Keep auth kind distinct from network — unreachable endpoint ≠ credential failure
  • APIS_HARNESS_NETWORK_COOLDOWN_SECONDS is a non-secret integer; invalid → 300 (never 0/negative)
  • Do not log, persist, or echo provider credentials on classify / cool-down paths

ToS / legal exposure

  • No customer ToS, DPA, or privacy-policy edits
  • Setup-time-only failover + mid-stream fail-fast preserves fix(apis): recover harness routing + Codex dispatch work stranded in the deploy checkout #361 safe-to-replay boundary (code comment required — already in Eng/UX)
  • Out of scope stays out of scope: treating Reconnecting... / stream disconnected as retryable without a provider no-side-effect guarantee
  • Escalation verdict: sign/ship as specified — risk analysis only, not legal advice

Checklist verdict: APPROVE — no [BLOCKING] legal items. Issue workflow gate_status.legal remains not_required.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workinglanius-triageIssue triaged by Lanius workflow coordinator

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions