Skip to content

Bug: A timed-out or refused upstream request is not flagged as an error on any abctl screen #1045

Description

@esnible

Summary

When an outbound request fails at the transport level — a timed-out LLM call, connection refused, DNS failure, TLS failure — the client correctly receives a 502, but no response-phase session event is recorded. The failure is therefore invisible on every abctl screen: it cannot be distinguished from a request that is still in flight.

Reproduction

A hanging upstream behind a proxy whose client times out (probe against authlib/listener/forwardproxy, an empty pipeline, Client: &http.Client{Timeout: 300ms}, upstream sleeping 3s):

client saw status: 502
events recorded: 1
  phase=request  status=0  host="127.0.0.1:59946"  error=<nil>

One event, phase=request. No response event, so nothing carries the 502 and nothing carries an error.

Cause

authbridge/authlib/usage/usage.go — the request event is appended at the end of the request phase, but the response event (the one carrying StatusCode and Error) is appended much later. The upstream failure path returns between the two:

authbridge/authlib/listener/forwardproxy/server.go:491-495

resp, err := client.Do(r)
if err != nil {
	http.Error(w, `{"error":"bad gateway"}`, http.StatusBadGateway)
	return
}

The 502 exists only on the wire to the client, never in the session store. reverseproxy's errorHandler (authlib/listener/reverseproxy/server.go:605-611) has the same shape.

What each screen shows

Screen Behaviour
Events table One req row with STATUS blank (statusCell returns "" for StatusCode == 0), DURATION blank, no paired resp row — identical to a request still in flight
Usage → ERRORS Not counted. usage.go:563 sets Errors = 1 only for StatusCode >= 400 or SessionDenied; a timeout is neither
Usage → breakdown by status Absent. The label is added only when StatusCode > 0, so the request falls into the (unlabelled) remainder
Usage → latency Excluded — a zero duration is treated as "not measured" rather than as a failure
Detail pane The request JSON, with no error field and no response

Why it matters

The request event is counted in Requests, so a burst of timeouts renders as ordinary traffic with a healthy error rate. The usage pane is least informative exactly when an operator is trying to find out whether their upstream is failing — and "STATUS blank" reads as "still waiting", which is the wrong conclusion.

SessionEvent already has an Error field (pipeline.DeriveError), but it is only ever populated on the response path.

Suggested fix

Record a response-phase event with StatusCode: 502 and a populated Error before the early return, in both listeners. That would surface the failure in the events table, the ERRORS metric, and the status breakdown at once, without any change to those consumers.

An Error value distinguishing why the upstream call failed (timeout vs refused vs TLS) would be more useful than the status code alone, since all of them surface as 502 today.

Scope of this report

Verified for proxy-sidecar's forward proxy — the laptop / Claude Code path. reverseproxy looks the same by inspection but was not tested, and envoy-sidecar's ext_proc path was not examined at all, so I cannot say whether it records a response event on upstream failure.

Operating system and architecture

Darwin arm64

Cortex version

dev (traced at b838de5f; both cited files are unmodified from main)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions