Summary
When an outbound request fails at the transport level — a timed-out LLM call, connection refused, DNS failure, TLS failure — the client correctly receives a 502, but no response-phase session event is recorded. The failure is therefore invisible on every abctl screen: it cannot be distinguished from a request that is still in flight.
Reproduction
A hanging upstream behind a proxy whose client times out (probe against authlib/listener/forwardproxy, an empty pipeline, Client: &http.Client{Timeout: 300ms}, upstream sleeping 3s):
client saw status: 502
events recorded: 1
phase=request status=0 host="127.0.0.1:59946" error=<nil>
One event, phase=request. No response event, so nothing carries the 502 and nothing carries an error.
Cause
authbridge/authlib/usage/usage.go — the request event is appended at the end of the request phase, but the response event (the one carrying StatusCode and Error) is appended much later. The upstream failure path returns between the two:
authbridge/authlib/listener/forwardproxy/server.go:491-495
resp, err := client.Do(r)
if err != nil {
http.Error(w, `{"error":"bad gateway"}`, http.StatusBadGateway)
return
}
The 502 exists only on the wire to the client, never in the session store. reverseproxy's errorHandler (authlib/listener/reverseproxy/server.go:605-611) has the same shape.
What each screen shows
| Screen |
Behaviour |
| Events table |
One req row with STATUS blank (statusCell returns "" for StatusCode == 0), DURATION blank, no paired resp row — identical to a request still in flight |
| Usage → ERRORS |
Not counted. usage.go:563 sets Errors = 1 only for StatusCode >= 400 or SessionDenied; a timeout is neither |
| Usage → breakdown by status |
Absent. The label is added only when StatusCode > 0, so the request falls into the (unlabelled) remainder |
| Usage → latency |
Excluded — a zero duration is treated as "not measured" rather than as a failure |
| Detail pane |
The request JSON, with no error field and no response |
Why it matters
The request event is counted in Requests, so a burst of timeouts renders as ordinary traffic with a healthy error rate. The usage pane is least informative exactly when an operator is trying to find out whether their upstream is failing — and "STATUS blank" reads as "still waiting", which is the wrong conclusion.
SessionEvent already has an Error field (pipeline.DeriveError), but it is only ever populated on the response path.
Suggested fix
Record a response-phase event with StatusCode: 502 and a populated Error before the early return, in both listeners. That would surface the failure in the events table, the ERRORS metric, and the status breakdown at once, without any change to those consumers.
An Error value distinguishing why the upstream call failed (timeout vs refused vs TLS) would be more useful than the status code alone, since all of them surface as 502 today.
Scope of this report
Verified for proxy-sidecar's forward proxy — the laptop / Claude Code path. reverseproxy looks the same by inspection but was not tested, and envoy-sidecar's ext_proc path was not examined at all, so I cannot say whether it records a response event on upstream failure.
Operating system and architecture
Darwin arm64
Cortex version
dev (traced at b838de5f; both cited files are unmodified from main)
Summary
When an outbound request fails at the transport level — a timed-out LLM call, connection refused, DNS failure, TLS failure — the client correctly receives a 502, but no response-phase session event is recorded. The failure is therefore invisible on every
abctlscreen: it cannot be distinguished from a request that is still in flight.Reproduction
A hanging upstream behind a proxy whose client times out (probe against
authlib/listener/forwardproxy, an empty pipeline,Client: &http.Client{Timeout: 300ms}, upstream sleeping 3s):One event,
phase=request. No response event, so nothing carries the 502 and nothing carries an error.Cause
authbridge/authlib/usage/usage.go— the request event is appended at the end of the request phase, but the response event (the one carryingStatusCodeandError) is appended much later. The upstream failure path returns between the two:authbridge/authlib/listener/forwardproxy/server.go:491-495The 502 exists only on the wire to the client, never in the session store.
reverseproxy'serrorHandler(authlib/listener/reverseproxy/server.go:605-611) has the same shape.What each screen shows
reqrow with STATUS blank (statusCellreturns""forStatusCode == 0), DURATION blank, no pairedresprow — identical to a request still in flightusage.go:563setsErrors = 1only forStatusCode >= 400orSessionDenied; a timeout is neitherStatusCode > 0, so the request falls into the(unlabelled)remaindererrorfield and no responseWhy it matters
The request event is counted in
Requests, so a burst of timeouts renders as ordinary traffic with a healthy error rate. The usage pane is least informative exactly when an operator is trying to find out whether their upstream is failing — and "STATUS blank" reads as "still waiting", which is the wrong conclusion.SessionEventalready has anErrorfield (pipeline.DeriveError), but it is only ever populated on the response path.Suggested fix
Record a response-phase event with
StatusCode: 502and a populatedErrorbefore the early return, in both listeners. That would surface the failure in the events table, the ERRORS metric, and the status breakdown at once, without any change to those consumers.An
Errorvalue distinguishing why the upstream call failed (timeout vs refused vs TLS) would be more useful than the status code alone, since all of them surface as 502 today.Scope of this report
Verified for proxy-sidecar's forward proxy — the laptop / Claude Code path.
reverseproxylooks the same by inspection but was not tested, and envoy-sidecar's ext_proc path was not examined at all, so I cannot say whether it records a response event on upstream failure.Operating system and architecture
Darwin arm64
Cortex version
dev (traced at
b838de5f; both cited files are unmodified frommain)