[yjjnls]

Fixes #3738 (DNS queries each wait up to 2 s for a new egress-policy fetch after a refresh fails)

Root cause

actorEgressPolicy.record is fail-static on refresh failures: it keeps the stale entry but never moves expires. So after the first failed refresh, every later query finds the entry expired and enters a fresh fetch (singleflight only deduplicates concurrent ones). The relay reads the actor's queries one at a time (internal/atunnel/dns/udphandler.go, processRequests), so each queued query waits out the fetch in front of it — up to egressPolicyFetchTimeout (2 s) per query — and each then starts another failing fetch. With #3575-style DeadlineExceeded refresh failures this stalls the relay for as long as the outage lasts.

Fix

While refreshes are failing (fetched && failing), EgressPolicy now:

• answers immediately with the stale entry (the documented fail-static behavior in internal/atunnel/dns/readme.md), and
• starts one background refresh per TTL at most (nextRetry, throttled in the same mutex as the rest of the cache state).

The background retry reuses the same singleflight fetch body (uncanceled 2 s deadline, record update), so a successful retry re-caches the policy as soon as it lands and clears the failure state; the next healthy refresh blocks again as before. The initial fetch, healthy refreshes, the logging levels, and "a failed refresh is not an error for a query once an entry is cached" are all unchanged.

On the maximum-staleness question raised in #3738 (T-26, "denies rather than serving stale policy indefinitely"): aggregate staleness is unchanged — today stale is also served indefinitely while refreshes fail, just with a 2 s wait per query. I deliberately left a hard maximum age out of this fix; happy to add one if maintainers want it.

Relation to #3729

No overlap: #3729 changes sendToUpstream to return an explicit error to the actor when sending to the last upstream fails (the error path for a query that needs a forwarded answer). This PR changes the policy-cache wait path in internal/atunnel/egresspolicy.go that decides whether a query waits on a fetch at all.

Testing

• TestActorEgressPolicyServesStaleWhileRefreshesFail (new): after the first failed refresh, three queries with every later fetch hanging all return the stale entry, each without waiting (1 s query timeout, total under 1 s); exactly one retry starts within the TTL; a retry that succeeds replaces the entry without a query waiting for it; the recovered entry then refreshes blocking, as in the healthy path. This test fails on main with context deadline exceeded on the first stale query.
• TestActorEgressPolicyLogsChanges (updated): failure logs now follow the once-per-TTL background retries (DEBUG per retry, WARN every 5 m, ERROR on the first failure of each failure run) instead of per-query; also covers a "no policy" entry failing static like any other.
• Existing tests unchanged and passing, including initial-fetch failure retry, singleflight sharing, and query-cancel behavior.

go test -count=3 -race -run 'TestActorEgressPolicy|TestGetEgressPolicy' ./internal/atunnel/  # ok
go test -count=1 ./internal/atunnel/...                                                      # ok
go build ./...                                                                               # ok

Disclosure

This PR was developed with AI assistance and has been reviewed and tested by me. The commit is signed off per the DCO.

[yjjnls]

Closing as promised on #3738: José (@glitch-ux)'s #3796 implements the same fix — stale answers once a refresh has failed, retries throttled to one per TTL, the first failure still bounded by the fetch timeout — with a tighter implementation than ours: the background retry rides the existing singleflight instead of a new field plus a goroutine, and it documents the behavior in dns/readme.md. I verified the equivalence point by point and ran its tests locally (go test -count=3 -race ./internal/atunnel/..., green).

If useful, two small coverage additions from this PR are up for grabs: a second failure/recovery cycle for a cached "no policy" entry in TestActorEgressPolicyLogsChanges, and an in-loop assertion that stale serves never return an error to the query. Happy to share the tests or cherry-pick them if wanted. Thanks José (@glitch-ux) for picking this up.