Repository navigation
atunnel: answer DNS from the stale egress policy while refreshes fail - #3796
José (glitch-ux) wants to merge 1 commit into
Conversation
A failed refresh never moved the cached policy's expiry, so each query waited on another fetch, up to egressPolicyFetchTimeout. Set the expiry a TTL ahead on failure and serve the stale entry in the meantime. Signed-off-by: José Maia <glitch-ux@users.noreply.github.com>
yjjnls
left a comment
There was a problem hiding this comment.
I compared this against #3784 before withdrawing ours, as promised on the issue. The semantics are equivalent, point by point:
- Once a refresh has failed, queries are answered from the cached entry without waiting on the next fetch (the early return before the
selectonresults; on our side, astaleWhileFailingcheck with the retry dispatched to a goroutine). - Retries are throttled to one per TTL while fetches keep failing — yours by moving
expiresa TTL forward inrecord, ours via an explicitnextRetryfield. Same bound, and the first failed refresh still waits up toegressPolicyFetchTimeoutin both, as does a first fetch with nothing cached. - Healthy refreshes are unchanged, and a recovery while a retry is hung takes effect on the next query without any query waiting on it.
Your implementation is the tighter of the two: the refresh is already dispatched through the singleflight by the time the stale early-return happens, so no new struct field and no goroutine management are needed. Updating dns/readme.md is also the right call — ours didn't touch the docs.
Verification: I checked out fe34f0c and ran go test -count=3 -race ./internal/atunnel/... (green), plus go vet and gofmt — clean. TestActorEgressPolicyDoesNotWaitOnRefreshesAfterAFailure is a stronger shape than the equivalent check in our suite: it asserts per query that nothing waits on the hung fetch and pins the fetch count at each step, with waitForRefresh joining the in-flight fetch deterministically instead of polling.
Two small coverage deltas from #3784, in case they're worth picking up (happy to share the tests or cherry-pick them):
- Our version of
TestActorEgressPolicyLogsChangesadds a second failure/recovery cycle on a cached "no policy" entry (fail →ERRORagain, recover →INFO), pinning that a cached "no policy" entry fails static through a full second cycle like any other entry. - The same test asserts in-loop that stale serves never surface an error to the query, not only that the log sequence matches.
Neither blocks anything — your suite already covers the behavior. Closing #3784 as promised; thanks for the quick follow-through here.
When a refresh of an actor's egress policy fails, atunnel keeps answering DNS queries from the cached policy, but each query still waited on a new fetch, up to the 2 s fetch timeout. A failed refresh never moved the cached entry's expiry. The relay handles an actor's queries one at a time, so the wait also held up the lookups queued behind it.
A failed refresh now moves the expiry a TTL forward. After that, queries get the stale entry and the retry runs in the background. The first failed refresh still waits for its fetch, up to 2 s, and a first fetch that fails with nothing cached still retries on the next query. Healthy refreshes are unchanged. After atelet recovers, the new policy can show up to one TTL later than before.
T-26 in
docs/security/threat-model.mdsays an enforcement point that can't get current policy denies rather than serving stale policy indefinitely. I left a maximum age out. Today the stale entry is also served for as long as refreshes fail, and retries now come once per TTL. Say if you want a cap.TestActorEgressPolicyDoesNotWaitOnRefreshesAfterAFailureis new and fails on main. I changedTestActorEgressPolicyFailsStaticOnRefreshFailureandTestActorEgressPolicyLogsChangesas well, because queries within a TTL of a failure no longer fetch and the query that starts a retry now gets the stale entry.#3784 also fixes this. Its author said on the issue they'd close it if I sent mine.
Fixes #3738
AI use: AI-assisted (Claude). I reviewed the diff and ran the tests.