Skip to content

atunnel: answer DNS from the stale egress policy while refreshes fail - #3796

Open
José (glitch-ux) wants to merge 1 commit into
agent-substrate:mainfrom
glitch-ux:atunnel-stale-policy-no-wait
Open

José (glitch-ux) wants to merge 1 commit into
agent-substrate:mainfrom
glitch-ux:atunnel-stale-policy-no-wait

Conversation

@glitch-ux

Copy link
Copy Markdown

When a refresh of an actor's egress policy fails, atunnel keeps answering DNS queries from the cached policy, but each query still waited on a new fetch, up to the 2 s fetch timeout. A failed refresh never moved the cached entry's expiry. The relay handles an actor's queries one at a time, so the wait also held up the lookups queued behind it.

A failed refresh now moves the expiry a TTL forward. After that, queries get the stale entry and the retry runs in the background. The first failed refresh still waits for its fetch, up to 2 s, and a first fetch that fails with nothing cached still retries on the next query. Healthy refreshes are unchanged. After atelet recovers, the new policy can show up to one TTL later than before.

T-26 in docs/security/threat-model.md says an enforcement point that can't get current policy denies rather than serving stale policy indefinitely. I left a maximum age out. Today the stale entry is also served for as long as refreshes fail, and retries now come once per TTL. Say if you want a cap.

TestActorEgressPolicyDoesNotWaitOnRefreshesAfterAFailure is new and fails on main. I changed TestActorEgressPolicyFailsStaticOnRefreshFailure and TestActorEgressPolicyLogsChanges as well, because queries within a TTL of a failure no longer fetch and the query that starts a retry now gets the stale entry.

#3784 also fixes this. Its author said on the issue they'd close it if I sent mine.

Fixes #3738

AI use: AI-assisted (Claude). I reviewed the diff and ran the tests.

  • Tests pass
  • Docs updated

A failed refresh never moved the cached policy's expiry, so each
query waited on another fetch, up to egressPolicyFetchTimeout. Set
the expiry a TTL ahead on failure and serve the stale entry in the
meantime.

Signed-off-by: José Maia <glitch-ux@users.noreply.github.com>

@yjjnls yjjnls left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I compared this against #3784 before withdrawing ours, as promised on the issue. The semantics are equivalent, point by point:

  • Once a refresh has failed, queries are answered from the cached entry without waiting on the next fetch (the early return before the select on results; on our side, a staleWhileFailing check with the retry dispatched to a goroutine).
  • Retries are throttled to one per TTL while fetches keep failing — yours by moving expires a TTL forward in record, ours via an explicit nextRetry field. Same bound, and the first failed refresh still waits up to egressPolicyFetchTimeout in both, as does a first fetch with nothing cached.
  • Healthy refreshes are unchanged, and a recovery while a retry is hung takes effect on the next query without any query waiting on it.

Your implementation is the tighter of the two: the refresh is already dispatched through the singleflight by the time the stale early-return happens, so no new struct field and no goroutine management are needed. Updating dns/readme.md is also the right call — ours didn't touch the docs.

Verification: I checked out fe34f0c and ran go test -count=3 -race ./internal/atunnel/... (green), plus go vet and gofmt — clean. TestActorEgressPolicyDoesNotWaitOnRefreshesAfterAFailure is a stronger shape than the equivalent check in our suite: it asserts per query that nothing waits on the hung fetch and pins the fetch count at each step, with waitForRefresh joining the in-flight fetch deterministically instead of polling.

Two small coverage deltas from #3784, in case they're worth picking up (happy to share the tests or cherry-pick them):

  1. Our version of TestActorEgressPolicyLogsChanges adds a second failure/recovery cycle on a cached "no policy" entry (fail → ERROR again, recover → INFO), pinning that a cached "no policy" entry fails static through a full second cycle like any other entry.
  2. The same test asserts in-loop that stale serves never surface an error to the query, not only that the log sequence matches.

Neither blocks anything — your suite already covers the behavior. Closing #3784 as promised; thanks for the quick follow-through here.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/network kind/bug Something isn't working / bugfixes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: DNS queries each wait up to 2 s for a new egress-policy fetch after a refresh fails

3 participants