Skip to content

fix: retry LookupAccountNameW with exponential backoff on ERROR_NONE_MAPPED - #221

Merged
jericht merged 2 commits into
OpenJobDescription:mainfrom
leon-li-inspire:fix/lookup-sid-retry-backoff
Jul 2, 2026
Merged

jericht merged 2 commits into
OpenJobDescription:mainfrom
leon-li-inspire:fix/lookup-sid-retry-backoff

Conversation

@leon-li-inspire

Copy link
Copy Markdown
Contributor

Summary

  • lookup_sid() calls LookupAccountNameW to resolve principal names to SIDs for DACL-based permission management
  • This fails intermittently with ERROR_NONE_MAPPED (1332) on freshly started EC2 instances where the LSA (Local Security Authority) service has not finished initializing
  • The failure is intermittent — re-runs pass because LSA is warm by then
  • Add retry with exponential backoff (up to 3 attempts, sleeping 1s then 2s) on error 1332, then re-raise if all attempts are exhausted

This is the Rust equivalent of openjd-adaptor-runtime-for-python#262.

Test plan

  • test_lookup_sid_succeeds_for_current_user: verifies lookup_sid succeeds through the retry loop for the current process user
  • test_lookup_sid_fails_for_nonexistent_user: verifies the error is surfaced after exhausting all retries (with backoff timing assertion ≥3s)

@leon-li-inspire
leon-li-inspire requested a review from a team as a code owner June 12, 2026 19:00
crowecawcaw
crowecawcaw previously approved these changes Jun 12, 2026
@leongdl

leongdl commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

Build and test failed, also there are merge conflicts that I can't rebase.

@leon-li-inspire

Copy link
Copy Markdown
Contributor Author

Thanks for the review! I've addressed all three issues:

1. Merge conflicts / branch behind main — Rebased onto the latest main. The branch is now MERGEABLE and no longer behind.

2. Build & Test (windows-latest) failure — The failure was error[E0603]: function lookup_sid is private. The new integration tests reference lookup_sid, which was pub(crate). I changed it to pub fn to mirror set_permissions in the same module (the enclosing module is only exposed under the test-utils feature, so it stays crate-private in normal builds).

3. Rustfmt failure — Reformatted per cargo fmt.

While in here I also tightened the retry logic so it only backs off on ERROR_NONE_MAPPED (1332) and fails fast on any other error, rather than burning the full backoff window on genuine failures. Extracted the single-attempt resolution into lookup_sid_once for clarity.

Verified locally against the x86_64-pc-windows-msvc target:

  • cargo check -p openjd-sessions --tests --features test-utils — passes
  • cargo clippy ... -- -D warnings — passes
  • cargo fmt --all -- --check — passes

CI for the new commit needs a maintainer to approve the workflow run.

@mwiebe
mwiebe force-pushed the fix/lookup-sid-retry-backoff branch from ee2061a to 374d89d Compare June 30, 2026 21:06
mwiebe
mwiebe previously approved these changes Jun 30, 2026
match lookup_sid_once(&name_w) {
Ok(sid) => return Ok(sid),
Err(e) => {
if is_lsa_not_ready(&e) && attempt < LOOKUP_MAX_RETRIES - 1 {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ERROR_NONE_MAPPED (1332) is overloaded: Windows returns it not only when the LSA service is still initializing, but also whenever a name genuinely does not map to any account ("No mapping between account names and security IDs was done"). So this retry path fires on every lookup of a non-existent/unmappable principal, not just the transient EC2-startup case.

Concretely, is_process_user() (session_user.rs:233) calls lookup_sid(&self.user) and falls back to string comparison when it fails. With this change, any user whose name does not resolve to a SID now blocks for ~3s (1s + 2s backoff) before that fallback runs — a behavior change for the common case of a non-existent or non-mappable user, on a path that previously returned immediately. The new test_lookup_sid_fails_for_nonexistent_user test confirms this by asserting the ≥3s delay.

If the goal is purely to absorb LSA-not-ready at instance startup, consider whether the retry can be scoped more narrowly (e.g. only on the first lookup after process start, or guarded so the steady-state "no such user" path is not penalized). At minimum, worth confirming this added latency is acceptable for callers like is_process_user.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch — this was a real regression. Fixed in 518e8ec.

ERROR_NONE_MAPPED (1332) is overloaded, so I now gate the backoff on whether the LSA is actually down rather than retrying on every 1332:

  • A new process-global LSA_AVAILABLE flag latches true the first time any lookup succeeds (LSA never restarts within a process lifetime).
  • On a 1332 failure when the flag isn't set yet, lsa_is_available() probes by resolving the current process user — which always maps once LSA is up. If that probe succeeds, the original 1332 must mean the name is unmapped, so we fail fast with no backoff.
  • Retry/backoff now only happens in the genuine LSA-still-initializing case at instance startup.

So the steady-state is_process_user() fallback path returns immediately again, exactly as before this PR. I inverted the corresponding test (test_lookup_sid_fails_fast_for_nonexistent_user) to assert the unmappable-name lookup completes in <1s once LSA is confirmed up.

Verified locally against x86_64-pc-windows-msvc: cargo check/clippy -D warnings/fmt --check all pass (and the Linux build too).

let probed = match crate::win32::get_process_user() {
Ok(user) => {
let name_w: Vec<u16> = user.encode_utf16().chain(std::iter::once(0)).collect();
lookup_sid_once(&name_w).is_ok()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The retry-on-ERROR_NONE_MAPPED feature only engages while lsa_is_available() returns false, and that probe resolves the current process user. But the helper/session orchestration typically runs as a Windows service under LocalSystem (S-1-5-18), a well-known SID that LookupAccountNameW can resolve very early — potentially before the LSA service has finished initializing enough to map normal (domain/local) accounts.

If that happens in the freshly-booted-EC2 window this PR targets:

  1. The real session-user lookup fails with ERROR_NONE_MAPPED (LSA not ready for that name).
  2. lsa_is_available() probes the process user (LocalSystem), which resolves, so it latches LSA_AVAILABLE = true.
  3. lsa_still_initializing is then false, so the lookup fails fast with no retry — defeating the feature.

Worth confirming the assumption that "the process user only resolves once LSA is fully up" actually holds for the well-known service identities this runs under. If it does not, the gate silently disables the backoff in precisely the scenario it is meant to handle.

@mwiebe
mwiebe force-pushed the fix/lookup-sid-retry-backoff branch from 518e8ec to 78d8a2c Compare July 2, 2026 20:11
…MAPPED

LookupAccountNameW fails intermittently with ERROR_NONE_MAPPED (1332) on
freshly started EC2 instances where the LSA service hasn't finished
initializing. Add retry with exponential backoff (up to 3 attempts,
sleeping 1s then 2s) to the lookup_sid() function, then re-raise if all
attempts are exhausted.

This is the Rust equivalent of the Python fix in
OpenJobDescription/openjd-adaptor-runtime-for-python#262 which addresses
the same intermittent failure in Windows integration tests across all DCC
repos that use LOGNAME=SYSTEM in their CodeBuild Windows environment.

Signed-off-by: Leon Li <2182521+leon-li-inspire@users.noreply.github.com>
…ng unmapped names

ERROR_NONE_MAPPED (1332) is overloaded: Windows returns it both when the LSA
service is still initializing (transient, only at instance startup) and when a
name genuinely does not map to an account (permanent). The previous retry fired
on every 1332, so steady-state lookups of a non-existent principal — notably
the WindowsSessionUser::is_process_user fallback path — blocked for ~3s before
returning, a regression for a path that previously returned immediately.

Gate the backoff on a process-global LSA-availability check: once any lookup
succeeds (or a probe of the current process user resolves), the LSA is known to
be up and a subsequent 1332 must mean the name is unmapped, so it fails fast.
The LSA never restarts within a process lifetime, so the result is cached.

Update test_lookup_sid_fails_for_nonexistent_user accordingly: it now asserts
the unmappable-name lookup fails fast (<1s) once LSA is confirmed up, instead of
asserting the old >=3s backoff.

Signed-off-by: Leon Li <2182521+leon-li-inspire@users.noreply.github.com>
@jericht
jericht force-pushed the fix/lookup-sid-retry-backoff branch from 78d8a2c to 1925434 Compare July 2, 2026 22:57
@jericht
jericht enabled auto-merge (squash) July 2, 2026 22:58
@jericht
jericht merged commit 0da9968 into OpenJobDescription:main Jul 2, 2026
22 checks passed
This was referenced Jul 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants