fix(server): retry stop lifecycle writes after concurrent updates

[pimlock]

Summary

openshell sandbox stop writes Stopping and then Stopped with one compare-and-swap each.

In HA, every gateway replica applies watch events for the terminating pod, and the per-replica lifecycle lock doesn't cover them.

If one of those writes lands between the re-read and the CAS, stop fails with Aborted: update sandbox lifecycle failed due to concurrent modification and the sandbox stays in Stopping. This PR retries both writes on conflict, the same way start already does.

Seen in Branch E2E Checks run 38004123998 ("Kubernetes HA E2E (Rust smoke)", scenario sandbox-lifecycle/stop-start/stop).

[run 7bzdq443t9][sandbox-lifecycle/stop-start/stop]
expected: sandbox 'ct-7bzdq443t9-ss' stop succeeds
actual: exit 1 after 3.0s
command: openshell sandbox stop ct-7bzdq443t9-ss
stdout:

stderr:
Error:   × code: 'The operation was aborted', message: \"update sandbox lifecycle\n  │ failed due to concurrent modification (current resource_version: 33)\"

Related Issue

No issue required. Small bug fix for an HA race seen in CI.

Changes

• stop_sandbox retries the Stopping write. Each retry re-reads the sandbox, checks the name, and re-runs the phase checks.
• New commit_sandbox_stopped retries the Stopped write while the sandbox is still Stopping. It returns early if the sandbox is already Stopped and aborts on any other phase.
• Startup recovery of sandboxes left in Stopping uses the same helper. It used to write against a copy read before the driver call, so any watch event during the stop made it fail.
• The retry limit is 5 attempts. The other limits are 3, but a terminating pod sends several watch updates to every replica.

Delete already retries its phase write. The automatic-restart claim backs off on conflict by design, so I left it alone.

Testing

• mise run pre-commit passes
• Unit tests added/updated
• E2E tests added/updated (if applicable)

The new test stop_retries_lifecycle_writes_after_concurrent_replica_write uses a SQLite trigger to add a conflicting write before each CAS. Stop still ends in Stopped. With retries disabled the test fails with the CI error. I didn't re-run the Kubernetes HA E2E job locally.

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)

[pimlock]

gator-agent

PR Review Status

This focused server fix retries lifecycle compare-and-swap writes that lose to concurrent HA watch updates. The initial code review found no blocking issues, but the required standard E2E workflow has not yet been dispatched with its new label.

Action required: A maintainer must open Actions run 38009458482 and click Re-run all jobs so test:e2e executes for the current head.

Blocking findings:

• No blocking findings remain

Carried findings:

• None

Non-blocking suggestions:

• None

Gator metadata

• Validation: Project-valid focused HA sandbox lifecycle race fix from a repository maintainer
• Docs: Not needed because this changes internal retry behavior without changing the user-facing contract
• Checks: Branch Checks, Helm Lint, Trivy Changes, DCO, and vouch gates are green; required standard E2E awaits dispatch
• E2E: test:e2e and test:e2e-kubernetes are applied; the label helper requires Re-run all jobs on run 38009458482
• Head SHA: 0667f74e12f89d5f60de66f32c68c7a33ea649dc
• Base SHA: 0ccc6b9a053fdaf47700e91c1b8ed858f2afed4f
• Merge base SHA: d789ec613b829044e4580f6cda7f5d4648a31285
• Patch ID: 28e3fe2fdaea4f7513129e6a006b02c58eed561c
• Gator payload: 11
• Review mode: initial
• Previous reviewed SHA: none
• Review budget exhausted: no
• Maintainer decision required: no
• Next state: gator:blocked
• Blocked reason: test_dispatch_required