Retry SQLITE_BUSY at DB open for up to 15 minutes instead of aborting

[MelvinBot]

Details

Bedrock treats a failed sqlite3_open_v2 at startup as fatal and immediately abort()s, with no retry. When a defunct predecessor process (e.g. one that was force-killed but is still dumping core) is still holding the exclusive HCTree file lock, the open returns SQLITE_BUSY (code 5) — a transient, self-healing condition — and Bedrock turns it into a hard crash loop that requires repeated manual/wrapper restarts until the old process finally releases the lock. This is exactly what happened on db2.rno/auth on 2026-07-01 (crashed twice before recovering on the third attempt).

This change makes SQLite::initializeDB poll every 5 seconds for up to 15 minutes when sqlite3_open_v2 returns SQLITE_BUSY, instead of aborting immediately. As requested, it does not parse the holding PID — it simply keeps retrying the open on the 5s interval until it either succeeds or the 15-minute cap is reached. Any other open error, or a SQLITE_BUSY that outlasts the 15-minute timeout, still falls through to the existing fatal SERROR/abort() so a genuinely wedged DB surfaces as a hard failure rather than hanging forever.

The 15-minute cap was sized against the one observed occurrence (stale lock persisted somewhere between ~4m43s and ~10m45s after the force-kill), leaving headroom since worst-case hold time scales with the predecessor's core-dump size. This is complementary to the Salt wrapper change (SIGKILL + wait-for-exit on stop, fuser lock-wait on start): the wrapper prevents the racy handoff, and this bedrock-side retry is defense-in-depth for when the predecessor takes minutes to actually release the FD.

Fixed Issues

$ https://github.com/Expensify/Expensify/issues/654403

Tests

// TODO: The human co-author must fill out the tests you ran before marking this PR as "ready for review".
// Please describe what tests you performed that validate your change worked.

---

Internal Testing Reminder: when changing bedrock, please compile auth against your new changes

[tylerkaraszewski]

@MelvinBot - remove the comment on lines 165-168

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: efb15fbeda

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Exit the retry loop on termination signals

When an HCTree lock persists while the starting process receives SIGTERM or SIGINT, this loop continues retrying for up to 15 minutes instead of stopping. SInitializeSignals() blocks these signals and records them in the dedicated signal thread, while main.cpp does not inspect the recorded termination state until after BedrockServer construction, so neither sleep() nor this condition reacts during initializeDB(). This makes a locked startup unresponsive to normal service-manager shutdown; include STerminationSignalCount() in the loop's exit condition so termination can proceed promptly.

Useful? React with 👍 / 👎.

[madmax330]

@tylerkaraszewski what do you think about this?