fix(storage): preserve failed writes through interrupted and concurrent recovery - #202
Merged
howethomas merged 4 commits intoSep 20, 2026
Conversation
… them A storage write that failed was logged and forgotten. The vCon never reached the backend, nothing queued it, and the chain still reported success. On BDS a ~100s outage of one backend silently lost every vCon written during it. Failures now land on DLQ:storage:<backend> with the vCon's Redis TTL extended from VCON_REDIS_EXPIRY (1h) to VCON_DLQ_EXPIRY (7d), so the body outlives the default retention and is still there to replay. Replaying re-attempts only the storage write: routing through the ingress DLQ instead would re-run the whole chain, including transcription that already succeeded. _process_storage still does not raise, so one failing backend does not stop the others or egress. Also retries the failures worth retrying in the vcon-mcp client: connection errors, read timeouts, and 429/502/503/504, with exponential backoff. POST is included in allowed_methods, which urllib3 excludes by default, because vcon-mcp upserts on the vCon uuid so a retried create converges rather than duplicating. CON-714 Co-authored-by: Cursor <cursoragent@cursor.com>
# Conflicts: # api/api.py # common/lib/queue.py # common/storage/vcon_mcp/__init__.py # common/tests/test_storage_dlq.py # conserver/main.py
howethomas
deleted the
thomashowe/con-714-storage-write-failures-are-silently-swallowed-vcons-are
branch
September 20, 2026 22:41
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Failed storage recovery could lose its queue entry when the API stopped after popping it, and enqueueing before extending body retention left a crash window. This follows #197 with acknowledgement only after a successful write and atomic body-retention/queue insertion. Longer TTLs and persistent bodies remain unchanged; missing bodies are reported as requiring manual recovery.
Replay runs blocking storage I/O in the threadpool and uses a Redis lock per backend to prevent overlapping snapshots from acknowledging a newer failure for the same UUID. The lock has no expiry; after a process crash an operator must verify all replay workers have stopped before removing it. HTTP sessions now close after use. The recovery and rollout procedure is in
docs/storage-recovery.md.Validation:
Delivery remains at least once and depends on Redis durability, available body retention, and a backend that tolerates repeated UUID writes. Production rollout needs both conserver and API images; no production outage was induced.
Tracks CON-714.