fix: increase shutdown delay to allow job re-queue on SIGTERM - #15
Open
Ramilito wants to merge 1 commit into
Open
fix: increase shutdown delay to allow job re-queue on SIGTERM#15Ramilito wants to merge 1 commit into
Ramilito wants to merge 1 commit into
Conversation
The ShutdownHandler delay (6s) runs concurrently with the ffmpeg shutdown hook (3s kill + force kill). By the time ffmpeg is killed and ApplicationShutdownException propagates to QueueService.repostJob(), there are only ~3 seconds left before Spring closes the context and destroys Redis connections. This causes repostJob() to silently fail, leaving jobs stuck in IN_PROGRESS with no re-queue. Increase the delay to 12s to ensure the full shutdown chain completes: ffmpeg kill (3s) + exception propagation + repostJob Redis save (~1s) + buffer.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When a worker receives SIGTERM (e.g., Kubernetes spot VM eviction), the
ShutdownHandlerdelay (6s) and the ffmpeg shutdown hook (3s kill) run concurrently as JVM shutdown hooks. By the time ffmpeg is killed andApplicationShutdownExceptionpropagates toQueueService.repostJob(), there are only ~3 seconds left before Spring closes the context and destroys Redis connections. This causesrepostJob()to silently fail, leaving jobs permanently stuck inIN_PROGRESS.This fix increases the
ShutdownHandlerdelay from 6s to 12s, giving enough time for the full shutdown chain: ffmpeg kill (3s) + exception propagation +repostJob()Redis save.Proof
Tested on AKS with spot VM eviction using
az vm simulate-eviction:Before (6s delay) — job stuck, never re-queued:
Job status stays
IN_PROGRESS, queue depth remains0.After (12s delay) — job successfully re-queued:
Job status transitions to
QUEUED, queue depth increases to1, and the job is picked up by the next worker.Test plan
IN_PROGRESSaz vm simulate-evictionon the noderepostJob()completingIN_PROGRESS→QUEUED