Skip to content

fix: increase shutdown delay to allow job re-queue on SIGTERM - #15

Open
Ramilito wants to merge 1 commit into
Eyevinn:releasefrom
Ramilito:fix/shutdown-delay-for-requeue
Open

fix: increase shutdown delay to allow job re-queue on SIGTERM#15
Ramilito wants to merge 1 commit into
Eyevinn:releasefrom
Ramilito:fix/shutdown-delay-for-requeue

Conversation

@Ramilito

Copy link
Copy Markdown

Summary

When a worker receives SIGTERM (e.g., Kubernetes spot VM eviction), the ShutdownHandler delay (6s) and the ffmpeg shutdown hook (3s kill) run concurrently as JVM shutdown hooks. By the time ffmpeg is killed and ApplicationShutdownException propagates to QueueService.repostJob(), there are only ~3 seconds left before Spring closes the context and destroys Redis connections. This causes repostJob() to silently fail, leaving jobs permanently stuck in IN_PROGRESS.

This fix increases the ShutdownHandler delay from 6s to 12s, giving enough time for the full shutdown chain: ffmpeg kill (3s) + exception propagation + repostJob() Redis save.

Proof

Tested on AKS with spot VM eviction using az vm simulate-eviction:

Before (6s delay) — job stuck, never re-queued:

08:29:21 - "Application is shutting down. Stopping encoding."
08:29:21 - "Delaying application shutdown"
08:29:24 - "Ffmpeg did not shut down in 3 seconds. Destroying forcibly."
08:29:27 - "Continue application shutdown"   ← context closes, no repost log

Job status stays IN_PROGRESS, queue depth remains 0.

After (12s delay) — job successfully re-queued:

08:29:21 - "Application is shutting down. Stopping encoding."
08:29:21 - "Delaying application shutdown"
08:29:24 - "Ffmpeg did not shut down in 3 seconds. Destroying forcibly."
08:29:27 - "Application was shut down. Will attempt to add back to queue"
08:29:27 - "Adding job to queue (repost on interrupt)"
08:29:27 - "Added job to queue (repost on interrupt)"
08:29:27 - "Stopping"

Job status transitions to QUEUED, queue depth increases to 1, and the job is picked up by the next worker.

Test plan

  • Deploy worker with fix on AKS spot node
  • Submit encoding job, wait for IN_PROGRESS
  • Run az vm simulate-eviction on the node
  • Verify worker logs show repostJob() completing
  • Verify Redis job status changes from IN_PROGRESSQUEUED
  • Verify queue depth increases (job available for next worker)

The ShutdownHandler delay (6s) runs concurrently with the ffmpeg
shutdown hook (3s kill + force kill). By the time ffmpeg is killed
and ApplicationShutdownException propagates to QueueService.repostJob(),
there are only ~3 seconds left before Spring closes the context and
destroys Redis connections. This causes repostJob() to silently fail,
leaving jobs stuck in IN_PROGRESS with no re-queue.

Increase the delay to 12s to ensure the full shutdown chain completes:
ffmpeg kill (3s) + exception propagation + repostJob Redis save (~1s)
+ buffer.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant