ISSUE-4357: Fix shared _temp race between cancelled and new run-service workers

[herbenderbler]

Description

Fixes #4357.

For run-service jobs, JobDispatcher.EnsureDispatchFinished cancelled the previous worker and returned without awaiting its exit. The new Runner.Worker then spawned while the previous worker was still tearing down. Both processes share <work>/_temp, so the exiting worker's TempDirectoryManager.CleanupTempDirectory() wiped the new worker's _runner_file_commands/* pipes mid-job, failing it with Missing file at path: .../set_output_<uuid> and exit code 102.

Implementation Details

• src/Runner.Listener/JobDispatcher.cs: in the _isRunServiceJob branch of EnsureDispatchFinished, after cancelling the previous worker, await Task.WhenAny(jobDispatch.WorkerDispatch, Task.Delay(45s)) and throw InvalidOperationException on timeout — matching the existing non-run-service branch in the same method.
• src/Test/L0/Listener/JobDispatcherL0.cs: add EnsureDispatchFinishedAwaitsPreviousWorkerForRunServiceJob regression test (drives the path with a controlled TaskCompletionSource; verified to fail without the fix and pass with it).

Checklist

• ./dev.sh format is a no-op on the diff.
• JobDispatcherL0 suite (14 tests) passes locally.
• Regression test added and verified to fail on main.
• Self-contained; no public API or _temp layout change.

[Copilot]

Pull request overview

Fixes a race in run-service job dispatch where a canceled worker could still be tearing down (and cleaning <work>/_temp) while a new worker starts, deleting the new worker’s _runner_file_commands/* files mid-job.

Changes:

• Update JobDispatcher.EnsureDispatchFinished for run-service jobs to cancel the previous worker and wait (up to 45s) for the previous WorkerDispatch to finish before proceeding.
• Add an L0 regression test that verifies EnsureDispatchFinished does not complete until the previous worker dispatch task completes.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
src/Runner.Listener/JobDispatcher.cs	Adds a 45s wait in the run-service cancellation path to prevent overlapping workers from racing on the shared _temp directory.
src/Test/L0/Listener/JobDispatcherL0.cs	Adds a regression test that drives the run-service EnsureDispatchFinished path and asserts it awaits the previous worker’s completion.