Fall back to local compilation when sccache has no workers - #387
Conversation
The distributed scheduler can remain reachable while reporting zero compilation servers. The setup check treated any SchedulerStatus as available, so cache misses waited on 7,140-second requests and retries until GitHub Actions canceled the ARM jobs at six hours. The existing local fallback was also exported from a child shell, so it could not reach compiler requests in the subsequent Docker build. Require a nonempty server list, reuse one status snapshot, and source the setup script from both Presto and Velox image builds. This preserves S3 caching while bypassing distributed compilation whenever the scheduler is unavailable. Add regression coverage for healthy, empty, missing, malformed, failed, explicitly disabled, and fallback-disabled scheduler states, and run it from PR checks.
simoneves
left a comment
simoneves
left a comment
There was a problem hiding this comment.
Looks plausible although I am by no means a power-user of this mechanism
| fi | ||
| } | ||
|
|
||
| healthy_status='{"SchedulerStatus":["https://arm64.example",{"servers":[{"id":"worker-1"}]}]}' |
There was a problem hiding this comment.
This is a test, the URL isn't used. Would you prefer a different name just to make the arch neutrality clear in the test case? x86_64 does follow the same zero-worker path, it isn't branched based on arch, if that alleviates your concern.
| fi | ||
| bash /sccache_setup.sh; | ||
| # Source the setup so its automatic local-fallback export reaches the build. | ||
| source /sccache_setup.sh; |
There was a problem hiding this comment.
Please explain the previous failing here in more detail
There was a problem hiding this comment.
Previously a child shell was executed which set a fallback environment variable (SCCACHE_NO_DIST_COMPILE=1), which is then lost when the process exits. Subsequent build commands didn't see it. By sourcing, it is executed in the current shell, so later steps inherit the fallback.
| runs-on: ubuntu-latest | ||
| outputs: | ||
| benchmark_data_tools: ${{ steps.filter.outputs.benchmark_data_tools }} | ||
| sccache: ${{ steps.filter.outputs.sccache }} |
There was a problem hiding this comment.
What does the sccache tag actually do here? (and below)
There was a problem hiding this comment.
This is a boolean flag that is set when any of the sccache related files change. By declaring it, it is exposed as an output of detect-changes, and test_sccache_setup checks the output later to decide whether to run. If we don't want an sccache related test on PRs, we can remove it.
3 participants
The distributed scheduler can remain reachable while reporting zero compilation servers. The setup check treated any SchedulerStatus as available, so cache misses waited on 7,140-second requests and retries until GitHub Actions canceled the ARM jobs at six hours.
The existing local fallback was also exported from a child shell, so it could not reach compiler requests in the subsequent Docker build.
Require a nonempty server list, reuse one status snapshot, and source the setup script from both Presto and Velox image builds. This preserves S3 caching while bypassing distributed compilation whenever the scheduler is unavailable.
Add regression coverage for healthy, empty, missing, malformed, failed, explicitly disabled, and fallback-disabled scheduler states, and run it from PR checks.