Default the scancode licence stage to off - #296
Merged
Conversation
scancode detects licences in first-party source. Dependency licences come declared from cdxgen and do not go through it, and neither path feeds vulnerability matching, so a deployment scanning for CVEs paid for the stage on every scan and read none of its output. The cost is not marginal. On a 2,080-repository corpus run its rule index reached 2.99 GB in one worker overnight, which was that worker's entire toolchain cache and enough to take a 26 GB root partition to 100%. SCANCODE_ENABLED now defaults to false and fails closed, matching scanoss_enabled: only true / 1 / yes enable it. Turning it off was already a supported non-fatal skip, so the pipeline behaviour when off is unchanged; what changes is which deployments get it without asking. Upgrade note: licence-compliance deployments must now set it explicitly. Detected first-party licences are how vendored or pasted third-party code is found, and dependency metadata never shows it.
The 2.99 GB attributed to scancode's rule index was pip's download cache. Both live under .cache, and scancode's own subdirectory measured 0 bytes; the disk growth continued with the stage disabled, which is what exposed the misattribution. That defect is fixed separately. The reason to default this off is unchanged and does not rest on the disk claim: the stage reads every eligible file in the tree to produce detected first-party licences, which do not feed vulnerability matching. The cost cited is now the per-scan wall clock the stage's own timeout, file ceiling and detection cap already exist to bound.
The test asserts a scancode_result artifact is persisted, which the stage only produces when it runs. With the default now off it has to request the stage explicitly. The off-path stays covered by the config unit tests.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
scancode detects licences in first-party source: code the organisation wrote. Dependency licences come declared from cdxgen and never go through this stage, and neither path feeds vulnerability matching. A deployment scanning for CVEs therefore pays for the stage on every scan and reads none of its output.
The cost is per-scan wall clock: the stage reads every eligible file in the tree. That is why it already carries its own timeout (default 600s), its own file ceiling, and its own detection cap, and why the public demo has always run with it off.
Change
SCANCODE_ENABLEDnow defaults tofalseand fails closed: onlytrue/1/yesenable it, anything else including typos reads as off. That is the same shape as the neighbouringscanoss_enabled, so the two switches now read the same way.This reverses the previous default-on / fail-open reasoning, which was that scancode is a pure-local pass with no egress and so costs nothing worth gating. Local is not the same as free.
Turning the stage off was already a supported, non-fatal path:
run_scancodeshort-circuits withScancodeDisabledand the pipeline continues on declared licences. Nothing about the off-behaviour changes here; what changes is which deployments get the stage without asking for it.A correction, recorded
The first version of this PR justified the change with a disk figure: 2.99 GB of cache attributed to scancode's rule index on a 2,080-repository corpus run. That attribution was wrong. The growth was pip's download cache. Both live under
~/.cache; scancode's own subdirectory measured 0 bytes, and the directory kept growing after this stage was disabled, which is what exposed the mistake. The real defect is fixed in #302.The reason to default this off never rested on that figure and is unchanged.
Upgrade note
A deployment that reads detected first-party licences must now set
SCANCODE_ENABLED=trueexplicitly. This is called out in the accessor docstring and in.env.example, alongside why you would want it: detected first-party licences are how vendored or pasted third-party code gets found, and dependency metadata never shows it. Licence-compliance deployments should turn it on.Test plan
tests/unit/test_scancode_config.pyrewritten for the new default: off by default,true/1/yesenable, andon/enable/ junk all read as off (those two are asserted deliberately, since they look like intent to enable and are not).tests/unit/integrations/test_scancode.pygains an autouse fixture enabling the stage. Those tests are about what the adapter does when it runs; the off-path is covered in the config test.pytest tests/unit(6,800+ tests). The only remaining failures are 2 pre-existing ones intest_admin_team_service.py, verified to reproduce unchanged with this branch stashed.mypy .clean over 823 files;ruff check .clean; em-dash lint clean.