Skip to content

Forward pip's cache controls into scan subprocesses - #302

Merged
haksungjang merged 1 commit into
mainfrom
pip-cache-passthrough
Sep 3, 2026
Merged

Forward pip's cache controls into scan subprocesses#302
haksungjang merged 1 commit into
mainfrom
pip-cache-passthrough

Conversation

@haksungjang

Copy link
Copy Markdown
Contributor

Problem

Dockerfile.worker sets PIP_NO_CACHE_DIR=1 so that resolving a Python project leaves no download cache behind. The subprocess env scrubber never forwarded that key, so the image's setting stopped at the worker process and every scan's Python resolution cached again under HOME.

Measured on a 2,080-repository corpus: 3.3 GB accumulated in one worker inside a day, which was most of what took a 26 GB root partition to 100% and stopped the run.

Worth recording how this was misdiagnosed. The growth showed up as ~/.cache, which is also where scancode keeps its rule index, so scancode was blamed first. It was not: with the scancode stage disabled the directory kept growing, and .cache/scancode-tk measured 0 bytes while .cache/pip held all 3.3 GB.

Change

PIP_NO_CACHE_DIR and PIP_CACHE_DIR join the cdxgen and prep allowlists.

The prep allowlist had no Python entries at all. Prep does not resolve Python dependencies itself, but the resolvers it runs can shell into pip, and the setting has to survive that hop or the cache it exists to prevent gets written anyway.

PIP_CACHE_DIR is included for the opposite deployment: the one that wants the cache and has somewhere with room to put it. Neither key is credential-bearing, so both are safe additions to a list whose purpose is keeping secrets away from hostile-clone subprocesses.

Test plan

  • Two new tests assert both keys survive scrubbed_env_for_cdxgen and scrubbed_env_for_prep
  • Mutation check: removing the pair from either allowlist fails the corresponding test
  • pytest tests/unit/integrations/ (973 passed)
  • ruff check . clean; em-dash lint clean

Dockerfile.worker sets PIP_NO_CACHE_DIR=1 so resolving a Python project
leaves no download cache behind. The subprocess env scrubber never
forwarded it, so the setting stopped at the worker process and every
scan's pip run cached again under HOME.

Measured on a 2,080-repository corpus: 3.3 GB in one worker inside a day,
which was most of what took a 26 GB root partition to 100%. The cache
directory was read as scancode's at first because both live under
.cache; scancode's own subdirectory was 0 bytes.

Adds PIP_NO_CACHE_DIR and PIP_CACHE_DIR to the cdxgen and prep
allowlists. PIP_CACHE_DIR rides along for the opposite deployment: the
one that wants the cache and has somewhere with room to put it.
@haksungjang
haksungjang merged commit 63bdc77 into main Sep 3, 2026
24 checks passed
@haksungjang
haksungjang deleted the pip-cache-passthrough branch September 3, 2026 04:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant