Skip to content

Collect EPSS scores, so the features that read them stop reading NULL - #317

Merged
haksungjang merged 3 commits into
mainfrom
epss-daily-sync
Sep 3, 2026
Merged

Collect EPSS scores, so the features that read them stop reading NULL#317
haksungjang merged 3 commits into
mainfrom
epss-daily-sync

Conversation

@haksungjang

Copy link
Copy Markdown
Contributor

vulnerabilities.epss_score and epss_percentile have existed since v2.4, and nothing ever wrote them. The scanner emits no EPSS on either path, measured on live runs at 88 SBOM findings and 107 image findings with zero EPSS keys between them, and every row the scanner created carried NULL. So the EPSS column and the min_epss filter on the Vulnerabilities tab showed nothing, the EPSS term in sort=priority never moved a ranking, the reports carried a blank column, and GATE_EPSS_THRESHOLD passed every build no matter how low it was set. The gate is the part worth naming: an absent gate is a gap somebody can see, while a gate that always passes looks like a verdict.

A daily beat now fetches FIRST's published CSV and writes the scores onto the CVEs this deployment has actually seen. It is off by default (EPSS_REFRESH_ENABLED), following the EOL and malicious-package feeds rather than KEV, because installing the product should not reach the public internet on its own and an air-gapped deployment can point EPSS_FEED_URL at an internal mirror. While it is off those surfaces stay empty, and the data-sources reference now says so plainly in both languages instead of implying the scores are always present. It also corrects a claim that was already wrong there: EPSS and KEV do not come from the scanner's database bundle, which is why each needs its own sync.

The bulk CSV is the only ingest path and the API is deliberately unused. FIRST's guidance is that their lookup API "should not be used for bulk downloads or to keep a local copy of all scores in sync", and using it that way would also pull the traffic under Services Terms whose licence grant is revocable and non-transferable, which is not a good footing for a redistributed product. Attribution sits in THIRD_PARTY_NOTICES.md, the data-sources page and the feed client's docstring, with the citation FIRST supplies. Nothing is redistributed: each installation downloads the file itself, and it is not vendored into the tree, the images or any release artifact.

A tick reads the deployment's own CVE ids first and drops feed rows outside that set as they stream past, so peak memory tracks the catalog rather than the 367,000-row document, and it writes only the rows whose score actually moved. A document that parses to implausibly few rows is refused before any write, so a truncated publish upstream cannot blank good scores, and nothing is ever cleared: a CVE that leaves the feed keeps its last known score. The daily mass UPDATE costs no audit rows, because worker sessions deliberately run without the audit listeners.

Tests use a captured excerpt of the real published file, twelve real rows with their real values, and assert the published numbers rather than merely that a score is present. That is the point: the defect survived a release because _extract_epss was unit-tested against hand-written dicts carrying an EPSS key the scanner never sends, so a fixture written from what the code expects proved the code correct and the feature dead. Removing the write from the task fails two of the integration tests. Schema change is one new status table on revision 0076 (0075 is head), with its GRANT and its entry in the app-role privileges fixture; the score columns and their index already existed.

vulnerabilities.epss_score and epss_percentile have existed since v2.4 and
nothing wrote them: the scanner emits no EPSS on either the SBOM or the
image path, so the tab column and min_epss filter, the sort=priority
ranking, the reports and GATE_EPSS_THRESHOLD were all reading empty
columns, and the gate passed every build at any threshold. A daily beat
now syncs FIRST's published CSV onto the CVEs this deployment has seen.

Off by default; the bulk CSV is the only ingest path because FIRST's
guidance forbids using their lookup API for bulk sync. Attribution is in
THIRD_PARTY_NOTICES, the data-sources page and the feed client.
THIRD_PARTY_NOTICES.md is mirrored into apps/backend, apps/frontend and
charts/trustedoss so each distributed artifact carries it, and a test
asserts the four are byte-identical. The EPSS entry went into the root
only.
@haksungjang
haksungjang merged commit 77baea7 into main Sep 3, 2026
33 checks passed
@haksungjang
haksungjang deleted the epss-daily-sync branch September 3, 2026 10:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant