Collect EPSS scores, so the features that read them stop reading NULL - #317
Merged
Conversation
vulnerabilities.epss_score and epss_percentile have existed since v2.4 and nothing wrote them: the scanner emits no EPSS on either the SBOM or the image path, so the tab column and min_epss filter, the sort=priority ranking, the reports and GATE_EPSS_THRESHOLD were all reading empty columns, and the gate passed every build at any threshold. A daily beat now syncs FIRST's published CSV onto the CVEs this deployment has seen. Off by default; the bulk CSV is the only ingest path because FIRST's guidance forbids using their lookup API for bulk sync. Attribution is in THIRD_PARTY_NOTICES, the data-sources page and the feed client.
THIRD_PARTY_NOTICES.md is mirrored into apps/backend, apps/frontend and charts/trustedoss so each distributed artifact carries it, and a test asserts the four are byte-identical. The EPSS entry went into the root only.
# Conflicts: # CHANGELOG.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
vulnerabilities.epss_scoreandepss_percentilehave existed since v2.4, and nothing ever wrote them. The scanner emits no EPSS on either path, measured on live runs at 88 SBOM findings and 107 image findings with zero EPSS keys between them, and every row the scanner created carried NULL. So the EPSS column and themin_epssfilter on the Vulnerabilities tab showed nothing, the EPSS term insort=prioritynever moved a ranking, the reports carried a blank column, andGATE_EPSS_THRESHOLDpassed every build no matter how low it was set. The gate is the part worth naming: an absent gate is a gap somebody can see, while a gate that always passes looks like a verdict.A daily beat now fetches FIRST's published CSV and writes the scores onto the CVEs this deployment has actually seen. It is off by default (
EPSS_REFRESH_ENABLED), following the EOL and malicious-package feeds rather than KEV, because installing the product should not reach the public internet on its own and an air-gapped deployment can pointEPSS_FEED_URLat an internal mirror. While it is off those surfaces stay empty, and the data-sources reference now says so plainly in both languages instead of implying the scores are always present. It also corrects a claim that was already wrong there: EPSS and KEV do not come from the scanner's database bundle, which is why each needs its own sync.The bulk CSV is the only ingest path and the API is deliberately unused. FIRST's guidance is that their lookup API "should not be used for bulk downloads or to keep a local copy of all scores in sync", and using it that way would also pull the traffic under Services Terms whose licence grant is revocable and non-transferable, which is not a good footing for a redistributed product. Attribution sits in
THIRD_PARTY_NOTICES.md, the data-sources page and the feed client's docstring, with the citation FIRST supplies. Nothing is redistributed: each installation downloads the file itself, and it is not vendored into the tree, the images or any release artifact.A tick reads the deployment's own CVE ids first and drops feed rows outside that set as they stream past, so peak memory tracks the catalog rather than the 367,000-row document, and it writes only the rows whose score actually moved. A document that parses to implausibly few rows is refused before any write, so a truncated publish upstream cannot blank good scores, and nothing is ever cleared: a CVE that leaves the feed keeps its last known score. The daily mass UPDATE costs no audit rows, because worker sessions deliberately run without the audit listeners.
Tests use a captured excerpt of the real published file, twelve real rows with their real values, and assert the published numbers rather than merely that a score is present. That is the point: the defect survived a release because
_extract_epsswas unit-tested against hand-written dicts carrying anEPSSkey the scanner never sends, so a fixture written from what the code expects proved the code correct and the feature dead. Removing the write from the task fails two of the integration tests. Schema change is one new status table on revision0076(0075is head), with its GRANT and its entry in the app-role privileges fixture; the score columns and their index already existed.