feat: add batch processing for Trivy vulnerabilities and new vulnerab… #5498

arjavdongaonkar · 2025-11-05T11:40:02Z

Description

New Batch Implementation Improvements (with approximate numbers)

Batch Mode Behavior

1. Component UUID lookups:
Instead of one lookup per component inside the loop → performed once
200 component queries → now 1 batched loop (≈200 ops total) (unchanged count but centralized)

2. Vulnerability existence checks:
Old: 3,000 “source + vulnId” lookups
New: Build a map of unique (source, vulnId) pairs, e.g. 3,000 unique →
Single getVulnerabilityByVulnIds() call
Replaces 3,000 DB queries with 1 batched query

3. Creating new vulnerabilities:
Still O(#missing vulns), but performed in one transaction scope.
Index event dispatched just once.

4. Adding vulnerability–component mappings:
Still requires writes, but all done in a tight loop inside one transaction.
No repeated component fetches
No repeated vulnerability lookups

Estimated totals (new code)

Operation	Old	New
Vulnerability lookups	3,000 queries	1 query
Component lookups	200 queries	200 lookups in a single block
Add vulnerability to component	3,000 writes	3,000 writes (unavoidable)
Severity updates	300 writes	300 writes
Index commits	N commits	1 commit

Grand total (new code)

~200 component lookups (no change)
1 DB query instead of 3,000 for vuln existence checks
~3,300 writes (same as old, but in one transaction block)
Significant transaction overhead reduction

Impact Summary

Metric	Old Code	New Code
Vulnerability lookups	~3,000 DB roundtrips	1 DB roundtrip
IndexEvent dispatches	many	1
Transactions opened	~200 (one per component)	1
Total DB operations	~6,500+	~3,500 but batched and far cheaper

Overall effect

Dramatically less DB round-trip latency
Higher throughput during large Trivy scans
Lower CPU + I/O load on the database
More predictable performance for large SBOMs

Addressed Issue

closes #5497

Checklist

I have read and understand the contributing guidelines
This PR fixes a defect, and I have provided tests to verify that the fix is effective
This PR implements an enhancement, and I have provided tests to verify that it works as intended
This PR introduces changes to the database model, and I have added corresponding update logic
This PR introduces new or alters existing behavior, and I have updated the documentation accordingly

owasp-dt-bot · 2025-11-05T11:40:13Z

✅ Snyk checks have passed. No issues have been found so far.

Status	Scanner	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

…ility retrieval method Signed-off-by: Arjav <[email protected]>

Signed-off-by: Arjav <[email protected]>

codacy-production · 2025-11-06T05:17:29Z

Coverage summary from Codacy

See diff coverage on Codacy

Coverage variation	Diff coverage
Report missing for `72dfb1a`¹	✅ 81.63% (target: 70.00%)

Coverage variation details

	Coverable lines	Covered lines	Coverage
Common ancestor commit (`72dfb1a`)	Report Missing	Report Missing	Report Missing
Head commit (`5a85de4`)	24144	19529	80.89%

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details

	Coverable lines	Covered lines	Diff coverage
Pull request (#5498)	98	80	81.63%

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

See your quality gate settings Change summary preferences

Codacy didn't receive coverage data for the commit, or there was an error processing the received data. Check your integration for errors and validate that your coverage setup is correct. ↩

nscuro

Thanks for the PR. A few change requests.

Also, it would be great if you could at least mention that AI was used for this, and to what capacity. The PR description was clearly written by AI, and makes various claims that are not reflected in the code.

nscuro · 2025-12-04T11:40:02Z

src/main/java/org/dependencytrack/tasks/scanners/TrivyAnalysisTask.java

+        handleAllComponentsBatchOptimized(vulnsByComponent);
+    }
+
+    private void handleAllComponentsBatchOptimized(final Map<Component, List<trivy.proto.common.Vulnerability>> vulnsByComponent) {


Can we use a method name that is a bit more concise? "Batch omtimized" is an implementation detail.

nscuro · 2025-12-04T11:41:16Z

src/main/java/org/dependencytrack/tasks/scanners/TrivyAnalysisTask.java

+                Component persistent = qm.getObjectByUuid(Component.class, uuid);
+                if (persistent != null) {
+                    persistentComponents.put(uuid, persistent);
+                }


Why not fetch the components in batches as well?

nscuro · 2025-12-04T11:43:02Z

src/main/java/org/dependencytrack/persistence/QueryManager.java

+    public List<Vulnerability> getVulnerabilityByVulnIds(Map<String, Set<String>> sourceToVulnIds,
+                                                         boolean includeVulnerableSoftware) {
+        return getVulnerabilityQueryManager().getVulnerabilitiesBySourceAndVulnIds(sourceToVulnIds, includeVulnerableSoftware);
+    }


The method names are different: getVulnerabilityByVulnIds vs. getVulnerabilitiesBySourceAndVulnIds.

nscuro · 2025-12-04T11:44:54Z

src/main/java/org/dependencytrack/tasks/scanners/TrivyAnalysisTask.java

+            List<Vulnerability> fetchedVulns = qm.getVulnerabilityByVulnIds(vulnIds, false);
+
+            for (Vulnerability vuln : fetchedVulns) {
+                existingVulns.put(vuln.getSource() + ":" + vuln.getVulnId(), vuln);


Use VulnIdAndSource as key instead.

nscuro · 2025-12-04T11:49:32Z

src/main/java/org/dependencytrack/persistence/VulnerabilityQueryManager.java

+
+        @SuppressWarnings("unchecked")
+        List<Vulnerability> fetched = (List<Vulnerability>) query.executeWithMap(paramValues);


Suggested change

@SuppressWarnings("unchecked")

List<Vulnerability> fetched = (List<Vulnerability>) query.executeWithMap(paramValues);

query.setNamedParameters(paramValues);

List<Vulnerability> fetched = executeAndCloseList(query);

nscuro · 2025-12-04T11:51:32Z

src/main/java/org/dependencytrack/persistence/VulnerabilityQueryManager.java

+        @SuppressWarnings("unchecked")
+        List<Vulnerability> detached = (List<Vulnerability>) pm.detachCopyAll(fetched);
+
+        // Safely apply aliases
+        for (Vulnerability vuln : detached) {
+            vuln.setAliases(getVulnerabilityAliases(vuln));
+        }


Aliases are not required for the Trivy analysis task. Detaching has a negative effect for the Trivy task because it then has to immediately attach the records back in order to perform persistence operations (e.g., adding a component) with them.

nscuro · 2025-12-04T11:52:35Z

src/main/java/org/dependencytrack/persistence/VulnerabilityQueryManager.java

+            Map<String, Set<String>> sourceToVulnIds,
+            boolean includeVulnerableSoftware) {


Consider accepting Collection<VulnIdAndSource> here instead, and omitting the includeVulnerableSoftware parameter. VulnerableSoftware records are not required for the Trivy analysis task.

arjavdongaonkar mentioned this pull request Nov 5, 2025

Trivy Analyzer Performs Excessive Repeated DB Queries During Vulnerability Ingestion #5497

Open

2 tasks

arjavdongaonkar force-pushed the optimize-trivy-analyzer branch 3 times, most recently from ab7d82c to 609d22f Compare November 6, 2025 05:04

arjavdongaonkar added 7 commits November 6, 2025 10:37

feat: add batch processing for Trivy vulnerabilities and new vulnerab…

2907c97

…ility retrieval method Signed-off-by: Arjav <[email protected]>

fix: change getAnalyses method visibility to public

58e25d1

Signed-off-by: Arjav <[email protected]>

remove suppression Expiration

99dfd92

Signed-off-by: Arjav <[email protected]>

Remove public

95bc03b

Signed-off-by: Arjav <[email protected]>

Remove public

6287400

Signed-off-by: Arjav <[email protected]>

add public

2bdd1d9

Signed-off-by: Arjav <[email protected]>

add removed query

5a85de4

Signed-off-by: Arjav <[email protected]>

arjavdongaonkar force-pushed the optimize-trivy-analyzer branch from 350b850 to 5a85de4 Compare November 6, 2025 05:08

nscuro added enhancement New feature or request integration/trivy Related to the Trivy integration labels Dec 4, 2025

nscuro requested changes Dec 4, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

feat: add batch processing for Trivy vulnerabilities and new vulnerab… #5498

feat: add batch processing for Trivy vulnerabilities and new vulnerab… #5498

Uh oh!

arjavdongaonkar commented Nov 5, 2025 •

edited

Loading

Uh oh!

owasp-dt-bot commented Nov 5, 2025 •

edited

Loading

Uh oh!

codacy-production bot commented Nov 6, 2025

Uh oh!

nscuro left a comment

Uh oh!

nscuro Dec 4, 2025

Uh oh!

nscuro Dec 4, 2025

Uh oh!

nscuro Dec 4, 2025

Uh oh!

nscuro Dec 4, 2025

Uh oh!

nscuro Dec 4, 2025

Uh oh!

nscuro Dec 4, 2025

Uh oh!

nscuro Dec 4, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants


		@SuppressWarnings("unchecked")
		List<Vulnerability> fetched = (List<Vulnerability>) query.executeWithMap(paramValues);

		Map<String, Set<String>> sourceToVulnIds,
		boolean includeVulnerableSoftware) {

Uh oh!

feat: add batch processing for Trivy vulnerabilities and new vulnerab… #5498

Are you sure you want to change the base?

feat: add batch processing for Trivy vulnerabilities and new vulnerab… #5498

Uh oh!

Conversation

arjavdongaonkar commented Nov 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

New Batch Implementation Improvements (with approximate numbers)

Batch Mode Behavior

Estimated totals (new code)

Grand total (new code)

Impact Summary

Overall effect

Addressed Issue

Checklist

Uh oh!

owasp-dt-bot commented Nov 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Snyk checks have passed. No issues have been found so far.

Uh oh!

codacy-production bot commented Nov 6, 2025

Coverage summary from Codacy

Footnotes

Uh oh!

nscuro left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

arjavdongaonkar commented Nov 5, 2025 •

edited

Loading

owasp-dt-bot commented Nov 5, 2025 •

edited

Loading