Reference chains for surviving live-heap samples via a bounded manual heap walk (PROF-15341) - #644
Draft
jbachorik wants to merge 43 commits into
Draft
Reference chains for surviving live-heap samples via a bounded manual heap walk (PROF-15341)#644jbachorik wants to merge 43 commits into
jbachorik wants to merge 43 commits into
Conversation
This comment has been minimized.
This comment has been minimized.
Contributor
CI Test ResultsRun: #30920900809 | Commit:
Status Overview
Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled Summary: Total: 32 | Passed: 31 | Failed: 0 | Cancelled: 1 Updated: 2026-08-04 17:53:27 UTC |
Contributor
Reliability & Chaos Results❌ 1 failure(s) detected Pipeline: https://gitlab.ddbuild.io/DataDog/java-profiler/-/pipelines/127431974 ❌ chaos: profiler gmalloc aarch64 21 0 3 temXchaos |
Contributor
Benchmark Results (commit cfec595)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/123822759 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 25 | 💥 0 ms (0 iters) | 💥 0 ms (0 iters) | — | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 1 | 1949 / 2055 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / · | ✅ / · | ✅ / · | ✅ / · | ✅ / · | ✅ / · |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 1 | 8727 / 8389 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 8166 / 8396 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 2 | 1292 / 1256 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 1 | 1288 / 1290 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 2830 / 3023 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 2 / ✅ | 2976 / 2821 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | · / ✅ | · / ✅ | · / 3 | · / 3555 | · / ✅ | · / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 3 | 3499 / 3485 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 1592 / 1697 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 1922 / 1854 | ✅ / ✅ | ✅ / ✅ |
jbachorik
added a commit
that referenced
this pull request
Jul 10, 2026
…racking Addresses review findings from PR #644 (reference chains for surviving live-heap samples): FrontierTable's shared-lock mutation race, a single-byte JFR event size prefix that silently truncates chains longer than 255 bytes, blocking sample-lock retries in writeReferenceChain() now bounded by a shared per-batch deadline with a drop counter, unvalidated referencechains sub-option values, a startThread()/ pthread_kill() race publishing _running before the thread handle is initialized, a null-JNIEnv leak in threadLoop(), releaseSearchTags() now surfacing GetObjectsWithTags() failures so restartSearch() never resets tag state prematurely, resolveLoadedClasses() skipping its per-class scan only when the loaded-class count is unchanged (not just non-decreasing), a spurious COMPLETED state after a failed first-pass FollowReferences call, and removal of a leftover debug helper in ExternalProcessReferenceChainTest. Adds regression test coverage for release-failure and negative-value option paths. Verified via the full ddprof-lib gtestDebug suite (149/149 tasks, 57/57 referenceChains_ut tests). Environment: Datadog workspace Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Contributor
Benchmark Results (commit 0714bd3)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124004132 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10184 ms (21 iters) | ✅ 10371 ms (21 iters) | ≈ +1.8% (±11.5%) | — / — |
| akka-uct | 25 | ✅ 8891 ms (24 iters) | ✅ 8865 ms (24 iters) | ≈ -0.3% (±10.2%) | — / — |
| finagle-chirper | 21 | ✅ 6003 ms (33 iters) | ✅ 6034 ms (33 iters) | ≈ +0.5% (±25.2%) | |
| finagle-chirper | 25 | ✅ 5507 ms (36 iters) | ✅ 5426 ms (36 iters) | ≈ -1.5% (±23.8%) | |
| fj-kmeans | 21 | ✅ 2834 ms (66 iters) | ✅ 2761 ms (67 iters) | 🟢 -2.6% | — / — |
| fj-kmeans | 25 | ✅ 2824 ms (66 iters) | ✅ 2838 ms (66 iters) | ≈ +0.5% (±2.7%) | — / — |
| future-genetic | 21 | ✅ 2113 ms (88 iters) | ✅ 2065 ms (89 iters) | ≈ -2.3% (±2.6%) | — / — |
| future-genetic | 25 | ✅ 2059 ms (90 iters) | ✅ 2102 ms (89 iters) | ≈ +2.1% (±2.6%) | — / — |
| naive-bayes | 21 | ✅ 1283 ms (134 iters) | ✅ 1260 ms (135 iters) | ≈ -1.8% (±32%) | — / — |
| naive-bayes | 25 | ✅ 1016 ms (169 iters) | ✅ 1017 ms (168 iters) | ≈ +0.1% (±31.5%) | — / — |
| reactors | 21 | ✅ 16658 ms (15 iters) | ✅ 16371 ms (16 iters) | ≈ -1.7% (±9%) | — / — |
| reactors | 25 | ✅ 18507 ms (15 iters) | ✅ 18056 ms (15 iters) | ≈ -2.4% (±5.6%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 1929 / 2034 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 1 | 2310 / 2113 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 5 | 8529 / 8852 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 8635 / 8390 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 5 | 1284 / 1281 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 5 / 2 | 1283 / 1285 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 2930 / 2864 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 2 / ✅ | 2963 / 2953 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 7 / 6 | 3558 / 3510 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 4 | 3463 / 3469 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 1564 / 1782 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1905 / 1681 | ✅ / ✅ | ✅ / ✅ |
Contributor
Benchmark Results (commit 700d838)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124275960 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10425 ms (21 iters) | ✅ 10259 ms (21 iters) | ≈ -1.6% (±11.1%) | — / — |
| akka-uct | 25 | ✅ 8748 ms (24 iters) | ✅ 8797 ms (24 iters) | ≈ +0.6% (±10.5%) | — / — |
| finagle-chirper | 21 | ✅ 5970 ms (33 iters) | ✅ 5982 ms (33 iters) | ≈ +0.2% (±25.3%) | |
| finagle-chirper | 25 | ✅ 5469 ms (36 iters) | ✅ 5468 ms (36 iters) | ≈ -0% (±24.3%) | |
| fj-kmeans | 21 | ✅ 2598 ms (73 iters) | ✅ 2715 ms (69 iters) | 🔴 +4.5% | — / — |
| fj-kmeans | 25 | ✅ 2784 ms (67 iters) | ✅ 2847 ms (66 iters) | ≈ +2.3% (±2.7%) | — / — |
| future-genetic | 21 | ✅ 2122 ms (87 iters) | ✅ 2126 ms (88 iters) | ≈ +0.2% (±2.7%) | — / — |
| future-genetic | 25 | ✅ 2035 ms (91 iters) | ✅ 2057 ms (91 iters) | ≈ +1.1% (±2.7%) | — / — |
| naive-bayes | 21 | ✅ 1304 ms (131 iters) | ✅ 1253 ms (136 iters) | ≈ -3.9% (±32.2%) | — / — |
| naive-bayes | 25 | ✅ 1015 ms (169 iters) | ✅ 1018 ms (168 iters) | ≈ +0.3% (±31.8%) | — / — |
| reactors | 21 | ✅ 16107 ms (15 iters) | ✅ 16847 ms (15 iters) | ≈ +4.6% (±7.9%) | — / — |
| reactors | 25 | ✅ 18469 ms (15 iters) | ✅ 19075 ms (14 iters) | ≈ +3.3% (±4.9%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 3 / ✅ | 1950 / 1991 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 2 | 2228 / 2453 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 6 / 2 | 8686 / 8417 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 8561 / 8290 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 1272 / 1289 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 2 / ✅ | 1287 / 1298 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 2912 / 3017 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 2907 / 2872 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 3 | 3515 / 3489 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 7 | 3498 / 3463 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 1732 / 1701 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1780 / 1924 | ✅ / ✅ | ✅ / ✅ |
jbachorik
added a commit
that referenced
this pull request
Jul 14, 2026
…racking Addresses review findings from PR #644 (reference chains for surviving live-heap samples): FrontierTable's shared-lock mutation race, a single-byte JFR event size prefix that silently truncates chains longer than 255 bytes, blocking sample-lock retries in writeReferenceChain() now bounded by a shared per-batch deadline with a drop counter, unvalidated referencechains sub-option values, a startThread()/ pthread_kill() race publishing _running before the thread handle is initialized, a null-JNIEnv leak in threadLoop(), releaseSearchTags() now surfacing GetObjectsWithTags() failures so restartSearch() never resets tag state prematurely, resolveLoadedClasses() skipping its per-class scan only when the loaded-class count is unchanged (not just non-decreasing), a spurious COMPLETED state after a failed first-pass FollowReferences call, and removal of a leftover debug helper in ExternalProcessReferenceChainTest. Adds regression test coverage for release-failure and negative-value option paths. Verified via the full ddprof-lib gtestDebug suite (149/149 tasks, 57/57 referenceChains_ut tests). Environment: Datadog workspace Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
jbachorik
force-pushed
the
jb/reference-chains
branch
from
July 14, 2026 12:03
700d838 to
9972bff
Compare
Contributor
Benchmark Results (commit 9972bff)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124511782 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
Contributor
Benchmark Results (commit a636398)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124700626 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
Contributor
Benchmark Results (commit f2d8978)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124747648 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
Contributor
Benchmark Results (commit a48899b)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124773626 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10376 ms (21 iters) | ✅ 10228 ms (21 iters) | ≈ -1.4% (±11%) | — / — |
| akka-uct | 25 | ✅ 8829 ms (24 iters) | ✅ 8859 ms (24 iters) | ≈ +0.3% (±10%) | — / — |
| finagle-chirper | 21 | ✅ 6001 ms (33 iters) | ✅ 5928 ms (33 iters) | ≈ -1.2% (±25.2%) | |
| finagle-chirper | 25 | ✅ 5469 ms (36 iters) | ✅ 5475 ms (36 iters) | ≈ +0.1% (±24.5%) | |
| fj-kmeans | 21 | ✅ 2687 ms (70 iters) | ✅ 2761 ms (68 iters) | 🔴 +2.8% | — / — |
| fj-kmeans | 25 | ✅ 2810 ms (66 iters) | ✅ 2805 ms (66 iters) | ≈ -0.2% (±2.6%) | — / — |
| future-genetic | 21 | ✅ 2047 ms (91 iters) | ✅ 2116 ms (88 iters) | 🔴 +3.4% | — / — |
| future-genetic | 25 | ✅ 2000 ms (93 iters) | ✅ 2083 ms (89 iters) | 🔴 +4.2% | — / — |
| naive-bayes | 21 | ✅ 1256 ms (136 iters) | ✅ 1253 ms (136 iters) | ≈ -0.2% (±32.8%) | — / — |
| naive-bayes | 25 | ✅ 1016 ms (169 iters) | ✅ 1016 ms (169 iters) | ≈ 0% (±31.3%) | — / — |
| reactors | 21 | ✅ 16870 ms (15 iters) | ✅ 16463 ms (15 iters) | ≈ -2.4% (±7.3%) | — / — |
| reactors | 25 | ✅ 18270 ms (15 iters) | ✅ 18333 ms (15 iters) | ≈ +0.3% (±4.5%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 2 | 1988 / 2024 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 3 | 2263 / 2395 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 6 | 8440 / 8679 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 2 | 8283 / 8410 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 1288 / 1260 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 4 / 4 | 1250 / 1295 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 2 | 2968 / 3003 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 2843 / 2902 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 5 | 3474 / 3510 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 2 | 3495 / 3488 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1726 / 1721 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 3 | 1893 / 1750 | ✅ / ✅ | ✅ / ✅ |
Contributor
Benchmark Results (commit 787f29e)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124811024 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10215 ms (21 iters) | ✅ 10475 ms (21 iters) | ≈ +2.5% (±11.9%) | — / — |
| akka-uct | 25 | ✅ 8843 ms (24 iters) | ✅ 8861 ms (24 iters) | ≈ +0.2% (±10.5%) | — / — |
| finagle-chirper | 21 | ✅ 5986 ms (33 iters) | ✅ 5971 ms (33 iters) | ≈ -0.3% (±25.5%) | |
| finagle-chirper | 25 | ✅ 5515 ms (36 iters) | ✅ 5479 ms (36 iters) | ≈ -0.7% (±24.2%) | |
| fj-kmeans | 21 | ✅ 2693 ms (70 iters) | ✅ 2836 ms (66 iters) | 🔴 +5.3% | — / — |
| fj-kmeans | 25 | ✅ 2820 ms (66 iters) | ✅ 2837 ms (66 iters) | ≈ +0.6% (±2.6%) | — / — |
| future-genetic | 21 | ✅ 2059 ms (90 iters) | ✅ 2070 ms (89 iters) | ≈ +0.5% (±2.6%) | — / — |
| future-genetic | 25 | ✅ 2089 ms (89 iters) | ✅ 2130 ms (87 iters) | ≈ +2% (±2.6%) | — / — |
| naive-bayes | 21 | ✅ 1284 ms (133 iters) | ✅ 1313 ms (131 iters) | ≈ +2.3% (±33%) | — / — |
| naive-bayes | 25 | ✅ 1009 ms (170 iters) | ✅ 988 ms (173 iters) | ≈ -2.1% (±31.4%) | — / — |
| reactors | 21 | ✅ 16540 ms (15 iters) | ✅ 16155 ms (15 iters) | ≈ -2.3% (±7.1%) | — / — |
| reactors | 25 | ✅ 18545 ms (15 iters) | ✅ 18816 ms (15 iters) | ≈ +1.5% (±4.2%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 2 | 1987 / 1990 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 4 | 2238 / 2323 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 8290 / 8529 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 8323 / 8386 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 1 | 1299 / 1277 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1274 / 1299 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 2 / ✅ | 2920 / 2982 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 4 | 2886 / 2864 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 4 / 1 | 3505 / 3499 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1593 / 1669 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 1849 / 1940 | ✅ / ✅ | ✅ / ✅ |
Contributor
Benchmark Results (commit fede524)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124833099 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
Contributor
Benchmark Results (commit ef0f5c6)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124961539 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
Contributor
Benchmark Results (commit 1299316)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124967829 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10336 ms (21 iters) | ✅ 10245 ms (21 iters) | ≈ -0.9% (±11.5%) | — / — |
| akka-uct | 25 | ✅ 8868 ms (24 iters) | ✅ 8789 ms (24 iters) | ≈ -0.9% (±10.7%) | — / — |
| finagle-chirper | 21 | ✅ 5968 ms (33 iters) | ✅ 5971 ms (33 iters) | ≈ +0.1% (±25.8%) | |
| finagle-chirper | 25 | ✅ 5510 ms (36 iters) | ✅ 5537 ms (36 iters) | ≈ +0.5% (±24.5%) | |
| fj-kmeans | 21 | ✅ 2696 ms (69 iters) | ✅ 2822 ms (66 iters) | 🔴 +4.7% | — / — |
| fj-kmeans | 25 | ✅ 2842 ms (66 iters) | ✅ 2746 ms (68 iters) | 🟢 -3.4% | — / — |
| future-genetic | 21 | ✅ 2068 ms (90 iters) | ✅ 2051 ms (90 iters) | ≈ -0.8% (±2.6%) | — / — |
| future-genetic | 25 | ✅ 2134 ms (87 iters) | ✅ 2059 ms (90 iters) | 🟢 -3.5% | — / — |
| naive-bayes | 21 | ✅ 1318 ms (130 iters) | ✅ 1253 ms (136 iters) | ≈ -4.9% (±31.8%) | — / — |
| naive-bayes | 25 | ✅ 998 ms (171 iters) | ✅ 1025 ms (167 iters) | ≈ +2.7% (±32.1%) | — / — |
| reactors | 21 | ✅ 16264 ms (15 iters) | ✅ 16625 ms (15 iters) | ≈ +2.2% (±7.4%) | — / — |
| reactors | 25 | ✅ 17926 ms (15 iters) | ✅ 18917 ms (15 iters) | ≈ +5.5% (±5.6%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 1917 / 1934 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 3 | 2285 / 2434 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 6 / 2 | 8503 / 8628 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 8331 / 8664 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 1287 / 1266 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 1277 / 1290 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 3 | 3037 / 2981 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 2897 / 2943 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 12 | 3507 / 3493 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 5 / 5 | 3446 / 3481 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 4 / ✅ | 1765 / 1665 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 1855 / 1920 | ✅ / ✅ | ✅ / ✅ |
jbachorik
added a commit
that referenced
this pull request
Jul 16, 2026
…racking Addresses review findings from PR #644 (reference chains for surviving live-heap samples): FrontierTable's shared-lock mutation race, a single-byte JFR event size prefix that silently truncates chains longer than 255 bytes, blocking sample-lock retries in writeReferenceChain() now bounded by a shared per-batch deadline with a drop counter, unvalidated referencechains sub-option values, a startThread()/ pthread_kill() race publishing _running before the thread handle is initialized, a null-JNIEnv leak in threadLoop(), releaseSearchTags() now surfacing GetObjectsWithTags() failures so restartSearch() never resets tag state prematurely, resolveLoadedClasses() skipping its per-class scan only when the loaded-class count is unchanged (not just non-decreasing), a spurious COMPLETED state after a failed first-pass FollowReferences call, and removal of a leftover debug helper in ExternalProcessReferenceChainTest. Adds regression test coverage for release-failure and negative-value option paths. Verified via the full ddprof-lib gtestDebug suite (149/149 tasks, 57/57 referenceChains_ut tests). Environment: Datadog workspace Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
jbachorik
force-pushed
the
jb/reference-chains
branch
from
July 16, 2026 11:31
1299316 to
baba070
Compare
Contributor
Benchmark Results (commit baba070)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124983919 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
Contributor
Benchmark Results (commit 35c49c2)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124997895 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10367 ms (21 iters) | ✅ 10371 ms (21 iters) | ≈ +0% (±11.3%) | — / — |
| akka-uct | 25 | ✅ 8917 ms (24 iters) | ✅ 8766 ms (24 iters) | ≈ -1.7% (±10.2%) | — / — |
| finagle-chirper | 21 | ✅ 5986 ms (33 iters) | ✅ 5972 ms (33 iters) | ≈ -0.2% (±25.3%) | |
| fj-kmeans | 21 | ✅ 2801 ms (67 iters) | ✅ 2750 ms (68 iters) | ≈ -1.8% (±2.6%) | — / — |
| fj-kmeans | 25 | ✅ 2842 ms (66 iters) | ✅ 2756 ms (68 iters) | 🟢 -3% | — / — |
| future-genetic | 21 | ✅ 2142 ms (87 iters) | ✅ 2080 ms (89 iters) | 🟢 -2.9% | — / — |
| future-genetic | 25 | ✅ 1991 ms (94 iters) | ✅ 2035 ms (91 iters) | ≈ +2.2% (±2.7%) | — / — |
| naive-bayes | 25 | ✅ 1025 ms (167 iters) | ✅ 1010 ms (169 iters) | ≈ -1.5% (±31.4%) | — / — |
| reactors | 21 | ✅ 16244 ms (15 iters) | ✅ 16474 ms (15 iters) | ≈ +1.4% (±7.9%) | — / — |
| reactors | 25 | ✅ 18288 ms (15 iters) | ✅ 18332 ms (15 iters) | ≈ +0.2% (±3.9%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 4 | 1950 / 1991 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 2303 / 2180 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 6 | 8767 / 8881 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 8034 / 8300 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 1316 / 1287 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 1289 / 1282 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 2 / ✅ | 2889 / 2882 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 1 | 2935 / 2838 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 8 / 2 | 3496 / 3495 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 4 / 4 | 3477 / 3510 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 1642 / 1586 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 3 | 1802 / 1759 | ✅ / ✅ | ✅ / ✅ |
Contributor
Benchmark Results (commit a9fb05f)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125008807 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10309 ms (21 iters) | ✅ 10453 ms (21 iters) | ≈ +1.4% (±11%) | — / — |
| akka-uct | 25 | ✅ 8814 ms (24 iters) | ✅ 8779 ms (24 iters) | ≈ -0.4% (±10.4%) | — / — |
| finagle-chirper | 21 | ✅ 6006 ms (33 iters) | ✅ 6047 ms (33 iters) | ≈ +0.7% (±25.8%) | |
| finagle-chirper | 25 | ✅ 5467 ms (36 iters) | ✅ 5455 ms (36 iters) | ≈ -0.2% (±24.8%) | |
| fj-kmeans | 21 | ✅ 2772 ms (68 iters) | ✅ 2687 ms (70 iters) | 🟢 -3.1% | — / — |
| fj-kmeans | 25 | ✅ 2810 ms (66 iters) | ✅ 2801 ms (66 iters) | ≈ -0.3% (±2.6%) | — / — |
| future-genetic | 21 | ✅ 2087 ms (88 iters) | ✅ 2055 ms (90 iters) | ≈ -1.5% (±2.7%) | — / — |
| future-genetic | 25 | ✅ 2088 ms (89 iters) | ✅ 2046 ms (91 iters) | ≈ -2% (±2.5%) | — / — |
| naive-bayes | 21 | ✅ 1233 ms (139 iters) | ✅ 1257 ms (136 iters) | ≈ +1.9% (±32.9%) | — / — |
| naive-bayes | 25 | 💥 0 ms (0 iters) | 💥 0 ms (0 iters) | — | — / — |
| reactors | 21 | ✅ 16947 ms (15 iters) | ✅ 15936 ms (15 iters) | ≈ -6% (±8.2%) | — / — |
| reactors | 25 | ✅ 18681 ms (15 iters) | ✅ 18731 ms (15 iters) | ≈ +0.3% (±5.3%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 2008 / 2140 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 2058 / 2262 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 3 | 8437 / 8783 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 3 | 8064 / 8228 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 6 / ✅ | 1275 / 1291 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 2 | 1250 / 1261 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 2 | 2876 / 3036 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 2 / ✅ | 2912 / 2869 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 4 | 3492 / 3491 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1892 / 1633 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 1914 / 1937 | ✅ / ✅ | ✅ / ✅ |
3 tasks
Contributor
Benchmark Results (commit 2033006)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125072159 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10363 ms (21 iters) | ✅ 10147 ms (21 iters) | ≈ -2.1% (±11.6%) | — / — |
| akka-uct | 25 | ✅ 8966 ms (24 iters) | ✅ 8873 ms (24 iters) | ≈ -1% (±10.2%) | — / — |
| finagle-chirper | 21 | ✅ 6007 ms (33 iters) | ✅ 5986 ms (33 iters) | ≈ -0.3% (±26.1%) | |
| finagle-chirper | 25 | ✅ 5488 ms (36 iters) | ✅ 5433 ms (36 iters) | ≈ -1% (±24.3%) | |
| fj-kmeans | 21 | ✅ 2726 ms (68 iters) | ✅ 2760 ms (68 iters) | ≈ +1.2% (±2.6%) | — / — |
| fj-kmeans | 25 | ✅ 2848 ms (66 iters) | ✅ 2743 ms (69 iters) | 🟢 -3.7% | — / — |
| future-genetic | 21 | ✅ 2062 ms (90 iters) | ✅ 2047 ms (90 iters) | ≈ -0.7% (±2.6%) | — / — |
| future-genetic | 25 | ✅ 1994 ms (93 iters) | ✅ 2057 ms (90 iters) | 🔴 +3.2% | — / — |
| naive-bayes | 21 | ✅ 1269 ms (135 iters) | ✅ 1258 ms (136 iters) | ≈ -0.9% (±32.5%) | — / — |
| naive-bayes | 25 | ✅ 1012 ms (169 iters) | ✅ 1016 ms (168 iters) | ≈ +0.4% (±31.6%) | — / — |
| reactors | 21 | ✅ 16554 ms (15 iters) | ✅ 16630 ms (15 iters) | ≈ +0.5% (±8.5%) | — / — |
| reactors | 25 | ✅ 18407 ms (15 iters) | ✅ 18308 ms (15 iters) | ≈ -0.5% (±4.1%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 2101 / 2026 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 4 / 3 | 2336 / 2365 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 3 | 8772 / 8497 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 8397 / 8470 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 1 | 1260 / 1274 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 3 | 1303 / 1307 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 2893 / 2882 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 2958 / 2912 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 3540 / 3527 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 5 / 6 | 3458 / 3485 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 1736 / 1754 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1990 / 1872 | ✅ / ✅ | ✅ / ✅ |
jbachorik
force-pushed
the
jb/reference-chains
branch
from
July 17, 2026 11:39
2033006 to
02a0a93
Compare
Contributor
Benchmark Results (commit 02a0a93)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125255579 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
Contributor
Bits has a CI fix ready🟢 Investigated · 🟢 Fix prepared · ⚪ Validation skipped · 🟠 Ready
View in Datadog | Reviewed commit ed6525a · Any feedback? Reach out in #deveng-pr-agent |
Contributor
Benchmark Results (commit 2874885)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/128197217 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10283 ms (21 iters) | ✅ 10369 ms (21 iters) | ≈ +0.8% (±11.2%) | — / — |
| akka-uct | 25 | ✅ 8936 ms (24 iters) | ✅ 8937 ms (24 iters) | ≈ +0% (±10.2%) | — / — |
| finagle-chirper | 21 | ✅ 5998 ms (33 iters) | ✅ 6010 ms (33 iters) | ≈ +0.2% (±25.9%) | |
| finagle-chirper | 25 | ✅ 5472 ms (36 iters) | ✅ 5465 ms (36 iters) | ≈ -0.1% (±24.4%) | |
| fj-kmeans | 21 | ✅ 2641 ms (72 iters) | ✅ 2647 ms (71 iters) | ≈ +0.2% (±2.5%) | — / — |
| fj-kmeans | 25 | ✅ 2845 ms (66 iters) | ✅ 2836 ms (66 iters) | ≈ -0.3% (±2.6%) | — / — |
| future-genetic | 21 | ✅ 2046 ms (91 iters) | ✅ 2102 ms (88 iters) | 🔴 +2.7% | — / — |
| future-genetic | 25 | ✅ 2004 ms (93 iters) | ✅ 2060 ms (90 iters) | 🔴 +2.8% | — / — |
| naive-bayes | 21 | ✅ 1249 ms (137 iters) | ✅ 1274 ms (134 iters) | ≈ +2% (±32.9%) | — / — |
| naive-bayes | 25 | ✅ 1057 ms (162 iters) | ✅ 1015 ms (169 iters) | ≈ -4% (±31.3%) | — / — |
| reactors | 21 | ✅ 15919 ms (15 iters) | ✅ 16127 ms (15 iters) | ≈ +1.3% (±8.6%) | — / — |
| reactors | 25 | ✅ 18579 ms (15 iters) | ✅ 18084 ms (15 iters) | ≈ -2.7% (±6.1%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 3 | 1920 / 1959 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 2376 / 2305 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 5 / 4 | 8575 / 8622 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 2 | 8481 / 8564 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 5 / 1 | 1286 / 1244 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 5 / ✅ | 1282 / 1306 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 2948 / 2925 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 2977 / 2856 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 7 | 3481 / 3513 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 3509 / 3455 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 1631 / 1742 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 1889 / 1893 | ✅ / ✅ | ✅ / ✅ |
…estarts shouldRunPass() ran the first whole-heap walk/tagging pass unconditionally even with no leak candidate to justify it; it now shares canAffordNewSearch() with the restart gate.
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 65 out of 71 changed files in this pull request and generated no new comments.
Suppressed comments (10)
utils/sweep-refchains-budgets.sh:94
- These
grep | awkpipelines can exit the script underset -euo pipefailwhen there are no safepoint matches (grep returns 1). The intent is to fall back to the JDK<=16 format and ultimately default to zeros, so both pipelines should tolerate "no matches" without aborting the sweep.
utils/sweep-refchains-all.sh:83 - These
grep | awkpipelines can exit the script underset -euo pipefailwhen there are no safepoint matches (grep returns 1). Since the code intends to fall back to the alternate format and default to zeros, both pipelines should be made non-fatal on empty matches.
utils/sweep-refchains-budgets.sh:81 - With
set -euo pipefail, thesegrep | cutpipelines will cause the whole sweep to exit if the[metrics]line is missing/malformed (grep returns 1). This defeats the later:-0defaults and makes the sweep brittle to transient repro failures.
This issue also appears on line 90 of the same file.
utils/compare-refchains-repro.sh:126
- The report column is labeled "delta (on vs. off)", but throughput delta is currently computed as
offrelative toon(reversed from every other metric). This makes the delta sign semantics inconsistent and easy to misread when comparing metrics.
utils/sweep-refchains-all.sh:76 - With
set -euo pipefail, thesegrep | cutpipelines will terminate the whole OFAT sweep if the[metrics]line is missing/malformed (grep returns 1). That makes the sweep brittle and prevents the later:-0defaults from taking effect.
This issue also appears on line 79 of the same file.
utils/compare-refchains-repro.sh:104
- With
set -euo pipefail, thesegrep | awkpipelines will exit the whole script when there are no safepoint matches (grep returns 1), so the fallback format and the:-0defaults never run. Making the pipelines non-fatal on empty matches will keep the comparison script robust.
ddprof-test/src/test/java/com/datadoghq/profiler/test/ProfilerTestRunner.java:121 - Splitting tag lists on
,without trimming whitespace means-Dtest.tags.include=slow, e2ewill try to match a tag named " e2e". Using a whitespace-tolerant split avoids surprising "tag not found" behavior.
ddprof-test/src/test/java/com/datadoghq/profiler/referencechains/ReferenceChainTestSeamsTest.java:141 assertTrue(!target.equals(null))is a confusing way to keeptargetlive, and it's easy for readers (and potentially the JIT) to treat as a redundant constant-true check. Prefer an explicit null assertion to both document intent and keep the last use obvious.
ddprof-lib/src/main/cpp/flightRecorder.cpp:2112rootKindName()hard-codes numericjvmtiHeapReferenceKindvalues (8, 21, 22, ...). This is brittle and obscures intent; using the JVMTI enum constants avoids mismatches if values ever change and makes the mapping self-documenting.
static const char *rootKindName(u8 root_kind) {
switch (root_kind) {
case 8:
return "static_field";
case 21:
ddprof-lib/src/main/cpp/flightRecorder.h:483
- This comment says
recordReferenceChain()is called fromReferenceChainTracker::pollWatchedTargets(), but the current implementation writes chains fromProfiler::dump()after snapshotting the resolved-chain cache. The comment should match the actual emission path to avoid misleading future changes/debugging.
// Mirrors recordReferenceChainAbandoned() above exactly, for
// ReferenceChainEvent instead. Called from Profiler::writeReferenceChain()
// (profiler.cpp), itself called from
// ReferenceChainTracker::pollWatchedTargets() (referenceChains.cpp) for
// each chain event discovered this poll cycle - unlike
Contributor
Benchmark Results (commit 1808111)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/128599211 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10311 ms (21 iters) | ✅ 10219 ms (21 iters) | ≈ -0.9% (±11.2%) | — / — |
| akka-uct | 25 | ✅ 8850 ms (24 iters) | ✅ 8955 ms (24 iters) | ≈ +1.2% (±10.2%) | — / — |
| finagle-chirper | 21 | ✅ 6002 ms (33 iters) | ✅ 5998 ms (33 iters) | ≈ -0.1% (±25%) | |
| finagle-chirper | 25 | ✅ 5461 ms (36 iters) | ✅ 5463 ms (36 iters) | ≈ +0% (±24.3%) | |
| fj-kmeans | 21 | ✅ 2718 ms (68 iters) | ✅ 2645 ms (71 iters) | 🟢 -2.7% | — / — |
| fj-kmeans | 25 | ✅ 2790 ms (67 iters) | ✅ 2833 ms (66 iters) | ≈ +1.5% (±2.6%) | — / — |
| future-genetic | 21 | ✅ 2134 ms (87 iters) | ✅ 2113 ms (87 iters) | ≈ -1% (±2.8%) | — / — |
| future-genetic | 25 | ✅ 2034 ms (92 iters) | ✅ 2051 ms (91 iters) | ≈ +0.8% (±2.6%) | — / — |
| naive-bayes | 21 | ✅ 1289 ms (133 iters) | ✅ 1262 ms (136 iters) | ≈ -2.1% (±31.9%) | — / — |
| naive-bayes | 25 | ✅ 1042 ms (164 iters) | ✅ 945 ms (180 iters) | ≈ -9.3% (±30.1%) | — / — |
| reactors | 21 | ✅ 16185 ms (15 iters) | ✅ 16119 ms (15 iters) | ≈ -0.4% (±8.4%) | — / — |
| reactors | 25 | ✅ 18862 ms (14 iters) | ✅ 18581 ms (15 iters) | ≈ -1.5% (±4.7%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 2 | 1960 / 1948 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 2196 / 2217 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 3 | 8464 / 8583 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 1 | 8078 / 8308 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 1296 / 1257 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 2983 / 3003 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 5 / 2 | 2887 / 2916 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 5 | 3485 / 3490 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 6 / 5 | 3460 / 3473 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 3 | 1618 / 1575 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1598 / 1920 | ✅ / ✅ | ✅ / ✅ |
…ection Fixes a false-positive see-saw pattern triggering reference-chain hunts without a real longer-term generation-count trend. Also fixes lock-free heap-floor ring memory ordering, gitignores review-tooling/OS cruft, and hardens staleLeaf_ut.cpp to actually exercise the counted-slot invariant.
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 68 out of 75 changed files in this pull request and generated no new comments.
Suppressed comments (8)
utils/run-chaos-harness.sh:209
- Same
findprecedence issue as the tcmalloc branch: without grouping,-maxdepth 4does not apply to the-name 'libjemalloc.dylib'branch. Group the name patterns so the search bounds are consistent and predictable.
ddprof-test/src/test/java/com/datadoghq/profiler/test/ProfilerTestRunner.java:121 - Same parsing issue as include-tags: whitespace and empty entries can cause unexpected behavior (e.g.
-Dtest.tags.exclude=slow, flakywon’t excludeflakybecause the tag becomes " flaky"). Trim/filter the list before building the TagFilter.
utils/sweep-refchains-budgets.sh:81 - With
set -euo pipefail, these pipelines will terminate the whole sweep when the stdout metrics line is missing or incomplete (e.g., grep returns exit code 1), even though the script is otherwise designed to default missing metrics to 0 later. The compare script already guards these extractions; this sweep should do the same so a single failed run doesn’t abort the entire matrix.
utils/sweep-refchains-all.sh:76 - Same issue as sweep-refchains-budgets.sh: with
set -euo pipefail, these metric-extraction pipelines will abort the whole OFAT sweep when the[metrics]line is missing or doesn’t contain a particular key. That turns one flaky run into a failed sweep instead of just recording zeros for that point.
utils/compare-refchains-repro.sh:124 - The throughput delta is computed with swapped arguments, which makes the percentage depend on the on value (denominator) instead of the off/baseline value. For example off=100, on=80 reports +25% instead of the expected +20% drop. Compute the throughput cost relative to the off baseline so the reported percentage matches the other metrics’ convention.
ddprof-test/src/test/java/com/datadoghq/profiler/referencechains/ReferenceChainTestSeamsTest.java:141 assertTrue(!target.equals(null))is an unusual way to keep a variable reachable and can invoke user-definedequalsif the type changes in the future. The local variable being in-scope is already sufficient; if you want an explicit reachability touch, use a null check instead.
utils/run-chaos-harness.sh:196- This
findcommand mixes-maxdepthwith-owithout grouping, so-maxdepth 4only applies to the first-namebranch. That can unexpectedly match a deep libtcmalloc.dylib elsewhere and makes the search less predictable. Wrap the-name ... -o -name ...in parentheses so-maxdepthapplies to both patterns.
This issue also appears on line 209 of the same file.
ddprof-test/src/test/java/com/datadoghq/profiler/test/ProfilerTestRunner.java:117
- The tag list parsing doesn’t trim whitespace or drop empty entries. A common value like
-Dtest.tags.include=slow, e2ewill try to match the tag " e2e" (leading space) and silently run nothing. Trim each tag (and ignore empties) before passing to JUnit’s TagFilter.
This issue also appears on line 118 of the same file.
…re docs
symbols_linux.cpp: compute _vaddr_diff via integer subtraction instead of
pointer arithmetic, avoiding a null-pointer UB that UBSan flags whenever a
non-PIE binary's PT_LOAD segment has p_vaddr == load base. Surfaced by the
earlier staleLeaf_ut.cpp fix now actually exercising Libraries::updateSymbols().
LiveHeapReferenceChains.md / reference-chains-collection-summary.md: correct
several claims that drifted from the shipped implementation (stale "Design
proposal" status, non-existent dedup set, understated budget-borrowing clamp,
misattributed PidController triple, removed function name, wrong enum name,
and drainPendingChainEvents()'s actual caller).
Drop the now-superseded planning/proposal docs from git tracking (kept
locally, not committed): JVMTI-IncrementalHeapWalk-Proposal.md and the
LiveHeapReferenceChains-{BenchmarkPlan,ImplementationPlan,RemainingWorkPlan,
VMStructsWalkDesign,VMStructsWalkImplementationPlan}.md docs all describe
approaches or work that were superseded or already shipped differently.
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 63 out of 70 changed files in this pull request and generated no new comments.
Suppressed comments (12)
utils/sweep-refchains-budgets.sh:93
- With
set -euo pipefail, thisgrep | grep | grep | awkpipeline will exit the script when there are no matching safepoint lines (grep returns 1). Since the code explicitly supports falling back to the JDK<=16 format when empty, the command substitution itself should not be fatal.
utils/sweep-refchains-budgets.sh:97 - Same pipefail issue as the primary safepoint parser: if the fallback pattern doesn’t match, this command substitution will exit the script before
stw_statscan be tested for emptiness.
utils/sweep-refchains-all.sh:82 - With
set -euo pipefail, this safepoint parser pipeline will terminate the entire sweep when there are no matches. Since the code already intends to treat “no matches” as “try the fallback format”, the command substitution should be non-fatal.
utils/sweep-refchains-all.sh:86 - Same pipefail issue as the primary safepoint parser: if the fallback pattern doesn’t match, the command substitution exits the script before
stw_statscan be checked for emptiness.
utils/run-chaos-harness.sh:209 - Same
findprecedence issue as the tcmalloc case: without parentheses,-maxdepth 4won’t apply to the second-namebranch. This can unintentionally select a deep/slow match.
ddprof-test/src/test/java/com/datadoghq/profiler/test/ProfilerTestRunner.java:121 - Same issue for excluded tags: whitespace and empty entries can lead to tags not being excluded as intended (or errors). Trim and filter before applying TagFilter.
utils/sweep-refchains-budgets.sh:81 - Because the script runs with
set -euo pipefail, thesegrep | cutpipelines will cause the whole sweep to exit if the[metrics]line is missing or malformed (grep returns 1). The later${var:-0}defaults won’t be reached. Make the extraction pipelines non-fatal (as compare-refchains-repro.sh already does).
This issue also appears in the following locations of the same file:
- line 90
- line 95
utils/sweep-refchains-all.sh:76
- Because the script runs with
set -euo pipefail, thesegrep | cutpipelines will exit the whole OFAT sweep if the[metrics]line is missing/malformed. The later${var:-0}defaults won’t be reached; make the parsing non-fatal.
This issue also appears in the following locations of the same file:
- line 79
- line 84
utils/compare-refchains-repro.sh:124
- The throughput delta is labeled as “delta (on vs. off)” but is currently computed as
(off - on) / onby swapping arguments topct_delta. This makes the magnitude use the wrong denominator and is inconsistent with the column heading and other metrics.
utils/run-chaos-harness.sh:196 - This
findexpression mixes-maxdepthwith-owithout parentheses, so-maxdepth 4only applies to the first-namebranch and the second branch can match at any depth. Group the name predicates so maxdepth applies uniformly.
This issue also appears on line 209 of the same file.
ddprof-test/src/test/java/com/datadoghq/profiler/test/ProfilerTestRunner.java:117
- Splitting tag lists with
includeTags.split(",")does not trim whitespace and can yield empty tag names (e.g. trailing comma). JUnit’s TagFilter treats tags literally, so this can silently not match or throw. Trim and drop empty entries before applying the filter.
This issue also appears on line 118 of the same file.
utils/refchains-report.py:46
- If the sweep CSV contains only a header row (e.g. REFCHAINS_SWEEP_VALUES is empty, or all runs failed before writing metrics),
rowswill be empty andmin(xs)/max(xs)will raiseValueError. Handle the empty-data case so the report can still be generated (even if it’s just an empty chart).
# Conflicts: # .github/workflows/nightly.yml # .github/workflows/release-validated.yml # ddprof-lib/src/main/cpp/counters.h # ddprof-lib/src/main/cpp/javaApi.cpp # ddprof-lib/src/main/java/com/datadoghq/profiler/JavaProfiler.java # ddprof-stresstest/src/chaos/java/com/datadoghq/profiler/chaos/Main.java
Contributor
Scan-Build Report
Bug Summary
Reports
|
||||||||||||||||||||||||||||||||||||
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 52 out of 59 changed files in this pull request and generated no new comments.
Suppressed comments (5)
utils/sweep-refchains-budgets.sh:81
- With
set -euo pipefail, these metric extractions will terminate the whole sweep if the[metrics]line is missing or malformed (thegrep -oE ...in the pipeline exits 1). Sincemetrics_lineis already best-effort (grep ... || true), the per-field parsing should also be best-effort and default to empty/0 instead of failing the entire run.
utils/sweep-refchains-all.sh:76 - With
set -euo pipefail, these parsing pipelines will cause the entire sweep to exit if the[metrics]summary line is missing or doesn't match (becausegrep -oE ...returns exit code 1 andpipefailpropagates it). Sincemetrics_lineis already optional here, the per-field parsing should also tolerate missing matches and fall back to empty/0.
utils/compare-refchains-repro.sh:125 - The script says this column is "delta (on vs. off)" and
pct_delta()is defined as (on-off)/off, but throughput is currently computed aspct_delta(off_entries, on_entries), which flips the sign semantics relative to every other metric and to the table header. This makes the output easy to misread (positive throughput delta here means lower throughput with refchains on). Consider computing throughput delta consistently as on-vs-off and document that for throughput a negative delta is worse (since higher is better).
ddprof-test/src/test/java/com/datadoghq/profiler/referencechains/ReferenceChainTestSeamsTest.java:145 assertTrue(!target.equals(null))is a no-op assertion (it will always pass) and it’s a confusing way to keep a local variable reachable. If you want an explicit reachability anchor, use a direct null-check; otherwise this line can be removed.
doc/reference-chains-design.md:5- This doc still has
**Jira:** TBD, but the PR is explicitly tied to PROF-15341. Updating the document’s Jira field makes it easier to trace the design back to its originating ticket.
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 52 out of 59 changed files in this pull request and generated no new comments.
Suppressed comments (12)
utils/sweep-refchains-budgets.sh:93
- Because the script runs with
set -euo pipefail, thisstw_stats=$(grep ... | ... | awk ...)assignment will abort the whole sweep when the first grep finds no matches (exit 1) — making the intended fallback format branch unreachable.
utils/sweep-refchains-budgets.sh:97 - Same issue as the JDK17+-format branch: without making the pipeline non-fatal, a missing match will terminate the script (and potentially skip emitting a row) instead of producing the documented 0/empty defaults.
utils/sweep-refchains-all.sh:82 - This
stw_stats=$(grep ... | ... | awk ...)pipeline can cause the sweep to exit early underset -euo pipefailwhen there are no matches for the JDK17+-style safepoint format (grep exit 1). That makes the fallback branch unreachable.
utils/sweep-refchains-all.sh:87 - Same pipefail/errexit issue as the JDK17+-format branch: if this grep finds no matches, the command substitution can terminate the script instead of producing default 0 STW metrics.
utils/compare-refchains-repro.sh:103 - Same issue as the JDK17+-format branch: without making this pipeline non-fatal, a missing match can abort the whole comparison run instead of falling back to
0 0 0 0.
utils/sweep-refchains-budgets.sh:81 - With
set -euo pipefail, these metric-extraction pipelines will cause the script to exit if the[metrics]line is missing or a field regex doesn’t match (grep returns non-zero + pipefail). The intent appears to be “default to 0 and keep going” (as in compare-refchains-repro.sh), so the pipelines should be made non-fatal.
This issue also appears in the following locations of the same file:
- line 90
- line 95
utils/sweep-refchains-all.sh:76
- With
set -euo pipefail, these grep-based extractions will terminate the whole OFAT sweep if the[metrics]line or any expected key is missing. The rest of the function already defaults missing values to 0, so these pipelines should be made non-fatal to match that contract.
This issue also appears in the following locations of the same file:
- line 79
- line 84
utils/compare-refchains-repro.sh:99
- With
set -euo pipefail, thisstw_stats=$(grep ... | ... | awk ...)assignment can terminate the script when the JDK17+-style safepoint format isn’t present (grep exit 1 under pipefail). That makes the intended fallback format branch unreachable.
This issue also appears on line 101 of the same file.
utils/refchains-report.py:131
- If the CSV has only a header (or is empty),
rowswill be empty andrender_chart()will crash onmin(xs)/max(xs). It’s better to fail gracefully and still write an HTML file explaining there was no data.
utils/refchains-report-multi.py:66 - If the input CSV is empty (or contains only headers),
rowsis empty and later code will crash (e.g.,rows[0],min(xs_num)). Writing a minimal HTML report and returning makes failures much easier to diagnose.
doc/reference-chains-design.md:5 - The PR and description are tied to PROF-15341, but the design doc header still says
**Jira:** TBD, which makes it harder to trace provenance later.
ddprof-test/src/test/java/com/datadoghq/profiler/referencechains/ReferenceChainTestSeamsTest.java:145 assertTrue(!target.equals(null))is a confusing way to keep the object reachable (and reads like it’s testingequals()semantics). A direct null check expresses the intent without invokingequals().
Three more call sites had the same class of bug as the earlier calcVirtualLoadAddress() fix: adding a non-zero offset directly to a pointer that base()/at() can legitimately return as NULL for a non-PIE (ET_EXEC) image or a p_vaddr==0 segment. Compute each in integer space and cast once, matching loadSymbolTable()'s existing correct pattern: - addRelocationSymbols()'s call site (.plt base) and its two internal base + r_offset relocation-import computations. - parseDwarfInfo()'s LOAD-segment end-address computation.
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 52 out of 59 changed files in this pull request and generated no new comments.
Suppressed comments (9)
utils/sweep-refchains-budgets.sh:93
- The
stw_stats=$(grep ... | ... | awk ...)pipelines can also abort the sweep underset -euo pipefailwhen grep finds no matches (exit=1). That prevents the intended fallback to the JDK<=16 log format and makes the script sensitive to minor unified-logging format differences.
utils/sweep-refchains-all.sh:82 - The
stw_stats=$(grep ... | ... | awk ...)pipelines can abort the script underset -euo pipefailif grep finds no matches (exit=1), preventing the fallback parsing path and making the sweep brittle across JDK unified-logging variations.
utils/sweep-refchains-budgets.sh:81 - With
set -euo pipefail, these metric-extraction pipelines will terminate the whole sweep if the[metrics]line is missing/malformed (grep exits 1 -> pipeline failure). That makes the sweep brittle in the presence of a failed/aborted JVM run (where you’d prefer the row to default to 0s and continue).
This issue also appears on line 89 of the same file.
utils/sweep-refchains-all.sh:76
- With
set -euo pipefail, these pipelines will terminate the entire OFAT sweep if the[metrics]line is missing/malformed (grep exits 1 -> pipeline failure). It’s better to tolerate missing metrics (default to 0s) so failed points don’t abort the whole run matrix.
This issue also appears on line 78 of the same file.
utils/compare-refchains-repro.sh:125
- The report table header says the delta column is “on vs. off”, but throughput delta is computed as
pct_delta(off, on), which flips the sign/denominator relative to every other metric. This makes throughput’s delta inconsistent and easy to misinterpret.
ddprof-lib/src/main/cpp/arguments.h:39 - This comment references design/plan docs that are not present in the repository (
doc/architecture/LiveHeapReferenceChains-ImplementationPlan.md,-BenchmarkPlan.md,-RemainingWorkPlan.md). That makes the in-code rationale hard to follow and creates dead links.
// Every constant below is a provisional default pending empirical
// tuning (see doc/architecture/LiveHeapReferenceChains-ImplementationPlan.md)
// - none of these values are backed by a benchmark run against this
// codebase. Each is chosen conservatively from cited precedent or from the
// shape of an existing, already-tuned subsystem, per the rationale below;
utils/refchains-report.py:46
render_chart()callsmin(xs)/max(xs)unconditionally, which throws on an empty CSV (e.g., if the sweep exits early or is invoked with an empty value list). The caller already toleratesrows == []for the table section, so charts should also handle empty input gracefully.
ddprof-test/src/test/java/com/datadoghq/profiler/referencechains/ReferenceChainTestSeamsTest.java:145assertTrue(!target.equals(null))is an unusual way to keep the reference alive; it also reads like a (misguided) null-check. Using a direct null check is clearer and still creates a use oftargetto keep it reachable to this point.
ddprof-lib/src/main/cpp/callTraceStorage.cpp:7#include <unistd.h>appears to be unused in this file (nousleep,getpid, etc.). Keeping includes minimal helps avoid portability surprises and reduces compile churn.
#include <unistd.h>
Contributor
Benchmark Results (commit 7fddcbc)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/128879382 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 25 | ✅ 8934 ms (24 iters) | ✅ 8841 ms (24 iters) | ≈ -1% (±9.8%) | — / — |
| finagle-chirper | 21 | ✅ 5987 ms (33 iters) | ✅ 5938 ms (33 iters) | ≈ -0.8% (±25.6%) | |
| finagle-chirper | 25 | ✅ 5449 ms (36 iters) | ✅ 5536 ms (36 iters) | ≈ +1.6% (±24.6%) | |
| fj-kmeans | 21 | ✅ 2758 ms (68 iters) | ✅ 2742 ms (68 iters) | ≈ -0.6% (±2.7%) | — / — |
| fj-kmeans | 25 | ✅ 2860 ms (66 iters) | ✅ 2779 ms (68 iters) | 🟢 -2.8% | — / — |
| future-genetic | 21 | ✅ 2157 ms (87 iters) | ✅ 2065 ms (90 iters) | 🟢 -4.3% | — / — |
| future-genetic | 25 | ✅ 2040 ms (91 iters) | ✅ 2128 ms (87 iters) | 🔴 +4.3% | — / — |
| naive-bayes | 21 | ✅ 1271 ms (135 iters) | ✅ 1276 ms (135 iters) | ≈ +0.4% (±32.4%) | — / — |
| naive-bayes | 25 | ✅ 1020 ms (168 iters) | ✅ 1014 ms (169 iters) | ≈ -0.6% (±31.5%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 4 | 2302 / 2364 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 7 | 8704 / 8781 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 8443 / 8475 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 5 / 1 | 1265 / 1284 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 5 | 1285 / 1298 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 4 | 3016 / 2863 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 3 | 2845 / 2988 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 3539 / 3522 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 4 / 2 | 3523 / 3496 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 3 | 1578 / 1605 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 1814 / 1799 | ✅ / ✅ | ✅ / ✅ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?:
Implements reference-chain reconstruction for live-heap samples that survive
past their allocation window (PROF-15341). A
ReferenceChainTrackerruns adedicated BFS thread that tags reachable objects with JVMTI object tags,
walks the heap incrementally across GC epochs, and records the referrer-type
chain back toward a GC root (labelled with its GC-root kind). It is bridged
to
LivenessTracker::selectLeakCandidates()'s population-slope leakdetection: when a still-live sampled object starts to look leak-shaped, its
chain is reconstructed from the frontier table and emitted to JFR.
flowchart TD LT["LivenessTracker::selectLeakCandidates (population-slope ranking)"] -->|"ranked klass candidates"| PWT["ReferenceChainTracker::pollWatchedTargets"] BFS["BFS thread: threadLoop"] -->|"shouldRunPass gate: GC epoch advanced or cadence elapsed"| RP["runPass"] RP -->|"every pass"| MW["runPassManualWalk: IterateOverReachableObjects seeds roots"] MW --> EF["expandFrontier: batched array-holder FollowReferences"] EF --> FT["FrontierTable: FRONTIER / EXPANDED / EDGE / ABANDONED"] FT -->|"tag already set on a leak-candidate instance"| PWT PWT -->|"buildChainEvent"| RC["cacheResolvedChain: one entry per klass id, cap 256"] RC -->|"Profiler::dump, snapshot without clearing"| DR["drainPendingChainEvents"] DR --> JFR["datadog.ReferenceChain / datadog.ReferenceChainAbandoned"]Heap-walk mechanism (the main design decision in this branch): every pass
takes a bounded manual walk (
runPassManualWalk()) — pure JVMTI(
IterateOverReachableObjectsfor root enumeration +expandFrontier()forincremental frontier expansion), resumable across passes, and bounded so its
safepoint pauses stay small on every collector, including ZGC. The walk
issues only JVMTI heap calls, which run inside the
VM_HeapWalkOperationsafepoint and honor ZGC's load barriers, so concurrent relocation cannot
corrupt it — it reads no raw oop.
expandFrontier()'s batching, concretely: instead of oneFollowReferencescall per frontier entry, up to
budgetpending tags are resolved in oneGetObjectsWithTagscall, packed into a single JNI array (holder), andexpanded via exactly one
FollowReferences(initial_object = holder)call —so one BFS level is discovered per VM-safepoint operation rather than one
safepoint per entry:
sequenceDiagram participant BFS as BFS thread participant JVMTI as JVMTI participant JNI as JNI holder array participant CB as heapReferenceCallback BFS->>BFS: pull up to budget tags from front of _pending_expand BFS->>JVMTI: GetObjectsWithTags, resolve which tags are still live JVMTI-->>BFS: live jobject references BFS->>JNI: EnsureLocalCapacity, then NewObjectArray to build holder alt exception, EnsureLocalCapacity failure, or null holder BFS->>BFS: ctx.truncated = true, retry this batch next pass else holder built successfully BFS->>JNI: SetObjectArrayElement per resolved object BFS->>JVMTI: FollowReferences, initial_object = holder JVMTI->>CB: heapReferenceCallback per outgoing edge CB-->>JVMTI: descend only for batch_tags boundary objects JVMTI-->>BFS: one BFS hop expanded for the whole batch BFS->>BFS: markExpanded, admitObject appends children to _pending_expand endexpandFrontier()'s JNI/JVMTI error handling is defensive by construction: anull
holder, a pending JNI exception afterNewObjectArray/SetObjectArrayElement, or anEnsureLocalCapacityfailure all setctx.truncated = true(retry next pass) instead of marking the batchpermanently
EXPANDED; a failedjava/lang/Objectclass resolution withpending work also forces
truncated = true, sorunPass()cannot mistakeit for
SearchState::COMPLETED.Termination and pacing — an unbounded traversal could otherwise stall a
GC safepoint or run forever:
stateDiagram-v2 direction LR [*] --> RUNNING RUNNING --> COMPLETED: frontier drained,<br/>no truncation this pass RUNNING --> ABANDONED: frontier-size cap hit,<br/>or wall-clock TTL exceeded<br/>with work still pending COMPLETED --> RUNNING: restartSearch ABANDONED --> RUNNING: restartSearch note right of ABANDONED SearchAbandonReason (FRONTIER_CAP/TTL) records which cutoff fired, for the T_REFERENCE_CHAIN_ABANDONED JFR event end note note right of RUNNING restartSearch only fires once a leak candidate is seen and PainBudget allows it end notetreated as immediate search abandonment, not per-pass truncation) —
configured via
referencechains=true:hops=N:budget=N:ttl=N:framecap=N(plus
firstpassbudget=Nfor the first pass's own budget). Negativehops/budget/framecapvalues are floored (an unfloored negativehopswould otherwise wrap to ~4e9 as au32, silently disabling the capit's meant to enforce); all three are also ceiling-clamped.
updatePacing(),pausetarget=Nms)adapts the effective per-pass budget and cadence toward a configured
safepoint-pause SLO.
restarted walk is very likely to still find the object) plus a
PainBudgetcooldown (painbudget=N, 0-100) — a leaky bucket overcumulative safepoint time spent, so a cheap search can restart sooner than
an expensive one. This closes a structural gap where a one-shot walk could
finish before population-trend detection had accumulated enough GC epochs
to flag a candidate, leaving anything allocated afterward permanently
undiscoverable.
LivenessTracker::selectLeakCandidates()) requiresa klass's population growth to clear both a magnitude bar and a floor-rise
bar for
LEAK_TREND_HYSTERESIS_BASE(5) consecutive qualifying epochs —lowered to
LEAK_TREND_HYSTERESIS_CORROBORATED(3) when the aggregatepost-GC heap floor is also rising — rather than trusting a single epoch's
positive slope. This closes a false-positive gap where an oscillating
("see-saw") population could otherwise trigger a search with no real
longer-term growth.
JFR persistence across dumps: a resolved chain is cached per klass id
(
_resolved_chains, capped at 256 entries, drop-not-evict once full — seeREFERENCE_CHAIN_EVENTS_DROPPED) and re-stamped into every subsequent dumpthe sample survives into (
drainPendingChainEvents()snapshots withoutclearing, mirroring how
LivenessTrackerre-emits live-object samples), so along-lived leak's chain is present in each JFR chunk rather than only the
chunk active when it was first reconstructed. Verified end-to-end in
ExternalProcessReferenceChainTest.Motivation:
PROF-15341 — give live-heap samples that survive long enough to look
leak-shaped an actual referrer chain, not just "this object is still alive",
without regressing safepoint-pause behaviour on low-pause collectors.
Additional Notes:
See
doc/reference-chains-design.mdanddoc/reference-chains-collection-summary.mdfor the full design and leak-detection mechanism. Per-pass
budget/hop-cap/TTL/frontier-cap/pause-target/pain-budget defaults are round,
unbenchmarked placeholders pending a measurement pass — no JMH harness for
this feature exists yet, though
utils/has repro, parameter-sweep, andJFR-report shell/Python tooling used to characterise pause behaviour ad hoc.
Earlier planning/proposal docs (implementation plan, remaining-work plan,
benchmark plan, and an alternative VMStructs-walk design) described approaches
or work that were superseded or shipped differently than planned, so they were
dropped from this PR (kept locally, not committed) rather than left to drift
further from the code.
How to test the change?:
./gradlew :ddprof-lib:gtestDebug— native unit suite, includingReferenceChainsTest/ReferenceChainsBfsTest/ReferenceChainsTagTest/FrontierTableTest/PollWatchedTargetsTest/ResolvedChainCacheTest,PainBudgetTest,SearchRestartTest(restart gate),ReferenceChainJfrRoundtripTest(JFR encode/decode),ArgumentsTest(
referencechains=sub-option parsing, includinghops/budget/framecapclamping behaviour), and the
LivenessTracker/SelectLeakCandidatesbridging tests.
./gradlew :ddprof-test:testDebug --tests "*ExternalProcessReferenceChainTest*"—genuinely separate-process end-to-end test: runs a real leaking-cache Java
app in a child JVM, asserts the reconstructed chain's leaf class, and
confirms the chain re-emits into three fresh JFR dumps (across-dumps
persistence).
./gradlew :ddprof-test:testDebug --tests "*ReferenceChainTrackingTest*"—in-process coverage for the walk engine, target-selection bridging, and
abandonment reporting.
For Datadog employees:
credentials of any kind, I've requested a security review (run the
dd:platform-security-reviewskill, or file a request via the PSEC review form).
bewairealso runs automatically on every PR.Unsure? Have a question? Request a review!
🤖 Generated with Claude Code