Carve slow/e2e tests into separate testSlow tasks - #700
Merged
Conversation
…re-release Tag long-running tests with @tag("slow"), exclude them from the default test tasks, and add testSlow${Config} Gradle tasks (wired to gtest deps) run only by the nightly and release-validated CI workflows. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Contributor
CI Test ResultsRun: #30831104518 | Commit:
Status Overview
Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled Summary: Total: 32 | Passed: 32 | Failed: 0 Updated: 2026-08-03 16:30:12 UTC |
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Contributor
Benchmark Results (commit 5502908)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/128197042 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10277 ms (21 iters) | ✅ 10216 ms (21 iters) | ≈ -0.6% (±11.6%) | — / — |
| akka-uct | 25 | ✅ 8828 ms (24 iters) | ✅ 8927 ms (24 iters) | ≈ +1.1% (±9.8%) | — / — |
| finagle-chirper | 21 | ✅ 5946 ms (33 iters) | ✅ 6015 ms (33 iters) | ≈ +1.2% (±25.2%) | |
| finagle-chirper | 25 | ✅ 5437 ms (36 iters) | ✅ 5470 ms (36 iters) | ≈ +0.6% (±23.9%) | |
| fj-kmeans | 21 | ✅ 2782 ms (67 iters) | ✅ 2657 ms (70 iters) | 🟢 -4.5% | — / — |
| fj-kmeans | 25 | ✅ 2778 ms (67 iters) | ✅ 2837 ms (66 iters) | ≈ +2.1% (±2.7%) | — / — |
| future-genetic | 21 | ✅ 2080 ms (89 iters) | ✅ 2075 ms (89 iters) | ≈ -0.2% (±2.6%) | — / — |
| future-genetic | 25 | ✅ 1971 ms (93 iters) | ✅ 2056 ms (90 iters) | 🔴 +4.3% | — / — |
| naive-bayes | 21 | ✅ 1241 ms (137 iters) | ✅ 1311 ms (131 iters) | ≈ +5.6% (±33.5%) | — / — |
| naive-bayes | 25 | ✅ 1012 ms (169 iters) | ✅ 1001 ms (171 iters) | ≈ -1.1% (±31.5%) | — / — |
| reactors | 21 | ✅ 17177 ms (15 iters) | ✅ 17395 ms (15 iters) | ≈ +1.3% (±7.5%) | — / — |
| reactors | 25 | ✅ 18447 ms (15 iters) | ✅ 18630 ms (15 iters) | ≈ +1% (±4.3%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 1995 / 2020 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 2 | 2171 / 2126 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 3 | 8768 / 8817 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 8597 / 8311 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 3 | 1268 / 1300 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 2 | 1250 / 1293 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 4 | 2937 / 2949 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 3 | 2872 / 2881 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 4 | 3506 / 3486 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 3502 / 3469 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1769 / 1817 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 1944 / 1918 | ✅ / ✅ | ✅ / ✅ |
jbachorik
marked this pull request as ready for review
July 31, 2026 15:01
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 734072b35d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Contributor
Benchmark Results (commit 734072b)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/128272132 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10402 ms (21 iters) | ✅ 10250 ms (21 iters) | ≈ -1.5% (±10.6%) | — / — |
| akka-uct | 25 | ✅ 8930 ms (24 iters) | ✅ 8826 ms (24 iters) | ≈ -1.2% (±10.4%) | — / — |
| finagle-chirper | 21 | ✅ 5953 ms (33 iters) | ✅ 5935 ms (33 iters) | ≈ -0.3% (±24.6%) | |
| finagle-chirper | 25 | ✅ 5494 ms (36 iters) | ✅ 5473 ms (36 iters) | ≈ -0.4% (±24.8%) | |
| fj-kmeans | 21 | ✅ 2727 ms (68 iters) | ✅ 2776 ms (67 iters) | ≈ +1.8% (±2.7%) | — / — |
| fj-kmeans | 25 | ✅ 2837 ms (66 iters) | ✅ 2828 ms (66 iters) | ≈ -0.3% (±2.6%) | — / — |
| future-genetic | 21 | ✅ 2133 ms (87 iters) | ✅ 2065 ms (90 iters) | 🟢 -3.2% | — / — |
| future-genetic | 25 | ✅ 2009 ms (92 iters) | ✅ 2077 ms (89 iters) | 🔴 +3.4% | — / — |
| naive-bayes | 21 | ✅ 1314 ms (130 iters) | ✅ 1234 ms (138 iters) | ≈ -6.1% (±32%) | — / — |
| naive-bayes | 25 | ✅ 1022 ms (167 iters) | ✅ 1012 ms (168 iters) | ≈ -1% (±31.8%) | — / — |
| reactors | 21 | ✅ 16450 ms (15 iters) | ✅ 15792 ms (15 iters) | ≈ -4% (±7.6%) | — / — |
| reactors | 25 | ✅ 18869 ms (15 iters) | ✅ 18773 ms (15 iters) | ≈ -0.5% (±5.1%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 2009 / 1922 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 2201 / 2301 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 5 / 4 | 8503 / 8481 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 8717 / 8472 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 3 | 1261 / 1255 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 1280 / 1266 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 1 | 2964 / 2991 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 2834 / 2958 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 1 | 3483 / 3477 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 3 | 3463 / 3437 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 1797 / 1623 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 1933 / 1917 | ✅ / ✅ | ✅ / ✅ |
…add copyright header Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 12 out of 12 changed files in this pull request and generated no new comments.
Suppressed comments (1)
ddprof-test/src/test/java/com/datadoghq/profiler/test/ProfilerTestRunner.java:126
- Tag filtering splits on "," but doesn’t trim whitespace or drop empty entries. This makes
-Dtest.tags.include=slow, e2ebehave unexpectedly (second tag becomes " e2e"), and a trailing comma can produce an empty tag. Consider trimming/filtering before building the TagFilter so tag-based CI selection is robust.
String includeTags = System.getProperty("test.tags.include");
if (includeTags != null && !includeTags.isEmpty()) {
requestBuilder.filters(TagFilter.includeTags(includeTags.split(",")));
}
String excludeTags = System.getProperty("test.tags.exclude");
This comment has been minimized.
This comment has been minimized.
|
Bits Code status: ✅ Done Comment @DataDog to request changes |
![dd-octo-sts[bot]](https://avatars.githubusercontent.com/in/1157446?s=60&v=4){kind=small}
Contributor
Benchmark Results (commit 0a04cee)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/128641133 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10304 ms (21 iters) | ✅ 10290 ms (21 iters) | ≈ -0.1% (±10.7%) | — / — |
| akka-uct | 25 | ✅ 8837 ms (24 iters) | ✅ 8907 ms (24 iters) | ≈ +0.8% (±10.8%) | — / — |
| finagle-chirper | 21 | ✅ 6054 ms (33 iters) | ✅ 5964 ms (33 iters) | ≈ -1.5% (±24.6%) | |
| finagle-chirper | 25 | ✅ 5458 ms (36 iters) | ✅ 5398 ms (36 iters) | ≈ -1.1% (±23.7%) | |
| fj-kmeans | 21 | ✅ 2743 ms (68 iters) | ✅ 2654 ms (71 iters) | 🟢 -3.2% | — / — |
| fj-kmeans | 25 | ✅ 2746 ms (69 iters) | ✅ 2809 ms (67 iters) | ≈ +2.3% (±2.8%) | — / — |
| future-genetic | 21 | ✅ 2061 ms (90 iters) | ✅ 2042 ms (90 iters) | ≈ -0.9% (±2.5%) | — / — |
| future-genetic | 25 | ✅ 2050 ms (91 iters) | ✅ 2127 ms (87 iters) | 🔴 +3.8% | — / — |
| naive-bayes | 21 | ✅ 1289 ms (133 iters) | ✅ 1288 ms (133 iters) | ≈ -0.1% (±32.4%) | — / — |
| naive-bayes | 25 | ✅ 1013 ms (169 iters) | ✅ 1008 ms (170 iters) | ≈ -0.5% (±31.3%) | — / — |
| reactors | 21 | ✅ 16078 ms (15 iters) | ✅ 16068 ms (15 iters) | ≈ -0.1% (±8.4%) | — / — |
| reactors | 25 | ✅ 18240 ms (15 iters) | ✅ 18638 ms (15 iters) | ≈ +2.2% (±4.7%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1957 / 2014 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 7 | 2182 / 2240 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 3 | 8697 / 8680 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 8701 / 8234 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 6 / ✅ | 1253 / 1288 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 3 | 1314 / 1277 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 2 | 2976 / 2875 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 2971 / 2902 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 7 / 3 | 3552 / 3468 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 6 | 3479 / 3458 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 1576 / 1555 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1870 / 1857 | ✅ / ✅ | ✅ / ✅ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?:
Tags long-running (45-70s+) tests with
@Tag("slow"), excludes them from the defaulttest${Config}Gradle tasks, and adds newtestSlow${Config}tasks (with the same gtest dependency wiring) that run only in the nightly and pre-release-validated CI workflows.Motivation:
Several tests (reference-chain tracking, class-unload/leak stress tests, dump-while-churning-threads) individually take 45-70+ seconds, inflating the regular per-PR test run. They belong in a slow/e2e suite that runs nightly and before releases, not on every PR.
Additional Notes:
ProfilerTestRunner(the musl Exec-task path) gained-Dtest.tags.include/-Dtest.tags.excludesystem-property-driven tag filtering, mirroring the native GradleTesttask'sincludeTags/excludeTags.testSlow${Config}tasks depend on the samegtest${Config}task as theirtest${Config}sibling, since each is an independent CI job.How to test the change?:
./gradlew :ddprof-test:tasksliststestDebug,testRelease,testSlowDebug,testSlowReleasewith correct descriptions../gradlew -q :ddprof-test:testSlowDebug --dry-runconfirms thegtestDebugtask graph is a dependency.nightly.ymlandrelease-validated.ymlvalidated as valid YAML.For Datadog employees:
credentials of any kind, I've requested a security review (run the
dd:platform-security-reviewskill, or file a request via the PSEC review form).
bewairealso runs automatically on every PR.Unsure? Have a question? Request a review!