fix(reliability): stop the reliability CI jobs from failing on every run - #697
Conversation
…env bugs Fail fast when the ddprof snapshot jar can't be fetched/unzipped instead of silently launching Java with an empty debug.lib path. Also fixes cgroup v2 CPU counting, the wrong mem_watch.sh path, and the self-referential RUNTIME CI variable. See PROF-15567.
.reliability_job only needed get-versions, so GitLab's DAG scheduling let it race ahead of deploy-artifact and try to fetch the ddprof snapshot before it was published. See PROF-15567.
…ty pom.xml mvn aborted with "Non-readable POM ... input contained no data" every time because it picked up the repo's stale, empty pom.xml before ever attempting resolution. Same workaround already used in download-snapshot-artifacts.sh.
Leftover from an old async-profiler publishing setup; unreferenced, and its presence at the repo root made plain `mvn` invocations fail outright.
![dd-octo-sts[bot]](https://avatars.githubusercontent.com/in/1157446?s=60&v=4){kind=small}
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8756ac7e10
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| timeout: 6h | ||
| variables: | ||
| RUNTIME: "${RUNTIME}" | ||
| # Default; a pipeline-schedule variable named RUNTIME overrides this. |
There was a problem hiding this comment.
This commit modifies .gitlab/reliability/.gitlab-ci.yml, jit_stability_check.sh, and memory_trend_check.sh, but none contains the required Datadog copyright notice; add a current-year notice to each modified file.
AGENTS.md reference: AGENTS.md:L372-L374
Useful? React with 👍 / 👎.
CI Test ResultsRun: #30460342877 | Commit:
Status Overview
Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled Summary: Total: 32 | Passed: 32 | Failed: 0 Updated: 2026-07-29 15:15:46 UTC |
deploy-artifact has when:never on release/* branches, but a scheduled reliability pipeline on such a branch has no branch restriction and would fail to build with a hard needs on a job that doesn't exist.
Benchmark Results (commit 8756ac7)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/127720275 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
Reliability & Chaos Results✅ All reliability & chaos checks passed Pipeline: https://gitlab.ddbuild.io/DataDog/java-profiler/-/pipelines/127732237 |
Benchmark Results (commit 6c41ccd)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/127732630 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
kaahos
left a comment
kaahos
left a comment
There was a problem hiding this comment.
It looks good to me! thanks for the fix!
What does this PR do?:
Fixes several independent bugs that were making the reliability pipeline (
memory_trend_check.sh,jit_stability_check.sh) fail on every run:pom.xmlat the repo root (leftover from an old async-profiler publishing setup) that made everymvninvocation abort with "Non-readable POM ... input contained no data" before it could resolve anything.deploy-artifactdependency to.reliability_job'sneeds:so the job can't start before the ddprof snapshot it depends on has actually been published.mvnoutput) after fetching the ddprof snapshot jar, so a fetch failure is reported clearly instead of unzip/find silently producing empty results downstream.count_cpus()to fall back to cgroup v2'scpuset.cpus.effective(and tonprocif neither cgroup path exists) instead of only reading the cgroup v1 path.mem_watch.shpath, which pointed at a nonexistent relative location.RUNTIME: "${RUNTIME}"default (which only resolved via a pipeline-schedule variable) with an actual default value.Motivation:
The reliability pipeline moved into this repo from
java-profiler-buildon 2026-04-03 and has been failing on every scheduled/manual run since, cascading through unzip/cgroup/library-load/env errors that all trace back to the issues above. See PROF-15567.Additional Notes:
None.
How to test the change?:
Validated by re-running the reliability pipeline on this branch and confirming the ddprof snapshot resolves and the jobs progress past artifact setup. CI on this PR (and the scheduled reliability pipeline) is the real validation, since these scripts only run in that environment.
For Datadog employees:
Unsure? Have a question? Request a review!