Skip to content

fix(reliability): stop the reliability CI jobs from failing on every run #697

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jbachorik merged 7 commits into from
Jul 29, 2026
Merged

fix(reliability): stop the reliability CI jobs from failing on every run #697

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
998588a
fix(ci): harden reliability jobs against missing ddprof artifact and …
jbachorik Jul 29, 2026
fe5cfa5
fix(ci): make reliability job wait for deploy-artifact
jbachorik Jul 29, 2026
494d398
fix(reliability): retry+capture mvn snapshot fetch instead of failing…
jbachorik Jul 29, 2026
3429d0a
fix(reliability): run mvn get from a temp dir to avoid the repo's emp…
jbachorik Jul 29, 2026
4f37a34
chore: remove stale empty pom.xml
jbachorik Jul 29, 2026
8756ac7
cleanup: drop temp-dir mvn workaround now that the stale pom.xml is gone
jbachorik Jul 29, 2026
6c41ccd
fix: make deploy-artifact dependency optional for release branches
jbachorik Jul 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions .gitlab/reliability/.gitlab-ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,13 @@ variables:
stage: reliability
timeout: 6h
variables:
RUNTIME: "${RUNTIME}"
# Default; a pipeline-schedule variable named RUNTIME overrides this.

@chatgpt-codex-connector chatgpt-codex-connector Bot Jul 29, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Add the required Datadog copyright headers

This commit modifies .gitlab/reliability/.gitlab-ci.yml, jit_stability_check.sh, and memory_trend_check.sh, but none contains the required Datadog copyright notice; add a current-year notice to each modified file.

AGENTS.md reference: AGENTS.md:L372-L374

Useful? React with 👍 / 👎.

RUNTIME: "3600"
needs:
- get-versions
- job: deploy-artifact
artifacts: false
Comment thread
jbachorik marked this conversation as resolved.
optional: true
rules:
- if: '$CI_PIPELINE_SOURCE == "schedule"'
when: always
Expand Down Expand Up @@ -39,6 +43,7 @@ variables:
- hs_err.log
- err.log
- out.log
- maven-get.log
reports:
dotenv: build.env
expire_in: 7 days
Expand All @@ -63,7 +68,8 @@ reliability-aarch64:
stage: reliability
timeout: 6h
variables:
RUNTIME: "${RUNTIME}"
# Default; a pipeline-schedule variable named RUNTIME overrides this.
RUNTIME: "3600"
needs:
- get-versions
- job: chaos:build
Expand Down
30 changes: 26 additions & 4 deletions .gitlab/reliability/jit_stability_check.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,15 +15,37 @@ source "/root/.sdkman/bin/sdkman-init.sh" 1>/dev/null 2>/dev/null

sdk install java 21.0.3-tem 1>/dev/null 2>/dev/null

mvn org.apache.maven.plugins:maven-dependency-plugin:2.1:get \
-DrepoUrl=https://central.sonatype.com/repository/maven-snapshots/ \
-Dartifact=com.datadoghq:ddprof:${CURRENT_VERSION} 1>/dev/null 2>/dev/null
DDPROF_JAR="/root/.m2/repository/com/datadoghq/ddprof/${CURRENT_VERSION}/ddprof-${CURRENT_VERSION}.jar"
# Capture mvn's actual output (not just presence/absence of the jar) so a
# genuine failure (auth, 404, blocked network) is visible instead of looking
# like a transient miss.
MVN_GET_LOG="${HERE}/../../maven-get.log"
: > "$MVN_GET_LOG"
for attempt in 1 2 3 4 5 6; do
echo "=== mvn get attempt ${attempt}/6 ===" >> "$MVN_GET_LOG"
mvn org.apache.maven.plugins:maven-dependency-plugin:2.1:get \
-DrepoUrl=https://central.sonatype.com/repository/maven-snapshots/ \
-Dartifact=com.datadoghq:ddprof:${CURRENT_VERSION} >> "$MVN_GET_LOG" 2>&1
[ -f "$DDPROF_JAR" ] && break
echo "ddprof ${CURRENT_VERSION} not yet resolvable (attempt ${attempt}/6), retrying in 20s"
sleep 20
done
if [ ! -f "$DDPROF_JAR" ]; then
echo "FAIL:Could not fetch ddprof ${CURRENT_VERSION} jar" >&2
echo "--- last mvn get output (see maven-get.log artifact for full history) ---" >&2
tail -n 40 "$MVN_GET_LOG" >&2
exit 1
fi

mkdir -p /var/lib/datadog/${CURRENT_VERSION}
rm -rf /var/lib/datadog/${CURRENT_VERSION}/*

unzip -q -d /var/lib/datadog/${CURRENT_VERSION} /root/.m2/repository/com/datadoghq/ddprof/${CURRENT_VERSION}/ddprof-${CURRENT_VERSION}.jar
unzip -q -d /var/lib/datadog/${CURRENT_VERSION} "$DDPROF_JAR"
AGENT_LIB=$(find /var/lib/datadog/${CURRENT_VERSION} -name 'libjavaProfiler.so' | fgrep '/linux-x64/')
if [ -z "$AGENT_LIB" ]; then
echo "FAIL:Could not locate libjavaProfiler.so for linux-x64" >&2
exit 1
fi

echo "Agent lib: ${AGENT_LIB}"
uname -a
Expand Down
42 changes: 36 additions & 6 deletions .gitlab/reliability/memory_trend_check.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,15 @@ ALLOCATOR=${3:-gmalloc}

# Function to count CPUs from ranges and individual numbers
count_cpus() {
local cpus=$(cat /sys/fs/cgroup/cpuset/cpuset.cpus)
local cpuset_file="/sys/fs/cgroup/cpuset/cpuset.cpus"
if [ ! -f "$cpuset_file" ] && [ -f /sys/fs/cgroup/cgroup.controllers ]; then
cpuset_file="/sys/fs/cgroup/cpuset.cpus.effective"
fi
if [ ! -f "$cpuset_file" ]; then
nproc
return
fi
local cpus=$(cat "$cpuset_file")
local cpu_count=0
local IFS=',' # Use comma as delimiter to split ranges and numbers

Expand All @@ -36,15 +44,37 @@ source "/root/.sdkman/bin/sdkman-init.sh" 1>/dev/null 2>/dev/null

sdk install java 21.0.3-tem 1>/dev/null 2>/dev/null

mvn org.apache.maven.plugins:maven-dependency-plugin:2.1:get \
-DrepoUrl=https://central.sonatype.com/repository/maven-snapshots/ \
-Dartifact=com.datadoghq:ddprof:${CURRENT_VERSION} 1>/dev/null 2>/dev/null
DDPROF_JAR="/root/.m2/repository/com/datadoghq/ddprof/${CURRENT_VERSION}/ddprof-${CURRENT_VERSION}.jar"
# Capture mvn's actual output (not just presence/absence of the jar) so a
# genuine failure (auth, 404, blocked network) is visible instead of looking
# like a transient miss.
MVN_GET_LOG="${HERE}/../../maven-get.log"
: > "$MVN_GET_LOG"
for attempt in 1 2 3 4 5 6; do
echo "=== mvn get attempt ${attempt}/6 ===" >> "$MVN_GET_LOG"
mvn org.apache.maven.plugins:maven-dependency-plugin:2.1:get \
-DrepoUrl=https://central.sonatype.com/repository/maven-snapshots/ \
-Dartifact=com.datadoghq:ddprof:${CURRENT_VERSION} >> "$MVN_GET_LOG" 2>&1
[ -f "$DDPROF_JAR" ] && break
echo "ddprof ${CURRENT_VERSION} not yet resolvable (attempt ${attempt}/6), retrying in 20s"
sleep 20
done
if [ ! -f "$DDPROF_JAR" ]; then
echo "FAIL:Could not fetch ddprof ${CURRENT_VERSION} jar" >&2
echo "--- last mvn get output (see maven-get.log artifact for full history) ---" >&2
tail -n 40 "$MVN_GET_LOG" >&2
exit 1
fi

mkdir -p /var/lib/datadog/${CURRENT_VERSION}
rm -rf /var/lib/datadog/${CURRENT_VERSION}/*

unzip -q -d /var/lib/datadog/${CURRENT_VERSION} /root/.m2/repository/com/datadoghq/ddprof/${CURRENT_VERSION}/ddprof-${CURRENT_VERSION}.jar
unzip -q -d /var/lib/datadog/${CURRENT_VERSION} "$DDPROF_JAR"
AGENT_LIB=$(find /var/lib/datadog/${CURRENT_VERSION} -name 'libjavaProfiler.so' | fgrep '/linux-x64/')
if [ -z "$AGENT_LIB" ]; then
echo "FAIL:Could not locate libjavaProfiler.so for linux-x64" >&2
exit 1
fi

echo "Agent lib: ${AGENT_LIB}"
uname -a
Expand All @@ -55,7 +85,7 @@ wget -q -O /var/lib/datadog/dd-java-agent.jar 'https://dtdg.co/latest-java-trace

CONTROL_FILE=".running"
touch $CONTROL_FILE
sh ./benchmarks/steps/mem_watch.sh $CONTROL_FILE ${HERE}/../../memwatch.log &
sh ${HERE}/../benchmarks/steps/mem_watch.sh $CONTROL_FILE ${HERE}/../../memwatch.log &

case $CONFIG in
profiler)
Expand Down
Empty file removed pom.xml
View file
Open in desktop
Empty file.
Loading