ansible: raise jenkins agent heap on mnx x64 workspace machines - #4493
Merged
ryanaslett merged 1 commit intoSep 28, 2026
Merged
Conversation
test-mnx-ubuntu2204-x64-1 (jenkins-workspace-9) hit a JVM heap
OutOfMemoryError in its Jenkins agent process on 2026-09-28, which
killed the JNLP connection thread and caused several node-test-linter
builds to time out and disconnect ('Agent went offline during the
build'). The agent runs with the default -Xmx128m from
roles/jenkins-worker/templates/systemd.service.j2.
Both mnx x64 workspace machines (jenkins-workspace-9 and
jenkins-workspace-10) have 16G of RAM and plenty of headroom, so raise
server_ram to 512m for them.
jenkins-workspace-6 (test-ibm-ubuntu2204-x64-3) is a much smaller
instance (3.8G RAM, 2 vCPUs) and its logs show no JVM heap OOM in over
two months of history, only an unrelated host-level OOM kill of a git
process. Left at the 128m default since there's no evidence it needs
more and the box can't spare the memory.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
richardlau
approved these changes
Sep 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
test-mnx-ubuntu2204-x64-1(jenkins-workspace-9) hit a JVM heapOutOfMemoryErrorin its Jenkins agent process on 2026-09-28:This killed the JNLP connection's reader thread, which the Jenkins controller saw as the agent going offline mid-build. Five
node-test-linterbuilds (and their upstreamnode-test-pull-requestruns) hung for 65–77 minutes before timing out and failing withClosedChannelException/ "Agent went offline during the build".The agent runs with
-Xmx{{ server_ram|default('128m') }}(ansible/roles/jenkins-worker/templates/systemd.service.j2), and no host currently overridesserver_ram, so all jenkins-workspace machines run with a 128m heap.Change
Set
server_ram: 512mfor the two mnx x64 workspace machines (jenkins-workspace-9 and jenkins-workspace-10). Both have 16G RAM and 72 cores, so there's ample headroom.Why not jenkins-workspace-6 too
jenkins-workspace-6(test-ibm-ubuntu2204-x64-3) is a much smaller instance — 3.8G RAM, 2 vCPUs. I checked its journal going back to 2026-07-17 and found no JVM heap OOM there; the only OOM event in that window was an unrelated host-level OOM-killer event that killed agitbuild process (a symptom of the whole box being short on RAM, not the agent's heap ceiling). Raising the agent's heap on that box wouldn't address a problem it's had, and would take RAM away from build processes on an already memory-constrained host, so it's left at the 128m default for now.🤖 Generated with Claude Code