Skip to content

Move the Linux CI jobs to OSDC runners - #6321

Draft
huydhn wants to merge 3 commits into
mainfrom
osdc/ci-linux-runners
Draft

huydhn wants to merge 3 commits into
mainfrom
osdc/ci-linux-runners

Conversation

@huydhn

@huydhn huydhn commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

Two changes that have to land together: the runner labels, and the container image they run in.

Labels

From pytorch/pytorch's .github/arc.yaml, applied across the CI and benchmark workflows and to generate_ci_matrix.py's host_machines table, which is where most of them come from:

linux.2xlarge               -> mt-l-x86iavx512-8-64
linux.4xlarge[.memory]      -> mt-l-x86iavx512-16-128
linux.12xlarge[.memory]     -> mt-l-x86iavx512-48-384
linux.24xlarge              -> mt-l-x86iavx512-94-192
linux.24xlarge.memory       -> mt-l-x86iavx512-94-768
linux.arm64.m7g.4xlarge     -> mt-l-arm64g3-16-62
linux.g5.4xlarge.nvidia.gpu -> mt-l-x86aavx2-11-41-a10g

The Graviton 3 choice is preserved — the arm jobs pick that instance for FP16FML, and arm64g3 is the same generation.

Image: amazonlinux:2023 -> almalinux:9

OSDC runs the k8s hook, which streams the workspace into the job container as a tar and extracts it there, rather than the host-side docker cp the EC2 runners used. So the image needs tar and find, and amazonlinux:2023 ships neither:

$ docker run --rm amazonlinux:2023 sh -c 'command -v tar; command -v find'
(neither)

Every job died in prepare-job before its first step — cpToPod timing out thirty times, then find: command not found. Not fixable from a step, since prepare-job runs first.

almalinux:9 is the closest thing to a drop-in:

tar find curl dnf libstdc++
amazonlinux:2023 no no yes yes GLIBCXX_3.4.29
almalinux:9 yes yes yes yes GLIBCXX_3.4.29

Same RPM family so the scripts' package handling is unchanged, and multi-arch so the arm jobs take the same line.

The pytorch/manylinux2_28-builder images were tried first and are the wrong tool here — they are AlmaLinux 8, and their older libstdc++ fails the conda-installed numpy with GLIBCXX_3.4.29 not found as soon as PyTorch is imported.

Scope

ROCm x86 build hosts move with the rest; the MI350 test jobs keep rocm/dev-ubuntu-22.04 and their own hardware. linux.gcp.a100 and linux.aws.h100 are commented-out future work. The wheel pipelines move in #6320.

@huydhn

huydhn commented Sep 17, 2026

Copy link
Copy Markdown
Contributor Author

The label mapping here is right, but these jobs cannot move as-is — the container image is the blocker, and it needs fixing before this lands.

amazonlinux:2023 is missing tar and find, both of which the OSDC k8s hook needs inside the job container:

$ docker run --rm amazonlinux:2023 sh -c 'command -v tar; command -v find'
(neither)

That produces the two failures on this PR:

  • execCpToPod: exec timed out after 300000ms (×30) — the hook streams the workspace in as a tar and extracts it in the container, the same constraint kubectl cp has. No tar, no copy.
  • sh: line 1: find: command not foundprepare-job script failed with exit code 1 — the post-copy hash verification (ACTIONS_RUNNER_COPY_VERIFY_ENABLED=true).

On EC2 this worked because the copy was a docker cp, where the tar work happens host-side and the image never needed either binary.

It cannot be fixed from a workflow step: prepare-job runs before step one, so dnf install findutils tar is too late. It has to be in the image.

Checked the alternatives:

image tar find
ubuntu:22.04, debian:bookworm-slim, almalinux:9 yes yes
quay.io/pypa/manylinux_2_28_x86_64, pytorch/manylinux2_28-builder yes yes
amazonlinux:2 no yes
amazonlinux:2023 no no

Separately, the ubuntu:22.04 jobs get past prepare-job and then fail at Setup Miniconda, after which every script hits conda: command not found. So a base-image swap alone is not enough; the conda bootstrap needs looking at too.

Suggested order: bake an FBGEMM CI image (with tar, findutils and conda preinstalled — which would also stop every job reinstalling the toolchain), then rebase this label swap on top. #6320 and #6319 do not depend on any of this.

Gotcha written up in pytorch/test-infra#8794.

@huydhn
huydhn force-pushed the osdc/ci-linux-runners branch from af5db6d to 784deab Compare September 17, 2026 22:00
@huydhn huydhn changed the title Move the Linux CI jobs to OSDC runners Move the Linux CI jobs to OSDC runners and the manywheel builder images Sep 17, 2026
@huydhn
huydhn force-pushed the osdc/ci-linux-runners branch from 784deab to cbf581a Compare September 17, 2026 23:08
@huydhn huydhn changed the title Move the Linux CI jobs to OSDC runners and the manywheel builder images Move the Linux CI jobs to OSDC runners Sep 17, 2026
@huydhn
huydhn marked this pull request as ready for review September 17, 2026 23:11
@huydhn
huydhn force-pushed the osdc/ci-linux-runners branch from cbf581a to add2049 Compare September 17, 2026 23:14
@huydhn
huydhn marked this pull request as draft September 17, 2026 23:17
@huydhn
huydhn force-pushed the osdc/ci-linux-runners branch 3 times, most recently from 2a21f30 to 818df34 Compare September 17, 2026 23:46
meta-codesync Bot pushed a commit that referenced this pull request Sep 23, 2026
Summary:
X-link: https://github.com/facebookresearch/FBGEMM/pull/3227

`__build_fbgemm_gpu_set_run_multicore` multiplies `Core(s) per socket` by `Socket(s)` from `lscpu`. That reports the **host's** physical topology — fine on a dedicated EC2 runner, wrong in a container on a shared node.

On an OSDC runner with 11 CPUs on a 192-core `g5.48xlarge`, it reads 48 × 2 and builds with `-j 96` inside a 41Gi container: ~0.4GiB per compile job against C++/CUDA TUs that want 1–2GiB each. The CUDA wheel build [OOM-killed](https://github.com/pytorch/FBGEMM/actions/runs/35268173569/job/105360670718) at ninja step 102/655:

```
core = 48
sockets = 2
[BUILD] Set multicore run option for setup.py: -j 96
##[error]Step failed: Container "job" was OOMKilled (exit code 137)
```

`nproc` respects the CPU affinity the process actually has — it printed `11` a few lines earlier in that same log — and is equally correct on EC2, where it returns the whole machine.

`BUILD_PARALLELISM` still overrides, and an unparseable count still leaves `run_multicore` empty, as before.

Split out of #6320 because `fbgemm_gpu_build.bash` is shared: #6320 and #6321 both hit this path, so landing it once avoids a conflicting duplicate in each.

Authored with Claude Code.

Pull Request resolved: #6361

Reviewed By: q10

Differential Revision: D121321143

Pulled By: huydhn

fbshipit-source-id: 35c95d3a6278ce3a02613253e542a7e504ac8cca
Swaps the EC2 labels for their OSDC equivalents across the CI and benchmark
workflows, and in generate_ci_matrix.py's host_machines table, which is where
most of them come from:

  linux.2xlarge            -> mt-l-x86iavx512-8-64
  linux.4xlarge[.memory]   -> mt-l-x86iavx512-16-128
  linux.12xlarge[.memory]  -> mt-l-x86iavx512-48-384
  linux.24xlarge           -> mt-l-x86iavx512-94-192
  linux.24xlarge.memory    -> mt-l-x86iavx512-94-768
  linux.arm64.m7g.4xlarge  -> mt-l-arm64g3-16-62
  linux.g5.4xlarge.nvidia.gpu -> mt-l-x86aavx2-11-41-a10g

Mapping from pytorch/pytorch's .github/arc.yaml. The Graviton 3 choice is
preserved: the arm CPU jobs pick that instance for FP16FML, and arm64g3 is the
same generation.

Three things about the container go with them.

Every job container becomes almalinux:9. OSDC runs the k8s hook, which streams
the workspace into the job container as a tar and extracts it there rather than
the host-side docker cp the EC2 runners used, so the image needs tar and find.
amazonlinux:2023 ships neither, and every job died in prepare-job before its
first step; it cannot be fixed from a step, because prepare-job runs first.
almalinux:9 is the closest drop-in: same RPM family, has tar, find and curl, is
multi-arch so the arm jobs take the same line, and its libstdc++ provides
GLIBCXX_3.4.29, matching what amazonlinux:2023 offered.

Two images were tried and rejected, both worth not repeating. The manylinux2_28
builders are AlmaLinux 8, and their older libstdc++ fails the conda-installed
numpy with "GLIBCXX_3.4.29 not found" as soon as PyTorch is imported.
ubuntu:22.04, which the ROCm build jobs already used, gets further and then
fails setup_miniconda: its /root/.bashrc opens with `[ -z "$PS1" ] && return`,
so the conda init block appended to the end never runs in a non-interactive
shell. RPM distros have no such guard. Those ROCm jobs move to almalinux:9 too,
so every container here is now the same image.

The CUDA container stops bind-mounting the host Docker socket. There is no
/var/run/docker.sock on an OSDC node -- containers run under containerd through
the hook -- so the mount aborts the job before it starts. Nothing in the
workflows or .github/scripts runs docker, so the mount is vestigial;
--privileged and --pid=host go with it.

And the host disk cleanup goes away. free_disk_space_on_host used nsenter -t 1
to escape the container and delete packages from the CI host, which its own
comment describes as being for containers on non-PyTorch-infra runners. A pod
has no host to clean -- its disk is the ephemeral storage the runner def asks
for -- and without --privileged the escape is refused outright with "nsenter:
reassociate to namespace 'ns/ipc' failed: Operation not permitted". The sibling
free_disk_space stays, because forks still run these workflows on GitHub-hosted
runners where those preinstall paths are real, but it no longer hard-codes sudo,
which almalinux:9 does not ship.

ROCm x86 build hosts move with the rest; the MI350 test jobs keep
rocm/dev-ubuntu-22.04 and their own hardware. linux.gcp.a100 and linux.aws.h100
are commented-out future work and are left alone.

The wheel pipelines are not touched here; they select runners through Nova's
matrix generator and move separately.

Authored with Claude Code.
Only amazonlinux:2023 needed to move: it is unavailable to OSDC runners. The
ROCm jobs were already on ubuntu:22.04, which works fine there and ships tar
and find for the container hook, so switching them to almalinux:9 bought
nothing and dragged the apt lines to yum with it.

Reverts both the image and the package manager in the two ROCm workflows. The
runner label change stays.
conda init appends its block to the end of ~/.bashrc, but Debian and Ubuntu
ship a ~/.bashrc that returns on line 6 when the shell is non-interactive:

  [ -z "$PS1" ] && return

CI shells are non-interactive, so sourcing it never reaches the appended block
and the conda calls further down setup_miniconda fail with

  .github/scripts/utils_base.bash: line 52: conda: command not found

This only surfaces where HOME already has a distro .bashrc. On EC2 HOME is
/github/home and conda init creates the file from scratch, so the block is the
whole file and sourcing works. On OSDC HOME is /root, where ubuntu:22.04 ships
its own .bashrc. almalinux hid the same bug -- the RHEL .bashrc has no such
guard, which is why only the ROCm jobs on ubuntu fail.

etc/profile.d/conda.sh is the file conda init exists to install, has no guard,
and works the same on every distro and on EC2. Verified in an ubuntu:22.04
container: sourcing ~/.bashrc leaves conda unresolved, sourcing conda.sh does
not.

Steps after this one were never affected -- setup_miniconda already persists
the prefix to GITHUB_PATH once it completes.
@huydhn
huydhn force-pushed the osdc/ci-linux-runners branch from 855ce0f to 76cc72e Compare September 23, 2026 23:39

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant