Conversation
|
The label mapping here is right, but these jobs cannot move as-is — the container image is the blocker, and it needs fixing before this lands.
That produces the two failures on this PR:
On EC2 this worked because the copy was a It cannot be fixed from a workflow step: Checked the alternatives:
Separately, the Suggested order: bake an FBGEMM CI image (with Gotcha written up in pytorch/test-infra#8794. |
af5db6d to
784deab
Compare
784deab to
cbf581a
Compare
cbf581a to
add2049
Compare
2a21f30 to
818df34
Compare
Summary: X-link: https://github.com/facebookresearch/FBGEMM/pull/3227 `__build_fbgemm_gpu_set_run_multicore` multiplies `Core(s) per socket` by `Socket(s)` from `lscpu`. That reports the **host's** physical topology — fine on a dedicated EC2 runner, wrong in a container on a shared node. On an OSDC runner with 11 CPUs on a 192-core `g5.48xlarge`, it reads 48 × 2 and builds with `-j 96` inside a 41Gi container: ~0.4GiB per compile job against C++/CUDA TUs that want 1–2GiB each. The CUDA wheel build [OOM-killed](https://github.com/pytorch/FBGEMM/actions/runs/35268173569/job/105360670718) at ninja step 102/655: ``` core = 48 sockets = 2 [BUILD] Set multicore run option for setup.py: -j 96 ##[error]Step failed: Container "job" was OOMKilled (exit code 137) ``` `nproc` respects the CPU affinity the process actually has — it printed `11` a few lines earlier in that same log — and is equally correct on EC2, where it returns the whole machine. `BUILD_PARALLELISM` still overrides, and an unparseable count still leaves `run_multicore` empty, as before. Split out of #6320 because `fbgemm_gpu_build.bash` is shared: #6320 and #6321 both hit this path, so landing it once avoids a conflicting duplicate in each. Authored with Claude Code. Pull Request resolved: #6361 Reviewed By: q10 Differential Revision: D121321143 Pulled By: huydhn fbshipit-source-id: 35c95d3a6278ce3a02613253e542a7e504ac8cca
Swaps the EC2 labels for their OSDC equivalents across the CI and benchmark workflows, and in generate_ci_matrix.py's host_machines table, which is where most of them come from: linux.2xlarge -> mt-l-x86iavx512-8-64 linux.4xlarge[.memory] -> mt-l-x86iavx512-16-128 linux.12xlarge[.memory] -> mt-l-x86iavx512-48-384 linux.24xlarge -> mt-l-x86iavx512-94-192 linux.24xlarge.memory -> mt-l-x86iavx512-94-768 linux.arm64.m7g.4xlarge -> mt-l-arm64g3-16-62 linux.g5.4xlarge.nvidia.gpu -> mt-l-x86aavx2-11-41-a10g Mapping from pytorch/pytorch's .github/arc.yaml. The Graviton 3 choice is preserved: the arm CPU jobs pick that instance for FP16FML, and arm64g3 is the same generation. Three things about the container go with them. Every job container becomes almalinux:9. OSDC runs the k8s hook, which streams the workspace into the job container as a tar and extracts it there rather than the host-side docker cp the EC2 runners used, so the image needs tar and find. amazonlinux:2023 ships neither, and every job died in prepare-job before its first step; it cannot be fixed from a step, because prepare-job runs first. almalinux:9 is the closest drop-in: same RPM family, has tar, find and curl, is multi-arch so the arm jobs take the same line, and its libstdc++ provides GLIBCXX_3.4.29, matching what amazonlinux:2023 offered. Two images were tried and rejected, both worth not repeating. The manylinux2_28 builders are AlmaLinux 8, and their older libstdc++ fails the conda-installed numpy with "GLIBCXX_3.4.29 not found" as soon as PyTorch is imported. ubuntu:22.04, which the ROCm build jobs already used, gets further and then fails setup_miniconda: its /root/.bashrc opens with `[ -z "$PS1" ] && return`, so the conda init block appended to the end never runs in a non-interactive shell. RPM distros have no such guard. Those ROCm jobs move to almalinux:9 too, so every container here is now the same image. The CUDA container stops bind-mounting the host Docker socket. There is no /var/run/docker.sock on an OSDC node -- containers run under containerd through the hook -- so the mount aborts the job before it starts. Nothing in the workflows or .github/scripts runs docker, so the mount is vestigial; --privileged and --pid=host go with it. And the host disk cleanup goes away. free_disk_space_on_host used nsenter -t 1 to escape the container and delete packages from the CI host, which its own comment describes as being for containers on non-PyTorch-infra runners. A pod has no host to clean -- its disk is the ephemeral storage the runner def asks for -- and without --privileged the escape is refused outright with "nsenter: reassociate to namespace 'ns/ipc' failed: Operation not permitted". The sibling free_disk_space stays, because forks still run these workflows on GitHub-hosted runners where those preinstall paths are real, but it no longer hard-codes sudo, which almalinux:9 does not ship. ROCm x86 build hosts move with the rest; the MI350 test jobs keep rocm/dev-ubuntu-22.04 and their own hardware. linux.gcp.a100 and linux.aws.h100 are commented-out future work and are left alone. The wheel pipelines are not touched here; they select runners through Nova's matrix generator and move separately. Authored with Claude Code.
Only amazonlinux:2023 needed to move: it is unavailable to OSDC runners. The ROCm jobs were already on ubuntu:22.04, which works fine there and ships tar and find for the container hook, so switching them to almalinux:9 bought nothing and dragged the apt lines to yum with it. Reverts both the image and the package manager in the two ROCm workflows. The runner label change stays.
conda init appends its block to the end of ~/.bashrc, but Debian and Ubuntu ship a ~/.bashrc that returns on line 6 when the shell is non-interactive: [ -z "$PS1" ] && return CI shells are non-interactive, so sourcing it never reaches the appended block and the conda calls further down setup_miniconda fail with .github/scripts/utils_base.bash: line 52: conda: command not found This only surfaces where HOME already has a distro .bashrc. On EC2 HOME is /github/home and conda init creates the file from scratch, so the block is the whole file and sourcing works. On OSDC HOME is /root, where ubuntu:22.04 ships its own .bashrc. almalinux hid the same bug -- the RHEL .bashrc has no such guard, which is why only the ROCm jobs on ubuntu fail. etc/profile.d/conda.sh is the file conda init exists to install, has no guard, and works the same on every distro and on EC2. Verified in an ubuntu:22.04 container: sourcing ~/.bashrc leaves conda unresolved, sourcing conda.sh does not. Steps after this one were never affected -- setup_miniconda already persists the prefix to GITHUB_PATH once it completes.
855ce0f to
76cc72e
Compare
Two changes that have to land together: the runner labels, and the container image they run in.
Labels
From pytorch/pytorch's
.github/arc.yaml, applied across the CI and benchmark workflows and togenerate_ci_matrix.py'shost_machinestable, which is where most of them come from:The Graviton 3 choice is preserved — the arm jobs pick that instance for FP16FML, and
arm64g3is the same generation.Image:
amazonlinux:2023->almalinux:9OSDC runs the k8s hook, which streams the workspace into the job container as a tar and extracts it there, rather than the host-side
docker cpthe EC2 runners used. So the image needstarandfind, andamazonlinux:2023ships neither:Every job died in
prepare-jobbefore its first step —cpToPodtiming out thirty times, thenfind: command not found. Not fixable from a step, sinceprepare-jobruns first.almalinux:9is the closest thing to a drop-in:tarfindcurldnfamazonlinux:2023almalinux:9Same RPM family so the scripts' package handling is unchanged, and multi-arch so the arm jobs take the same line.
The
pytorch/manylinux2_28-builderimages were tried first and are the wrong tool here — they are AlmaLinux 8, and their older libstdc++ fails the conda-installed numpy withGLIBCXX_3.4.29 not foundas soon as PyTorch is imported.Scope
ROCm x86 build hosts move with the rest; the MI350 test jobs keep
rocm/dev-ubuntu-22.04and their own hardware.linux.gcp.a100andlinux.aws.h100are commented-out future work. The wheel pipelines move in #6320.