build(evaluator): install NeMo Fabric from published wheels via a Linux fabric extra

[SandyChapman]

Status: parked (draft). Waiting on NeMo Fabric to cut a non-prerelease release before merge — see the context comment below. The only prereleases remaining in the lock are Fabric-owned codex pins that clear when Fabric bumps.

What

NeMo Fabric now publishes wheels to PyPI, so the Fabric agent-eval runtime no longer needs a source build from a local checkout on Linux. This adds a Linux-gated fabric extra on nemo-evaluator-sdk that installs the published nemo-fabric[claude,codex,deepagents,runtime] from the lock.

Changes

• fabric extra (Linux-gated) on nemo-evaluator-sdk. The native nemo-fabric-runtime wheel (which provides the nemo_fabric module) ships manylinux only — no macOS wheel yet, so the sys_platform == 'linux' marker keeps macOS lock resolution intact.
• nemo-relay 0.4.x → 0.5.x (stable, all-platform wheels) to align with Fabric's relay line. Mirrored in the bundle-package block (packages/nemo_platform) and the vendored-SDK block (sdk/python/nemo-platform). Observability imports (AtifConfig, AtofConfig, ComponentSpec, ObservabilityConfig) verified on 0.5.0.
• Explicit prerelease constraints for the fabric codex subtree (openai-codex, openai-codex-cli-bin) — same pattern as the existing safetensors>=0.8.0rc0 line; no global --prerelease=allow. (sqlite-vec was initially constrained too, but it's a plain LangGraph transitive dep — langgraph-checkpoint-sqlite>=0.1.6 — so it resolves to stable 0.1.9.)
• script/dev-install-fabric.sh reworked (macOS-native dev path):
  • Installs nemo-fabric[codex,relay,runtime] — the runtime extra provides the importable nemo_fabric after Fabric's metapackage split ([codex,relay] alone no longer does).
  • Downloads the prebuilt nemo-relay gateway from the NeMo-Relay GitHub releases (checksum-verified, pinned via NEMO_RELAY_VERSION) instead of a cargo install from a local NeMo-Relay checkout — no NeMo-Relay clone or Rust build needed. Source-build fallback for platforms with no prebuilt asset (Intel macOS) or NEMO_RELAY_REPO.
• New fabric-wheel-smoke CI job (gated on the deps path filter, registered in ci-status): installs the extra from the lock on Linux and imports the nemo_fabric surface runtime.py uses. This is the only CI coverage that actually resolves/installs/imports the real fabric tree (unit tests use a hermetic nemo_fabric fake) and validates the manylinux_2_39 runtime wheel installs on the runner glibc.

Verification

• uv lock --check clean; branch rebased onto current main.
• macOS uv sync pulls zero fabric packages (all Linux-marker-gated) — only bumps relay 0.4.0 → 0.5.0.
• Fabric unit tests are hermetic; live tests remain gated behind requires_live_fabric.
• fabric-wheel-smoke green on Linux (real import nemo_fabric).

Known limitations / follow-ups

• Parked on Fabric prerelease (see status above): merge once nemo-fabric ships a non-prerelease and its adapters/codex drops the openai-codex beta pin → then drop the prerelease constraints and re-lock.
• hermes omitted: hermes-agent pins requests==2.33.0 exactly, conflicting with the workspace's requests>=2.33.1 floor. Re-add once upstream loosens it.
• Gateway not on PyPI: the nemo-relay gateway daemon (required for live ATIF capture on out-of-process agents like Codex) is published as prebuilt GitHub-release binaries for all platforms incl. macOS arm64, but not as a pip wheel — the Relay team confirmed they don't plan to wheel it, so the dev script / containers install it as a per-platform binary.
• No macOS nemo-fabric-runtime wheel yet (upstream ask on Fabric).
• Vendored-SDK dep block was hand-mirrored; a full make update-sdk regen can follow at merge time.

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features
  • Added optional Fabric SDK support with Linux-only dependency handling for Fabric and Codex.
  • Updated tool constraints to include prerelease Codex packages when Fabric is enabled (no-op on platforms where Fabric isn’t used).
• Bug Fixes
  • Improved Fabric + NeMo-Relay development installation across OS/architecture, including verified prebuilt gateway downloads with checksum checks and a source-build fallback.
• Tests
  • Added a CI smoke job to validate Fabric wheel installation and confirm nemo_fabric imports successfully.

[github-actions]

Suite	Lines Covered	Line Rate	Branch Rate
Unit Tests	26384/34035	77.5%	61.8%
Integration Tests	15177/32660	46.5%	18.7%

[coderabbitai]

📝 Walkthrough

Walkthrough

Fabric dependency constraints, development installation, relay provisioning, and Linux CI smoke validation were updated. Aggregate CI status now includes the Fabric wheel smoke job.

Changes

Fabric installation flow

Layer / File(s)	Summary
Fabric dependency contracts packages/nemo_evaluator_sdk/pyproject.toml, packages/nemo_platform/pyproject.toml, pyproject.toml	Adds the Linux-gated fabric extra, advances the nemo-relay constraint, and constrains prerelease Codex dependencies.
Development installation and relay provisioning script/dev-install-fabric.sh	Updates Fabric package installation and uninstall behavior, verifies imports, and adds platform-aware relay binary download, checksum verification, and source-build fallback.
CI wheel smoke validation .github/workflows/ci.yaml	Installs the SDK with the fabric extra from the lockfile, verifies nemo_fabric imports, and includes the job in aggregate CI status.

Sequence Diagram(s)

sequenceDiagram
  participant CI
  participant uv
  participant nemo_evaluator_sdk
  participant nemo_fabric
  CI->>uv: Sync nemo-evaluator-sdk[fabric] with frozen lockfile
  uv->>nemo_evaluator_sdk: Resolve Fabric extra
  nemo_evaluator_sdk->>nemo_fabric: Provide Fabric package
  CI->>nemo_fabric: Import module and Fabric symbols

Loading

Possibly related PRs

• NVIDIA-NeMo/nemo-platform#648: Updates the related Fabric smoke and development-install flows for the newer Fabric SDK API.

Suggested labels: ci

Suggested reviewers: crookedstorm, ironcommit

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: adding a Linux-gated fabric extra to install NeMo Fabric from published wheels.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fabric-published-wheels/schapman

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/ci.yaml:
- Around line 909-910: Update the actions/checkout step in the CI workflow to
set persist-credentials to false, preventing GITHUB_TOKEN from being stored in
.git/config while preserving the existing checkout configuration.

In `@script/dev-install-fabric.sh`:
- Line 78: Update the installation flow around RELAY_BIN_DIR so the installed
nemo-relay binary is discoverable by the live integration test. Either install
it into a directory already on PATH or print a shell PATH export that prepends
RELAY_BIN_DIR after installation, while preserving support for custom CARGO_HOME
values.
- Around line 79-80: Update the existing nemo-relay detection in the
provisioning flow to honor NEMO_RELAY_VERSION: obtain the installed binary’s
version, compare it with the requested version, and only skip installation when
they match. When they differ, replace the binary or fail explicitly instead of
treating any PATH match as already provisioned.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: b76ccb50-3940-4bfc-8f7b-258b56a9b7d3

📥 Commits

Reviewing files that changed from the base of the PR and between 848ea33 and 1e85de4.

⛔ Files ignored due to path filters (2)

• sdk/python/nemo-platform/pyproject.toml is excluded by !sdk/**
• uv.lock is excluded by !**/*.lock

📒 Files selected for processing (5)

• .github/workflows/ci.yaml
• packages/nemo_evaluator_sdk/pyproject.toml
• packages/nemo_platform/pyproject.toml
• pyproject.toml
• script/dev-install-fabric.sh

[SandyChapman]

Holding this as a draft until NeMo Fabric ships a public, non-prerelease release.

The change itself is complete and CI is green, but it currently pulls Fabric's daily alpha wheels, which drag two Fabric-owned prerelease pins into the workspace lock (openai-codex==0.1.0b3 → openai-codex-cli-bin==0.137.0a4, both exact-pinned by nemo-fabric-adapters-codex). We don't want to lock non-Fabric prereleases into the platform.

Those clear automatically once Fabric cuts a stable release and its adapters/codex bumps off the openai-codex beta. At that point the plan is: point the extra at the stable nemo-fabric, drop the openai-codex/openai-codex-cli-bin prerelease constraints from the root pyproject.toml, re-lock, confirm fabric-wheel-smoke passes on the stable wheels, and mark ready for review.

Parking rather than merging also avoids interim CI flakiness from Fabric alpha wheels being garbage-collected from the index.