NERSC adapter: custom_attributes not translated to sbatch flags

[osmiumzero]

Summary

The PSI/J JobAttributes.custom_attributes field accepts key-value pairs but the NERSC adapter does not translate them to corresponding sbatch flags. This prevents users from passing Slurm-specific options like -C gpu (constraint).

Reproduction

POST /api/v1/compute/job/6d00f875-dfc1-4a41-9309-456c5f2048df
{
    "executable": "/path/to/script.sh",
    "resources": {"node_count": 1, "gpu_cores_per_process": 4},
    "attributes": {
        "queue_name": "regular",
        "account": "m3792_g",
        "duration": 5400,
        "custom_attributes": {"constraint": "gpu"}
    }
}

Expected: Job submitted with sbatch -C gpu
Actual: Constraint not passed to sbatch. The custom_attributes field has no effect.

Tested key formats

Both of these were tested; neither produces any sbatch flags:

• {"constraint": "gpu"} — no effect
• {"slurm.constraint": "gpu"} — no effect (PSI/J convention for Slurm-specific attributes)

Impact

On Perlmutter, GPU jobs require -C gpu when using user-facing QOS names like regular. Without the constraint, Slurm routes to the CPU QOS variant which has different wall time limits.

This is partially masked by #49 (API forces gpu_debug QOS which has GPU constraint built into the QOS definition), but will block any other custom sbatch flags users need to pass.

Note: gpu_cores_per_process in ResourceSpec also does NOT automatically add -C gpu to sbatch — this might be expected behavior, but worth noting since GPU resources without GPU constraints will fail.

Environment

• NERSC Perlmutter
• API: https://api.iri.nersc.gov/api/v1
• Date: 2026-03-03

[gabor-lbl]

Thanks for logging this issue.

If "custom_attributes": {"constraint": "gpu"} is specified, the code will generate a job script with:

#SBATCH --constraint=gpu

and then call srun.

Is this incorrect? Thanks!

[osmiumzero]

Thanks for the quick response! You may be correct — our testing of custom_attributes was confounded by #49 (API forcing gpu_debug QOS, which has the GPU constraint built-in). Since gpu_debug always includes -C gpu, we couldn't isolate whether custom_attributes was having any effect.

We tested both {"constraint": "gpu"} and {"slurm.constraint": "gpu"} but couldn't confirm either way because the constraint was always present via the QOS.

Once #49 is resolved (so we can use regular or gpu_regular QOS), we'll retest and update this issue. It's possible this is not a bug — apologies if so!

The more critical issue for us is #49 — queue_name being ignored and all jobs forced to gpu_debug QOS. That 30-minute wall time cap blocks our full pipeline (~35 min on 4×A100). Would really appreciate a look at that one.

[gabor-lbl]

No problem. #49 should be fixed now.

[gabor-lbl]

As discussed on slack, you also need to set gpu_cores_per_process to a value 1 or greater.