Repository navigation
Intermittent KeyError when polling Slurm job arrays #3719
Description
Description
ReFrame intermittently raises a KeyError shortly after starting independent job arrays on the same Slurm partition. It is most likely to reproduce when running the same check with multiple programming environments.
Steps to reproduce
Save the following files and run:
reframe --nocolor -c checks.py -C config.yaml -rThe failure is timing-dependent, so the command may need to be run multiple times.
checks.py
import reframe as rfm
import reframe.utility.sanity as sn
@rfm.simple_test
class SlurmJobArray(rfm.RunOnlyRegressionTest):
"""Reproducer for two job arrays polled on the same partition."""
valid_systems = ["generic-cluster:compute"]
valid_prog_environs = ["env-a", "env-b"]
num_array_tasks = 32
num_tasks = 1
num_tasks_per_node = 1
@run_after("setup")
def set_job_array(self):
self.job.options += [f"--array=0-{self.num_array_tasks - 1}"]
# Stagger elements so both environment-specific arrays overlap in polling.
executable = "bash"
executable_opts = [
"-c",
"'sleep $((SLURM_ARRAY_TASK_ID % 4 + 1)); echo Hello world'",
]
@sanity_function
def validate(self):
count = sn.count(sn.findall(r"Hello world", self.stdout))
return sn.assert_eq(
count,
self.num_array_tasks,
msg="expected {1} Hello world lines, but found {0}",
)config.yaml
systems:
- name: generic-cluster
descr: Generic Slurm reproducer
hostnames: ['.*']
partitions:
- name: compute
descr: Generic compute partition
scheduler: slurm
launcher: srun
environs: [env-a, env-b]
max_jobs: 10
environments:
- name: env-a
- name: env-bError output
[ReFrame Setup]
version: 4.10.3
command: '/ssd0/data/users/sivaramp/.local/bin/reframe --nocolor -c checks.py -C config.yaml -r'
launched by: sivaramp@generic-login1
working directory: '/ssd0/data/users/sivaramp'
settings files: '<builtin>', '/ssd0/data/users/sivaramp/config.yaml'
selected system: 'generic-cluster'
check search path: '/ssd0/data/users/sivaramp/checks.py'
stage directory: '/ssd0/data/users/sivaramp/stage'
output directory: '/ssd0/data/users/sivaramp/output'
log files: '/tmp/rfm-nqw1n9ov.log'
results database: [off] '/ssd0/data/users/sivaramp/.reframe/reports/results.db'
[==========] Running 1 check(s)
[==========] Started on Tue Sep 1 17:57:22 2026+0000
[----------] start processing checks
[ RUN ] SlurmJobArray /4c932770 @generic-cluster:compute+env-a
[ RUN ] SlurmJobArray /4c932770 @generic-cluster:compute+env-b
[ PASSED ] Ran 0/2 test case(s) from 1 check(s) (0 failure(s), 0 expected failure(s), 0 skipped, 0 aborted)
[==========] Finished on Tue Sep 1 17:57:24 2026+0000
ERROR: run session stopped: key error: '44153'
ERROR: Traceback (most recent call last):
File "/ssd0/data/users/sivaramp/.local/share/uv/tools/reframe-hpc/lib/python3.14/site-packages/reframe/frontend/cli.py", line 1780, in main
runner.runall(testcases, restored_cases)
~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/ssd0/data/users/sivaramp/.local/share/uv/tools/reframe-hpc/lib/python3.14/site-packages/reframe/core/logging.py", line 1139, in _fn
return fn(*args, **kwargs)
File "/ssd0/data/users/sivaramp/.local/share/uv/tools/reframe-hpc/lib/python3.14/site-packages/reframe/frontend/executors/__init__.py", line 731, in runall
self._runall(testcases)
~~~~~~~~~~~~^^^^^^^^^^^
File "/ssd0/data/users/sivaramp/.local/share/uv/tools/reframe-hpc/lib/python3.14/site-packages/reframe/frontend/executors/__init__.py", line 824, in _runall
self._policy.exit()
~~~~~~~~~~~~~~~~~^^
File "/ssd0/data/users/sivaramp/.local/share/uv/tools/reframe-hpc/lib/python3.14/site-packages/reframe/frontend/executors/policies.py", line 430, in exit
self._poll_tasks()
~~~~~~~~~~~~~~~~^^
File "/ssd0/data/users/sivaramp/.local/share/uv/tools/reframe-hpc/lib/python3.14/site-packages/reframe/frontend/executors/policies.py", line 480, in _poll_tasks
sched.poll(*jobs)
~~~~~~~~~~^^^^^^^
File "/ssd0/data/users/sivaramp/.local/share/uv/tools/reframe-hpc/lib/python3.14/site-packages/reframe/core/schedulers/slurm.py", line 561, in poll
self._cancel_if_blocked(jobs)
~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
File "/ssd0/data/users/sivaramp/.local/share/uv/tools/reframe-hpc/lib/python3.14/site-packages/reframe/core/schedulers/slurm.py", line 605, in _cancel_if_blocked
pending_job = pending_jobs[jobid]
~~~~~~~~~~~~^^^^^^^
KeyError: '44153'
Log file(s) saved in '/tmp/rfm-nqw1n9ov.log'
Metadata
Metadata
Assignees
Type
Projects
- Status
Activity
vkarak commentedon Sep 14, 2026
I reproduced it in the tutorial's Docker compose setup after I ran it a couple of times:
vkarak commentedon Sep 15, 2026
@pranav-sivaraman Could you try #3737 that fixes this?
pranav-sivaraman commentedon Sep 16, 2026
Hi @vkarak I can confirm this fixes the issue on my end