OOM kills in sys containers are not reported as `OOMKilled`

[Nevon]

On a Kubernetes/CRI-O node using cgroup v2 with the systemd cgroup manager, when a sysbox container exceeds its memory limit the kernel OOM-kills it correctly, but Kubernetes reports the container as terminated with reason: Error rather than reason: OOMKilled.

I traced this to conmon being unable to read memory.events for the container. conmon derives its OOM-watch path from /proc/<container-pid>/cgroup, which for sysbox containers points at the init.scope leaf cgroup that sysbox-runc creates. That leaf has no memory.* files, because the memory controller is not enabled in the parent scope's cgroup.subtree_control.

I may well be misunderstanding the intended design here or how cgroup controller delegation is meant to work for sys containers, so I'd appreciate a sanity check on whether this is expected behaviour, a configuration mistake on our side, or an actual bug.

Environment

Component	Version
sysbox-runc	0.7.0 (CE), commit a4dd414f7b9b7455c0fbf0d5e5db7bcfe30645bc
CRI-O	1.34.5
conmon	2.1.13
cgroups	v2 (cgroup2fs), systemd cgroup manager
Kubernetes	managed cluster, containers created via runtimeClassName → sysbox-runc handler

CRI-O runtime configuration:

[crio.runtime]
  cgroup_manager = "systemd"
  conmon_cgroup = "system.slice"

  [crio.runtime.runtimes.sysbox-runc]
    allowed_annotations = ["io.kubernetes.cri-o.userns-mode"]
    monitor_path = "/usr/local/bin/crio-conmon"
    runtime_path = "/usr/bin/sysbox-runc"
    runtime_type = "oci"

What I observe

1. The kernel performs a real cgroup OOM kill

memory: usage 6291456kB, limit 6291456kB, failcnt 127750
<workload> invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=903
oom-kill:constraint=CONSTRAINT_MEMCG,...,oom_memcg=/kubepods.slice/kubepods-burstable.slice/
  kubepods-burstable-pod<uid>.slice/crio-<ctr>.scope
Tasks in .../crio-<ctr>.scope are going to be killed due to memory.oom.group set
Memory cgroup out of memory: Killed process <pid> (tini) ...

memory.oom.group=1, so all processes in the container cgroup were killed, including the container's PID 1. So far so good.

2. conmon cannot open memory.events

conmon <ctr> <ninfo>: container <pid> exited with status 137
conmon <ctr> <nwarn>: Failed to open cgroups file:
  /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod<uid>.slice/
  crio-<ctr>.scope/init.scope/memory.events

3. Kubernetes therefore records a plain error exit

kubelet generic.go:356 "Generic (PLEG): container finished" containerID="<ctr>" exitCode=137

kubernetes.containers.state.terminated reports reason: error. No OOMKilled is ever surfaced, in the pod status or in metrics.

Why memory.events is missing

What I measured

The container scope has the memory controller available, but enables nothing for its children, and the leaf has no memory-controller files:

$ cat <scope>/cgroup.controllers
cpuset cpu io memory hugetlb pids misc

$ cat <scope>/cgroup.subtree_control
                                  # empty

$ ls <scope>/init.scope
cgroup.controllers  cgroup.events  cgroup.freeze  cgroup.kill  cgroup.max.depth
cgroup.max.descendants  cgroup.pressure  cgroup.procs  cgroup.stat
cgroup.subtree_control  cgroup.threads  cgroup.type  cpu.pressure  cpu.stat
cpu.stat.local  io.pressure  memory.pressure

$ cat <scope>/init.scope/memory.events
cat: .../init.scope/memory.events: No such file or directory

(memory.pressure, cpu.pressure, io.pressure and cpu.stat are present irrespective of controller enablement, so the leaf has zero memory-controller files.)

All container processes live in the leaf, none in the parent:

PIDs in <scope>          : 0
PIDs in <scope>/init.scope: 1
$ cat /proc/<container-pid>/cgroup
0::/kubepods.slice/.../crio-<ctr>.scope/init.scope

The memory limit and memory.oom.group are set on the parent scope, so enforcement still works:

$ cat <scope>/memory.max ; cat <scope>/memory.oom.group
134217728
1

Same measurement on a crun container (same node, same kernel, same CRI-O)

$ cat <crun-scope>/cgroup.subtree_control
cpuset cpu io memory hugetlb pids misc

$ ls <crun-scope>/container | grep ^memory
memory.current  memory.events  memory.events.local  memory.high  memory.low
memory.max  memory.min  memory.numa_stat  memory.oom.group  memory.peak
memory.pressure  memory.reclaim  memory.stat  memory.swap.current
memory.swap.events  memory.swap.high  memory.swap.max  memory.swap.peak
memory.zswap.current  memory.zswap.max  memory.zswap.writeback

$ cat <crun-scope>/container/memory.events
low 0 high 0 max 0 oom 0 oom_kill 0 oom_group_kill 0

That container is also a leaf-in-scope arrangement with 0 processes in the parent and 1 in the leaf. The only relevant difference I can find is cgroup.subtree_control.

Based on this, these are the conclusions I can draw:

A controller is only available in a child cgroup if the parent has enabled it in cgroup.subtree_control. Combined with the comparison above, I read the empty subtree_control as the reason the leaf has no memory.events.

Comparing the two runtimes, both runtimes put container processes in a leaf cgroup but one in one of them does the leaf receive controllers.

	sysbox-runc	crun
PID 1 cgroup	<scope>/init.scope	<scope>/container
PIDs in parent / leaf	0 / 1	0 / 1
cgroup.subtree_control on scope	(empty)	cpuset cpu io memory hugetlb pids misc
<leaf>/memory.events	missing	present
conmon OOM detection	fails	works

I read through parts of sysbox-runc to see if I could understand if this was intentional. This comment where the leaf is created and the container's init is placed into it explains a lot about keeping the container's cgroup root free of processes so that inner sub-cgroups don't become "domain invalid"

What I could not find is any point where +memory (or the other controllers) is written into the parent scope's cgroup.subtree_control so that the leaf actually receives them. The leaf-creation logic in the cgroupfs manager looks similar, so it's possible the same applies there, but I've only tested the systemd manager.

Questions

1. Is the leaf cgroup expected to have the memory controller enabled? If so, should sysbox-runc write +memory … to the parent scope's cgroup.subtree_control after creating init.scope?
2. Or is the expectation that something inside the sys container must enable the controllers on the now-chowned cgroup.subtree_control? Is a non-systemd sys container a supported configuration in this respect?
3. Is there a CRI-O or sysbox setting we've missed that would make the leaf inherit controllers?
4. More broadly, is OOM reporting to the container runtime a supported property for sys containers, or a known limitation of the leaf-cgroup design?

Let me know if there's any more information I can provide.