Skip to content

system, check: warn when a benchmark ran under a cgroup CPU quota - #277

Open
manunicholasjacob wants to merge 1 commit into
psf:mainfrom
manunicholasjacob:throttle-check
Open

manunicholasjacob wants to merge 1 commit into
psf:mainfrom
manunicholasjacob:throttle-check

Conversation

@manunicholasjacob

Copy link
Copy Markdown

Problem

pyperf system show runs ten operations and pyperf check warns on four statistical conditions, but nothing in either looks at whether the benchmark was throttled while it ran.

A cgroup CPU quota (docker run --cpus, a Kubernetes CPU limit, systemd CPUQuota=) throttles the benchmark, and throttling does not look like noise. A quota that binds for the whole run slows every value by about the same factor, so the mean shifts but the relative spread does not, and the result comes back looking statistically stable while being wrong. The advice check offers on an unstable result (more runs and values, pyperf system tune) does not help, because the machine was not jittery, it was capped. This is the common case on CI runners and in containers. Follows up #274 and reuses the cpu_quota detection from #265.

What this does

  • system show: a new "CPU quota" operation, next to "Power supply", that reports a binding quota (via get_cgroup_cpu_quota()) and, from cgroup v2 cpu.stat, whether the benchmark cgroup was throttled. It reports and advises only, the same shape as the power supply check, because nothing pyperf can tune fixes an externally imposed quota.
  • check: warns when a saved result carries cpu_quota metadata, so a stored result is judged after the fact instead of reading as stable.
  • _collect_metadata.py: adds get_cgroup_cpu_throttling() beside get_cgroup_cpu_quota(). It walks the same cgroup hierarchy and reads nr_throttled / throttled_usec (cgroup v2) or throttled_time (cgroup v1), taking the largest counters across the hierarchy because the throttling is recorded where the limit binds while the root stays at zero. It returns None when the counters cannot be read, so "could not be checked" stays distinct from zero.

Scope

This is the quota-detection piece. The two design questions in #274 are left as follow-ups: capturing the per-run cpu.stat delta around the benchmark (and whether to store throttle counters in metadata), and thermal throttling (core_throttle_count on x86, vcgencmd get_throttled on the Pi), where "could not be checked" has to stay distinct from zero.

Tests

  • test_metadata.py: cgroup v2 throttled, zero-is-not-None, cgroup v1 nanosecond conversion, and no-cgroup returning None.
  • test_cli.py: check warns on cpu_quota metadata and does not warn without it.
  • test_system.py: the CPU quota operation is silent with no quota, reports a set quota, and reports throttling.

Checked on a Raspberry Pi 5

Under systemd-run --user --scope -p CPUQuota=25%, released pyperf reports "The benchmark seems to be stable" on a run that was throttled for hundreds of milliseconds, and system show says nothing about the quota. With this change check warns that the benchmark ran under a CPU quota, and system show reports the quota and the throttling.

pyperf system show registers ten operations and pyperf check warns on
four statistical conditions, but nothing looks at whether the run was
throttled. A cgroup CPU quota (docker run --cpus, a Kubernetes CPU
limit, systemd CPUQuota=) throttles the benchmark: it moves the mean
and leaves the relative spread alone, so a capped run comes back looking
statistically stable while being wrong, and the advice offered on an
unstable result (more runs, system tune) does not help because the
machine was not jittery, it was capped.

Add a "CPU quota" operation next to "Power supply" that reports a
binding quota and, from cgroup v2 cpu.stat, whether the benchmark cgroup
was throttled. Reuse get_cgroup_cpu_quota() from the cpu_quota metadata
(psfgh-265) and add get_cgroup_cpu_throttling() beside it. pyperf check
emits the same warning from the cpu_quota metadata of a saved result, so
an already stored result can be judged after the fact.

Signed-off-by: Manu Nicholas Jacob <manunicholasjacob@gmail.com>
return stat if found else None


def get_cgroup_cpu_throttling():

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please don't duplicate code, there is already get_cgroup_cpu_quota(). Why not reusing it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants