Skip to content

Commit 0e67be0

Browse files
authored
Add callgrind harness (#576)
I want to measure callgrind instructions too, which can be more stable than pure wall time.
1 parent ab81550 commit 0e67be0

6 files changed

Lines changed: 296 additions & 2 deletions

File tree

‎README.md‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -336,6 +336,7 @@ You can find several test harnesses in the `harness/` directory:
336336
* `perf` - a simplified harness that runs for exactly the hinted number of iterations
337337
* `ractor` - a harness that runs the benchmark inside Ractors to measure scaling
338338
* `gc` - the default harness plus GC timing, count and heap statistics
339+
* `callgrind` - a harness to profile fixed iterations with Valgrind Callgrind, excluding setup and warmup
339340
* `stackprof` - a harness to profile the benchmark with stackprof
340341
* `stats` - count method calls and loop iterations
341342
* `vernier` - a harness to profile the benchmark with vernier
@@ -367,6 +368,50 @@ When using `run_benchmarks.rb`, you can specify a harness with the `--harness` o
367368

368369
There is also a robust but complex CI harness in [the yjit-metrics repo](https://github.com/Shopify/yjit-metrics).
369370

371+
### Callgrind harness
372+
373+
Requires Valgrind with Callgrind support and Ruby 3.4 or newer for the
374+
[`valgrind` gem](https://rubygems.org/gems/valgrind). Like the other profiler
375+
harnesses, it installs the gem globally if missing; you can install it ahead
376+
of time with `gem install valgrind` (building it requires Valgrind headers).
377+
378+
With the runner, `--harness=callgrind` automatically launches each benchmark
379+
under `valgrind --tool=callgrind --instr-atstart=no`:
380+
381+
```sh
382+
./run_benchmarks.rb --harness=callgrind --warmup=10 --bench=1 fib
383+
```
384+
385+
Profiles go in the runner's output directory as `callgrind-<benchmark>.out.<pid>`
386+
(with numbered suffixes for explicit dumps). Ruby flags, pre-init code, and
387+
CPU pinning are preserved. Valgrind must be on `PATH`.
388+
389+
When running a benchmark directly, launch Ruby under Callgrind explicitly:
390+
391+
```sh
392+
WARMUP_ITRS=10 MIN_BENCH_ITRS=1 HARNESS=callgrind \
393+
valgrind --tool=callgrind --instr-atstart=no \
394+
--callgrind-out-file="$PWD/data/callgrind.out.%p" \
395+
ruby benchmarks/fib.rb
396+
```
397+
398+
Setup and `WARMUP_ITRS` iterations (default: 10) run without instrumentation.
399+
The harness then starts instrumentation, zeros counters, profiles exactly
400+
`MIN_BENCH_ITRS` iterations (default: the benchmark's iteration hint), stops
401+
instrumentation, and dumps stats. Unlike sampling profilers, this harness does
402+
not use `MIN_BENCH_TIME`: even one iteration can be slow under Callgrind.
403+
On a measured-iteration failure it still dumps a partial profile and propagates
404+
the exception without writing results.
405+
406+
The dump argument is a description, **not a filename**. Valgrind controls the
407+
output path via `--callgrind-out-file`; use an absolute path because some
408+
benchmarks change directories. Explicit dumps normally have a numbered suffix
409+
(e.g. `data/callgrind.out.12345.1`); open that dump with `callgrind_annotate`,
410+
KCachegrind, or QCachegrind. Without Valgrind the gem's requests are no-ops and
411+
no profile is produced. If `RESULT_JSON_PATH` is set, the harness writes timing
412+
samples for runner compatibility; these are instrumented times, not normal
413+
benchmark performance measurements.
414+
370415
### Chain harness
371416

372417
The `chain` harness allows you to combine multiple harnesses together. This is useful when you want to run a benchmark through an analysis tool, such as a profiler, while another harness drives the iterations.

‎harness/callgrind.rb‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
# frozen_string_literal: true
2+
3+
# Profile only the measured iterations with Callgrind (not setup or warmup).
4+
# Usage:
5+
# WARMUP_ITRS=10 MIN_BENCH_ITRS=1 HARNESS=callgrind \
6+
# valgrind --tool=callgrind --instr-atstart=no ruby benchmarks/fib.rb
7+
# Use --callgrind-out-file=/absolute/path/callgrind.out.%p to choose the output.
8+
9+
require_relative "../lib/harness"
10+
require_relative "../lib/harness/extra"
11+
require "rbconfig"
12+
13+
# Valgrind client requests are silent no-ops outside Valgrind, so without
14+
# this check a run without Callgrind would succeed with no profile.
15+
raise "callgrind unsupported" unless RbConfig::CONFIG["host_os"].include?("linux")
16+
# Valgrind fakes argv and /proc/self/exe, but the kernel's process name is
17+
# the tool binary (e.g. "callgrind-amd64").
18+
unless File.read("/proc/self/comm").start_with?("callgrind-")
19+
raise "harness/callgrind.rb must run under `valgrind --tool=callgrind` " \
20+
"(e.g. ./run_benchmarks.rb --harness=callgrind)"
21+
end
22+
23+
# Load before benchmark Bundler setup can hide globally installed gems.
24+
ensure_global_gem("valgrind")
25+
require "valgrind"
26+
27+
# Also exclude setup when the caller forgot --instr-atstart=no.
28+
Valgrind.callgrind_stop_instrumentation
29+
30+
def run_benchmark(num_itrs_hint, **, &block)
31+
warmup_itrs = Integer(ENV.fetch('WARMUP_ITRS', 10))
32+
bench_itrs = Integer(ENV.fetch('MIN_BENCH_ITRS', num_itrs_hint))
33+
raise ArgumentError, 'WARMUP_ITRS must be non-negative' if warmup_itrs < 0
34+
raise ArgumentError, 'MIN_BENCH_ITRS must be positive' if bench_itrs <= 0
35+
36+
warmup = []
37+
bench = []
38+
Valgrind.callgrind_stop_instrumentation
39+
warmup_itrs.times do
40+
start = get_time
41+
block.call
42+
warmup << get_time - start
43+
end
44+
45+
begin
46+
Valgrind.callgrind_start_instrumentation
47+
Valgrind.callgrind_zero_stats
48+
bench_itrs.times do
49+
start = get_time
50+
block.call
51+
bench << get_time - start
52+
end
53+
ensure
54+
# Preserve a partial profile on failure, but never leave instrumentation on.
55+
Valgrind.callgrind_stop_instrumentation
56+
Valgrind.callgrind_dump_stats(benchmark_name)
57+
end
58+
59+
puts "Callgrind stats dumped for #{benchmark_name} (output controlled by --callgrind-out-file)."
60+
return_results(warmup, bench) if ENV['RESULT_JSON_PATH']
61+
end

‎lib/argument_parser.rb‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -101,7 +101,7 @@ def parse(argv)
101101
args.skip_zjit = true
102102
end
103103

104-
opts.on("--harness=HARNESS_NAME", "which harness to use (default: default, options: once, bips, perf, ractor, gc, stackprof, vernier, warmup, stats, continuous, chain, mplr)") do |v|
104+
opts.on("--harness=HARNESS_NAME", "which harness to use (default: default, options: once, bips, perf, callgrind, ractor, gc, stackprof, vernier, warmup, stats, continuous, chain, mplr)") do |v|
105105
# Accept the old directory names (harness, harness-<name>) for backward compatibility
106106
v = v == 'harness' ? 'default' : v.sub(/^harness-/, '')
107107
args.harness = v

‎lib/benchmark_suite.rb‎

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -207,7 +207,8 @@ def run_single_benchmark(script_path, result_json_path, ruby, cmd_prefix, env, b
207207
ENV["RESULT_JSON_PATH"] = result_json_path
208208

209209
# Set up the benchmarking command
210-
cmd = (cmd_prefix + [
210+
profiler_prefix = benchmark_harness == "callgrind" ? callgrind_cmd(script_path) : []
211+
cmd = (cmd_prefix + profiler_prefix + [
211212
*ruby,
212213
*pre_init,
213214
*harness_args_for(benchmark_harness),
@@ -226,6 +227,19 @@ def run_single_benchmark(script_path, result_json_path, ruby, cmd_prefix, env, b
226227
end
227228
end
228229

230+
# Keep output outside benchmark directories, which may chdir during setup.
231+
# The child PID makes profiles unique across benchmarks and Ruby configurations.
232+
def callgrind_cmd(script_path)
233+
name = File.basename(script_path, '.rb')
234+
name = File.basename(File.dirname(script_path)) if name == 'benchmark'
235+
profile_dir = File.expand_path(out_path)
236+
FileUtils.mkdir_p(profile_dir)
237+
[
238+
"valgrind", "--tool=callgrind", "--instr-atstart=no",
239+
"--callgrind-out-file=#{File.join(profile_dir, "callgrind-#{name}.out.%p")}",
240+
]
241+
end
242+
229243
def benchmark_harness_for(benchmark_name)
230244
return harness if harness_explicit
231245

‎test/benchmark_suite_test.rb‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -713,6 +713,58 @@ def base_cmd_for(ruby_description, benchmark_name: 'simple', harness: 'default',
713713
end
714714
end
715715

716+
describe 'Callgrind command wrapping' do
717+
def capture_benchmark_command(harness, script: 'benchmarks/simple.rb', status: 0)
718+
suite = BenchmarkSuite.new(categories: [], name_filters: [], out_path: @out_path,
719+
harness: harness, harness_explicit: true, no_pinning: true)
720+
command = nil
721+
child_env = nil
722+
check_call = lambda do |cmd, **options|
723+
command = cmd
724+
child_env = options[:env]
725+
File.write(ENV.fetch('RESULT_JSON_PATH'), JSON.generate('warmup' => [], 'bench' => [1.0])) if status == 0
726+
{ success: status == 0, status: Struct.new(:exitstatus).new(status) }
727+
end
728+
result = nil
729+
BenchmarkRunner.stub(:check_call, check_call) do
730+
result = suite.send(:run_single_benchmark, script, File.join(@out_path, 'result.json'),
731+
['/path with spaces/ruby', '--yjit'], ['taskset', '-c', '2'], {}, harness)
732+
end
733+
[command, child_env, result]
734+
end
735+
736+
it 'wraps Ruby after CPU pinning and preserves Ruby flags and the harness' do
737+
command, env, result = capture_benchmark_command('callgrind')
738+
assert_equal ['taskset', '-c', '2', 'valgrind', '--tool=callgrind', '--instr-atstart=no'], command.take(6)
739+
assert_equal "--callgrind-out-file=#{@out_path}/callgrind-simple.out.%p", command[6]
740+
assert_equal ['/path with spaces/ruby', '--yjit', '-r', File.join(BenchmarkSuite::HARNESS_DIR, 'callgrind'),
741+
File.expand_path('benchmarks/simple.rb')], command.drop(7)
742+
assert_equal 'callgrind', env['HARNESS']
743+
assert_equal command, Shellwords.split(result[:command])
744+
assert_nil ENV['RESULT_JSON_PATH']
745+
end
746+
747+
it 'uses the benchmark directory name for macrobenchmarks' do
748+
command, = capture_benchmark_command('callgrind', script: 'benchmarks/railsbench/benchmark.rb')
749+
assert_equal "--callgrind-out-file=#{@out_path}/callgrind-railsbench.out.%p", command[6]
750+
end
751+
752+
it 'does not wrap other harnesses' do
753+
command, env, = capture_benchmark_command('once')
754+
assert_equal ['taskset', '-c', '2', '/path with spaces/ruby', '--yjit'], command.take(5)
755+
refute_includes command, 'valgrind'
756+
assert_equal 'once', env['HARNESS']
757+
end
758+
759+
it 'propagates Valgrind failure and restores the result path' do
760+
ENV['RESULT_JSON_PATH'] = 'original.json'
761+
_, _, result = capture_benchmark_command('callgrind', status: 7)
762+
refute result[:success]
763+
assert_equal 7, result[:status].exitstatus
764+
assert_equal 'original.json', ENV['RESULT_JSON_PATH']
765+
end
766+
end
767+
716768
describe '#run_benchmark' do
717769
it 'returns data hash on success' do
718770
suite = BenchmarkSuite.new(

‎test/callgrind_harness_test.rb‎

Lines changed: 122 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,122 @@
1+
require_relative 'test_helper'
2+
require 'open3'
3+
require 'tmpdir'
4+
require 'json'
5+
require 'rbconfig'
6+
require 'fileutils'
7+
8+
class CallgrindHarnessTest < Minitest::Test
9+
ROOT = File.expand_path('..', __dir__)
10+
11+
# Pretend to be a Linux process running under Callgrind (or not).
12+
def fake_platform(host_os: 'linux', comm: 'callgrind-amd64')
13+
<<~STUB
14+
require 'rbconfig'
15+
RbConfig::CONFIG['host_os'] = #{host_os.inspect}
16+
File.singleton_class.prepend(Module.new do
17+
def read(path, *args, **kwargs)
18+
path == '/proc/self/comm' ? #{(comm + "\n").inspect} : super
19+
end
20+
end)
21+
STUB
22+
end
23+
24+
def load_harness(**platform)
25+
script = fake_platform(**platform) + "require #{File.join(ROOT, 'harness/callgrind').inspect}"
26+
env = { 'RUBYOPT' => nil, 'BUNDLE_GEMFILE' => nil, 'BUNDLER_SETUP' => nil }
27+
Open3.capture3(env, RbConfig.ruby, '-e', script, chdir: ROOT)
28+
end
29+
30+
def run_harness(env = {}, fail_at: nil)
31+
Dir.mktmpdir do |dir|
32+
# Fake a global gem so tests do not install a native extension.
33+
gem_lib = File.join(dir, 'gems', 'lib')
34+
FileUtils.mkdir_p(gem_lib)
35+
File.write(File.join(gem_lib, 'valgrind.rb'), <<~RUBY)
36+
module Valgrind
37+
def self.callgrind_stop_instrumentation; $events << 'stop'; end
38+
def self.callgrind_start_instrumentation; $events << 'start'; end
39+
def self.callgrind_zero_stats; $events << 'zero'; end
40+
def self.callgrind_dump_stats(name); $events << "dump:" + name; end
41+
end
42+
RUBY
43+
script = fake_platform + <<~RUBY
44+
require 'json'
45+
$events = []
46+
require #{File.join(ROOT, 'lib/harness/extra').inspect}
47+
def ensure_global_gem(name)
48+
raise 'wrong gem' unless name == 'valgrind'
49+
$LOAD_PATH << #{gem_lib.inspect}
50+
end
51+
require #{File.join(ROOT, 'harness/callgrind').inspect}
52+
def return_results(warmup, bench)
53+
$events << ['results', warmup.size, bench.size]
54+
end
55+
calls = 0
56+
begin
57+
run_benchmark(3, ignored_keyword: true) do
58+
calls += 1
59+
$events << 'work'
60+
raise 'workload failed' if calls == #{fail_at.inspect}
61+
end
62+
rescue => e
63+
$events << e.message
64+
end
65+
puts JSON.generate($events)
66+
RUBY
67+
clean_env = {
68+
'RUBYOPT' => nil, 'BUNDLE_GEMFILE' => nil, 'BUNDLER_SETUP' => nil,
69+
'WARMUP_ITRS' => '2', 'MIN_BENCH_ITRS' => nil,
70+
'RESULT_JSON_PATH' => File.join(dir, 'results.json')
71+
}.merge(env)
72+
out, err, status = Open3.capture3(clean_env, RbConfig.ruby, '-I', dir, '-e', script, chdir: ROOT)
73+
assert status.success?, "#{out}\n#{err}"
74+
JSON.parse(out.lines.last)
75+
end
76+
end
77+
78+
def test_profiles_hint_iterations_after_warmup
79+
assert_equal ['stop', 'stop', 'work', 'work', 'start', 'zero',
80+
'work', 'work', 'work', 'stop', 'dump:-e', ['results', 2, 3]], run_harness
81+
end
82+
83+
def test_iteration_overrides_and_zero_warmup
84+
assert_equal ['stop', 'stop', 'start', 'zero', 'work', 'stop', 'dump:-e', ['results', 0, 1]],
85+
run_harness({ 'WARMUP_ITRS' => '0', 'MIN_BENCH_ITRS' => '1' })
86+
end
87+
88+
def test_failure_stops_and_dumps_without_results
89+
assert_equal ['stop', 'stop', 'work', 'work', 'start', 'zero', 'work',
90+
'stop', 'dump:-e', 'workload failed'], run_harness({}, fail_at: 3)
91+
end
92+
93+
def test_warmup_failure_never_starts_profiling
94+
assert_equal ['stop', 'stop', 'work', 'workload failed'], run_harness({}, fail_at: 1)
95+
end
96+
97+
def test_invalid_counts_fail_before_workload
98+
assert_equal ['stop', 'MIN_BENCH_ITRS must be positive'], run_harness({ 'MIN_BENCH_ITRS' => '0' })
99+
assert_equal ['stop', 'WARMUP_ITRS must be non-negative'], run_harness({ 'WARMUP_ITRS' => '-1' })
100+
events = run_harness({ 'MIN_BENCH_ITRS' => 'invalid' })
101+
assert_equal 2, events.size
102+
assert_match(/invalid value for Integer/, events.last)
103+
end
104+
105+
def test_direct_execution_does_not_write_results
106+
refute run_harness({ 'RESULT_JSON_PATH' => nil }).any? { |event| event.is_a?(Array) }
107+
end
108+
109+
def test_rejects_non_linux
110+
_, err, status = load_harness(host_os: 'darwin24')
111+
refute status.success?
112+
assert_match(/callgrind unsupported/, err)
113+
end
114+
115+
def test_rejects_linux_without_callgrind
116+
%w[ruby memcheck-amd64-].each do |comm|
117+
_, err, status = load_harness(comm: comm)
118+
refute status.success?, comm
119+
assert_match(/must run under `valgrind --tool=callgrind`/, err)
120+
end
121+
end
122+
end

0 commit comments

Comments
 (0)