Fix same benchmark comparisson on cheap machines

[RafaelGSS]

https://github.com/RafaelGSS/bench-node/blob/main/test/env.js#L67

[jdmarshall]

Is this a memory problem?

[RafaelGSS]

No, it's actually something to investigate. It does work as expected when the machine is ok, but on cheap machines it seems to be inconsistent. We just need to figure out if that's a hardware expected variance (due to CPU scheduler plus other resources) or it's a Node.js/V8 variance - which should be something we must fix.

I expect this to be a hardware variance, but we need to check.

Some ways to investigate it would be:

• Create a benchmark suite with one benchmark function and run it multiple times, but as a separate process. e.g: `for i in {1..5}; do node --allow-natives-syntax index.js; done
• Calculate the CV (do it multiple times)
• Do the same with other benchmark tools
  • Alternatively, test bench-node without the DoNotOptimize plugin

[H4ad]

If I remember correctly, I once read about v8/chrome/puppeteer being slower with 256 MB of memory and much faster with 1025 MB of memory. In a blog post, they were talking about generating PR thumbnails using AWS Lambda.

I don't know if that applies here, but just to add another thought.

[jdmarshall]

If it’s external, I would guess noisy neighbors. But it could also be cpu slicing? Which would be testable with docker as well.

[RafaelGSS]

@riosje could you provide the exact hardware details of the hosted-runner so we can investigate?

[jdmarshall]

Just running some of my benchmarks, set up to compare two copies of the same test, I'm seeing variability go as high as 13%.

That's on podman, configured to allocate 5 CPUs and 8GB of RAM. So there's a lot of jitter here anyway even before you get to tiny VMs. Sufficient to randomly fail your builds.

It does look like if you slice a container very thin (--cpus=0.5) I can get the variability to go a lot higher:

constructors ⇒ new Registry() ⇒ prom-client@latest (baseline)
constructors ⇒ new Registry() ⇒ prom-client@duplicate (1.18x faster)
constructors ⇒ new Counter() ⇒ prom-client@latest (baseline)
constructors ⇒ new Counter() ⇒ prom-client@duplicate (1.32x slower)

I don't think the docker cpuslices were ever envisioned to be the sort of 'fair' that a benchmarking tool requires in order to do A/B comparisons. Specifically: sequential A/B testing. I suspect if the tests were staggered you might be able to get more realistic outputs.

I have no idea how AWS handles this, but by default Docker uses a period of 100ms and then you get throttled. That would definitely blow a benchmark up.

[RafaelGSS]

We should define a minimal hardware for a "fair" Node.js benchmark using bench-node somehow. It's also very important to compare how other benchmark libraries behave on those tiny machines (of course, when DisableOptimization is enabled)

[jdmarshall]

What do you think about pausing between samples or adding your own canaries to detect jank?

[RafaelGSS]

I'll try to provision a few dedicated machines on AWS and see how it behaves, and also compare it with other benchmark libraries (that have the %NeverOptimize feature)

[riosje]

Hi @RafaelGSS the vm we're using is c6i.xlarge this is the spec
Amazon EC2 C6i and C6id instances are powered by 3rd Generation Intel Xeon Scalable processors (code named Ice Lake) with an all-core turbo frequency of 3.5 GHz, offer up to 15% better compute price performance over C5 instances, and always-on memory encryption using Intel Total Memory Encryption (TME).
vCPU: 4
RAM: 8GB

Since those VM's uses shared hardware you could see variance on the perf, due to several factors including noisy neighbors

[jdmarshall]

At cpus=0.25 the situation is brutal. I've seen up to 2.67x for identical benchmarks.

I'm experimenting with putting delays in the setup and teardown section to try to smooth out the throttling ripples. But so far the worst run I've observed was with my smoothing code in place:

constructors ⇒ new Counter() ⇒ prom-client@latest (baseline)
constructors ⇒ new Counter() ⇒ prom-client@duplicate (3.49x faster)

I've seen 2.67x without the smoothing. Clearly there's a ton of jitter here.

With 1/4 of a CPU, any benchmark that takes longer than 25ms, there's not much you can do. The timer.start/stop loop becomes something to trip over instead of a help here, because it assumes the clock must be running for the entire loop. You'd need a pause interface in order to take a breather between runs, and then I think you're into the same sort of issues I tried to explain to the maintainer of tinybench. You're amortizing the poor clock resolution in NodeJS (because of the event loop) across multiple calls instead of trying to sample on every single iteration, where the clock jitter is tragicomic.

[jdmarshall]

from https://medium.com/@freey/how-to-test-out-the-cpu-throttling-issue-on-linux-kernel-docker-and-kubernetes-environment-c5246a206f69

cat /sys/fs/cgroup/cpu.stat

usage_usec 12906225
user_usec 9955443
system_usec 2950782
core_sched.force_idle_usec 0
nr_periods 490
nr_throttled 315
throttled_usec 31525559
nr_bursts 0
burst_usec 0

Emphasis is mine, the numbers that move when I 'break' bench-node

With no --cpu* directives on docker run:

usage_usec 9113799
user_usec 7854664
system_usec 1259134
core_sched.force_idle_usec 0
nr_periods 0
nr_throttled 0
throttled_usec 0
nr_bursts 0
burst_usec 0

This might help you at least diagnose your issue, but I wonder if it's worth grabbing the data at the end of setup and check it at the beginning of teardown to see if the stats are bogus.

[RafaelGSS]

That's good @jdmarshall, thank you. Have you tried with other benchmark libraries?

I will work on a reproducible repo that tests (and produces diagnostic output) in different environments and tests against other benchmark libraries, I just couldn't find time yet.

[jdmarshall]

tinybench showing impressive variability:

049a79eba4de:/test# node tiny.js 
┌─────────┬─────────────┬────────────────────┬───────────────────┬────────────────────────┬────────────────────────┬─────────┐
│ (index) │ Task name   │ Latency avg (ns)   │ Latency med (ns)  │ Throughput avg (ops/s) │ Throughput med (ops/s) │ Samples │
├─────────┼─────────────┼────────────────────┼───────────────────┼────────────────────────┼────────────────────────┼─────────┤
│ 0       │ 'task'      │ '7779.1 ± 105.87%' │ '1042.0 ± 417.00' │ '891298 ± 0.87%'       │ '959693 ± 426360'      │ 12855   │
│ 1       │ 'same task' │ '18353 ± 112.50%'  │ '2250.0 ± 250.00' │ '444883 ± 0.83%'       │ '444444 ± 51053'       │ 5454    │
└─────────┴─────────────┴────────────────────┴───────────────────┴────────────────────────┴────────────────────────┴─────────┘
049a79eba4de:/test# node tiny.js 
┌─────────┬─────────────┬────────────────────┬───────────────────┬────────────────────────┬────────────────────────┬─────────┐
│ (index) │ Task name   │ Latency avg (ns)   │ Latency med (ns)  │ Throughput avg (ops/s) │ Throughput med (ops/s) │ Samples │
├─────────┼─────────────┼────────────────────┼───────────────────┼────────────────────────┼────────────────────────┼─────────┤
│ 0       │ 'task'      │ '3597.6 ± 98.62%'  │ '792.00 ± 84.00'  │ '1196176 ± 0.28%'      │ '1262626 ± 149803'     │ 31054   │
│ 1       │ 'same task' │ '6372.3 ± 112.81%' │ '917.00 ± 125.00' │ '982615 ± 0.54%'       │ '1090513 ± 172114'     │ 15693   │
└─────────┴─────────────┴────────────────────┴───────────────────┴────────────────────────┴────────────────────────┴─────────┘
049a79eba4de:/test# node tiny.js 
┌─────────┬─────────────┬────────────────────┬───────────────────┬────────────────────────┬────────────────────────┬─────────┐
│ (index) │ Task name   │ Latency avg (ns)   │ Latency med (ns)  │ Throughput avg (ops/s) │ Throughput med (ops/s) │ Samples │
├─────────┼─────────────┼────────────────────┼───────────────────┼────────────────────────┼────────────────────────┼─────────┤
│ 0       │ 'task'      │ '5292.5 ± 106.19%' │ '750.00 ± 83.00'  │ '1169528 ± 0.47%'      │ '1333333 ± 165917'     │ 19192   │
│ 1       │ 'same task' │ '9592.5 ± 158.95%' │ '958.00 ± 167.00' │ '913410 ± 0.75%'       │ '1043841 ± 218785'     │ 10425   │
└─────────┴─────────────┴────────────────────┴───────────────────┴────────────────────────┴────────────────────────┴─────────┘
049a79eba4de:/test# node tiny.js 
┌─────────┬─────────────┬────────────────────┬───────────────────┬────────────────────────┬────────────────────────┬─────────┐
│ (index) │ Task name   │ Latency avg (ns)   │ Latency med (ns)  │ Throughput avg (ops/s) │ Throughput med (ops/s) │ Samples │
├─────────┼─────────────┼────────────────────┼───────────────────┼────────────────────────┼────────────────────────┼─────────┤
│ 0       │ 'task'      │ '12053 ± 110.32%'  │ '1875.0 ± 208.00' │ '609017 ± 1.00%'       │ '533333 ± 53257'       │ 8297    │
│ 1       │ 'same task' │ '9801.1 ± 115.60%' │ '1209.0 ± 541.00' │ '817208 ± 0.95%'       │ '827130 ± 365662'      │ 10203   │
└─────────┴─────────────┴────────────────────┴───────────────────┴────────────────────────┴────────────────────────┴─────────┘

Notice how much the overall numbers dropped in the last run.

[jdmarshall]

I propose that perhaps this is a documentation and resource management situation. We can and do split testing into separate tasks that run with differing amounts of resources. If the benchmark is important, then it should not be left up to running on low quality hardware.

I’ve seen first hand the sorts of externalities that can come from running CI on underprovisioned machines. It’s a false economy that can result in real and excessive costs elsewhere in the system. Better to run CI on beefier hardware but either amortize the cost across multiple projects or allocate and deallocate them on demand.

I think it’s reasonable in this case to encourage teams to run your code in unambiguous situations.