Publish scripts to run benchmarks #15
Conversation
edeandrea
force-pushed
the
open-benchmarks
branch
from
63ce4af to
0b8b23b
Compare
edeandrea
force-pushed
the
open-benchmarks
branch
from
171834a to
0233622
Compare
|
@holly-cummins @franz1981 @willr3 would you all mind reviewing? I still need to add documentation |
|
@Sanne can you check if we are happy already with postgres as it is configured? For local and remote it doesn't bound the CPU set nor the mount of full cores to use whilst would be a good idea to have both I think. |
|
By running $ ./run-benchmarks.sh -q 3.28.4 -s 3.5.6I got: [jbang] Resolving dependencies...
[jbang] io.hyperfoil.tools:qDup:0.10.6
[jbang] Dependencies resolved
05:48:29.541 Running qDup version 0.10.6
05:48:30.817 output path = /tmp//20251020_054829
05:48:30.817 stream logging enabled
05:48:30.853 json server listening at fedora.fritz.box:31337
05:48:30.887 test-runner-setup@target-host. failed to start shell process: java.io.IOException: Cannot run program "script" (in directory "/home/forked_franz"): error=2, No such file or directory
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1170)
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1089)
at io.hyperfoil.tools.qdup.shell.LocalShell.connectShell(LocalShell.java:66)
at io.hyperfoil.tools.qdup.shell.AbstractShell.connect(AbstractShell.java:240)
at io.hyperfoil.tools.qdup.shell.AbstractShell.getShell(AbstractShell.java:127)
at io.hyperfoil.tools.qdup.Run.lambda$queueSetupScripts$17(Run.java:755)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.io.IOException: error=2, No such file or directory
at java.base/java.lang.ProcessImpl.forkAndExec(Native Method)
at java.base/java.lang.ProcessImpl.<init>(ProcessImpl.java:295)
at java.base/java.lang.ProcessImpl.start(ProcessImpl.java:225)
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1126)
... 9 more
05:48:30.887 test-cgroups-setup@target-host. failed to start shell process: java.io.IOException: Cannot run program "script" (in directory "/home/forked_franz"): error=2, No such file or directory
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1170)
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1089)
at io.hyperfoil.tools.qdup.shell.LocalShell.connectShell(LocalShell.java:66)
at io.hyperfoil.tools.qdup.shell.AbstractShell.connect(AbstractShell.java:240)
at io.hyperfoil.tools.qdup.shell.AbstractShell.getShell(AbstractShell.java:127)
at io.hyperfoil.tools.qdup.Run.lambda$queueSetupScripts$17(Run.java:755)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.io.IOException: error=2, No such file or directory
at java.base/java.lang.ProcessImpl.forkAndExec(Native Method)
at java.base/java.lang.ProcessImpl.<init>(ProcessImpl.java:295)
at java.base/java.lang.ProcessImpl.start(ProcessImpl.java:225)
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1126)
... 9 more
05:48:30.888 test-runner-setup@target-host. failed to connect to target-host LOCAL
05:48:30.888 test-cgroups-setup@target-host. failed to connect to target-host LOCAL
05:48:30.888 setup failed to connect LOCAL
05:48:30.888 setup failed to connect LOCAL
05:48:30,888 [ :- @ ] failed to connect all ssh sessions for setup
05:48:30.889 Skipping cleanup - Abort has been defined to not run any cleanup scripts
Finished in 00.015 at /tmp//20251020_054829
05:48:30.895 ignoring stop call when already stopped |
| config.repo.branch: main | ||
| config.repo.url: https://github.com/quarkusio/spring-quarkus-perf-comparison.git | ||
|
|
||
| RUN.WRK_BIN: jbang wrk@hyperfoil |
There was a problem hiding this comment.
This is what is reported by wrk while extracting manually the report:
Some notes:
- calibration phase duration is hardcoded in the command (which is not great)
- the report is pretty boring per se: latencies are useless (due to coordinated omission) and, as expected, throughput is stable
Which makes the report per se not as interesting as I was hoping (due to the simplicity of the workload...)
In short: I don't think we need to extract the report if we want to keep this test as simple as this.
|
One additional note re #15 (comment) I was interested into getting (not now, but in a future), the time to peak throughput metric. 
by:
To be fair this is gross (it relies on a sampled throughput w 1 second granularity, which is not so wow) and imprecise, but is easy and give a rough idea of the time required to stabilize. The downside re Probability the best way (for Hotspot at least) to extract the mentioned metric is to perform a run with JFR events re compilation on (see https://sap.github.io/SapMachine/jfrevents/21#compilation) and consider the server in steady state only when there are none (excluding the server's stop which will likely cause deoptimizations and new compilation due to the newly taken branches). |
|
@edeandrea re #15 (comment) I've commented at Hyperfoil/qDup#250 |
Works on my machine :D (although I'm not using |
|
@edeandrea what happen if you run it locally? |
I haven't tried yet because I'm on a mac....I'm focusing on remote linux first, since thats the use case for running in the perf lab. |
Just as a note - this is exactly what the benchmark was doing in the lab before we started any of this (well, different app, but read-only on immutable data). If it makes sense to change the benchmark to something else, then lets investigate what we'd like to do. |
edeandrea
force-pushed
the
open-benchmarks
branch
from
21f00b2 to
7f4a9b6
Compare
benchmarks/helpers/requirements.yml
Outdated
| - set-state: PROFILER_JVM_ARGS -XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints | ||
|
|
||
| ensure-gh-cli-linux: | ||
| - script: sudo |
There was a problem hiding this comment.
A sudo script requires users to interact via a prompt.
Indeed, locally I got a
05:34:22,742 [ test-runner:setup-725 @ target-host ] ensure-gh-cli-linux
05:39:22,845 [ test-runner:setup-725 @ target-host ] [sudo] password for forked_franz:
05:39:22,845 [ test-runner:setup-725 @ target-host ] sudo: timed out reading password
05:39:24,590 [ test-runner:setup-725 @ target-host ] sudo: a password is required
which make it unable to be installed.
On the server maybe we can, but still...
There was a problem hiding this comment.
A requirement is that you have to have passwordless sudo privs set up on the host machine. Kinda sucks, but it is what it is (I don't know another way around it).
|
i'm starting to feel like there are lots (of valid) comments on the PR. I'm also not a benchmarking expert, so there are probably specific things with regard to wrk that I may not be 100% accurate here. Would it make more sense to merge this PR and then add specific & targeted issues to fix things up? WDYT? |
edeandrea
changed the title
Unless either of you have any objections, I'm going to merge this. |
|
Sorry, this review request got sent into spam. (Thanks, email provider.) But I'm +1 to merge-and-iterate. |
|
I think the 'benchmarks' folder name could make the purpose clearer (since there are benchmark-y scripts in the top level, too. But that's something we can iterate. I guess we'll need to update the perf-lab when we do, so it's not super-trivial. |
Let's solve the naming thing now, as the perf lab changes haven't yet been implemented. Maybe Or Or |
edeandrea
force-pushed
the
open-benchmarks
branch
from
2cb3f8d to
2206641
Compare
Ah, in that case, yes, let's figure it out.
I think putting it into scripts is probably logic, since all the new stuff is a script.
I think 'acceptable' might make sense only to me. :) But actually maybe it still does, because the idea is you're running the perf lab scripts locally. Let me make sure I understand the layout. There's three modes of running, right? "Quick and dirty", "you want to do it right but only have a laptop", and "you have access to a perf lab (basically, you're John O'Hara in disguise)". Q&D lives in the top-level in scripts, with instructions in the main readme. YWTDIR (you want to do it right) and YAJOH (you are John O'Hara) share all common scripts? But YWTDIR needs extra instructions in a readme to explain all of what's going on, because you're not John O'Hara, and YAJOH is triggered from elsewhere, using the scripts source-controlled here? If that's right, I think I vote for
|
That would be tidier, but it means the getting started experience is worse. So tradeoffs. Having to type |
Sorry, that's what I was trying to say. I know Q&D is distinct. So If it was a Venn diagram, Q&D would be one circle and the YWTDIR and YAJOH circles would overlap completely. |
|
Ok so just to complete this thought, we want to move everything in |
Yes, I think so :) |
👍 |
edeandrea
force-pushed
the
open-benchmarks
branch
from
60f0128 to
79334c9
Compare
3 participants
Publishing all the scripts & documentation to run official benchmarks.