performance.html

<!doctype html>
<html lang="en"><head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Where the time goes</title>
<meta name="description" content="Every benchmark, the ones Grasm loses, and the measurement rules each mistake bought.">
<link rel="icon" href="favicon.svg" type="image/svg+xml">
<link rel="preload" href="vendor/fonts/figtree.woff2" as="font" type="font/woff2" crossorigin>
<link rel="preload" href="vendor/fonts/sora.woff2" as="font" type="font/woff2" crossorigin>
<link rel="stylesheet" href="style.css">
<script src="vendor/motion.js" defer></script>
<script src="vendor/lottie.js" defer></script>
<script type="module" src="app.js"></script>
</head><body data-page="performance">
<header class="topbar">
  <a class="brand" href="index.html"><svg viewBox="0 0 48 48" width="20" height="20" fill="none" stroke="currentColor" stroke-width="6" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><path d="M9 6H39"/><path d="M24 14A13 13 0 1 0 37 27H26"/></svg><span>Grasm</span></a>
  <span class="ver">v1.0.0</span>
  <nav>
    <a href="different.html">Why Grasm</a>
    <a href="start.html">Docs</a>
    <a href="security.html">Security</a>
    <a href="achieved.html">Scorecard</a>
    <a href="playground.html">Playground</a>
  </nav>
  <a class="cta" href="start.html">Get started</a>
</header>
<div class="frame">
<nav class="rail">
<a class="mark" href="index.html"><svg viewBox="0 0 48 48" width="18" height="18" fill="none" stroke="currentColor" stroke-width="6" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><path d="M9 6H39"/><path d="M24 14A13 13 0 1 0 37 27H26"/></svg><b>grasm</b><span>a systems language</span></a>
<h2>Start</h2>
<a href="different.html">Why Grasm</a>
<a href="start.html">Getting started</a>
<a href="install.html">Install</a>
<a href="playground.html">Playground</a>
<h2>Learn</h2>
<a href="tutorial.html">Tutorial</a>
<a href="tour.html">Tour</a>
<a href="effects.html">Effects</a>
<a href="tiers.html">Ownership</a>
<a href="errors.html">Errors</a>
<a href="concurrency.html">Concurrency</a>
<a href="security.html">Security</a>
<h2>Reference</h2>
<a href="syntax.html">Syntax</a>
<a href="language.html">Language</a>
<a href="library.html">Library</a>
<a href="packages.html">Packages</a>
<a href="spec.html">Specification</a>
<a href="architecture.html">Architecture</a>
<h2>More</h2>
<a href="examples.html">Examples</a>
<a href="achieved.html">Scorecard</a>
<a href="performance.html" aria-current="page">Performance</a>
<a href="about.html">About</a>
<a href="status.html">Status</a>
<a href="roadmap.html">Roadmap</a>
</nav>
<main>

<p class="eyebrow">More</p>
<h1>Where the time goes.</h1>
<p class="lede">Grasm does not beat Rust. The geometric mean over the
microbenchmark suite is <strong>0.965×</strong> of it —
0.955× with the harness's own floor taken out of both sides
— which clears the 0.95× goal and is not the same as a win, and the
systems suite below loses a benchmark by 3.6×. Both tables
here are rendered from the files the runs wrote, not copied onto the page.</p>

<h2>The microbenchmark suite</h2>
<p>Eight programs, against Rust, C, Zig and Go. Five are at or above parity:
<code>wordfreq</code> 1.67×, <code>sort</code> 1.12×, <code>sieve</code>
1.03×, <code>fib</code> 1.01× (also ahead of Zig) and
<code>parwords</code> 1.01×. <code>matmul</code> at 0.92× is behind Rust
and ahead of Zig and Go.</p>
<p><code>tree</code> at <strong>0.51×</strong> is what the
geomean carries, and the cause is the representation rather than any operation:
a list of value structs is held as a box per node. Rust written <em>the way
Grasm represents it</em> — an <code>Rc</code> per node — runs 2.4×
slower than Grasm, so the gap to idiomatic Rust is the boxes and nothing else.
Storing those nodes inline was built and measured at <em>1.33× slower</em>:
a node's release becomes a runtime call where it was one inlined decrement,
which costs more than the box it removes. The release has to get cheaper
first.</p>
<p class="note">The eighth program, <code>parmap</code>, has no ratio here on
purpose. Parallel rows on this machine swing about ±10% run to run, so a
single number for one would be a number about the machine. A serial proxy is
used instead when a change needs measuring.</p>

<h2>The systems suite</h2>
<p>Five realistic workloads — a chat fan-out, an ECS update loop, a
pointer-heavy graph build and drop, a 1 GiB ETL decode, and a TCP proxy —
against Rust and Go. This is the recorded table:</p>
<div class="scroll"><table><thead><tr><th>benchmark</th><th>metric</th><th>Grasm</th><th>Rust</th><th>Go</th></tr></thead><tbody><tr><td>chat</td><td>deliveries/s (median of 3)</td><td>590,078</td><td><strong>693,534</strong></td><td>131,735</td></tr><tr><td>chat</td><td>peak RSS MB</td><td><strong>8.4</strong></td><td>69.4</td><td>264.0</td></tr><tr><td>ecs (1M × 100)</td><td>wall ms</td><td>277</td><td><strong>238</strong></td><td>313</td></tr><tr><td>ecs</td><td>RSS MB</td><td>47.3</td><td><strong>47.2</strong></td><td>50.1</td></tr><tr><td>dirtyheap (5M)</td><td>wall ms</td><td>1,860</td><td><strong>1,240</strong></td><td>2,720</td></tr><tr><td>dirtyheap</td><td>drop ms</td><td><strong>266</strong></td><td>326</td><td>83 (GC, concurrent)</td></tr><tr><td>dirtyheap</td><td>RSS MB</td><td><strong>718</strong></td><td>789</td><td>914</td></tr><tr><td>etl (1 GiB)</td><td>wall ms</td><td>10,260</td><td><strong>2,854</strong></td><td>10,429</td></tr><tr><td>etl</td><td>RSS MB</td><td>5.7</td><td><strong>4.0</strong></td><td>10.6</td></tr><tr><td>proxy (256 conns)</td><td>req/s (median of 3)</td><td>42,045</td><td>40,159</td><td><strong>47,563</strong></td></tr><tr><td>proxy</td><td>p99 ms (median of 3)</td><td>29.1</td><td>27.0</td><td><strong>18.5</strong></td></tr><tr><td>proxy</td><td>RSS MB</td><td><strong>7.2</strong></td><td>8.3</td><td>37.1</td></tr></tbody></table></div>
<p><strong>Grasm wins memory everywhere</strong> — 8×
on <code>chat</code>, 8.4 MB against Rust's
69.4, and <code>dirtyheap</code> now uses less than Rust's
arena while dropping faster than it. It loses <code>etl</code>,
<code>dirtyheap</code>, <code>ecs</code> and <code>chat</code> on time, and the
<code>proxy</code> tail to both.</p>
<p>What is behind each loss, measured rather than guessed. <code>etl</code>:
Grasm makes a value per string field and validates UTF-8 per slice where
Rust's decoder borrows from the line — borrowed string and bytes views are
the next lever, and that row is 10,260 ms against
2,854. <code>dirtyheap</code>: the URL is built as a string and
then copied into the field, where Rust formats it into place.
<code>ecs</code>: the update loop is at parity and the setup loop is behind on
instructions retired. The <code>proxy</code> tail is
<strong>unexplained</strong>, which is the honest state of it — see
below.</p>

<h3>Two published conclusions that a better measurement removed</h3>
<p>The <code>proxy</code> row was first taken as the median of three
<em>sequential</em> runs, and this machine drifts further than that between
them: the same binary, five times in a row, gave 52,157 down to 35,893 requests
a second. Re-run <em>interleaved</em> — grasm, go, rust, grasm, go, rust,
five rounds — two recorded conclusions did not survive:</p>
<div class="scroll"><table><thead><tr><th></th><th>Grasm</th><th>Rust</th><th>Go</th></tr></thead><tbody><tr><td>req/s</td><td>50,366</td><td>52,308</td><td><strong>53,128</strong></td></tr><tr><td>p99 ms</td><td>16.7</td><td><strong>12.8</strong></td><td>14.1</td></tr></tbody></table></div>
<p>Throughput is <strong>parity</strong>, not 12% behind Go. <strong>Rust leads
the tail</strong>, where the recorded row had it behind Grasm. What stands is
that Grasm's tail is last — 1.18× Go's and 1.30× Rust's, a real gap
and a much smaller one than the 1.57× that was published.</p>
<p>And the explanation that had been given for that tail is <em>wrong</em>.
<code>proxy/README.md</code> attributed it to connection affinity: a worker
owns the clients it accepted, so a busy worker stalls them while others idle.
That predicts the tail improves as workers are added, and it is disprovable
without building a work-stealing scheduler, because <strong>one worker per
connection is perfect distribution by construction</strong>. 32 workers against
256, for 256 connections: 28.7 ms and 27.1 ms at the p99. No change. So work
stealing would have bought nothing, and the cause is now unknown rather than
known-and-unfixed — which is a worse-sounding and more useful place to
be.</p>

<h2>Compile speed</h2>
<p><strong>535–547k lines a second</strong>
through parse and typecheck, against a 500k goal, and
<strong>p50 21 ms / p95 40 ms</strong>
edit-to-diagnostic through <code>grasm lsp</code> on the largest program here.
Four things got it there, all found by measuring: three superlinear defects
(a field chain that was exponential, a method chain that was exponential, a
nested tuple literal that was cubic); memoized inference, added for those
chains and worth <strong>2.2× on ordinary code too</strong>; <em>not</em>
freeing the syntax tree at exit, which was a third of <code>grasm check</code>
walking a tree node by node immediately before the operating system reclaimed
the whole address space in one step; and a non-cryptographic hasher for tables
keyed by byte offsets in a file the compiler is already reading.</p>
<p>The incremental figure is met at this repository's scale and the goal is
written for a million lines. There is no query engine: the server re-checks the
whole program on every change, and at 3,500 lines that is fast enough to hide
the problem.</p>

<h2>Binary size and task scaling</h2>
<p>A hello-world went <strong>351.1 KB →
85.7 KB</strong>, and the scheduler went
<strong>40,000 tasks in 199 s →
1,000,000 in 22 s</strong>.
<a href="achieved.html">Both are on the scorecard with what moved them →</a></p>

<h2>How any of this is measured</h2>
<p>Every rule below exists because of a specific mistake, and the mistake is
worth more than the rule.</p>
<div class="scroll"><table>
<tr><th>Rule</th><th>The mistake that bought it</th></tr>
<tr><td><b>Interleave the A and B runs</b></td><td>Sequential runs drift about 2% on this machine. Alternating the two binaries turned a reported 1.036× and 0.982× into 1.010× and 1.001× — both of the original readings were the drift.</td></tr>
<tr><td><b>Refuse to benchmark a busy machine</b></td><td><code>bench.sh</code> stops above a third of the cores being busy. The same build measured 478k LOC/s at load 9 and 571k at load 3; a number without its load is not a number.</td></tr>
<tr><td><b>Price the ceiling before building</b></td><td>Counting instructions overestimated three optimisations by 6–20×. The way to bound a gain is to <em>break</em> the thing — remove the work and measure what is left — not to model it.</td></tr>
<tr><td><b>A profiler's leaves lie about cost</b></td><td>A leaf worth 28% of <code>sample</code>'s samples was worth 1% of the clock, twice. Price a symbol by removing it, not by its share of samples.</td></tr>
<tr><td><b>Static frequency is not cost</b></td><td>Reference counting is 43% of every runtime call the compiler emits — 210,661 call sites to <code>grasm_val_dec</code> across 256 programs — and it does not appear in a profile at all. Those sites are cold. Allocation and thread-local lookups are the real cost, which is the same answer every measurement gives.</td></tr>
<tr><td><b>Subtract the harness</b></td><td>A constant added to every cell compresses every ratio toward 1. A 21 ms one made a losing suite read as parity, and it was published in three places before it was found in one.</td></tr>
<tr><td><b>macOS charges for a new executable</b></td><td>About 450 ms the first time a freshly built binary runs, flat with its size. Reuse the inode or measure that tax instead of the program.</td></tr>
</table></div>
<p class="note"><strong>And a load artefact looks exactly like a defect.</strong>
A native run killed under load exits <code>-9</code> with empty output, which
reads as a mismatch. Seven fuzzer disagreements, two cross-compile failures and
a hung concurrency test were all that, and all passed on re-run. The rule is to
re-run before debugging — which is also why the <code>crosstest</code> figure
on the <a href="achieved.html">scorecard</a> is annotated with the load it was
taken at.</p>

</main>
</div>
<footer class="sitefoot">
  <div class="inner">
    <div class="cols">
      <div class="brandcol">
        <a class="brand" href="index.html"><svg viewBox="0 0 48 48" width="20" height="20" fill="none" stroke="currentColor" stroke-width="6" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><path d="M9 6H39"/><path d="M24 14A13 13 0 1 0 37 27H26"/></svg><span>Grasm</span></a>
        <span>Four engines, one answer.</span>
      </div>
      <div class="links">
<div><span>Start</span><a href="different.html">Why Grasm</a><a href="start.html">Getting started</a><a href="install.html">Install</a><a href="playground.html">Playground</a></div>
<div><span>Learn</span><a href="tutorial.html">Tutorial</a><a href="tour.html">Tour</a><a href="effects.html">Effects</a><a href="tiers.html">Ownership</a><a href="errors.html">Errors</a><a href="concurrency.html">Concurrency</a><a href="security.html">Security</a></div>
<div><span>Reference</span><a href="syntax.html">Syntax</a><a href="language.html">Language</a><a href="library.html">Library</a><a href="packages.html">Packages</a><a href="spec.html">Specification</a><a href="architecture.html">Architecture</a></div>
<div><span>More</span><a href="examples.html">Examples</a><a href="achieved.html">Scorecard</a><a href="performance.html">Performance</a><a href="about.html">About</a><a href="status.html">Status</a><a href="roadmap.html">Roadmap</a></div>
      </div>
    </div>
    <p>Counts here are computed from the repository as the page is written;
    measurements are read from the document that recorded them.</p>
  </div>
</footer>
</body></html>