perf: compare each benchmark in adjacent app processes

.github/PERFORMANCE.md

@@ -1,8 +1,10 @@
 # Release performance CI
 
 `Nitro Performance` builds the dedicated `apps/benchmark` Release/Hermes app.
-Each platform runs base → head → head → base on the same machine, reversing case
-order for the second pair. Each case gets a fresh app process; installation,
+Each platform installs base and head side by side on the same machine. For each
+benchmark it calibrates base in a discarded process, measures base, then measures
+head immediately, before moving to the next benchmark. There is one AB pair,
+with five warmup batches and twenty measured batches per process. Each case gets a fresh app process; installation,
 startup, transport and process restarts are outside timing. There is no automatic
 third pair. Manual reruns are retained as identifiable workflow attempts.
 
@@ -22,14 +24,15 @@ tail latency. The report shows every observed change of at least 5%, including
 Promise cases. This is a presentation threshold, not a calibrated regression
 budget. Expand the report for all metrics, individual process medians, matched
 pair changes, and sample MAD relative to p50. Matching pooled medians do not prove
-equal performance. Two process pairs do not justify confidence intervals.
+equal performance. One pair cannot establish repeatability between launches or justify confidence
+intervals. Repeat measurement jobs or same-revision runs to investigate variation.
 
 Performance is currently report-only. Build, execution and malformed-result
 failures still fail CI. Turning observed differences into a regression gate needs
 empirical validation on unchanged commits and intentional slowdowns on each
 unchanged suite/testbed. No Promise case is permanently exempt. Scheduled/manual
 runs with the same base and head SHA measure baseline variation explicitly.
-Changed benchmark definitions run two head-only measurements as a new baseline,
+Changed benchmark definitions run one head-only measurement per case as a new baseline,
 without executing the old base app or publishing an invented paired baseline.
 This also handles the first rollout of a new runner protocol.
 
@@ -40,6 +43,9 @@ collection. Each measurement job downloads the immutable app artifact ID produce
 by its build job; base and head still run together on one machine. A changed suite
 builds only head. An identical base/head SHA reuses the same binary for both sides.
 Otherwise each revision is built from its own checkout with the same build script.
+Base uses `com.margelo.nitrobenchmark`; head uses `com.margelo.nitrobenchmark.head`.
+Android keeps its Java namespace and fully qualified activity name unchanged.
+For identical SHAs, both roles launch the single installed head binary.
 
 Use GitHub's **Re-run job and dependent jobs** on `measure-android` or
 `measure-ios` to repeat measurements without rebuilding successful ancestors.
@@ -52,8 +58,10 @@ The app artifact includes base/head SHAs, suite hashes, Release configuration,
 architecture and toolchain metadata. iOS apps are tar archives to preserve
 permissions and symlinks. Gradle's basic cache is the sole Android cache owner;
 Gradle still checks source/task inputs, while exact app reuse is by artifact ID.
-There is no new iOS compiler cache. First-run speed or CI stability improvements
-have not been measured; app reuse specifically avoids build work on manual reruns.
+There is no new iOS compiler cache. App reuse avoids build work on manual reruns. With 46 cases, a comparable suite
+uses 138 fresh processes (46 calibration + 46 base + 46 head), down from 230.
+Closer comparisons reduce time separation, but fixed base-first order can still
+introduce bias; same-revision runs are needed to assess that on each testbed.
 
 ## Artifacts and publishing
 

.github/workflows/performance.yml

@@ -130,7 +130,7 @@ jobs:
             --platform android --base-root base --head-root head \
             --architecture x86_64 --toolchain "$(java -version 2>&1 | head -1) / NDK 29.0.14206865" \
             --output apps/build.json
-          bash head/scripts/performance/build-android.sh "$GITHUB_WORKSPACE/head"
+          bash head/scripts/performance/build-android.sh "$GITHUB_WORKSPACE/head" com.margelo.nitrobenchmark.head
           cp head/apps/benchmark/android/app/build/outputs/apk/release/app-release.apk apps/head.apk
           if [[ "$(jq -r .baseSha apps/build.json)" == "$(jq -r .headSha apps/build.json)" ]]; then
             cp apps/head.apk apps/base.apk
@@ -260,7 +260,7 @@ jobs:
             --platform ios --base-root base --head-root head \
             --architecture arm64 --toolchain "$(xcodebuild -version | tr '\n' ' ')" \
             --output apps/build.json
-          bash head/scripts/performance/build-ios.sh "$GITHUB_WORKSPACE/head"
+          bash head/scripts/performance/build-ios.sh "$GITHUB_WORKSPACE/head" com.margelo.nitrobenchmark.head
           # Tar preserves the executable bits and symlinks inside each .app.
           tar -czf apps/head.app.tar.gz -C head/apps/benchmark/ios/build-benchmark/Build/Products/Release-iphonesimulator NitroBenchmark.app
           if [[ "$(jq -r .baseSha apps/build.json)" == "$(jq -r .headSha apps/build.json)" ]]; then

apps/benchmark/README.md

@@ -14,8 +14,8 @@ Debug benchmark or publish anything to Bencher.
 From the repository root:
 
 ```sh
-bash scripts/performance/build-android.sh "$PWD"
-bash scripts/performance/build-ios.sh "$PWD"
+bash scripts/performance/build-android.sh "$PWD" com.margelo.nitrobenchmark.head
+bash scripts/performance/build-ios.sh "$PWD" com.margelo.nitrobenchmark.head
 ```
 
 Both platforms use the normal `Release` configuration, an embedded optimized
@@ -25,18 +25,20 @@ permits cleartext only to `127.0.0.1` and `localhost` for the host receiver.
 
 ## Run locally
 
-For an already booted Android API 36 emulator, after building the APK:
+For an already booted Android API 36 emulator, compare two fresh launches of the
+same built APK to check measurement variation:
 
 ```sh
-bun scripts/performance/run-device.ts \
+bun scripts/performance/run-sequence.ts \
   --platform android \
-  --app apps/benchmark/android/app/build/outputs/apk/release/app-release.apk \
-  --output /tmp/nitro-benchmark.json \
+  --base-app apps/benchmark/android/app/build/outputs/apk/release/app-release.apk \
+  --head-app apps/benchmark/android/app/build/outputs/apk/release/app-release.apk \
+  --base-root "$PWD" \
+  --head-root "$PWD" \
+  --output-directory /tmp/nitro-benchmark \
   --device-id "$(adb get-serialno)" \
-  --run-id android-local-1 \
-  --reverse false \
-  --commit-sha "$(git rev-parse HEAD)" \
-  --suite-hash "$(bun scripts/performance/suite-hash.ts .)" \
+  --base-sha "$(git rev-parse HEAD)" \
+  --head-sha "$(git rev-parse HEAD)" \
   --device 'Local emulator' \
   --os-version 'Android 16 / API 36' \
   --architecture x86_64 \
@@ -49,10 +51,15 @@ The host installs each binary once, then launches a fresh process for each case
 and assembles their results. This releases Nitro's runtime-scoped JSI reference
 bookkeeping between cases; GC alone cannot clear that cache. Each process posts
 one result only after its timing is complete. Per-case raw results are kept beside
-the combined output in a `*-cases/` directory. Reversing the suite reverses the
-case launch order too. Startup, transport, and process restarts are not timed.
+the combined output in `base-1-cases/`, `head-1-cases/`, and
+`calibration-base-cases/` directories. For each case, calibration exits before
+base and head run back to back. Identical SHAs reuse one installed binary. Startup, transport, and process restarts are not timed.
 For iOS, use `--platform ios`, a simulator UDID for `--device-id`, and the built
-`NitroBenchmark.app` for `--app`, with matching simulator/toolchain metadata.
+`NitroBenchmark.app` for `--base-app` and `--head-app`, with matching
+simulator/toolchain metadata.
+To compare different revisions, build base from its own checkout using the default
+app ID (omit the second build-script argument), and build head with the `.head`
+ID shown above. Pass the corresponding app paths, source roots, and commit SHAs.
 Local runs do not upload results.
 
 Each metric targets 150 ms of timed work per sample (roughly

apps/benchmark/android/app/build.gradle

@@ -83,7 +83,7 @@ android {
 
     namespace = "com.margelo.nitrobenchmark"
     defaultConfig {
-        applicationId "com.margelo.nitrobenchmark"
+        applicationId = providers.gradleProperty("nitroBenchmarkApplicationId").getOrElse("com.margelo.nitrobenchmark")
         minSdkVersion rootProject.ext.minSdkVersion
         targetSdkVersion rootProject.ext.targetSdkVersion
         versionCode 1

scripts/performance/build-android.sh

@@ -1,8 +1,9 @@
 #!/usr/bin/env bash
 set -euo pipefail
-cd "${1:?Usage: build-android.sh <checkout-root>}"
+cd "${1:?Usage: build-android.sh <checkout-root> [application-id]}"
 # Keep both revisions on the NDK version recorded by this CI testbed.
 grep -Fq 'ndkVersion = "29.0.14206865"' apps/benchmark/android/build.gradle
 bun install --frozen-lockfile
 cd apps/benchmark/android
-./gradlew :app:assembleRelease --no-daemon --build-cache -PreactNativeArchitectures=x86_64
+./gradlew :app:assembleRelease --no-daemon --build-cache -PreactNativeArchitectures=x86_64 \
+  "-PnitroBenchmarkApplicationId=${2:-com.margelo.nitrobenchmark}"

scripts/performance/build-artifacts.test.ts

@@ -51,7 +51,7 @@ for (const platform of ['android', 'ios'] as const) {
           script,
           `#!/bin/bash
 set -eu
-echo "$(basename "$1")" >> "$GITHUB_WORKSPACE/builds"
+echo "$(basename "$1") \${2:-com.margelo.nitrobenchmark}" >> "$GITHUB_WORKSPACE/builds"
 if [[ '${platform}' == android ]]; then
   mkdir -p "$1/apps/benchmark/android/app/build/outputs/apk/release"
   echo "$(basename "$1")" > "$1/apps/benchmark/android/app/build/outputs/apk/release/app-release.apk"
@@ -93,7 +93,14 @@ fi
         })
         expect(
           (await Bun.file(path.join(root, 'builds')).text()).trim().split('\n')
-        ).toEqual(mode === 'paired' ? ['head', 'base'] : ['head'])
+        ).toEqual(
+          mode === 'paired'
+            ? [
+                'head com.margelo.nitrobenchmark.head',
+                'base com.margelo.nitrobenchmark',
+              ]
+            : ['head com.margelo.nitrobenchmark.head']
+        )
         const base = path.join(
           root,
           `apps/base.${platform === 'ios' ? 'app.tar.gz' : 'apk'}`

scripts/performance/build-ios.sh

@@ -1,6 +1,6 @@
 #!/usr/bin/env bash
 set -euo pipefail
-cd "${1:?Usage: build-ios.sh <checkout-root>}"
+cd "${1:?Usage: build-ios.sh <checkout-root> [application-id]}"
 bun install --frozen-lockfile
 cd apps/benchmark
 bundle install
@@ -14,6 +14,7 @@ xcodebuild \
   -configuration Release \
   -sdk iphonesimulator \
   -destination 'generic/platform=iOS Simulator' \
+  "PRODUCT_BUNDLE_IDENTIFIER=${2:-com.margelo.nitrobenchmark}" \
   ARCHS=arm64 \
   ONLY_ACTIVE_ARCH=YES \
   CODE_SIGNING_ALLOWED=NO \

scripts/performance/isolated-cases.test.ts

@@ -1,6 +1,6 @@
 import { describe, expect, test } from 'bun:test'
 import type { BenchmarkRunResult } from '../../apps/benchmark/src/benchmarks/types'
-import { runIsolatedCases } from './isolated-cases'
+import { combineIsolatedCases } from './isolated-cases'
 
 function result(index: number): BenchmarkRunResult {
   return {
@@ -44,13 +44,8 @@ function result(index: number): BenchmarkRunResult {
 }
 
 describe('fresh-process benchmark cases', () => {
-  test('assembles every case in order without changing samples or revision metadata', async () => {
-    const calls: number[] = []
-    const combined = await runIsolatedCases(async (index) => {
-      calls.push(index)
-      return result(index)
-    })
-    expect(calls).toEqual([0, 1, 2])
+  test('assembles every case in order without changing samples or revision metadata', () => {
+    const combined = combineIsolatedCases([0, 1, 2].map(result))
     expect(combined.metrics.map((m) => m.id)).toEqual(
       [0, 1, 2].map((i) => result(i).metrics[0]!.id)
     )
@@ -61,19 +56,14 @@ describe('fresh-process benchmark cases', () => {
     expect(combined.durationMs).toBe(300)
   })
 
-  test('stops immediately on a missing process result', async () => {
-    const calls: number[] = []
-    await expect(
-      runIsolatedCases(async (index) => {
-        calls.push(index)
-        if (index === 1) throw new Error('app timed out')
-        return result(index)
-      })
-    ).rejects.toThrow('app timed out')
-    expect(calls).toEqual([0, 1])
+  test('rejects incomplete suites', () => {
+    expect(() => combineIsolatedCases([])).toThrow()
+    expect(() => combineIsolatedCases([result(0), result(1)])).toThrow(
+      'Incomplete'
+    )
   })
 
-  test('rejects duplicate cases, wrong revisions, changed settings, and wrong indices', async () => {
+  test('rejects duplicate cases, wrong revisions, changed settings, and wrong indices', () => {
     const changes: ((r: BenchmarkRunResult) => void)[] = [
       (r) => {
         r.metrics[0]!.id = result(0).metrics[0]!.id
@@ -95,27 +85,20 @@ describe('fresh-process benchmark cases', () => {
       },
     ]
     for (const change of changes) {
-      await expect(
-        runIsolatedCases(async (index) => {
-          const r = result(index)
-          if (index === 1) change(r)
-          return r
-        })
-      ).rejects.toThrow()
+      const runs = [0, 1, 2].map(result)
+      change(runs[1]!)
+      expect(() => combineIsolatedCases(runs)).toThrow()
     }
   })
 
-  test('rejects an unbounded suite or a non-isolated first result before launching more', async () => {
+  test('rejects an unbounded suite or a non-isolated result', () => {
     for (const count of [0, 101]) {
-      await expect(
-        runIsolatedCases(async () => ({ ...result(0), benchmarkCount: count }))
-      ).rejects.toThrow()
+      expect(() =>
+        combineIsolatedCases([{ ...result(0), benchmarkCount: count }])
+      ).toThrow()
     }
-    await expect(
-      runIsolatedCases(async () => ({
-        ...result(0),
-        metrics: [result(0).metrics[0]!, result(1).metrics[0]!],
-      }))
-    ).rejects.toThrow('unexpected cases')
+    const runs = [0, 1, 2].map(result)
+    runs[0]!.metrics.push(result(1).metrics[0]!)
+    expect(() => combineIsolatedCases(runs)).toThrow('unexpected cases')
   })
 })

scripts/performance/isolated-cases.ts

@@ -1,16 +1,17 @@
 import type { BenchmarkRunResult } from '../../apps/benchmark/src/benchmarks/types'
 import { validateBenchmarkRun, validateExpectedRun } from './schema'
 
-/** Install once; the caller starts and terminates a fresh app process per case. */
-export async function runIsolatedCases(
-  runCase: (index: number) => Promise<BenchmarkRunResult>
-): Promise<BenchmarkRunResult> {
-  const first = validateBenchmarkRun(await runCase(0))
+/** Combine a complete suite without discarding any per-process raw samples. */
+export function combineIsolatedCases(
+  runs: readonly BenchmarkRunResult[]
+): BenchmarkRunResult {
+  const first = validateBenchmarkRun(runs[0])
   const count = first.benchmarkCount!
-  const runs: BenchmarkRunResult[] = []
+  if (runs.length !== count)
+    throw new Error('Incomplete isolated benchmark suite.')
   const ids = new Set<string>()
   for (let index = 0; index < count; index++) {
-    const run = index === 0 ? first : validateBenchmarkRun(await runCase(index))
+    const run = validateBenchmarkRun(runs[index])
     const { work: _firstWork, ...sharedConfiguration } = first.configuration
     validateExpectedRun(run, { ...sharedConfiguration, benchmarkIndex: index })
     if (
@@ -31,7 +32,6 @@ export async function runIsolatedCases(
       )
     }
     ids.add(run.metrics[0]!.id)
-    runs.push(run)
   }
   const configuration = { ...first.configuration }
   delete configuration.benchmarkIndex

scripts/performance/report-markdown.ts

@@ -124,7 +124,7 @@ export function renderPerformanceReportMarkdown(
   const lines = [
     '## Performance Report',
     '',
-    '> **Report only:** Measurements do not fail this PR. Process pairs describe this run; they do not establish statistical confidence.',
+    '> **Report only:** Measurements do not fail this PR. Each benchmark has one base/head process pair. Samples describe within-process variation; they do not establish repeatability between launches or statistical confidence.',
   ]
   if (options.baseSha === options.headSha) {
     lines.push(
@@ -153,7 +153,7 @@ export function renderPerformanceReportMarkdown(
     lines.push(
       '',
       '<details>',
-      '<summary>All benchmarks and process variation</summary>',
+      '<summary>All benchmarks and sample variation</summary>',
       '',
       table(platform.comparisons, platform.platform),
       '',