Skip to content

ci: try Blacksmith runners for performance benchmarks #1614

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mrousavy merged 2 commits into from
Sep 7, 2026
Merged

ci: try Blacksmith runners for performance benchmarks #1614

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
16367b7
ci: try Blacksmith runners for performance benchmarks
mrousavy Sep 7, 2026
ae2d72d
Merge branch 'main' into perf/blacksmith
mrousavy Sep 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion .github/PERFORMANCE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,14 @@ consume its new manifest. Merge the complete producer/publisher stack before rel
there is no permanent old-schema reporting path. Existing raw app results may
contain extra summary fields, which the raw parser ignores.

## Android host requirements
## CI runners

Platform builds and measurements use Blacksmith: `blacksmith-4vcpu-ubuntu-2404`
for Android and `blacksmith-6vcpu-macos-26` for iOS. Raw result device descriptions
identify the requested runner tier. Preparation,
collection and trusted publishing remain on GitHub-hosted runners. Changing
runner hardware requires checking same-code variation again; absolute timings
from different hosts are not evidence of a Nitro performance change.

The API 36 x86_64 emulator requires KVM. CI checks `/dev/kvm`, verifies acceleration
before boot and uses `-accel on`; it must not silently use software CPU emulation.
Expand Down
12 changes: 6 additions & 6 deletions .github/workflows/performance.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,7 @@ jobs:
build-android:
needs: prepare
if: needs.prepare.outputs.relevant == 'true'
runs-on: ubuntu-24.04
runs-on: blacksmith-4vcpu-ubuntu-2404
timeout-minutes: 40
outputs:
artifact_id: ${{ steps.upload.outputs.artifact-id }}
Expand Down Expand Up @@ -150,7 +150,7 @@ jobs:

measure-android:
needs: [prepare, build-android]
runs-on: ubuntu-24.04
runs-on: blacksmith-4vcpu-ubuntu-2404
timeout-minutes: 75
outputs:
artifact_id: ${{ steps.upload.outputs.artifact-id }}
Expand Down Expand Up @@ -213,7 +213,7 @@ jobs:
--base-sha "$BASE_SHA" --head-sha "$HEAD_SHA"
--output-directory performance-android
--device-id "$(adb get-serialno)"
--device "Pixel 7 emulator / KVM" --os-version "Android 16 / API 36"
--device "Pixel 7 emulator / KVM / Blacksmith 4 vCPU" --os-version "Android 16 / API 36"
--architecture x86_64 --toolchain "$(jq -r .toolchain apps/build.json)"
- name: Upload Android results
id: upload
Expand All @@ -228,7 +228,7 @@ jobs:
build-ios:
needs: prepare
if: needs.prepare.outputs.relevant == 'true'
runs-on: macos-26
runs-on: blacksmith-6vcpu-macos-26
timeout-minutes: 60
outputs:
artifact_id: ${{ steps.upload.outputs.artifact-id }}
Expand Down Expand Up @@ -281,7 +281,7 @@ jobs:

measure-ios:
needs: [prepare, build-ios]
runs-on: macos-26
runs-on: blacksmith-6vcpu-macos-26
timeout-minutes: 90
outputs:
artifact_id: ${{ steps.upload.outputs.artifact-id }}
Expand Down Expand Up @@ -328,7 +328,7 @@ jobs:
--build-metadata apps/build.json \
--base-sha "$BASE_SHA" --head-sha "$HEAD_SHA" \
--output-directory performance-ios --device-id "$DEVICE_ID" \
--device 'iPhone 17 Pro simulator' --os-version 'iOS 26.5' \
--device 'iPhone 17 Pro simulator / Blacksmith 6 vCPU' --os-version 'iOS 26.5' \
--architecture arm64 --toolchain "$(jq -r .toolchain apps/build.json)"
- name: Delete simulator
if: always() && env.DEVICE_ID != ''
Expand Down