Skip to content

Build RPC CI Ingestion Load Test#741

Open
cjonas9 wants to merge 49 commits into
mainfrom
apply-load
Open

Build RPC CI Ingestion Load Test#741
cjonas9 wants to merge 49 commits into
mainfrom
apply-load

Conversation

@cjonas9
Copy link
Copy Markdown
Contributor

@cjonas9 cjonas9 commented May 15, 2026

What

TODO: Fill this out.

This is a PR implementing a repeatable CI ingestion load test on a full database of 7 days of ledgers. The approximate design is here:
Screenshot 2026-05-14 at 12 59 14 PM

This test, currently, is triggered on pushes to the branch apply-load, but will later be modified to trigger on any merge to main or on PR comments stating "run load test"

For proper credit attribution, some preliminary work from George's branch load-testing was migrated here. To avoid difficulties with migrating that branch to RPC's origin/head under the time crunch of this project, this branch was made.

Why

CI testing of RPC ingestion performance; benchmarking.

Known limitations

N/A

Copilot AI review requested due to automatic review settings May 15, 2026 22:40
@github-actions
Copy link
Copy Markdown
Contributor

⏳ Load test launching on i-063ed1e3a29f001e3 (commit 241bdf833edbfc18d3c312a7109168d171324aa1).
Workflow run: https://github.com/stellar/stellar-rpc/actions/runs/25944865928
Posting results when the run finishes (~15 min).

@github-actions
Copy link
Copy Markdown
Contributor

⏳ Load test launching on i-00512a51d1658eebf (commit ffb3299ea026f7d5678b8b3e2ae178bad6256691).
Workflow run: https://github.com/stellar/stellar-rpc/actions/runs/25972993065
Posting results when the run finishes.

@github-actions
Copy link
Copy Markdown
Contributor

❌ Load test did not produce results within 2700 s.

Instance: i-00512a51d1658eebf
Workflow run: https://github.com/stellar/stellar-rpc/actions/runs/25972993065

Last 40 lines of /var/log/user-data.log:

go: downloading cloud.google.com/go/pubsub v1.50.1
go: downloading github.com/felixge/httpsnoop v1.0.4
go: downloading google.golang.org/genproto/googleapis/api v0.0.0-20251029180050-ab9386a59fda
go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20251029180050-ab9386a59fda
go: downloading cloud.google.com/go/monitoring v1.24.3
go: downloading github.com/GoogleCloudPlatform/opentelemetry-operations-go/internal/resourcemapping v0.54.0
go: downloading github.com/GoogleCloudPlatform/opentelemetry-operations-go/detectors/gcp v1.30.0
go: downloading cloud.google.com/go/auth/oauth2adapt v0.2.8
go: downloading go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.63.0
go: downloading golang.org/x/net v0.47.0
go: downloading github.com/go-logr/logr v1.4.3
go: downloading go.opentelemetry.io/otel/metric v1.38.0
go: downloading github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.12
go: downloading github.com/jmespath/go-jmespath v0.4.0
go: downloading github.com/google/s2a-go v0.1.9
go: downloading go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.63.0
go: downloading golang.org/x/time v0.14.0
go: downloading go.opencensus.io v0.24.0
go: downloading github.com/fsnotify/fsnotify v1.6.0
go: downloading github.com/sagikazarmark/slog-shim v0.1.0
go: downloading github.com/mitchellh/mapstructure v1.5.0
go: downloading github.com/spf13/afero v1.10.0
go: downloading github.com/spf13/cast v1.5.1
go: downloading github.com/googleapis/enterprise-certificate-proxy v0.3.6
go: downloading github.com/go-logr/stdr v1.2.2
go: downloading go.opentelemetry.io/auto/sdk v1.2.1
go: downloading github.com/envoyproxy/go-control-plane/envoy v1.35.0
go: downloading cloud.google.com/go/pubsub/v2 v2.0.0
go: downloading github.com/hashicorp/hcl v1.0.0
go: downloading github.com/subosito/gotenv v1.6.0
go: downloading gopkg.in/ini.v1 v1.67.0
go: downloading github.com/magiconair/properties v1.8.7
go: downloading github.com/pelletier/go-toml/v2 v2.1.0
go: downloading github.com/spiffe/go-spiffe/v2 v2.6.0
go: downloading github.com/cncf/xds/go v0.0.0-20251031190108-5cf4b1949528
go: downloading golang.org/x/text v0.31.0
go: downloading github.com/go-jose/go-jose/v4 v4.1.3
go: downloading cel.dev/expr v0.25.0
go: downloading github.com/envoyproxy/protoc-gen-validate v1.2.1
go: downloading golang.org/x/crypto v0.45.0

@github-actions
Copy link
Copy Markdown
Contributor

⏳ Load test launching on i-078bb1a6cf48ebd5c (commit bae2fdcc976b16d71a9d31a2a0e846798628f765).
Workflow run: https://github.com/stellar/stellar-rpc/actions/runs/25976180010
Posting results when the run finishes.

@github-actions
Copy link
Copy Markdown
Contributor

❌ Load test did not produce results within 2700 s.

Instance: i-078bb1a6cf48ebd5c
Workflow run: https://github.com/stellar/stellar-rpc/actions/runs/25976180010

Last 40 lines of /var/log/user-data.log:

go: downloading cloud.google.com/go/pubsub v1.50.1
go: downloading github.com/felixge/httpsnoop v1.0.4
go: downloading github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.12
go: downloading google.golang.org/genproto/googleapis/api v0.0.0-20251029180050-ab9386a59fda
go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20251029180050-ab9386a59fda
go: downloading cloud.google.com/go/monitoring v1.24.3
go: downloading github.com/GoogleCloudPlatform/opentelemetry-operations-go/internal/resourcemapping v0.54.0
go: downloading github.com/GoogleCloudPlatform/opentelemetry-operations-go/detectors/gcp v1.30.0
go: downloading cloud.google.com/go/auth/oauth2adapt v0.2.8
go: downloading go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.63.0
go: downloading github.com/go-logr/logr v1.4.3
go: downloading golang.org/x/net v0.47.0
go: downloading go.opentelemetry.io/otel/metric v1.38.0
go: downloading github.com/jmespath/go-jmespath v0.4.0
go: downloading github.com/google/s2a-go v0.1.9
go: downloading go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.63.0
go: downloading golang.org/x/time v0.14.0
go: downloading github.com/fsnotify/fsnotify v1.6.0
go: downloading github.com/mitchellh/mapstructure v1.5.0
go: downloading github.com/sagikazarmark/slog-shim v0.1.0
go: downloading github.com/spf13/afero v1.10.0
go: downloading github.com/spf13/cast v1.5.1
go: downloading go.opencensus.io v0.24.0
go: downloading github.com/googleapis/enterprise-certificate-proxy v0.3.6
go: downloading github.com/go-logr/stdr v1.2.2
go: downloading go.opentelemetry.io/auto/sdk v1.2.1
go: downloading github.com/envoyproxy/go-control-plane/envoy v1.35.0
go: downloading github.com/subosito/gotenv v1.6.0
go: downloading github.com/hashicorp/hcl v1.0.0
go: downloading gopkg.in/ini.v1 v1.67.0
go: downloading github.com/magiconair/properties v1.8.7
go: downloading github.com/pelletier/go-toml/v2 v2.1.0
go: downloading cloud.google.com/go/pubsub/v2 v2.0.0
go: downloading golang.org/x/text v0.31.0
go: downloading github.com/spiffe/go-spiffe/v2 v2.6.0
go: downloading github.com/cncf/xds/go v0.0.0-20251031190108-5cf4b1949528
go: downloading github.com/go-jose/go-jose/v4 v4.1.3
go: downloading github.com/envoyproxy/protoc-gen-validate v1.2.1
go: downloading cel.dev/expr v0.25.0
go: downloading golang.org/x/crypto v0.45.0

@github-actions
Copy link
Copy Markdown
Contributor

⏳ Load test launching on i-03fd0df99bc9f26ab (commit 70eb9dc7e4401ac88fefe521202bea4c27336077).
Workflow run: https://github.com/stellar/stellar-rpc/actions/runs/25978515373
Posting results when the run finishes.

@github-actions
Copy link
Copy Markdown
Contributor

Ingest load test failed (run 25978515373 on 70eb9dc7e4401ac88fefe521202bea4c27336077)

benchmark failed:
    test.go:649: rpc="daemon" time="2026-05-17T02:45:22.330Z" level=info msg="starting request" host="localhost:34463" ip="127.0.0.1:56550" method=POST path=/ pid=21767 req=ip-172-31-22-206/H6kXavv6R0-044009 subsys=http useragent=Go-http-client/1.1
    test.go:649: rpc="daemon" time="2026-05-17T02:45:22.330Z" level=info msg="starting JSONRPC request" json_req=22005 method=getHealth pid=21767 req=44010 subsys=jsonrpc
    test.go:649: rpc="daemon" time="2026-05-17T02:45:22.355Z" level=info msg="finished JSONRPC request" duration=24.617106ms json_req=44010 pid=21767 req=44010 status=ok subsys=jsonrpc
    test.go:649: rpc="daemon" time="2026-05-17T02:45:22.355Z" level=info msg="finished request" bytes=137 duration=24.844428ms ip="127.0.0.1:56550" method=POST path=/ pid=21767 req=ip-172-31-22-206/H6kXavv6R0-044009 route=/ status=200 subsys=http
    test.go:649: rpc="daemon" time="2026-05-17T02:45:22.355Z" level=info msg="starting request" host="localhost:34463" ip="127.0.0.1:56550" method=POST path=/ pid=21767 req=ip-172-31-22-206/H6kXavv6R0-044011 subsys=http useragent=Go-http-client/1.1
    test.go:649: rpc="daemon" time="2026-05-17T02:45:22.355Z" level=info msg="starting JSONRPC request" json_req=22006 method=getHealth pid=21767 req=44012 subsys=jsonrpc
    test.go:649: rpc="daemon" time="2026-05-17T02:45:22.388Z" level=info msg="finished JSONRPC request" duration=33.156138ms json_req=44012 pid=21767 req=44012 status=ok subsys=jsonrpc
    test.go:649: rpc="daemon" time="2026-05-17T02:45:22.389Z" level=info msg="finished request" bytes=137 duration=33.395593ms ip="127.0.0.1:56550" method=POST path=/ pid=21767 req=ip-172-31-22-206/H6kXavv6R0-044011 route=/ status=200 subsys=http
    test.go:649: rpc="daemon" time="2026-05-17T02:45:22.389Z" level=info msg="starting request" host="localhost:34463" ip="127.0.0.1:56550" method=POST path=/ pid=21767 req=ip-172-31-22-206/H6kXavv6R0-044013 subsys=http useragent=Go-http-client/1.1
    test.go:649: rpc="daemon" time="2026-05-17T02:45:22.389Z" level=info msg="starting JSONRPC request" json_req=22007 method=getHealth pid=21767 req=44014 subsys=jsonrpc
    test.go:649: rpc="daemon" time="2026-05-17T02:45:22.413Z" level=info msg="finished JSONRPC request" duration=23.991986ms json_req=44014 pid=21767 req=44014 status=ok subsys=jsonrpc
    test.go:649: rpc="daemon" time="2026-05-17T02:45:22.413Z" level=info msg="finished request" bytes=137 duration=24.22574ms ip="127.0.0.1:56550" method=POST path=/ pid=21767 req=ip-172-31-22-206/H6kXavv6R0-044013 route=/ status=200 subsys=http
    test.go:649: rpc="daemon" time="2026-05-17T02:45:22.413Z" level=info msg="starting request" host="localhost:34463" ip="127.0.0.1:56550" method=POST path=/ pid=21767 req=ip-172-31-22-206/H6kXavv6R0-044015 subsys=http useragent=Go-http-client/1.1
    test.go:649: rpc="daemon" time="2026-05-17T02:45:22.414Z" level=info msg="starting JSONRPC request" json_req=22008 method=getHealth pid=21767 req=44016 subsys=jsonrpc
    test.go:649: rpc="daemon" time="2026-05-17T02:45:22.439Z" level=info msg="finished JSONRPC request" duration=25.77648ms json_req=44016 pid=21767 req=44016 status=ok subsys=jsonrpc
    test.go:649: rpc="daemon" time="2026-05-17T02:45:22.440Z" level=info msg="finished request" bytes=137 duration=26.009048ms ip="127.0.0.1:56550" method=POST path=/ pid=21767 req=ip-172-31-22-206/H6kXavv6R0-044015 route=/ status=200 subsys=http
    test.go:649: rpc="daemon" time="2026-05-17T02:45:22.440Z" level=info msg="starting request" host="localhost:34463" ip="127.0.0.1:56550" method=POST path=/ pid=21767 req=ip-172-31-22-206/H6kXavv6R0-044017 subsys=http useragent=Go-http-client/1.1
    test.go:649: rpc="daemon" time="2026-05-17T02:45:22.440Z" level=info msg="starting JSONRPC request" json_req=22009 method=getHealth pid=21767 req=44018 subsys=jsonrpc
    test.go:649: rpc="daemon" time="2026-05-17T02:45:22.463Z" level=info msg="finished JSONRPC request" duration=22.702325ms json_req=44018 pid=21767 req=44018 status=ok subsys=jsonrpc
    test.go:649: rpc="daemon" time="2026-05-17T02:45:22.463Z" level=info msg="finished request" bytes=137 duration=22.91114ms ip="127.0.0.1:56550" method=POST path=/ pid=21767 req=ip-172-31-22-206/H6kXavv6R0-044017 route=/ status=200 subsys=http
    test.go:649: rpc="daemon" time="2026-05-17T02:45:22.463Z" level=info msg="starting request" host="localhost:34463" ip="127.0.0.1:56550" method=POST path=/ pid=21767 req=ip-172-31-22-206/H6kXavv6R0-044019 subsys=http useragent=Go-http-client/1.1
    test.go:649: rpc="daemon" time="2026-05-17T02:45:22.463Z" level=info msg="starting JSONRPC request" json_req=22010 method=getHealth pid=21767 req=44020 subsys=jsonrpc
    test.go:649: rpc="daemon" time="2026-05-17T02:45:22.487Z" level=info msg="finished JSONRPC request" duration=23.921722ms json_req=44020 pid=21767 req=44020 status=ok subsys=jsonrpc
    test.go:649: rpc="daemon" time="2026-05-17T02:45:22.487Z" level=info msg="finished request" bytes=137 duration=24.137282ms ip="127.0.0.1:56550" method=POST path=/ pid=21767 req=ip-172-31-22-206/H6kXavv6R0-044019 route=/ status=200 subsys=http
    test.go:649: rpc="daemon" time="2026-05-17T02:45:22.488Z" level=info msg="starting request" host="localhost:34463" ip="127.0.0.1:56550" method=POST path=/ pid=21767 req=ip-172-31-22-206/H6kXavv6R0-044021 subsys=http useragent=Go-http-client/1.1
    test.go:649: rpc="daemon" time="2026-05-17T02:45:22.488Z" level=info msg="starting JSONRPC request" json_req=22011 method=getHealth pid=21767 req=44022 subsys=jsonrpc
    test.go:649: rpc="daemon" time="2026-05-17T02:45:22.513Z" level=info msg="finished JSONRPC request" duration=25.161404ms json_req=44022 pid=21767 req=44022 status=ok subsys=jsonrpc
    test.go:649: rpc="daemon" time="2026-05-17T02:45:22.513Z" level=info msg="finished request" bytes=137 duration=25.381143ms ip="127.0.0.1:56550" method=POST path=/ pid=21767 req=ip-172-31-22-206/H6kXavv6R0-044021 route=/ status=200 subsys=http
    test.go:649: rpc="daemon" time="2026-05-17T02:45:22.513Z" level=info msg="starting request" host="localhost:34463" ip="127.0.0.1:56550" method=POST path=/ pid=21767 req=ip-172-31-22-206/H6kXavv6R0-044023 subsys=http useragent=Go-http-client/1.1
    test.go:649: rpc="daemon" time="2026-05-17T02:45:22.513Z" level=info msg="starting JSONRPC request" json_req=22012 method=getHealth pid=21767 req=44024 subsys=jsonrpc
    test.go:649: rpc="daemon" time="2026-05-17T02:45:22.533Z" level=info msg="Ingesting ledger 62540206" pid=21767
    test.go:649: rpc="daemon" time="2026-05-17T02:45:22.536Z" level=info msg="finished JSONRPC request" duration=22.833463ms json_req=44024 pid=21767 req=44024 status=ok subsys=jsonrpc
    test.go:649: rpc="daemon" time="2026-05-17T02:45:22.536Z" level=info msg="finished request" bytes=137 duration=23.029756ms ip="127.0.0.1:56550" method=POST path=/ pid=21767 req=ip-172-31-22-206/H6kXavv6R0-044023 route=/ status=200 subsys=http
    test.go:649: rpc="daemon" time="2026-05-17T02:45:22.537Z" level=info msg="starting request" host="localhost:34463" ip="127.0.0.1:56550" method=POST path=/ pid=21767 req=ip-172-31-22-206/H6kXavv6R0-044025 subsys=http useragent=Go-http-client/1.1
    test.go:649: rpc="daemon" time="2026-05-17T02:45:22.537Z" level=info msg="starting JSONRPC request" json_req=22013 method=getHealth pid=21767 req=44026 subsys=jsonrpc
    test.go:649: rpc="daemon" time="2026-05-17T02:45:22.562Z" level=info msg="finished JSONRPC request" duration=24.831344ms json_req=44026 pid=21767 req=44026 status=ok subsys=jsonrpc
    test.go:649: rpc="daemon" time="2026-05-17T02:45:22.562Z" level=info msg="finished request" bytes=137 duration=25.082474ms ip="127.0.0.1:56550" method=POST path=/ pid=21767 req=ip-172-31-22-206/H6kXavv6R0-044025 route=/ status=200 subsys=http
    test.go:649: rpc="daemon" time="2026-05-17T02:45:22.562Z" level=info msg="starting request" host="localhost:34463" ip="127.0.0.1:56550" method=POST path=/ pid=21767 req=ip-172-31-22-206/H6kXavv6R0-044027 subsys=http useragent=Go-http-client/1.1
    test.go:649: rpc="daemon" time="2026-05-17T02:45:22.562Z" level=info msg="starting JSONRPC request" json_req=22014 method=getHealth pid=21767 req=44028 subsys=jsonrpc
    test.go:649: rpc="daemon" time="2026-05-17T02:45:22.588Z" level=info 

@github-actions
Copy link
Copy Markdown
Contributor

⏳ Load test launching on i-0c4b28ca3d70de2d9 (commit 705edecf2e1fc28c81cf2b5dbe9ac20e082dafea).
Workflow run: https://github.com/stellar/stellar-rpc/actions/runs/25982721783
Posting results when the run finishes.

@github-actions
Copy link
Copy Markdown
Contributor

❌ Load test did not produce results within 7200 s.

Instance: i-0c4b28ca3d70de2d9
Workflow run: https://github.com/stellar/stellar-rpc/actions/runs/25982721783

Last 40 lines of /var/log/user-data.log:

__DEBUG_TAIL_UNAVAILABLE__

@github-actions
Copy link
Copy Markdown
Contributor

⏳ Load test launching on i-0f14b45d6e547cdec (commit caec3f4347b2ff935f6d03e75a1f27d4f678b54a).
Workflow run: https://github.com/stellar/stellar-rpc/actions/runs/26017754277
Posting results when the run finishes.

@github-actions
Copy link
Copy Markdown
Contributor

📈 Ingest load test — caec3f4

Metric Value
Ledgers replayed 1000
Throughput 1.3949769047712368 ledgers/sec
Ingest wall-clock 716.857746232s
Per-ledger p50 / p95 / p99 500.001 / 724.999 / 974.999 ms
Golden DB fetch+decompress 1445s
Total benchmark time 1519s
stellar-core Warning: running non-release version v26.0.0-40-g6284d3fa6 of stellar-core
Workflow run #26017754277

@github-actions
Copy link
Copy Markdown
Contributor

⏳ Load test launching on i-03bab76ac4e7160c7 (commit b32181d083cf41e235ba52c9173bcba64a850de3).
Workflow run: https://github.com/stellar/stellar-rpc/actions/runs/26067674134
Posting results when the run finishes.

@github-actions
Copy link
Copy Markdown
Contributor

Ingest load test failed (run 26067674134 on b32181d083cf41e235ba52c9173bcba64a850de3)

volume throttle could not be confirmed; refusing to run benchmark on an unverified volume throughput

@github-actions
Copy link
Copy Markdown
Contributor

⏳ Load test launching on i-073d0854300060a29 (commit 3a0571f7affa6c1060f54eddcacd1e3ec589b317).
Workflow run: https://github.com/stellar/stellar-rpc/actions/runs/26070055454
Posting results when the run finishes.

@github-actions
Copy link
Copy Markdown
Contributor

📈 Ingest load test — 3a0571f

Metric Value
Ledgers replayed 1000
Throughput 1.2738550742284451 ledgers/sec
Ingest wall-clock 785.018657327s
Per-ledger p50 / p95 / p99 549.998 / 950 / 1275 ms
Golden DB fetch+decompress 1447s
Total benchmark time 1755s
stellar-core Warning: running non-release version v26.0.0-40-g6284d3fa6 of stellar-core
Workflow run #26070055454

@github-actions
Copy link
Copy Markdown
Contributor

⏳ Load test launching on i-0a9eae25e0e5c4ff4 (commit 2c4a004089cd96eba3f500620aa2c61be64110b8).
Workflow run: https://github.com/stellar/stellar-rpc/actions/runs/26192960169
Posting results when the run finishes.

@github-actions
Copy link
Copy Markdown
Contributor

📈 Ingest load test — 2c4a004

Metric Value
Ledgers replayed 1000
Throughput 1.2945867670661413 ledgers/sec
Ingest wall-clock 772.447259187s
Per-ledger p50 / p95 / p99 525 / 875 / 1175 ms
Golden DB fetch+decompress 1542s
Total benchmark time 1930s
stellar-core Warning: running non-release version v26.0.0-40-g6284d3fa6 of stellar-core
Workflow run #26192960169

@github-actions
Copy link
Copy Markdown
Contributor

⏳ Load test launching on i-092d3326c0fa398eb (commit d620450c898418cd8ed13223fc6c4e1e3844ae47).
Workflow run: https://github.com/stellar/stellar-rpc/actions/runs/26524753526
Posting results when the run finishes.

@github-actions
Copy link
Copy Markdown
Contributor

Ingest load test failed (run 26524753526 on d620450c898418cd8ed13223fc6c4e1e3844ae47)

golden DB hash mismatch: expected [2026-05-27T16:37:20Z] no sha256-raw on s3://stellar-rpc-ci-load-test/current/golden.sqlite.zst; skipping golden DB checksum, got 68d7569e57f924efaddd6629112d116a2a1d2a650f0d0e0cc329c00d35be7ada

@github-actions
Copy link
Copy Markdown
Contributor

⏳ Load test launching on i-082c03a655a2a6b35 (commit 41529f46b293a83d2feb55c923405dc4455c73d4).
Workflow run: https://github.com/stellar/stellar-rpc/actions/runs/26526623820
Posting results when the run finishes.

@github-actions
Copy link
Copy Markdown
Contributor

📈 Ingest load test — 41529f4

Metric Value
Ledgers replayed 1000
Throughput 0.9680488746003295 ledgers/sec
Ingest wall-clock 1033.005694483s
Per-ledger p50 / p95 / p99 750 / 1350 / 1825 ms
Golden DB fetch+decompress 1553s
stellar-core v26.0.0-40-g6284d3fa6
Workflow run #26526623820

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants