Target
Treat Try Playwright as a pet distributed system and actually scale it.
Load to design for: ~1M concurrent users, ~3 runs/user/minute → ~50k new runs/s. If a run lasts ~15s, that is ~750k concurrent executions.
Today we are nowhere near that: one control replica, exclusive RabbitMQ reply queues, in-memory SSE, warm one-shot pods (~1 CPU / 1Gi, RestartPolicy: Never). That model is correct for isolation, and cannot hit this load. The work is to keep isolation while moving cost from “a Kubernetes pod per snippet” to “a dense, recycled sandbox plus a horizontally scaled control plane.”
Related: live logs (#320 / log-watch SSE) pin streams to one control process. That has to go before control HPA is real.
Non-goals (until the steps below)
- Multi-region (do this last).
- Live screenshot streaming.
- Perfect fairness across languages on day one.
Steps
1. Name the bottleneck, then split the control plane from the request path
Why: handleRun still borrows a pod, talks AMQP, and (for SSE) holds state in process. Exclusive reply queues already mean control cannot HPA. K8s Pods.Create on the user path will not do 50k/s.
Do: Control becomes a stateless API: auth, admit, enqueue runId, return 202. Logs/SSE read a shared stream. A pool manager (not the API) creates/deletes sandboxes.
|
|
| Pro |
API replicas scale; failures don’t drop every in-flight AMQP consumer. |
| Con |
Two extra moving parts (queue + run store). |
2. Shared run log bus (replace in-memory hub + exclusive AMQP replies)
Why: Live logs and multi-replica control need the same story: runId → log lines → done.
Do: Redis Streams or NATS JetStream keyed by runId (job payload + log/done). SSE gateways XREAD/subscribe that key. Keep RabbitMQ only if we outgrow “simple” before we outgrow one broker box—don’t run two brokers.
|
|
| Pro |
Any replica can serve /log-watch; control HPA works. |
| Con |
New ops surface; must TTL/trim streams or they become a disk bomb. |
3. Admission, queues, and fairness (before more hardware)
Why: 50k/s of Chromium is a budget problem first. Unbounded enqueue is how you melt the cluster.
Do: Global + per-user token buckets; max in-flight; queue with delay SLO (e.g. p99 queue wait 2s, else 429). Shed load at the edge (Caddy/CDN) before Go.
|
|
| Pro |
Predictable cost; protects workers. |
| Con |
Users wait or get 429; need a clear UX. |
4. Dense workers, not one k8s pod per run
Why: 750k pods is not a cluster, it’s a career. Isolation still matters (arbitrary code).
Do, in order:
- Warm pool + KEDA on current one-shot pods — good to ~hundreds of concurrent runs, teaches autoscaling. Not the 1M design.
- Long-lived VM/microVM sandboxes (Firecracker/gVisor/Kata), recycle per run (wipe FS, kill leftover browsers). Pool sized by in-flight runs, not by “API replicas.”
- Pack N slots per node with hard cgroup + network egress via the proxy mesh.
|
|
| Pro |
10–100× density vs pod-per-run; still throw away the world after each snippet. |
| Con |
Biggest engineering chunk; recycling bugs = escape/cross-talk. |
Skip “process-level multi-tenant Node” without a VM boundary at this scale.
5. Autoscale every stateless tier the same way
HPA/KEDA on:
- frontend/Caddy (static + reverse proxy; CDN the assets)
- control API (after 1–2)
- SSE/log gateways (separate from POST /run so a million EventSources don’t block admission)
- file-service → become a thin validator; presign PUT/GET to object storage, don’t stream all bytes through one Go deploy
- egress proxy (Squid is a single choke point): fleet of proxies, or eBPF/envoy egress, with per-run identity so one user can’t eat the NAT
etcd for shares will not take this write rate → Redis/S3 for snippet blobs.
|
|
| Pro |
Each piece scales on its own signal (QPS, stream count, egress bandwidth). |
| Con |
More services to page on; need one load-test harness or you scale the wrong replica. |
6. Multi-region last
Why: Browsers + artifact uploads hate one continent. Only pays off after a dense worker pool exists.
Do: Regional worker+proxy+log-bus; global admission (or regional tokens); run id stays in-region.
|
|
| Pro |
Latency and blast radius. |
| Con |
Consistency for share URLs; 3× ops. |
Suggested order of attack
1 → 2 → 3 (control actually scales, load can be refused) → 5 on edge/files/proxy (cheap wins) → 4 (the real capacity) → 6.
How we know we’re not lying
A load profile, not vibes: N concurrent SSE clients, R runs/s, queue wait p99, $/run. First milestone can be 100 concurrent runs with HPA control; the 1M-user math is the north star so we don’t “autoscaling” a pod-per-run control loop and call it done.
Target
Treat Try Playwright as a pet distributed system and actually scale it.
Load to design for: ~1M concurrent users, ~3 runs/user/minute → ~50k new runs/s. If a run lasts ~15s, that is ~750k concurrent executions.
Today we are nowhere near that: one control replica, exclusive RabbitMQ reply queues, in-memory SSE, warm one-shot pods (~1 CPU / 1Gi,
RestartPolicy: Never). That model is correct for isolation, and cannot hit this load. The work is to keep isolation while moving cost from “a Kubernetes pod per snippet” to “a dense, recycled sandbox plus a horizontally scaled control plane.”Related: live logs (#320 / log-watch SSE) pin streams to one control process. That has to go before control HPA is real.
Non-goals (until the steps below)
Steps
1. Name the bottleneck, then split the control plane from the request path
Why:
handleRunstill borrows a pod, talks AMQP, and (for SSE) holds state in process. Exclusive reply queues already mean control cannot HPA. K8sPods.Createon the user path will not do 50k/s.Do: Control becomes a stateless API: auth, admit, enqueue
runId, return202. Logs/SSE read a shared stream. A pool manager (not the API) creates/deletes sandboxes.2. Shared run log bus (replace in-memory hub + exclusive AMQP replies)
Why: Live logs and multi-replica control need the same story:
runId → log lines → done.Do: Redis Streams or NATS JetStream keyed by
runId(job payload +log/done). SSE gatewaysXREAD/subscribethat key. Keep RabbitMQ only if we outgrow “simple” before we outgrow one broker box—don’t run two brokers./log-watch; control HPA works.3. Admission, queues, and fairness (before more hardware)
Why: 50k/s of Chromium is a budget problem first. Unbounded enqueue is how you melt the cluster.
Do: Global + per-user token buckets; max in-flight; queue with delay SLO (e.g. p99 queue wait 2s, else 429). Shed load at the edge (Caddy/CDN) before Go.
4. Dense workers, not one k8s pod per run
Why: 750k pods is not a cluster, it’s a career. Isolation still matters (arbitrary code).
Do, in order:
Skip “process-level multi-tenant Node” without a VM boundary at this scale.
5. Autoscale every stateless tier the same way
HPA/KEDA on:
etcd for shares will not take this write rate → Redis/S3 for snippet blobs.
6. Multi-region last
Why: Browsers + artifact uploads hate one continent. Only pays off after a dense worker pool exists.
Do: Regional worker+proxy+log-bus; global admission (or regional tokens); run
idstays in-region.Suggested order of attack
1 → 2 → 3(control actually scales, load can be refused) →5on edge/files/proxy (cheap wins) →4(the real capacity) →6.How we know we’re not lying
A load profile, not vibes: N concurrent SSE clients, R runs/s, queue wait p99, $/run. First milestone can be 100 concurrent runs with HPA control; the 1M-user math is the north star so we don’t “autoscaling” a pod-per-run control loop and call it done.