feat(aggregator): scaffold plugin-registry aggregator#971
Merged
Conversation
…ndler The registry-cli bundler resolves the plugin descriptor by looking for a default export (function or pre-resolved descriptor) or a named `createPlugin` export. The named-only `marketplaceTestPlugin` export wasn't discoverable, so `emdash-registry bundle --dir packages/plugins/marketplace-test` failed with INVALID_PLUGIN_FORMAT. Add `export default marketplaceTestPlugin` to match the convention used by other plugins (e.g. `forms`, `color`).
Lands the apps/aggregator project skeleton plus the shared
@emdash-cms/atproto-test-utils package skeleton. No business logic yet —
this PR establishes the structure so subsequent PRs each touch one
component.
apps/aggregator
- Wrangler config: D1, Records Queue (+ DLQ), Records DO, 6h reconciliation
Cron, vars (JETSTREAM_URL, CONSTELLATION_URL, WANTED_COLLECTIONS).
- Cloudflare Vite plugin for dev/build (`vite dev` / `vite build`),
`wrangler deploy` after build for ship.
- Initial D1 migration `0001_init.sql` lands every v1 table at once
(packages, releases + FK + idx, release_duplicate_attempts,
mirrored_artifacts, labels, label_state + partial enforce idx,
labellers, packages_fts + triggers, ingest_state, known_publishers).
Slices 2 and 3 read these tables but don't add new ones, so this is
the only DDL we expect to ship before NSID stabilisation.
- Worker entrypoint exports RecordsJetstreamDO + a no-op default with
fetch/queue/scheduled stubs; subsequent PRs fill them in.
- Test rig uses @cloudflare/vitest-pool-workers v0.16's cloudflareTest()
plugin. Migrations are read at config time and piped into the worker
isolate via the TEST_MIGRATIONS binding; tests apply them in beforeAll.
Smoke test proves migrations apply, INSERT round-trips, FTS5 trigger
fires, and the FK rejects orphan releases.
packages/atproto-test-utils (skeleton)
- Private workspace package consumed by both registry-cli (later) and
apps/aggregator. PR 1b lands the real-crypto MockPds + MockJetstream
+ MockDidResolver + createFakePublisher helper.
Workspace
- apps/* added to pnpm-workspace.yaml.
- Catalog entries added for @atcute/{car,cbor,cid,crypto,firehose,
jetstream,mst,repo,xrpc-server,xrpc-server-cloudflare},
@cloudflare/{vite-plugin,vitest-pool-workers}, vite.
- vitest bumped to ^4.1.5 — required by vitest-pool-workers v0.16.
Lands the in-memory atproto fixtures so the aggregator's verification path exercises the same code in tests as in production. The load-bearing claim: a record signed by a FakeRepo, fetched via MockPds.handle as a sync.getRecord CAR, and fed into @atcute/repo's verifyRecord round-trips cleanly. Mocks that skip signing would let verification regressions slip through; this rules that out. Components: - FakeRepo wraps @atproto/repo's Repo + MemoryBlockstore for one DID. Real P-256 keypair, real signed commits, real MST construction. getRecordCar uses @atproto/repo's getRecords provider (same path the cirrus PDS reference uses). - MockPds is multi-tenant (mounts many FakeRepos), implements FetchHandlerObject for @atcute/client, and serves both publish-side endpoints (applyWrites, putRecord, repo.getRecord) and aggregator-side endpoints (sync.getRecord-as-CAR with application/vnd.ipld.car, listRecords). Response shapes mirror cirrus. - MockJetstream is a driveable async iterable; tests emit commit events, subscribers receive filtered events, history replays on reconnect with cursor. - MockDidResolver maps DIDs to DID documents with the publisher's signing multikey + PDS endpoint. - createFakePublisherFixture wires all four together: createPublisher registers a new keypair-backed repo with the PDS and resolver in one call. Tests (11): record round-trips for profile + release records, signature mismatch rejection, exclusion-proof CAR for missing records, JSON listRecords shape, Jetstream filtering + history replay, DID resolution with PDS endpoint extraction. Workspace: adds @atproto/repo + @atproto/crypto to catalog as test-only deps. Production aggregator still uses @atcute/repo for verification — the heavyweight @atproto package is private to the test fixture package.
PR 1a bumped vitest in the catalog from 4.0.18 to 4.1.5 because `@cloudflare/vitest-pool-workers@0.16` requires it. The two related packages had separate version pins outside the catalog and stayed at 4.0.x, producing "Running mixed versions is not supported" warnings on every admin/core test run. Bump both to 4.1.5 to match. The bump to @vitest/browser-playwright 4.1.5 also tightens playwright's strict-mode role inference: <input type="file" aria-label="Upload file"> is now treated as having an accessible name that matches a `/Upload/` regex, which collides with the actual Upload button in MediaPickerModal. Anchor the test's regex (`/^Upload$/`) so it only matches the button's exact accessible name.
|
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
emdash-i18n | db26582 | May 09 2026, 01:34 PM |
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
emdash-perf-coordinator | db26582 | May 09 2026, 01:33 PM |
Contributor
PR template validation failedPlease fix the following issues by editing your PR description:
See CONTRIBUTING.md for the full contribution policy. |
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
docs | db26582 | May 09 2026, 01:34 PM |
Contributor
Scope checkThis PR changes 3,981 lines across 28 files. Large PRs are harder to review and more likely to be closed without review. If this scope is intentional, no action needed. A maintainer will review it. If not, please consider splitting this into smaller PRs. See CONTRIBUTING.md for contribution guidelines. |
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
emdash-playground | db26582 | May 09 2026, 01:35 PM |
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
emdash-demo-cache | db26582 | May 09 2026, 01:35 PM |
@emdash-cms/admin
@emdash-cms/auth
@emdash-cms/blocks
@emdash-cms/cloudflare
emdash
create-emdash
@emdash-cms/gutenberg-to-portable-text
@emdash-cms/x402
@emdash-cms/plugin-ai-moderation
@emdash-cms/plugin-atproto
@emdash-cms/plugin-audit-log
@emdash-cms/plugin-color
@emdash-cms/plugin-embeds
@emdash-cms/plugin-forms
@emdash-cms/plugin-webhook-notifier
commit: |
Contributor
Overlapping PRsThis PR modifies files that are also changed by other open PRs:
This may cause merge conflicts or duplicated work. A maintainer will coordinate. |
Merged
Contributor
There was a problem hiding this comment.
Pull request overview
This PR lays the groundwork for the upcoming plugin-registry “aggregator” Worker by adding a new apps/aggregator project (Wrangler + D1 schema + workers-pool smoke tests), introducing a shared @emdash-cms/atproto-test-utils package for real-crypto atproto fixtures, and aligning Vitest-related dependencies needed for @cloudflare/vitest-pool-workers.
Changes:
- Add
apps/aggregatorscaffold: Wrangler config, initial D1 migration (tables + FTS), Worker/DO entry stubs, and workers-pool smoke tests. - Add
packages/atproto-test-utils: FakeRepo + MockPds/MockJetstream/MockDidResolver + round-trip verification tests. - Workspace/dependency wiring updates: include
apps/*in pnpm workspace, add catalog entries, fix marketplace-test default export, and bump Vitest UI / browser-playwright versions.
Reviewed changes
Copilot reviewed 26 out of 28 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| pnpm-workspace.yaml | Include apps/* in the workspace and add catalog entries (atproto/atcute + Cloudflare workers test tooling + Vite/Vitest). |
| packages/plugins/marketplace-test/src/index.ts | Add a default export so descriptor bundling can resolve the plugin consistently. |
| packages/core/package.json | Align @vitest/ui with the newer Vitest version used by the workspace. |
| packages/atproto-test-utils/package.json | New private test-utils package definition and dependencies. |
| packages/atproto-test-utils/tsconfig.json | New TS config for the test-utils package. |
| packages/atproto-test-utils/vitest.config.ts | New Vitest config for the test-utils package. |
| packages/atproto-test-utils/src/index.ts | Public exports for the atproto test utilities. |
| packages/atproto-test-utils/src/types.ts | Shared atproto DID type re-export. |
| packages/atproto-test-utils/src/fake-repo.ts | In-memory repo wrapper built on @atproto/repo for generating signed CAR proofs. |
| packages/atproto-test-utils/src/mock-pds.ts | Multi-tenant in-memory PDS mock implementing XRPC endpoints and CAR responses. |
| packages/atproto-test-utils/src/mock-jetstream.ts | Driveable Jetstream async-iterable mock with replay/cursor support. |
| packages/atproto-test-utils/src/mock-did-resolver.ts | In-memory DID document resolver with PDS + signing-key helpers. |
| packages/atproto-test-utils/src/fake-publisher.ts | High-level fixture wiring FakeRepo + MockPds + MockDidResolver + MockJetstream. |
| packages/atproto-test-utils/tests/round-trip.test.ts | Round-trip tests proving CARs signed by FakeRepo verify via @atcute/repo. |
| packages/admin/package.json | Align @vitest/browser-playwright version with Vitest 4.1.x. |
| packages/admin/tests/components/MediaPickerModal.test.tsx | Anchor the “Upload” locator to avoid strict-mode ambiguity. |
| apps/aggregator/package.json | New aggregator app package scripts and dependencies/devDependencies. |
| apps/aggregator/tsconfig.json | New TS config for the aggregator app. |
| apps/aggregator/vite.config.ts | Cloudflare Vite plugin config for dev/build. |
| apps/aggregator/vitest.config.ts | Workers-pool Vitest config that reads and exposes D1 migrations to tests. |
| apps/aggregator/wrangler.jsonc | Wrangler config (D1, Queue, DO, cron trigger, env vars) for the aggregator worker. |
| apps/aggregator/migrations/0001_init.sql | Initial D1 schema: packages/releases, labels, mirror tracking, FTS, ingest state. |
| apps/aggregator/src/env.ts | Env binding types + Records queue job shape (DI seam). |
| apps/aggregator/src/index.ts | Worker entrypoint stub for fetch/queue/scheduled handlers and DO export. |
| apps/aggregator/src/records-do.ts | Skeleton Durable Object class for Jetstream connection (to be implemented later). |
| apps/aggregator/test/env.d.ts | Type references for @cloudflare/vitest-pool-workers test environment. |
| apps/aggregator/test/smoke.test.ts | Smoke tests proving migrations apply + FTS trigger works + FK enforcement rejects orphans. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Bump catalog wrangler from ^4.80.0 to ^4.83.0. @cloudflare/vite-plugin (pulled in transitively via @astrojs/cloudflare) requires this; the outdated catalog version broke demos/cloudflare typecheck and the Cloudflare-template smoke tests on CI. infra/cache-demo had already pinned ^4.83.0, so this brings the catalog in line with the highest in-tree pin. - Drop redundant `| undefined` from FakeRepo.getRecordValue's return type; `unknown` already includes undefined (typescript-eslint:no-redundant-type-constituents). - Replace deep import `@atproto/repo/dist/sync/provider.js` with the package's public `getRecords` export. Same function, no exports-map brittleness on upgrade. (Per Copilot review.) - Validate did:plc:/did:web: prefix in MockPds parseDid instead of accepting any did:* and casting. Matches the AtprotoDid type's runtime guarantee — invalid methods (e.g. did:key:) now reject at the boundary instead of slipping through tests. (Per Copilot review.) - Drop rootDir from apps/aggregator and packages/atproto-test-utils tsconfigs; both `include` test directories that sit outside `./src`, which trips TS6059 in some tooling. Without rootDir, TS infers from include and tests live cleanly alongside source. (Per Copilot review.)
Six bugs/gaps found in adversarial review of the mock infrastructure. Each fix lands with a regression test so the same bug can't slip back. 1. MockJetstream cursor was off-by-one. The replay filter used `event.time_us < cursor`, which redelivered the cursor event itself. Real Jetstream treats the cursor as "last seen, don't redeliver" — replay must be strict-after. Fixed to `<=` and added a test that emits two events, subscribes with cursor = first event's time_us, and asserts only the second is delivered. 2. MockJetstreamSubscription.cursor returned the global last event time, not the per-subscriber position. A subscriber that had consumed only event 3 of 10 read sub.cursor → time of event 10; reconnecting with that value would silently lose events 4-9. Now tracks lastDeliveredTimeUs per subscriber and exposes it through the cursor getter. Added a test that emits two events, consumes one, and asserts sub.cursor reflects the consumed event. 3. MockJetstreamSubscription.next() now throws when called concurrently from two consumers (previous behaviour silently orphaned the first resolver). Concurrent next() is legal for AsyncIterator; this mock doesn't support it, so failing loudly beats hanging. 4. MockPds dispatched on URL pathname only, ignoring HTTP method — `GET applyWrites` returned 200. Now switches on `(method, pathname)` and returns 405 MethodNotAllowed for known endpoints used with the wrong verb. Added a test asserting GET on applyWrites returns 405. 5. MockPds.repoApplyWrites only handled #create. PR 3's tests will need update + delete flows to model profile updates and tombstoning. Added FakeRepo.updateRecord + deleteRecord that drive @atproto/repo's applyWrites with the right WriteOpAction, and dispatched on $type in MockPds. Added a round-trip test that publishes a profile, updates its license, then deletes it. 6. FakePublisher.publishProfile defaulted authors to [] and let security end up empty if neither securityEmail nor securityUrl was passed — both fields require minLength: 1 in the lexicon. Now throws when no security contact is provided, and defaults authors to a single entry derived from the publisher's handle. Added two tests: one asserts the throw, the other asserts the default authors entry uses the handle. Also reverted a misguided `cursor: undefined` addition to listRecords — JSON.stringify drops undefined keys, which IS the cirrus end-of-stream shape (the `cursor` field is optional). The reviewer's concern there was misread; updated the comment to spell out the contract.
ascorbic
changed the title
A literal placeholder ID makes wrangler think the binding is already configured and skip provisioning on first deploy, then fail when trying to use a non-existent database. Omit the field entirely so auto-provision fires.
Per the project's CLAUDE.md and wrangler's own guidance, the Env type should come from `worker-configuration.d.ts` (generated by `wrangler types`) rather than being maintained by hand. The hand-rolled shape would drift from the actual bindings any time wrangler.jsonc changed. - Add `worker-configuration.d.ts` (generated, committed alongside the similar files in packages/marketplace, infra/perf-monitor, the cloudflare templates, etc.). - Drop the manual `Env` interface from src/env.ts; keep only the project-specific `RecordsJob` type. - Drop `@cloudflare/workers-types` from devDependencies + tsconfig `types` array. Wrangler now ships the runtime types in the generated d.ts; the workers-types package is superseded. - Drop `test/env.d.ts` — its single triple-slash for vitest-pool-workers is now covered by the tsconfig's `types` entry. - Update records-do.ts and index.ts to consume the global `Env`. Re-run `wrangler types` after editing wrangler.jsonc to keep the generated file in sync.
…R refs WANTED_COLLECTIONS is part of the protocol contract, not a per-deployment tunable. Move to apps/aggregator/src/constants.ts where the type system can keep it honest, and drop it from wrangler.jsonc `vars`. Production, staging, dev, and self-hosted instances all subscribe to the same NSIDs. Also strip slice/PR scaffolding language from comments — those refs mean nothing to anyone reading the code outside the immediate dev context. Replace with descriptions of what the placeholder will become (e.g. "PDS-verified ingest will land here") rather than which sub-PR will land it. Regenerated worker-configuration.d.ts after the wrangler.jsonc change.
This was referenced May 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Lands the foundational scaffold for an experimental plugin-registry aggregator. This is Slice 1, sub-PRs 1a + 1b — no business logic yet, but every subsequent PR (Records DO, Records Consumer, read endpoints, reconciliation) builds on this foundation.
Structured as four commits to keep the review surface small:
feat(plugin-marketplace-test): add default export for registry-cli bundler— one-line fix; the bundler resolves descriptors via default export, the test plugin only had a named export. Same convention asformsandcolor.feat(aggregator): scaffold plugin-registry aggregator (Slice 1 PR 1a)— theapps/aggregatorWrangler project, full D1 schema migration, worker entrypoint stubs, Cloudflare Vite plugin for dev/build, vitest-pool-workers for tests, smoke tests proving migrations apply + FTS5 trigger fires + FK rejects orphan releases.feat(atproto-test-utils): real-crypto MockPds + mocks (Slice 1 PR 1b)— new shared test fixture package: real-crypto FakeRepo built on@atproto/repo, MockPds (multi-tenant, serves CAR viasync.getRecordexactly as a real PDS does), MockJetstream (driveable, replay on reconnect with cursor), MockDidResolver,createFakePublisherFixture. The load-bearing claim: a record signed by FakeRepo, fetched as a CAR, and fed into@atcute/repo'sverifyRecordround-trips cleanly. 11 tests cover signature mismatch rejection, exclusion proofs, JSON listRecords shape, jetstream filtering + replay, and DID resolution.chore: align @vitest/browser-playwright + @vitest/ui with vitest 4.1.5—@cloudflare/vitest-pool-workers@0.16requires vitest 4.1+. Bumping vitest in the catalog left two related packages out of sync; this aligns them. Also fixes one admin test (MediaPickerModal) where playwright 4.1's tightened strict-mode role inference made/Upload/match both the button and a hidden file input — anchored to/^Upload$/.Subsequent slices land sequentially:
@atcute/repo'sverifyRecord)first-primaryon label-dependent pathsEMDASH_REGISTRY_URLat the deployed aggregatorCloses #
Type of change
Checklist
pnpm typecheckpassespnpm lintpasses (no new warnings introduced)pnpm testpasses (4500+ tests across the workspace)pnpm formathas been runapps/aggregatorandpackages/atproto-test-utilsare private)AI-generated code disclosure
Test output
Notes for review
The biggest design decision baked in here is D1 + read replicas (vs DO SQLite). DO SQLite was tempting for simplicity, but the aggregator's workload is asymmetric — one region writes via cron + Jetstream, reads come from everywhere — which is what D1's read-replication model is built for. With Sessions API
first-primaryon label-dependent reads, takedowns propagate immediately while non-label-dependent reads (none today, but reserved for future use) can land on the nearest replica.The other notable choice is
@atproto/repo(Bluesky's reference) as a test-only dependency in@emdash-cms/atproto-test-utils. The production aggregator still uses@atcute/repofor verification (lighter, Workers-friendly). The test fixture wants the canonical reference implementation so any divergence between our mocks and a real PDS is the test's problem, not the mock's. Verified by the round-trip test that signs with@atproto/repoand verifies with@atcute/repo.