Name migrations for when they were written, and prove nobody can collide again - #131
Merged
Conversation
…ide again Yesterday two branches both took `0032` — `0032_feed_refetch_signals.sql` (#127) and `0032_queue_hourly.sql` (#128). Git raised no conflict, because they are different filenames, so nothing failed and nothing warned. Writing the test that now guards this found it had happened twice before and nobody had ever noticed: 0006 feed_items_created_idx, feeds_sitemap_index 0018 crawl_log, queue, scraped_feeds <- three files 0032 feed_refetch_signals, queue_hourly It has cost nothing so far because `migrate()` keys `_migrations` on the filename rather than the number, so each file still ran exactly once. That is luck, not design: it holds only while the colliding files commute, and nothing was checking that they did. The first order-dependent pair would apply in whichever order lexical sort happened to pick. So new migrations are named `YYYYMMDDHHMMSS_what_it_does.sql`, as Supabase does it. Two people would have to write one in the same second to collide. The honest trade, written down in the README rather than discovered later: timestamps order by when a migration was *written*, not when it was *merged*, so a fresh database can replay in a different order from the one production used. That is not a regression — sequential numbering had the same divergence and the silent collisions on top — but it means migrations must stay independent of each other, and `_migrations.applied_at` is the authority when the two disagree. Nothing is renamed. `migrate()` keys on the filename, so renaming an applied migration makes the runner treat it as new and run it again — survivable for a purely additive file, since ALREADY_APPLIED swallows "duplicate column name", but 0001, 0013 and 0017 also carry inserts and updates. The old files sort before every timestamp anyway ('0' precedes '2'), so history keeps its order untouched, and the test asserts that rather than assuming it. The test is the point. A convention in a README would not have stopped any of the three collisions above, because whoever merges second is never prompted to renumber — the tool that would prompt them sees two unrelated new files. This is the prompt. Verified both ways: it rejects a new `0033_*` and it rejects two files sharing a timestamp. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The migration guard in this branch is a test, and nothing in this repository ran tests. The only checks on a pull request were Socket's dependency scans, which say nothing about whether the code works — so eleven workspaces of tests existed and were run by whoever remembered to. That is not a side issue here, it is the same issue. A guard nobody runs is a comment, and the three migrations that quietly claimed a taken number went unnoticed for exactly that reason. Node 22 to match apps/*/Dockerfile rather than the newest version `engines` would allow, and pnpm from `packageManager` so the version is not written down in two places that can drift. The build runs too. The tests cover packages and route handlers; a JSX or import error in a page only surfaces when Next compiles it, which is how the about page rewrite could have shipped broken. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The first run of the new workflow cancelled all four tests in request-timeout.test.js with "Promise resolution is still pending but the event loop has already resolved". They pass every time on a quiet laptop, and the suite had never run anywhere else, so this had been latent since #111. The stub is at fault, not `withTimeout`. It answered only by listening: init.signal.addEventListener('abort', () => reject(init.signal.reason)) A real fetch handed a signal that has *already* fired rejects immediately. An abort listener added afterwards hears nothing — the event has been and gone — so the promise never settles, the file is torn down, and the three tests behind it are cancelled by their parent. On a quiet machine the 30ms deadline never beats the stub to its first line. On a loaded two-core runner it does. Forced both ways to be sure, with a deadline already expired on arrival: old (listener only) -> HUNG new (checks aborted) -> rejected TimeoutError after 40ms So the stub now checks `signal.aborted` first, which is what it was always claiming to imitate. Shared between the two tests that need it rather than fixed twice. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ndoned The previous commit fixed a real defect in this stub and not the one that was breaking CI. The tests were still cancelled, identically, and the reason it looked like a loaded-runner race was that it reproduces on Node 22 *in isolation on an idle machine* — CI runs Node 22 to match apps/*/Dockerfile, while local development is on Node 24. `AbortSignal.timeout()` schedules an **unref'd** timer. That is deliberate: a pending deadline must not hold a program open. Verified identical on both versions — a script whose only pending work is such a deadline exits in 1ms under 22 and 24 alike. In production nothing notices, because the real in-flight fetch holds a socket open and the loop stays alive until the deadline fires. A stub that returns a promise and does nothing else gives the loop no reason to stay awake at all, so it drains and the runner reports "Promise resolution is still pending but the event loop has already resolved" and cancels the file. Node 24's test runner happens to keep the loop alive and Node 22's does not, which is the whole of the difference between green locally and red in CI. So the stub now holds a ref'd timer for the life of the request it is standing in for, cleared on every exit. Verified on both: 4 passed, 0 cancelled under Node 22 and Node 24, and the full workspace suite is clean under Node 22. The `aborted` check from the previous commit stays. It was a genuine fault — a listener added after the event has fired never hears it — and would have bitten as soon as a deadline beat the stub to its first line. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ralyodio
added a commit
that referenced
this pull request
Aug 19, 2026
`0033_author_searches.sql` took the next sequential number, and there is no next sequential number any more: #131 froze the scheme at 0032 and added a test that says so, because 0006, 0018 and 0032 were each claimed by two or more branches before anybody noticed. Git never flags it — the files have different names — so the guard is the only thing that would. Renamed to the moment the file was actually written, 2026-08-19 08:18:24 UTC, per `packages/db/migrations/README.md`. Nothing referenced it by name, and it has never been applied anywhere, so this is a rename rather than a second migration. The failure was also hiding the rest of the suite: `pnpm -r` stops at the first package that fails, so packages/db going red meant ingest, auth, notify, translate, discover, web and poller never ran in CI at all. On Node 22, which is what CI uses: 1,067 pass, 0 fail, 0 cancelled, and `pnpm build` clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ralyodio
added a commit
that referenced
this pull request
Aug 19, 2026
#133) * Find the author when the blog names nobody, by reading the address /authors/pierre-felgines is a name and nothing else: no email, no site, no accounts. The pass that produced it was not broken. It fetched felginep.github.io, found no rel="me", no h-card, no JSON-LD and exactly one outbound link — to the Jekyll theme its author happened to use — correctly concluded there was nobody to credit, and stamped the feed as checked. Meanwhile the author's GitHub account was named in the hostname the feed is served from, and one request to it returns "Pierre Felgines". That is the gap this closes, and it is not a corner. Of 369,056 feeds only 3,275 have ever been enriched, and 77% of the identity we hold came free out of feed documents rather than from visiting anything. Reading what publishers marked up works beautifully on the part of the small web that marks itself up; this reads the two things nobody had to publish. **The host.** A blog on <user>.github.io, <user>.substack.com, medium.com/@user and seven more names its owner's account in the address. Deriving it is string arithmetic on URLs already in hand, so a feed on an unrecognised platform costs nothing extra at all. **The profile behind it.** GitHub returns a real name, an avatar, a homepage and sometimes an email its owner chose to publish; GitLab a name and public email; Codeberg the same through Gitea; and a fediverse account returns its profile fields with the instance's own rel="me" verification already performed — the same handshake enrichFeedAuthors spends up to three fetches proving, arriving done. Three rules keep this from inventing people, which is the expensive direction: - An organisation is not a person. GitHub and Gitea serve both from one endpoint, so jekyll.github.io resolves to an account whose name is a product. The account is still stored as a link — a link is not a claim about who somebody is — but no author row is created. - A derived account is evidence, not proof, and sits below the 0.6 publishing floor until a profile answers. A 404 leaves no trace rather than half a person. - `verified` keeps meaning what the schema says. It is set when the profile links *back* at the site being enriched, which is the IndieWeb handshake in the other direction, and never merely because an account exists. safeFetch grew a headers option rather than being bypassed: a profile URL is built from a hostname read out of somebody else's feed, so it is exactly as untrusted as a page URL and must keep the private-address guard and the timeout. GITHUB_TOKEN is wired through the poller and is not optional in practice — the anonymous GitHub API allows 60 requests an hour per IP, which one batch spends, against 5,000 with a token. Measured against the live APIs while building, not assumed: GitLab's unauthenticated user search answers a reduced object (name and public_email only, the contact fields needing a token), and Mastodon bios are rendered HTML, where stripping every tag to a space turns "<b>InfoSec</b>." into "InfoSec ." — the test caught that before it reached anybody's page. * Read the file whose only job is to say who made this Every source the enrichment had was markup carrying identity as a side effect: a link with a rel attribute, a microformat class, a byline in a feed document. humans.txt is the one convention written to answer the question directly, and the pass was walking past it. Parsed rather than scraped, because the format inverts everything else: the key is the role and the value is the person ("Chef: Jane Doe"), and the contact lines that follow belong to the name above them. A parser that collected links globally would hand Jane's Mastodon to Bob, which is worse than finding nothing — it is a wrong contact address published as a right one. Two sections of the file are not this blog's authors. /* THANKS */ credits other people's work, so reading it as authorship attributes a blog to whoever its author admires; /* SITE */ describes the build. Both are skipped, and the role filters still apply, so "Developer: the web team" names nobody. Fetched only when the ordinary pages named nobody. Most sites do not publish one, so asking every site costs a request per feed across the directory to help a minority; asking after the pages come back empty spends it exactly where it decides between an author and no author. It is also read as text, not HTML — plenty of servers answer every path with their 404 page, and parsing that would turn a stylesheet reference into somebody's website. /now and /uses join the page list for the same reason humans.txt earns a request: they are conventions of exactly the population this directory indexes, written in the first person, and a blog that has one often has no /about. One fix found while testing: a "Site:" line is the most useful in the file and classifyLink returns null for it, because it rightly refuses to file arbitrary links as somebody's website. Here the key has already said that is what it is, so the fallback is scoped to the keys that said so — Standards: and Language: still produce nothing. * Buy a search for the people who left no trail, on a budget that cannot be reset The free sources read what a publisher put where we could find it. Some people put nothing anywhere, and for them a search engine is the only remaining route. It is also the one that can spend money and the one that can invent a person, so almost all of this is what it refuses to do. **It cannot run over the directory.** The credits come from CrawlProof's 25,000 a month, already shared with CrawlProof's own outreach runner, against 369,056 feeds — one query each would be fifteen times the monthly allowance. So it takes a key, a non-zero budget and an explicit switch, and is off without all three. **The budget survives a restart, because it is written down.** A counter in the process would be reset by every deploy, which is not a budget but a rate limit with a hole in it. `author_searches` is the ledger and the authority, and it is counted from the provider's own cycle — this account resets on the 13th, not the 1st, so a calendar month would let the allowance be spent twice across a reset. The ledger is also the audit trail: which people we spent money looking for, and whether it found anything, is how the gate gets tuned rather than guessed at. **It cannot invent a person.** Every query is scoped to the author's own domain as well as to one network, because the name is the ambiguous part — "Jane Doe" site:linkedin.com/in returns every Jane Doe, and the blog she writes is what distinguishes her from them. Results are filtered through classifyLink, so a company page, a job posting and an article about somebody are all discarded where a profile is kept. Nothing found this way is ever marked `verified`: that column means the IndieWeb handshake, and a search engine's opinion that two strings co-occur is not it. Every link is stamped `web-search` so a consumer can exclude the class outright. The gate is mean on purpose — confident it is a person, more than one word of name, publishes here, and currently unreachable — and it is stated twice, in the SQL that selects and in `worthSearching` that re-checks, so a drift between them costs nothing rather than money. On LinkedIn, since it is the thing that was actually asked for: a profile URL found this way is stored, because it is a public address the search engine has already indexed. The profile behind it is not fetched — auth-walled, 999 to anything automated, and its terms forbid scraping. We can say where somebody's LinkedIn is; we cannot say what is on it. Two things the tests caught rather than the reader: `authors` has no `site` column (it is `site_url`), and `addAuthorLinks` already existed and batches its inserts into one round trip, so the second one I wrote was deleted rather than kept. * Stop losing a publisher for ninety days because their site was down once Two things in the same area, both found while looking for a queue that turned out not to be needed. **A failure was recorded as a miss.** Every fetch in the enrichment fails softly — a dead host, a timeout and a 503 all come back as "no page" rather than as an exception — so a site that was simply down looked identical to a site that names nobody, and both were stamped as checked. That cost the publisher their enrichment for the whole ninety-day recheck cycle on the strength of one bad afternoon, and on a pass that has so far reached 3,275 of 369,056 feeds it quietly loses everyone on a flaky host. The pass now tracks whether it got an answer out of the publisher at all. Reached and found nobody: stamped, left alone until the recheck. Never reached: stamped to come back in a few days, because nothing was learned and the next attempt may well work. Still stamped either way, which is what keeps a permanently broken feed off the head of the queue. Done by back-dating the stamp rather than adding an attempts column, and that is a trade rather than a shortcut: writes here serialize and the crawl is already write-bound, so the fix that costs one UPDATE beats the tidier one that costs a migration and a second column on every read. **The pass had no row on /crawlstats.** It is the only job on the board that was invisible, which is a large part of why it looked like it had never started. It now shows its backlog, how many publishers it looked at in the last hour, and how far through the directory it is — read off the partial index 0024 already built, and counted as the stamped set rather than its complement, because 3,275 index entries is a cheap question and 369,056 rows is not. The queue rebuild this started as is deliberately not here. dueForAuthors already scopes to active feeds, so the pass has never been walking the 280,360 pending ones, and at the rate now observed the 84,398 active feeds are done in under a fortnight. A claim/lease table would have been machinery for a problem the numbers say does not exist. An existing test caught the new row before it shipped: every job on the board must report when it last ran, and one that emits an event nobody records is a row that reads as permanently stalled. * Name the author-search migration for when it was written `0033_author_searches.sql` took the next sequential number, and there is no next sequential number any more: #131 froze the scheme at 0032 and added a test that says so, because 0006, 0018 and 0032 were each claimed by two or more branches before anybody noticed. Git never flags it — the files have different names — so the guard is the only thing that would. Renamed to the moment the file was actually written, 2026-08-19 08:18:24 UTC, per `packages/db/migrations/README.md`. Nothing referenced it by name, and it has never been applied anywhere, so this is a rename rather than a second migration. The failure was also hiding the rest of the suite: `pnpm -r` stops at the first package that fails, so packages/db going red meant ingest, auth, notify, translate, discover, web and poller never ran in CI at all. On Node 22, which is what CI uses: 1,067 pass, 0 fail, 0 cancelled, and `pnpm build` clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Yes — Supabase-style timestamps are the right call, and the investigation turned up more than expected.
It has happened three times, not once
Yesterday two branches both took
0032(#127 and #128). Git raised no conflict, because they are different filenames. Writing the test that now guards this found it had happened twice before and nobody ever noticed:It has cost nothing so far, because
migrate()keys_migrationson the filename rather than the number, so each file still ran exactly once. That is luck rather than design: it holds only while the colliding files commute, and nothing was checking that they did. The first order-dependent pair would apply in whichever order lexical sort happened to pick.The change
New migrations are named
YYYYMMDDHHMMSS_what_it_does.sql:Two people would have to write one in the same second to collide.
The honest trade, written down rather than discovered later
Timestamps order by when a migration was written, not when it was merged. Write one Monday, merge it Friday, and a colleague's Wednesday migration ran first in production while a fresh database replays yours first.
This is not a regression — sequential numbering had the same divergence and the silent collisions on top — but it means:
add column/create index if not existscommute; anything order-dependent belongs in one file)_migrations.applied_atis the authority when a fresh replay and production disagreeNothing is renamed
migrate()keys on the filename, so renaming an applied migration makes the runner treat it as new and run it again. Survivable for a purely additive file —ALREADY_APPLIEDswallowsduplicate column name/already exists— but0001_init.sql,0013_feed_category.sqland0017_crawl_hourly.sqlalso carryinsertandupdatestatements, which are not idempotent.The old files sort before every timestamp anyway (
'0'precedes'2'), so history keeps its order for good. The test asserts that rather than assuming it.Why a test and not just a README
A convention in a README would not have stopped any of the three collisions above, because whoever merges second is never prompted to renumber — the tool that would prompt them sees two unrelated new files. This is the prompt.
packages/db/test/migration-names.test.jsenforces: valid filename shape, no two files claiming a slot (the seven already-shipped duplicates grandfathered by name, with the reason), no new sequential numbers past0032, timestamps that name a real moment that has passed, and that history still sorts first.Verified it actually fires, both ways:
One caveat on how much this protects you: the repo has no
.github/workflows, so nothing runs the suite automatically. This guard fires onpnpm -r test, not on push. Worth a CI workflow separately if you want it enforced at merge.Testing
All 11 workspace suites green.
🤖 Generated with Claude Code