Topics follow the feed, not only the posts

[ralyodio]

feeds.category has always been re-derived on every successful crawl. Topics were not: they were gated on publishedSomethingNew || existingTopics === 0.

That asks the wrong question. topicsFrom reads the channel's own categories, title and description as well as its items, so a publisher can change what their feed is about without publishing anything — retag it, rename it, rewrite the standfirst. For a blog that then goes quiet the gate was never true again, and the directory filed it under whatever it was about on the last day it posted.

Why this is affordable

Re-deriving on every crawl is cheap because of the diff, not in spite of it:

• the three reads it needs are already issued whenever auxiliary writes are on, so this adds no round trip;
• topicsFrom is pure computation over a document that is already parsed;
• keywordDiffStatements emits nothing when the extracted set matches what is stored — which for a quiet feed is every time.

The second test asserts that last point directly rather than through the rows, because feed_keywords carries no timestamp, so "rewritten to the same values" is indistinguishable from "left alone" — the same indistinguishability that hid this cost in the first place.

What this does not change

The write cost that forced CRAWL_AUXILIARY_WRITES=0 in production is first crawls, where every topic is a genuine insert and no diff can help. That is a property of the backlog, not of this gate. This change is inert until that switch goes back on.

Tests

Full workspace suite green (11 packages, 0 failures). Two new cases in packages/ingest/test/recrawl.test.js:

• a feed retagged without publishing has its topics revised;
• an unchanged re-crawl emits zero topic write statements.

🤖 Generated with Claude Code