Take subscription lists as they come: 200k lines, and .txt everywhere

[ralyodio]

Why

A 110,000-line list of feeds should be an ordinary thing to submit. Three things stopped it.

The 50k number was never a submission cap. RAW_INPUT_LINE_LIMIT bounds the copy of an upload stored to name it on its status page — the real ceiling is MAX_UPLOAD_FEEDS, in the tens of millions. But "only the first 50,000 lines of the upload are kept" reads as a limit however it is meant, and 50k is below the size of the lists people actually bring. Raised to 200,000.

A .txt of one feed per line was silently discarded by every client that isn't our uploader. The upload field is named opml and the endpoint took the name for the truth, so a plain list was parsed as XML, matched no outlines and returned no-feeds-in-opml having imported nothing — with JS off, from curl, or from an agent. A 4.8 MB list is under the 10 MB inline limit, so it reached that path. It is sniffed now, by the same call the browser already made.

A large submission arriving in one piece was crawled inside the request carrying it. importFeeds reads every feed URL and slug in the directory, then one round trip per 500 rows — over 200 of them for 108k entries, under a 5-minute ceiling. Past 5,000 entries it is now staged and released to the poller, exactly as the batched uploader does it.

What changed

• RAW_INPUT_LINE_LIMIT 50,000 → 200,000.
• sniffKind reads content before name: <opml in the head means OPML whatever the file is called — the case its own comment promised and the old order lost.
• /api/submit sniffs uploaded files instead of assuming OPML, and stages anything over 5,000 entries rather than importing it inline. Below the threshold nothing changes: one URL still redirects to the blog it just added.
• Progress counted only queued feeds, so the entire handover — now most of a large import's life — showed a full bar and the word "finished" under a heading reading "Import in progress". Staged entries now count towards the total in both the page and the SSE stream, and the poll interval is sized from them too (it had been putting the biggest uploads on the fastest tick).
• The uploader switched its progress panel on the sniffed kind, so a .txt chosen in the file form drew its bar under the paste box. It switches on which form started it now.
• Copy on /submit says what the field actually takes.

Verified

End to end against a real 108,000-line, 5.4 MB file, app running on SQLite:

path	result
multipart .txt, JS off	303 to the status page, 108,000 staged in 1.0s
browser uploader	2.3s, begin + 54 stage + finish, 108,000 recorded
20k-entry OPML	still read as OPML, titles and & intact
OPML living in a .txt	now sniffed as OPML — 6,000 clean entries
poller drain	50,000 feeds in 6.1s, unique slugs
status page	0% — 0 of 108,000 — working (was 100% — finished)

pnpm -r test 1,035 pass / 0 fail; pnpm build clean.

Known, not addressed

Storing 200k lines means raw_input can hold up to the 10 MB the inline endpoint allows, and /submissions/<id> reads that column on every load while auto-refreshing every 30s. The worst case was already ~6 MB before this, so it is an amplification rather than a new cost — but if it matters, the fix is a bounded read (substr(raw_input, 1, …) plus length(...)), which needs a decision about what the panel's "total" should then mean.

🤖 Generated with Claude Code