Skip to content

Fix Kakuyomu Next.js crawling - #643

Merged
kanasimi merged 3 commits into
kanasimi:masterfrom
AlexbeatsZ:agent/kakuyomu-nextjs-support
Jul 18, 2026
Merged

kanasimi merged 3 commits into
kanasimi:masterfrom
AlexbeatsZ:agent/kakuyomu-nextjs-support

Conversation

@AlexbeatsZ

@AlexbeatsZ AlexbeatsZ commented Jul 18, 2026

Copy link
Copy Markdown
Contributor

What changed

  • Parse Kakuyomu work metadata and tableOfContentsV2 from the current Next.js __NEXT_DATA__ Apollo state.
  • Keep the existing episode-body parser while making its title pattern local under strict mode.
  • Use https-proxy-agent for Kakuyomu so HTTPS proxy CONNECT requests include SNI, with support for both proxy environment variables and the existing proxy= CLI argument.
  • Treat regenerate as per-run state instead of inheriting it from cached work JSON.
  • Ignore generated dependency and Kakuyomu download directories, and document the verified workflow in AIREADME.md.

Why

Kakuyomu's work page no longer exposes the legacy widget-toc-* markup, so the crawler detected zero chapters. CeJS 4.5.10's built-in HTTPS proxy agent also opens TLS without SNI, which CloudFront rejects with a TLS handshake failure.

After an EPUB regeneration, cached regenerate: true was inherited on the next normal run. The crawler then resumed from the last downloaded chapter and rewrote the OPF manifest and spine with only that chapter, making readers show only the final episode.

Validation

  • node --check novel.ja-JP/kakuyomu.js
  • npm ls cejs https-proxy-agent --depth=0
  • Downloaded Kakuyomu work 16818093080247863087: 136 of 136 episodes.
  • Rebuilt the EPUB from cache: 137 XHTML manifest entries and 137 spine entries (TOC plus 136 episodes).
  • Re-ran the normal download command after regeneration; the EPUB SHA-256 and modification time remained unchanged.
  • Confirmed 136 unique chapter files in the EPUB navigation document and sampled the first, middle, and final episode titles.
  • 7z t reported Everything is Ok for the generated EPUB.
  • Re-ran with proxy=http://127.0.0.1:7897 and confirmed the completed work was detected correctly.

@AlexbeatsZ
AlexbeatsZ force-pushed the agent/kakuyomu-nextjs-support branch from faa2dcf to c30ec46 Compare July 18, 2026 19:22
Do not inherit per-run regenerate state from cached work JSON, which otherwise repacks only the last chapter on a normal rerun.
@kanasimi
kanasimi marked this pull request as ready for review July 18, 2026 23:15
@kanasimi
kanasimi merged commit 6b2a671 into kanasimi:master Jul 18, 2026
4 of 8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants