Skip to content

File-backed scratchpad assets: store images as sidecar files#31

Merged
molefrog merged 6 commits into
mainfrom
claude/scratchpad-image-storage-qdz6kc
Jul 8, 2026
Merged

File-backed scratchpad assets: store images as sidecar files#31
molefrog merged 6 commits into
mainfrom
claude/scratchpad-image-storage-qdz6kc

Conversation

@molefrog

@molefrog molefrog commented Jul 7, 2026

Copy link
Copy Markdown
Owner

Summary

Move scratchpad image bytes from inline base64 data URLs in .moi/.scratchpad.json to content-addressed files in .moi/.scratchpad/. This prevents megabytes of pixels from being re-serialized on every autosave and shipped over every GET/PUT, while maintaining backward compatibility with legacy snapshots.

Key Changes

  • New module server/scratchpad-assets.ts: Implements file-backed asset storage with:

    • Content-addressed file naming (SHA256 hash of bytes) for automatic deduplication
    • MIME type to file extension mapping
    • Validation of asset sources to prevent directory traversal attacks
    • Orphan asset cleanup with a grace period to handle async uploads
    • Migration of legacy inline base64 data URLs to files on save

  • Browser asset store (client/components/Scratchpad.tsx):

    • Implements TLAssetStore to POST image bytes to /scratchpad/assets on paste/drop
    • Resolves asset: file references back to serving URLs at render time
    • Transparently handles legacy data URLs (passed through unchanged)

  • Server API routes (server/api.ts):

    • POST /scratchpad/assets — accepts raw image bytes, stores as content-addressed file, returns asset: src
    • GET /scratchpad/assets/:file — serves stored assets with immutable cache headers

  • Scratchpad save pipeline (server/scratchpad.ts):

    • Calls extractInlineAssets() before every save to migrate any base64 blobs to files
    • Calls sweepOrphanAssets() after save to reclaim unreferenced files
    • Updated comments to reflect new asset model

  • Server-side image processing (server/scratchpad-executor.ts):

    • add-image now stores resized WebP bytes as a file asset instead of embedding as base64
    • Returns asset: src on the asset record

  • Image reading (server/scratchpad.ts):

    • readScratchpadImage() resolves asset: file references back to data URLs for CLI compatibility
    • Handles legacy inline data URLs and remote https URLs transparently

  • Tests: Comprehensive test suite covering:

    • Content addressing and deduplication
    • Source validation and security
    • Inline asset extraction and migration
    • Orphan asset cleanup with grace period
    • End-to-end save/load cycle

Implementation Details

  • Content addressing: File names are asset-<sha256>.<ext>, making every write idempotent and enabling automatic deduplication when identical images are pasted multiple times
  • Backward compatibility: Legacy snapshots with inline base64 are transparently migrated on the next save; stale browser tabs holding base64 in their live store are handled the same way
  • Grace period: Orphan sweep waits 5 minutes before deleting unreferenced files, allowing for the async upload → asset record save window
  • Security: Strict validation of asset file names prevents directory traversal and only serves files we created
  • Immutable caching: Assets are served with Cache-Control: public, max-age=31536000, immutable since content addressing guarantees they never change

https://claude.ai/code/session_01T3HuarjiNPZL3FiwvCTaXf

claude added 5 commits July 8, 2026 08:15
@claude
Store scratchpad images as files instead of base64 in the snapshot
732199c 
Images pasted into the Scratchpad (or added via `moi scratch add image`)
were inlined as base64 data URLs inside asset records in
.moi/.scratchpad.json — megabytes re-serialized on every ~500ms autosave
and shipped whole over every snapshot GET/PUT and tab reload.

Image bytes now live as content-addressed files in
.moi/scratchpad-assets/ (<sha256>.<ext>), referenced from asset records
by `asset:` srcs — one of tldraw's native src protocols:

- The browser gets a TLAssetStore: upload POSTs pasted/dropped bytes to
  the new POST /scratchpad/assets route; resolve maps `asset:` srcs to
  the immutable-cached GET route.
- The server's `add image` writes the resized webp straight to a file.
- Legacy snapshots migrate lazily: every save funnels through
  saveScratchpadDoc, which extracts any inline base64 it finds (also
  covering stale tabs that still PUT blobs — content addressing makes
  re-extraction converge on the same file).
- A post-save sweep deletes files the document no longer references,
  with an mtime grace window covering the upload-to-autosave gap.
- `clear` now drops asset records too, so the sweep reclaims the files.
- `moi scratch read-image` resolves `asset:` refs back to a data URL, so
  the CLI protocol is unchanged; `read` surfaces the short `asset:` src.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T3HuarjiNPZL3FiwvCTaXf
@claude
Move scratchpad asset files to .moi/.scratchpad/
7f78d82 
Hidden sidecar dir next to .moi/.scratchpad.json, matching the
snapshot's dotfile convention — both are moi-internal.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T3HuarjiNPZL3FiwvCTaXf
@claude
Prefix scratchpad asset files with asset-
9361a76 
Files in .moi/.scratchpad/ are now asset-<sha256>.<ext>, so the sidecar
dir stays unambiguous if it ever holds anything else. Srcs remain
`asset:` + file name; unprefixed names are no longer valid.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T3HuarjiNPZL3FiwvCTaXf
@claude
Handle dangling asset references and protect .bak-referenced files
bf6b170 
Two edge cases around "asset record exists, file is gone":

- `moi scratch read` now flags such shapes with `missing: true`, so the
  agent learns the reference is dangling up front instead of from a
  failing `read-image` (which already errors with the expected
  location). The browser side needs nothing: the served URL 404s and
  tldraw shows its broken-image placeholder.
- The orphan sweep now also keeps files referenced by
  .scratchpad.json.bak — the schema-change backup is the manual escape
  hatch after a downgrade, and restoring it should still render its
  images. Raw regex scan of the .bak (no JSON parse; it may be a huge
  legacy file), cached by mtime since it only changes on schema
  upgrades.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T3HuarjiNPZL3FiwvCTaXf
@claude
Fix scratchpad asset sweep data loss and hardening from code review
f47a8df 
Addresses correctness findings in the file-backed scratchpad assets work:

- Sweep race / clear resurrection (data loss): the orphan sweep keyed its
  grace window on file mtime, so a long-referenced image got zero grace the
  instant it became unreferenced (e.g. `clear` racing an open tab's pending
  autosave), letting the sweep permanently delete a file the surviving
  snapshot still pointed at. Anchor the grace to when a file is FIRST observed
  unreferenced instead, tracked in-memory. This also subsumes the dedup
  mtime-not-refreshed case (mtime no longer factors into the decision).

- `delete`/re-add leaked asset files: `delete` removed only the shape and its
  bindings, leaving the asset record to pin its file against the sweep forever
  (asymmetric with the `clear` fix). Drop the asset record when the deleted or
  replaced shape was its last user (shared assets are preserved).

- Atomic asset writes: write via a temp sibling + rename so a crash or full
  disk mid-write can't leave a truncated file at the content-addressed path
  that the exists() skip would then serve forever.

- Stored-XSS hardening: serve assets with Content-Disposition: attachment and
  X-Content-Type-Options: nosniff so a pasted/uploaded SVG can't execute as
  script in the app origin on direct navigation (img/video/fetch ignore these).

- assetStore held in a ref, not useMemo (which React may discard): a fresh
  `assets` identity makes <Tldraw> rebuild its store and remount the editor,
  dropping unsaved edits and resetting the camera.

- Flag `missing` in `read` for shapes whose asset RECORD is absent, not only
  when the file is gone, so the agent isn't handed a healthy-looking image.

- Sync the CLI's read-image extension map with the server (avif/apng/video),
  so those assets save with a real extension instead of `.bin`.

- Derive ASSET_FILE_RE and the .bak scanner from one shared pattern so they
  can't drift and delete backup-referenced files; add image/apng to MIME_EXT.

Tests updated for the new grace semantics; new regression tests cover the
clear-race data loss and the delete reclaim path.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HCbMMxrNwyPAtJP5qryT6W
@molefrog molefrog force-pushed the claude/scratchpad-image-storage-qdz6kc branch from 8f05b1a to f47a8df Compare July 8, 2026 08:17
@claude
Accept browser image drops up to 32MB and rescale to fit
69f41e4 
tldraw's defaults reject dropped/pasted assets over 10MB and never rescale, so a
phone photo either bounced or (once admitted) bloated the snapshot's sidecar
files at full resolution. Pass `maxAssetSize` (32MB) and `maxImageDimension`
(2048px) to <Tldraw>: the size gate runs before the rescale, so a large image is
admitted and then shrunk to fit before upload. The downscaled bytes land well
under the server's existing 50MB asset cap, so no server change is needed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HCbMMxrNwyPAtJP5qryT6W
@molefrog molefrog merged commit c9f34c4 into main Jul 8, 2026
1 check passed
@molefrog molefrog deleted the claude/scratchpad-image-storage-qdz6kc branch July 8, 2026 08:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@molefrog @claude