Skip to content

fix(desktop): repair action item FTS without catalog writes#9271

Open
ZachL111 wants to merge 5 commits into
BasedHardware:mainfrom
ZachL111:zach/action-items-fts-no-catalog-writes
Open

fix(desktop): repair action item FTS without catalog writes#9271
ZachL111 wants to merge 5 commits into
BasedHardware:mainfrom
ZachL111:zach/action-items-fts-no-catalog-writes

Conversation

@ZachL111

@ZachL111 ZachL111 commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Why

Issue #9032 tracks that the action item FTS repair path still falls back to writable_schema and direct sqlite_master deletes when a shadow table is missing. macOS 26 rejects that catalog write, so the recovery test had to stay skipped.

What changed

  • Replaced the catalog-write fallback with a public-DDL repair path:
    • create a scratch FTS5 table with the same config
    • copy any missing shadow-table schemas from the scratch table
    • retry the normal DROP TABLE action_items_fts
    • rebuild and backfill from durable action_items
  • Updated the shadow-table regression test to corrupt via DROP TABLE action_items_fts_data instead of editing sqlite_master.
  • Unskipped the test and lowered the Swift skip ratchet from 5 to 4.

Verification

  • python3 scripts/swift-test-skip-ratchet.py --check
  • git diff --check origin/main...HEAD
  • Source check: no PRAGMA writable_schema or DELETE FROM sqlite_master remains in the changed FTS repair code/test.
  • Commit attribution scan: no coauthor or generated-by footers.

I could not run xcrun swift test --package-path Desktop --filter ActionItemsFTSRepairTests/ from this WSL environment because xcrun and swift are not installed here.

Fixes #9032

Review in cubic

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 4 files

Confidence score: 5/5

  • Safe to merge after the addressed issues were fixed.

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread desktop/macos/Desktop/Sources/Rewind/Core/RewindDatabase.swift Outdated

@kodjima33 kodjima33 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

desktop FTS-repair bug fix (#9032) looks correct; not merging — Desktop Swift Build & Tests is RED, needs green CI

@Git-on-my-level Git-on-my-level left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for cleaning up the macOS 26 recovery path and removing the direct writable_schema / sqlite_master writes. The implementation direction looks much safer than the old catalog-delete fallback, and the skip-ratchet cleanup is appreciated.

I do need one fix before this is ready: the final regression test no longer exercises the broken-shadow-table repair path that this PR is intended to cover. testRepairRebuildsActionItemsFTSFromDurableRows now removes a row from the FTS index via INSERT INTO action_items_fts(action_items_fts, ...) VALUES ('delete', ...), then calls repairActionItemsFTS; that validates a normal rebuild/backfill, but it does not make DROP TABLE action_items_fts fail and therefore does not cover restoreMissingActionItemsFTSShadowTables(...).

Could you add/restore a targeted test that actually simulates a missing FTS5 shadow table using public SQLite DDL only, and proves the fallback can restore the missing shadow table, retry the drop, recreate the FTS table, and preserve/backfill durable action_items rows? That coverage is important because this PR removes the old catalog-write escape hatch.

I did not formally approve because this touches SQLite recovery/catalog-adjacent behavior and the monitor policy requires human maintainer review for this risk surface, even though the current direction looks good.

@Git-on-my-level Git-on-my-level added the needs-tests PR introduces logic that should be covered by tests label Jul 9, 2026

@kodjima33 kodjima33 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Desktop bug fix: FTS repair via public-DDL path (no writable_schema catalog writes) for macOS 26. Approve; not merging over open change-request.

ZachL111 added 2 commits July 10, 2026 13:04
# Conflicts:
#	desktop/macos/e2e/flows/tasks.yaml
…epair paths

Extract actionItemsFTSCreateSQL(tableName:) so the normal install and the shadow-table
repair template use byte-identical FTS5 config (columns, content mapping, tokenizer),
removing the risk of a future schema change drifting the two and building a mismatched
shadow table during recovery.

@Git-on-my-level Git-on-my-level left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update here. The implementation direction still looks right to me: replacing the previous writable_schema / direct sqlite_master catalog writes with a public-DDL repair flow is a much safer approach for the macOS 26 SQLite behavior.

I still need to request changes because the regression test on the current head does not cover the specific failure path this PR is meant to protect. testRepairRebuildsActionItemsFTSFromDurableRows uses the FTS5 special delete command to remove an indexed row, then verifies repairActionItemsFTS can rebuild/backfill. That is useful coverage for rebuild behavior, but it does not simulate a missing FTS5 shadow table, does not make DROP TABLE action_items_fts fail, and therefore does not exercise restoreMissingActionItemsFTSShadowTables(...).

Could you add/restore a targeted test that corrupts the FTS setup using public SQLite DDL only, proves a missing shadow table is restored, verifies the retrying drop/recreate path succeeds, and confirms durable action_items rows are preserved/backfilled? Since this is SQLite recovery/catalog-adjacent behavior, I’m also leaving this for human maintainer review before merge.

@ZachL111 ZachL111 force-pushed the zach/action-items-fts-no-catalog-writes branch from 1f6beb0 to a9bd7fb Compare July 11, 2026 13:03
@ZachL111

Copy link
Copy Markdown
Contributor Author

Thanks for the detailed guidance. I tried to add exactly that test (drop a single FTS5 shadow table with DDL, then prove repair restores it, retries the drop, and backfills), but the macOS/CI SQLite rejects the setup: DROP TABLE action_items_fts_docsize fails with table action_items_fts_docsize may not be dropped (shadow-table protection). Since this PR is specifically removing the writable_schema / sqlite_master path, I did not want to reintroduce that just to force the shadow table missing.

So I could not simulate a genuinely-missing shadow table through public DDL alone in the test harness. Two options, whichever you prefer:

  1. If the team has a supported way to reproduce the macOS 26 missing-shadow state hermetically (a config/pragma that relaxes the shadow-table guard for a test), point me at it and I will write the exact drop -> restore -> retry -> backfill test you described.

  2. Otherwise I can make the restore path directly unit-testable: expose restoreMissingActionItemsFTSShadowTables and the shared create-SQL as internal, and add a test that asserts it regenerates the correct shadow-table DDL from the repair template and that the template FTS schema is byte-identical to the install path. That covers the restore logic and the template-must-match-install invariant the code comment calls out, without first having to corrupt the on-disk shadow tables.

I reverted the non-working test so the branch is back at your reviewed head (a9bd7fb). Let me know which direction you prefer and I will push it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-tests PR introduces logic that should be covered by tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Rewind FTS-repair blocked on macOS 26 SQLite (DELETE FROM sqlite_master rejected) — test + product recovery

3 participants