Replace predicate merges with equality-delete row deltas

[nicosuave]

Summary

• pin the Sidequery PyIceberg fork at 968fb967, including equality-delete planning, Arrow reads, writes, and delete-only keys
• route both dlt merge strategy names through one Iceberg equality-delete row delta
• commit hard-delete keys and replacement rows in the same snapshot
• remove composite predicate construction and all row batching; the deprecated merge_batch_size option is accepted but ignored
• document reader, partitioning, concurrency, and orphan-file tradeoffs

This supersedes the batching approach from closed PR #24. The merge size is no longer governed by an arbitrary row count.

Why

The previous paths generated a large OR(AND(...), ...) expression for composite delete-insert and PyIceberg upsert generated match predicates while scanning/re-writing target data. Production observed a native SIGBUS around 1,018 composite-key rows with PyIceberg 0.11.1 on macOS arm64. Making the batches smaller only avoided the failure and weakened snapshot atomicity; it did not fix the merge design.

The new path writes the incoming keys as Iceberg v2 equality-delete files and appends replacement data in the same row-delta snapshot. Work grows with key data and files rather than expression-tree size, and no target-table scan is required to construct the commit.

Atomicity and compatibility

• Readers see either the previous snapshot or both the equality deletes and replacement data.
• Physical data/delete files are written before metadata commit; failed commits can leave files for normal orphan-file cleanup.
• Concurrent writes follow Iceberg sequence semantics: a later equality delete can hide matching rows from an earlier commit, while later rows survive.
• Tables must use Iceberg format v2+ and all readers must implement equality deletes.
• Partitioned tables require every partition source column in the merge key; evolved partition specs are currently rejected.
• The pinned fork reports PyIceberg 0.12.0, requires Python 3.10+, and resolves pyiceberg-core 0.10.1. dlt-iceberg already requires Python 3.11+.
• The exact Git SHA makes the behavior reproducible, but installing this revision requires Git/network access until the fork is released as an artifact.

Validation

• uv run pytest tests/ -q: 190 passed
• local SQL catalog: 15,000 composite-key source rows, 25,000 final rows, one merge snapshot; also proven with legacy merge_batch_size=1 to verify it has no effect
• Lakekeeper REST + MinIO: 50,000 composite-key source rows, 125,000 final rows, one merge snapshot
• PyIceberg fork unit suite: 3,970 passed, 3 skipped before the delete-only API addition; focused upsert suite after addition: 29 passed
• PyIceberg Docker REST/MinIO/Spark interoperability: 200,000-row source merge and 300,000-row final table passed

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.