Replace predicate merges with equality-delete row deltas

[nicosuave]

Summary

• pin the Sidequery PyIceberg fork at 968fb967, including equality-delete planning, Arrow reads, writes, and delete-only keys
• route both dlt merge strategy names through one Iceberg equality-delete row delta
• commit hard-delete keys and replacement rows in the same snapshot
• remove composite predicate construction and all row batching; the deprecated merge_batch_size option is accepted but ignored
• document reader, partitioning, concurrency, and orphan-file tradeoffs

This supersedes the batching approach from closed PR #24. The merge size is no longer governed by an arbitrary row count.

Why

The previous paths generated a large OR(AND(...), ...) expression for composite delete-insert and PyIceberg upsert generated match predicates while scanning/re-writing target data. Production observed a native SIGBUS around 1,018 composite-key rows with PyIceberg 0.11.1 on macOS arm64. Making the batches smaller only avoided the failure and weakened snapshot atomicity; it did not fix the merge design.

The new path writes the incoming keys as Iceberg v2 equality-delete files and appends replacement data in the same row-delta snapshot. Work grows with key data and files rather than expression-tree size, and no target-table scan is required to construct the commit.

Atomicity and compatibility

• Readers see either the previous snapshot or both the equality deletes and replacement data.
• Physical data/delete files are written before metadata commit; failed commits can leave files for normal orphan-file cleanup.
• Concurrent writes follow Iceberg sequence semantics: a later equality delete can hide matching rows from an earlier commit, while later rows survive.
• Tables must use Iceberg format v2+ and all readers must implement equality deletes.
• Partitioned tables require every partition source column in the merge key; evolved partition specs are currently rejected.
• The pinned fork reports PyIceberg 0.12.0, requires Python 3.10+, and resolves pyiceberg-core 0.10.1. dlt-iceberg already requires Python 3.11+.
• The exact Git SHA makes the behavior reproducible, but installing this revision requires Git/network access until the fork is released as an artifact.

Validation

• uv run pytest tests/ -q: 190 passed
• local SQL catalog: 15,000 composite-key source rows, 25,000 final rows, one merge snapshot; also proven with legacy merge_batch_size=1 to verify it has no effect
• Lakekeeper REST + MinIO: 50,000 composite-key source rows, 125,000 final rows, one merge snapshot
• PyIceberg fork unit suite: 3,970 passed, 3 skipped before the delete-only API addition; focused upsert suite after addition: 29 passed
• PyIceberg Docker REST/MinIO/Spark interoperability: 200,000-row source merge and 300,000-row final table passed