Bound composite-key merge predicates

[nicosuave]

Summary

• bound composite-key PyIceberg match/delete predicates with a dedicated merge_composite_key_batch_size setting (default 500)
• preserve the existing merge_batch_size=500000 behavior for single-key merges, whose match filter is a flat IN expression
• stage all upsert, delete-insert, and hard-delete batches in one PyIceberg transaction per table
• validate duplicate upsert keys across the complete incoming table before batching
• update the development lockfile to PyIceberg 0.11.1 / pyiceberg-core 0.8.0

Evidence and rationale

Production on macOS arm64 with dlt-iceberg 0.6.2, PyIceberg 0.11.1, and AWS S3 Tables exited with SIGBUS when a growing table received roughly 1,018 composite-key rows in one merge transaction; 515 rows succeeded.

PyIceberg builds a single-key match as an In predicate, but composite keys become a nested Or of per-row And predicates. The general 500,000-row batch limit therefore remains appropriate for the flatter single-key path, while composite expressions get a separate conservative cap below the observed failure boundary.

PyIceberg 0.11.1 is currently the latest stable release, and its patch release changes do not include a fix for this merge-expression failure mode. The transaction APIs used here are available in both the project's declared minimum PyIceberg 0.10.0 and current 0.11.1.

Upstream implementation references:

• https://github.com/apache/iceberg-python/blob/pyiceberg-0.11.1/pyiceberg/table/upsert_util.py
• https://github.com/apache/iceberg-python/blob/pyiceberg-0.11.1/pyiceberg/table/__init__.py

Atomicity and performance

All batches for one table are staged through Transaction.delete, Transaction.upsert, and Transaction.append, then published by one catalog metadata commit. Readers of the current table reference therefore see either the pre-merge state or the completed merge state. A failure before transaction exit does not publish a partial batch.

Atomicity remains per table, as before. It does not cover multiple tables in one dlt load, and prior table creation/schema-evolution commits are separate. Failed writes may leave unreferenced data files for normal Iceberg orphan-file cleanup.

The composite safety cap trades additional scans and staged snapshot updates for bounded native expression size. Operators can tune merge_composite_key_batch_size upward after workload-specific testing; merge_batch_size continues to act as the overall ceiling.

Validation

• uv run pytest -q — 184 passed, 15 skipped (PyIceberg 0.11.1)
• focused regression suite — 11 passed on PyIceberg 0.11.1
• focused regression suite — 11 passed on PyIceberg 0.10.0
• real local SQLite-catalog upsert and delete-insert tests at 1,018 composite-key rows on both supported PyIceberg versions
• uv lock --check
• git diff --check