Add writeExistingDataManifest for manifest rewrites - #39
Open
ryo-rm wants to merge 2 commits into
Open
Conversation
The Avro reader decodes Iceberg stat maps as arrays of {key, value}
records rather than plain objects; boundForField in prune.js already
handles both shapes on the read side. encodeMap only handled the object
form, so writeExistingDeleteManifest threw "expected bigint value" out of
avroWrite whenever it carried over an entry that had come back from a
manifest with stats on it.
Real delete files carry those stats: the Spark and Java position delete
files under test/files/hyperparam-iceberg/*/bunnies both have
lower_bounds, and the Java equality delete file has value_counts as well.
Icebird alone never reached this, since v3 refuses to write new position
delete files and v2 refuses to write deletion vectors, so the two only
coexist on an upgraded table or one another engine wrote. The added test
covers that case.
Rewriting manifests without touching the data files they reference — merging many small fast-append manifests into one — needs data entries written with status EXISTING and explicit per-entry snapshot_id, sequence_number and file_sequence_number. writeDataManifest hard-codes status ADDED and leaves the sequence numbers null for inheritance, which would move carried-over files forward to the rewriting snapshot's sequence number and change which delete files apply to them. Mirrors writeExistingDeleteManifest, which already covers the delete side for deletion vector replacement. Exporting manifestEntrySchema instead would pin the Avro schema builder's signature, manifestContent argument included, as public API.
ryo-rm
force-pushed
the
feat/write-existing-data-manifest
branch
from
August 18, 2026 08:37
d0f48a7 to
9a606c1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Depends on #38 — that fix is the first commit here.
It would be useful to be able to run the standard "rewrite manifests"
maintenance operation with icebird: merging many small fast-append manifests
into one, without touching the data files. A writer that commits frequently
accumulates manifests quickly, and scans slow down with the count.
Most of it is already reachable —
icebergManifests,buildSnapshotUpdatewith
skipPriorManifestPaths,buildPartitionSummaries. The gap is the write:carried-over files need status
EXISTINGand explicit per-entrysnapshot_id,sequence_numberandfile_sequence_number, whilewriteDataManifesthard-codes
ADDEDand leaves the sequence numbers null for inheritance — whichwould move those files to the rewriting snapshot's sequence number and change
which delete files apply to them.
So this adds
writeExistingDataManifest, mirroringwriteExistingDeleteManifest.Exporting
manifestEntrySchemawould work too, though it pins the Avro schemabuilder's signature as public API. Happy to go whichever way you prefer.
Rewrite manifests built on this
The call site — the rest is assembling the manifest-list entry and committing,
which
buildSnapshotUpdateandfileCatalogCommitalready cover:Run against a v2 table with four data manifests and one position delete file,
merging the data manifests into one and leaving the delete manifest alone:
Building the same merge on
writeDataManifestinstead resurrects the deletedrow: the carried-over files take the rewriting snapshot's sequence number, so
the delete file no longer applies to them.