Skip to content

Add writeExistingDataManifest for manifest rewrites - #39

Open
ryo-rm wants to merge 2 commits into
hyparam:masterfrom
ryo-rm:feat/write-existing-data-manifest
Open

Add writeExistingDataManifest for manifest rewrites#39
ryo-rm wants to merge 2 commits into
hyparam:masterfrom
ryo-rm:feat/write-existing-data-manifest

Conversation

@ryo-rm

@ryo-rm ryo-rm commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Depends on #38 — that fix is the first commit here.

It would be useful to be able to run the standard "rewrite manifests"
maintenance operation with icebird: merging many small fast-append manifests
into one, without touching the data files. A writer that commits frequently
accumulates manifests quickly, and scans slow down with the count.

Most of it is already reachable — icebergManifests, buildSnapshotUpdate
with skipPriorManifestPaths, buildPartitionSummaries. The gap is the write:
carried-over files need status EXISTING and explicit per-entry snapshot_id,
sequence_number and file_sequence_number, while writeDataManifest
hard-codes ADDED and leaves the sequence numbers null for inheritance — which
would move those files to the rewriting snapshot's sequence number and change
which delete files apply to them.

So this adds writeExistingDataManifest, mirroring writeExistingDeleteManifest.
Exporting manifestEntrySchema would work too, though it pins the Avro schema
builder's signature as public API. Happy to go whichever way you prefer.

Rewrite manifests built on this

The call site — the rest is assembling the manifest-list entry and committing,
which buildSnapshotUpdate and fileCatalogCommit already cover:

const entries = (await icebergManifests({ metadata, resolver }))
  .filter(l => dataManifestPaths.has(l.url))
  .flatMap(l => l.entries)
  .filter(e => e.status !== 2)

const writer = resolver.writer(manifestPath)
await writeExistingDataManifest({ writer, schema, partitionSpec, entries, formatVersion })

Run against a v2 table with four data manifests and one position delete file,
merging the data manifests into one and leaving the delete manifest alone:

data manifests before: 4 | delete manifests kept: 1
merged manifest: min_sequence_number=1 sequence_number=6
manifests after: 1 data + 1 delete

rows before rewrite: 0,1,3,4,5,99
rows after rewrite:  0,1,3,4,5,99

Building the same merge on writeDataManifest instead resurrects the deleted
row: the carried-over files take the rewriting snapshot's sequence number, so
the delete file no longer applies to them.

merged manifest: min_sequence_number=6 sequence_number=6
rows after rewrite:  0,1,2,3,4,5,99

ryo-rm added 2 commits August 18, 2026 16:49
The Avro reader decodes Iceberg stat maps as arrays of {key, value}
records rather than plain objects; boundForField in prune.js already
handles both shapes on the read side. encodeMap only handled the object
form, so writeExistingDeleteManifest threw "expected bigint value" out of
avroWrite whenever it carried over an entry that had come back from a
manifest with stats on it.

Real delete files carry those stats: the Spark and Java position delete
files under test/files/hyperparam-iceberg/*/bunnies both have
lower_bounds, and the Java equality delete file has value_counts as well.
Icebird alone never reached this, since v3 refuses to write new position
delete files and v2 refuses to write deletion vectors, so the two only
coexist on an upgraded table or one another engine wrote. The added test
covers that case.
Rewriting manifests without touching the data files they reference —
merging many small fast-append manifests into one — needs data entries
written with status EXISTING and explicit per-entry snapshot_id,
sequence_number and file_sequence_number. writeDataManifest hard-codes
status ADDED and leaves the sequence numbers null for inheritance, which
would move carried-over files forward to the rewriting snapshot's
sequence number and change which delete files apply to them.

Mirrors writeExistingDeleteManifest, which already covers the delete side
for deletion vector replacement. Exporting manifestEntrySchema instead
would pin the Avro schema builder's signature, manifestContent argument
included, as public API.
@ryo-rm
ryo-rm force-pushed the feat/write-existing-data-manifest branch from d0f48a7 to 9a606c1 Compare August 18, 2026 08:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant