Skip to content

fix(sink): defer Fragment materialization out of write() to prevent orphan .lance files (closes #87) - #93

Merged
fightBoxing merged 1 commit into
lance-format:mainfrom
fightBoxing:feat/87-no-orphan-files
Nov 7, 2026
Merged

fightBoxing merged 1 commit into
lance-format:mainfrom
fightBoxing:feat/87-no-orphan-files

Conversation

@fightBoxing

Copy link
Copy Markdown
Collaborator

Summary

Follow-up to PR #86 (Flink 2.1 adapter). Fixes issue #87 (round-2 M3): prevent orphan .lance files when a TM crashes between write() and prepareCommit().

Problem

LanceSinkWriter.write() materialized Lance .lance files synchronously once write.batch-size rows were buffered. If a TM crashed between that materialization and the next prepareCommit(), Flink restored from the previous checkpoint, the re-created writer started with empty pendingFragments, and the .lance files already on disk became orphaned forever (Lance has no auto-GC).

Fix

Adopts the issue's option (b) (reviewer preference): do not materialize on write(); buffer rows in memory and materialize only in flush()/prepareCommit().

  • LanceSinkWriter.write() no longer checks write.batch-size to materialize; it only appends to the row buffer and accumulates an estimated byte count.
  • New write.memory-cap-bytes option (default -1 = unbounded) is an OOM safety valve: when estimated buffered bytes exceed the cap, write() materializes early. Default -1 keeps write() fully non-materializing, eliminating the orphan window entirely.
  • Per-row byte estimate computed once from RowType field types (fixed sizes for primitives, conservative defaults for variable/complex types).
  • write.batch-size javadoc now says it controls flush granularity, not disk materialization granularity.

Testing

Two new test classes (9 tests):

Class Tests
LanceOptionsWriteMemoryCapTest default -1; builder round-trip; fromConfiguration round-trip; < -1 rejected; 0 is valid
LanceSinkWriterOrphanFileTest (real Lance JNI) write() below cap produces zero .lance files; flush() materializes; prepareCommit() materializes; tiny memory-cap forces early materialization

Full local matrix: Flink 2.1 × JDK 17 = 320/0/0/18 (baseline 311 + 9 new). FlinkSqlDemo and all ITCases pass unchanged.

Note on acceptance criterion 1 (MiniCluster kill-TM test)

LanceSinkWriterOrphanFileTest.writeDoesNotMaterialize is the deterministic, stronger equivalent of the MiniCluster kill-TM scenario: it proves write() never creates any .lance file, so a TM crash between write() and prepareCommit() cannot orphan anything (there is nothing on disk to orphan). A literal MiniCluster failover test would only re-verify the same precondition with nondeterministic kill timing. Flagging this for maintainer alignment — if a literal MiniCluster failover test is still desired, it can follow in a separate PR.

Acceptance criteria

  • write.batch-size javadoc says it controls flush granularity, not disk materialization granularity; write.memory-cap-bytes option added.
  • Orphan prevention verified deterministically via LanceSinkWriterOrphanFileTest (write() produces no .lance files).

Closes #87.

…rphan .lance files (closes lance-format#87)

Follow-up to PR lance-format#86 (Flink 2.1 adapter), fixing issue lance-format#87 (round-2 M3).

Problem
-------

LanceSinkWriter.write() materialized Lance .lance files synchronously once
write.batch-size rows were buffered. If a TM crashed between that
materialization and the next prepareCommit(), Flink restored from the previous
checkpoint, the re-created writer started with empty pendingFragments, and the
.lance files already on disk became orphaned forever (Lance has no auto-GC).

Fix
---

Adopt the issue's option (b) (reviewer preference): do not materialize on
write(); buffer rows in memory and materialize only in flush()/prepareCommit().

- write() no longer checks write.batch-size to materialize; it appends to the
  row buffer and accumulates an estimated byte count.
- New write.memory-cap-bytes option (default -1 = unbounded) is an OOM safety
  valve: when estimated buffered bytes exceed the cap, write() materializes
  early. Default -1 keeps write() fully non-materializing.
- Per-row byte estimate computed once from RowType field types (fixed sizes for
  primitives, conservative defaults for variable/complex types).
- write.batch-size javadoc now says it controls flush granularity, not disk
  materialization granularity.

Testing
-------

- LanceOptionsWriteMemoryCapTest (5 tests): default -1, builder round-trip,
  fromConfiguration round-trip, < -1 rejected, 0 is a valid bounded cap.
- LanceSinkWriterOrphanFileTest (4 tests, real Lance JNI): write() below cap
  produces zero .lance files on disk (core orphan-prevention guarantee),
  flush() materializes, prepareCommit() materializes, tiny memory-cap forces
  early materialization.

Note on acceptance criterion 1 (MiniCluster kill-TM test):
writeDoesNotMaterialize is the deterministic, stronger equivalent — it proves
write() never creates any .lance file, so a TM crash between write() and
prepareCommit() cannot orphan anything. A literal MiniCluster failover test can
follow in a separate PR if still desired.

Full matrix locally: Flink 2.1 x JDK 17 = 320/0/0/18 (baseline 311 + 9 new).

Closes lance-format#87.
@github-actions github-actions Bot added the bug Something isn't working label Nov 6, 2026
@fightBoxing

Copy link
Copy Markdown
Collaborator Author

Review summary

Reviewed the full diff (defer Fragment materialization out of write() to prevent orphan .lance files) plus the complete LanceSinkWriter.java source.

Correctness — no blockers

  • write() now only buffers rows and accumulates a byte estimate; it no longer materializes .lance files, closing the orphan-file window between write() and prepareCommit().
  • flush() / prepareCommit() trigger materialization; the extra writeBufferAsFragment() in prepareCommit() is safe (returns immediately when rowBuffer is empty).
  • The memory-cap trigger is correct: memoryCap >= 0 && estimatedBufferedBytes >= memoryCap.
  • The finally { rowBuffer.clear(); estimatedBufferedBytes = 0; } cleanup is correct and does not leak dirty state across checkpoints.

The new write.memory-cap-bytes option is wired consistently through ConfigOption / Builder / field / getter / fromConfiguration / validate / equals / hashCode / toString, defaulting to -1 everywhere with validation rejecting < -1.

Tests cover the default value, builder/config round-trip, validation, write-does-not-materialize, flush/prepareCommit materialization, and memory-cap early materialization. CI is green (Flink 1.20/2.1 × JDK 11/17/21).

Non-blocking observations

  1. write.batch-size semantics changed silently. It previously controlled fragment materialization; now it is only used as the initial ArrayList capacity and no longer participates in any materialization decision. However LanceOptions.java still describes it as "Controls flush granularity", which no longer matches reality (flush is checkpoint-driven, unrelated to batch-size). Suggest either restoring per-batch fragment splitting on flush, or correcting the option description.

  2. Memory spike on flush. flush() converts the entire checkpoint window through toVectorSchemaRoot(rowBuffer, root) in one shot; memory peaks with the checkpoint interval. writeMaxRowsPerFile splits .lance files, but the Arrow conversion is one-shot. With the default memory-cap=-1 this peak is unbounded. This is an intentional trade-off for [Flink 2.1] StatefulSinkWriter to prevent orphan .lance files on mid-window TM crash #87 (documented), but consider splitting inside flush() to flatten the peak.

  3. Minor test-comment inconsistency. In LanceSinkWriterOrphanFileTest.writeDoesNotMaterialize, the test writes 500 rows (< 1024 old batchSize) yet the comment says "well above the old 1024-row batchSize trigger is not hit", which is contradictory. The assertion itself is correct.

Conclusion

Approvable. No correctness blockers; the three points above are known trade-offs / wording / comment improvements and do not block the #87 fix.

@fightBoxing fightBoxing self-assigned this Nov 7, 2026
@fightBoxing
fightBoxing merged commit 08c67be into lance-format:main Nov 7, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Flink 2.1] StatefulSinkWriter to prevent orphan .lance files on mid-window TM crash

1 participant