Repository navigation
fix(sink): defer Fragment materialization out of write() to prevent orphan .lance files (closes #87) - #93
Conversation
…rphan .lance files (closes lance-format#87) Follow-up to PR lance-format#86 (Flink 2.1 adapter), fixing issue lance-format#87 (round-2 M3). Problem ------- LanceSinkWriter.write() materialized Lance .lance files synchronously once write.batch-size rows were buffered. If a TM crashed between that materialization and the next prepareCommit(), Flink restored from the previous checkpoint, the re-created writer started with empty pendingFragments, and the .lance files already on disk became orphaned forever (Lance has no auto-GC). Fix --- Adopt the issue's option (b) (reviewer preference): do not materialize on write(); buffer rows in memory and materialize only in flush()/prepareCommit(). - write() no longer checks write.batch-size to materialize; it appends to the row buffer and accumulates an estimated byte count. - New write.memory-cap-bytes option (default -1 = unbounded) is an OOM safety valve: when estimated buffered bytes exceed the cap, write() materializes early. Default -1 keeps write() fully non-materializing. - Per-row byte estimate computed once from RowType field types (fixed sizes for primitives, conservative defaults for variable/complex types). - write.batch-size javadoc now says it controls flush granularity, not disk materialization granularity. Testing ------- - LanceOptionsWriteMemoryCapTest (5 tests): default -1, builder round-trip, fromConfiguration round-trip, < -1 rejected, 0 is a valid bounded cap. - LanceSinkWriterOrphanFileTest (4 tests, real Lance JNI): write() below cap produces zero .lance files on disk (core orphan-prevention guarantee), flush() materializes, prepareCommit() materializes, tiny memory-cap forces early materialization. Note on acceptance criterion 1 (MiniCluster kill-TM test): writeDoesNotMaterialize is the deterministic, stronger equivalent — it proves write() never creates any .lance file, so a TM crash between write() and prepareCommit() cannot orphan anything. A literal MiniCluster failover test can follow in a separate PR if still desired. Full matrix locally: Flink 2.1 x JDK 17 = 320/0/0/18 (baseline 311 + 9 new). Closes lance-format#87.
Review summaryReviewed the full diff (defer Fragment materialization out of Correctness — no blockers
The new Tests cover the default value, builder/config round-trip, validation, write-does-not-materialize, flush/prepareCommit materialization, and memory-cap early materialization. CI is green (Flink 1.20/2.1 × JDK 11/17/21). Non-blocking observations
ConclusionApprovable. No correctness blockers; the three points above are known trade-offs / wording / comment improvements and do not block the #87 fix. |
Summary
Follow-up to PR #86 (Flink 2.1 adapter). Fixes issue #87 (round-2 M3): prevent orphan
.lancefiles when a TM crashes betweenwrite()andprepareCommit().Problem
LanceSinkWriter.write()materialized Lance.lancefiles synchronously oncewrite.batch-sizerows were buffered. If a TM crashed between that materialization and the nextprepareCommit(), Flink restored from the previous checkpoint, the re-created writer started with emptypendingFragments, and the.lancefiles already on disk became orphaned forever (Lance has no auto-GC).Fix
Adopts the issue's option (b) (reviewer preference): do not materialize on
write(); buffer rows in memory and materialize only inflush()/prepareCommit().LanceSinkWriter.write()no longer checkswrite.batch-sizeto materialize; it only appends to the row buffer and accumulates an estimated byte count.write.memory-cap-bytesoption (default-1= unbounded) is an OOM safety valve: when estimated buffered bytes exceed the cap,write()materializes early. Default-1keepswrite()fully non-materializing, eliminating the orphan window entirely.RowTypefield types (fixed sizes for primitives, conservative defaults for variable/complex types).write.batch-sizejavadoc now says it controls flush granularity, not disk materialization granularity.Testing
Two new test classes (9 tests):
LanceOptionsWriteMemoryCapTest-1; builder round-trip;fromConfigurationround-trip;< -1rejected;0is validLanceSinkWriterOrphanFileTest(real Lance JNI)write()below cap produces zero.lancefiles;flush()materializes;prepareCommit()materializes; tiny memory-cap forces early materializationFull local matrix: Flink 2.1 × JDK 17 = 320/0/0/18 (baseline 311 + 9 new).
FlinkSqlDemoand all ITCases pass unchanged.Note on acceptance criterion 1 (MiniCluster kill-TM test)
LanceSinkWriterOrphanFileTest.writeDoesNotMaterializeis the deterministic, stronger equivalent of the MiniCluster kill-TM scenario: it proveswrite()never creates any.lancefile, so a TM crash betweenwrite()andprepareCommit()cannot orphan anything (there is nothing on disk to orphan). A literal MiniCluster failover test would only re-verify the same precondition with nondeterministic kill timing. Flagging this for maintainer alignment — if a literal MiniCluster failover test is still desired, it can follow in a separate PR.Acceptance criteria
write.batch-sizejavadoc says it controls flush granularity, not disk materialization granularity;write.memory-cap-bytesoption added.LanceSinkWriterOrphanFileTest(write() produces no.lancefiles).Closes #87.