Skip to content

fix(table): register the documented tuning options on the table factory - #99

Merged
fightBoxing merged 1 commit into
lance-format:mainfrom
fightBoxing:docs/refresh-docs-and-register-tuning-options
Nov 9, 2026
Merged

fightBoxing merged 1 commit into
lance-format:mainfrom
fightBoxing:docs/refresh-docs-and-register-tuning-options

Conversation

@fightBoxing

Copy link
Copy Markdown
Collaborator

Summary

While refreshing docs/src for 0.2.0 I found that eight tuning options were documented and present in LanceOptions, but never declared on LanceDynamicTableFactory. A DDL using any of them failed at translation:

ValidationException: Unsupported options found for 'lance'.
  read.limit
  write.memory-cap-bytes
  index.num-bits
  index.max-level
  index.m
  index.ef-construction
  vector.ef
  vector.refine-factor

So the fix and the docs refresh land together.

Code fix

  • Register the eight keys in optionalOptions().
  • Thread them into the LanceOptions builder in buildLanceOptions().
  • Add DocumentedOptionsTest, which pins the contract:
    • every documented key is registered on the factory;
    • one DDL using all of them survives translation (explainSql runs the factory's option validation);
    • the values actually reach LanceOptions (read limit, memory cap, PQ/HNSW index knobs, vector search knobs).
  • Extend LanceSqlITCase#testOptionalOptions to the full documented key set.

Docs refresh (docs/src)

Page Change
index.md Supported version matrix (1.20 / 2.1), note that 1.18 / 1.19 are no longer built from 0.2.0, and the new capability bullets (upsert/delete, schema evolution, Hadoop filesystems)
install.md Both Maven coordinates, per-module build commands, lance-namespace dependency, JDK 17 --add-opens note
config.md write.memory-cap-bytes; the overwrite-once-per-job semantics; primary-key changelog modes; hadoop.* path options
operations/ddl/create-catalog.md ALTER TABLE is supported on the lance catalog (add / rename / drop column) and rejected on lance-namespace; type changes rejected; RENAME TABLE matrix per catalog
operations/ddl/create-table.md Primary-key tables; per-index-type tuning knobs
operations/dml/insert-into.md Corrected write modes, memory cap option, and the upsert/delete story (previously marked "not supported")
operations/dql/select.md Aggregate pushdown section
operations/dql/time-travel.md Version-retention note for overwrite
performance.md Memory cap in the write-amplification table

Verification

mvn clean package (JDK 17) → BUILD SUCCESS

Module Tests
lance-flink-1.20 Tests run: 278, Failures: 0, Errors: 0, Skipped: 18
lance-flink-2.1 Tests run: 337, Failures: 0, Errors: 0, Skipped: 18

New: DocumentedOptionsTest (4 cases) and LanceSqlITCase#testDocumentedTuningOptionsAreAcceptedByTheFactory.

The docs and LanceOptions exposed eight tuning knobs that the dynamic table
factory never declared, so a DDL using them failed at translation with
'ValidationException: Unsupported options found for lance':
read.limit, write.memory-cap-bytes, index.num-bits, index.max-level, index.m,
index.ef-construction, vector.ef and vector.refine-factor.

Register them in optionalOptions() and thread them into the LanceOptions
builder, then add DocumentedOptionsTest to keep the docs and the factory in
sync: it asserts every documented key is registered, that one DDL using all of
them survives translation, and that the values actually reach LanceOptions.

Refresh docs/src for 0.2.0:
- index: supported version matrix, upsert/delete, schema evolution and
  Hadoop-filesystem reads;
- install: both artifacts, per-module builds, JDK 17 add-opens note, and a note
  that 1.18 / 1.19 are no longer built;
- config: write.memory-cap-bytes, the overwrite once-per-job semantics, primary
  key changelog modes, and hadoop.* path options;
- create-catalog: ALTER TABLE is supported on the lance catalog (add / rename /
  drop column) but not on the namespace catalog, and type changes are rejected;
- create-table: primary key tables and the index tuning knobs per index type;
- insert-into: the corrected write modes and the upsert/delete story;
- select / performance / time-travel: aggregate pushdown, the memory cap, and
  the version-retention note.

Co-authored-by: Cursor <cursoragent@cursor.com>
@github-actions github-actions Bot added the bug Something isn't working label Nov 9, 2026
@fightBoxing
fightBoxing merged commit 24f6534 into lance-format:main Nov 9, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant