Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 49 additions & 1 deletion docs/src/config.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,9 +27,15 @@ catalog types are `'lance'` (directory/S3) and `'lance-namespace'` (dir/rest).

| Option | Required | Default | Description |
|---|---|---|---|
| `write.batch-size` | ❌ | 1024 | Write batch size |
| `write.batch-size` | ❌ | 1024 | Rows buffered per writer before a flush. Controls flush granularity, not disk layout: rows are materialized on `flush()` / `prepareCommit()` |
| `write.mode` | ❌ | append | `append` or `overwrite` |
| `write.max-rows-per-file` | ❌ | 1000000 | Maximum rows per data file |
| `write.memory-cap-bytes` | ❌ | -1 | Estimated in-memory buffer size that forces an early fragment materialization. `-1` means unbounded — `write()` never materializes, which removes the orphan-file window but makes memory usage grow with the checkpoint interval |

> **`write.mode = overwrite`**: the dataset is replaced **once per job**. The decision lives in the
> serial committer, which tags every version it commits with the `flink.job-id` transaction
> property: the first commit issues `Overwrite`, later checkpoint commits issue `Append`. OVERWRITE
> jobs are therefore safe with parallelism > 1.

### Vector index

Expand All @@ -54,6 +60,30 @@ catalog types are `'lance'` (directory/S3) and `'lance-namespace'` (dir/rest).
| `vector.ef` | ❌ | 100 | HNSW search width |
| `vector.refine-factor` | ❌ | — | Refine factor for recall |

## Primary key and changelog mode

Declare a primary key to switch the sink from append-only to upsert:

```sql
CREATE TABLE users (
id BIGINT,
name STRING,
embedding ARRAY<FLOAT>,
PRIMARY KEY (id) NOT ENFORCED
) WITH (
'connector' = 'lance',
'path' = '/data/users'
);
```

| Declared schema | Changelog mode accepted by the sink |
|---|---|
| no primary key | `INSERT` only |
| `PRIMARY KEY` | `INSERT`, `UPDATE_AFTER`, `DELETE` (`UPDATE_BEFORE` is dropped) |

Upserts are applied by `Dataset.mergeInsert` in the serial committer; rows are `keyBy` the primary
key so all events for one key reach the same subtask. Composite primary keys are supported.

## Catalog options (`type = 'lance'`)

Directory or S3 warehouse.
Expand All @@ -77,3 +107,21 @@ Directory or S3 warehouse.
| `root` | ❌ | — | Root path for directory namespace |
| `uri` | ❌ | — | URI for REST namespace |
| `default-database` | ❌ | default | Default database name |

## Hadoop-family filesystem options

Any option prefixed with `hadoop.` is stripped and injected into the Hadoop `Configuration` used to
resolve the dataset path, so `hdfs://` / `tbdsfs://` / `viewfs://` locations can be reached without
a `core-site.xml` on the node:

```sql
CREATE TABLE hdfs_table (
id BIGINT,
embedding ARRAY<FLOAT>
) WITH (
'connector' = 'lance',
'path' = 'tbdsfs:///path/to/dataset',
'hadoop.tbdsfs.meta' = 'zk://namenode:2181/tbdsfs',
'hadoop.fs.defaultFS' = 'tbdsfs:///'
);
```
15 changes: 15 additions & 0 deletions docs/src/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,18 +9,33 @@ and vector search workloads.
By using the Flink Connector for Lance, you can run Flink's stream/batch processing, SQL querying,
and stateful pipelines directly on Lance datasets, including native vector search.

## Supported versions

| Module | Flink version | Required JDK |
|---|---|---|
| `lance-flink-1.20` | Apache Flink 1.20 | JDK 11+ |
| `lance-flink-2.1` | Apache Flink 2.1 | JDK 17+ |

Flink 1.18 and 1.19 reached EOL before the Flink 2.1 support landed and are no longer built or
released from 0.2.0 on.

## Features

The connector is built on the Flink Table API (`DynamicTableSource` / `DynamicTableSink`) plus
`CatalogFactory`. Specifically, you can use the Flink Connector for Lance to:

* **Read & Write Lance Datasets**: append and overwrite datasets via Flink SQL or the DataStream API.
* **Upsert & Delete**: declare a `PRIMARY KEY ... NOT ENFORCED` (single or composite) and ingest
`+I` / `+U` / `-D` changelogs through `Dataset.mergeInsert`.
* **Schema Evolution**: `ALTER TABLE` add / rename / drop columns through the `lance` catalog.
* **Column, Filter, Limit & Aggregate Pushdown**: push projections, `WHERE` predicates, limits and
aggregations down to Lance for efficient scans.
* **Vector Search**: KNN search over `ARRAY<FLOAT>` columns with `L2`, `Cosine`, and `Dot` metrics,
via the `LanceVectorSearchFunction` table function.
* **Vector Index Building**: create `IVF_PQ`, `IVF_HNSW`, and `IVF_FLAT` indexes.
* **Time Travel**: read a historical version via `read.version` or `read.as-of-timestamp`.
* **Hadoop-compatible filesystems**: read `hdfs://`, `tbdsfs://` and `viewfs://` paths through the
Hadoop `FileSystem` API.
* **Catalog Support**: a directory/S3 `lance` catalog and a `lance-namespace` catalog (dir / rest).

## Quick Start
Expand Down
24 changes: 21 additions & 3 deletions docs/src/install.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,9 @@ The connector ships one artifact per supported Flink minor version:

## Dependencies

The connector depends on `org.lance:lance-core` (7.0.0) and Apache Arrow (18.3.0). These are pulled
in transitively; you only need to add the connector artifact for your Flink version.
The connector depends on `org.lance:lance-core` (7.0.0), `lance-namespace` (0.7.5) and Apache Arrow
(18.3.0). These are pulled in transitively; you only need to add the connector artifact for your
Flink version.

Lance's Java bindings ship a platform-specific JNI native library
(`liblance_jni.so` / `liblance_jni.dylib`) inside the `lance-core` jar. Ensure you run on a
Expand All @@ -25,6 +26,14 @@ supported platform (linux-x86-64, linux-aarch64, darwin-aarch64).
## Maven

```xml
<!-- Flink 1.20 -->
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>lance-flink-1.20</artifactId>
<version>0.2.0</version>
</dependency>

<!-- Flink 2.1 -->
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>lance-flink-2.1</artifactId>
Expand All @@ -35,7 +44,12 @@ supported platform (linux-x86-64, linux-aarch64, darwin-aarch64).
## Build from source

```bash
mvn clean verify
# Both modules (requires JDK 17+, since the Flink 2.1 module targets Java 17)
mvn clean package

# A single module
mvn -pl lance-flink-1.20 test
mvn -pl lance-flink-2.1 test
```

The build produces a fat jar per module (e.g. `lance-flink-2.1/target/lance-flink-2.1-*.jar`).
Expand All @@ -46,3 +60,7 @@ catalog types in SQL.

The Arrow allocator defaults are set at runtime; if you see classloader-related SPI issues in
tests, set the system property `arrow.memory.allocator.type=Netty`.

On JDK 17+, Arrow reaches `java.nio.Buffer.address` reflectively, so add
`--add-opens=java.base/java.nio=ALL-UNNAMED` to the JVM arguments (the Maven build already does
this through `argLine`).
32 changes: 25 additions & 7 deletions docs/src/operations/ddl/create-catalog.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,10 +71,28 @@ CREATE CATALOG my_lance WITH (

## Supported DDL

| Statement | Status |
|---|---|
| `CREATE DATABASE` / `DROP DATABASE` / `ALTER DATABASE` | ✅ |
| `SHOW DATABASES` / `SHOW TABLES` | ✅ |
| `CREATE TABLE` / `DROP TABLE` / `RENAME TABLE` | ✅ |
| `ALTER TABLE` | ❌ — not supported (structure immutable) |
| `CREATE INDEX` | ❌ — no SQL DDL; configure `index.*` on the table |
| Statement | `lance` (directory/S3) | `lance-namespace` |
|---|---|---|
| `CREATE DATABASE` / `DROP DATABASE` / `ALTER DATABASE` | ✅ | ✅ |
| `SHOW DATABASES` / `SHOW TABLES` | ✅ | ✅ |
| `CREATE TABLE` / `DROP TABLE` | ✅ | ✅ |
| `RENAME TABLE` | ✅ | ❌ — not supported |
| `ALTER TABLE` (add / rename / drop column) | ✅ | ❌ — not supported yet |
| `CREATE INDEX` | ❌ — no SQL DDL; configure `index.*` on the table | ❌ |

### ALTER TABLE

On the `lance` catalog, `ALTER TABLE` performs schema evolution on the underlying dataset:

| Change | Supported | Notes |
|---|---|---|
| `ADD COLUMN` | ✅ | New column is added to the Arrow schema |
| `DROP COLUMN` | ✅ | Column is removed from the Arrow schema |
| `RENAME COLUMN` | ✅ | Metadata-only rename |
| Change column type | ❌ | Rejected — `castTo` in Lance Java SDK 7.0.0 is a silent no-op, so widening would corrupt data silently |

```sql
ALTER TABLE vectors ADD (source STRING);
ALTER TABLE vectors RENAME COLUMN content TO body;
ALTER TABLE vectors DROP source;
```
32 changes: 32 additions & 0 deletions docs/src/operations/ddl/create-table.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,16 +30,48 @@ CREATE TABLE doc_embeddings (
'index.column' = 'embedding',
'index.num-partitions' = '256',
'index.num-sub-vectors' = '16',
'index.num-bits' = '8',
'vector.metric' = 'COSINE'
);
```

Index tuning knobs (all optional, shown with defaults):

| `index.type` | Options |
|---|---|
| `IVF_PQ` | `index.num-partitions` (256), `index.num-sub-vectors` (auto), `index.num-bits` (8) |
| `IVF_HNSW` | `index.num-partitions` (256), `index.max-level` (7), `index.m` (16), `index.ef-construction` (100) |
| `IVF_FLAT` | `index.num-partitions` (256) |

Building the index itself is done through `LanceIndexBuilder` (see
[Performance](../../performance.md)); the `index.*` table options carry the configuration.

## Required options

| Option | Description |
|---|---|
| `path` | Path to the Lance dataset |

## Primary key tables

Add `PRIMARY KEY ... NOT ENFORCED` to switch the table from append-only to upsert:

```sql
CREATE TABLE users (
id BIGINT,
name STRING,
embedding ARRAY<FLOAT>,
PRIMARY KEY (id) NOT ENFORCED
) WITH (
'connector' = 'lance',
'path' = '/data/users'
);
```

Composite primary keys are supported, and the constraint is persisted with the dataset so it
survives a catalog round-trip. See [INSERT INTO](../dml/insert-into.md) for the changelog modes
this enables.

## Type mapping

| Lance / Arrow type | Flink type |
Expand Down
54 changes: 47 additions & 7 deletions docs/src/operations/dml/insert-into.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,18 @@ the `write.mode` option.
| `write.mode` | Behaviour |
|---|---|
| `append` (default) | Append rows to the existing dataset |
| `overwrite` | Replace the dataset on first write |
| `overwrite` | Replace the dataset once per job |

`overwrite` is decided by the serial committer, which tags every version it commits with the
`flink.job-id` transaction property:

- the first commit of a job issues `Overwrite` and replaces whatever the dataset held before;
- later checkpoint commits of the same job issue `Append`, so rows written by other subtasks or by
earlier checkpoints are preserved.

A new submission gets a new job id and therefore replaces the previous result again. Because the
decision no longer happens in the writer, OVERWRITE jobs are safe with parallelism &gt; 1 — a
late-starting subtask never deletes files that a peer already materialized.

## Example

Expand All @@ -24,16 +35,45 @@ INSERT INTO vectors VALUES
| `write.batch-size` | 1024 | Rows buffered before a flush |
| `write.mode` | `append` | `append` or `overwrite` |
| `write.max-rows-per-file` | 1000000 | Rows per data file |
| `write.memory-cap-bytes` | -1 | Buffered-bytes threshold that forces an early fragment materialization (`-1` = unbounded) |

## Primary key: upsert and delete

Declaring a primary key switches the sink to keyed upsert mode:

```sql
CREATE TABLE users (
id BIGINT,
name STRING,
embedding ARRAY<FLOAT>,
PRIMARY KEY (id) NOT ENFORCED
) WITH (
'connector' = 'lance',
'path' = '/data/users'
);

-- +I: insert
INSERT INTO users VALUES (1, 'Alice', ARRAY[0.1, 0.2]);

-- +U / -D arrive from any changelog source and are applied via mergeInsert
INSERT INTO users SELECT id, name, embedding FROM upstream_cdc;
```

| Declared schema | Accepted changelog |
|---|---|
| no primary key | `INSERT` only |
| `PRIMARY KEY` (single or composite) | `INSERT`, `UPDATE_AFTER`, `DELETE` |

Rows are `keyBy` the primary key so that every event for a key reaches the same subtask; the
committer then applies `Dataset.mergeInsert` serially.

## Current limitations
## Statement support

| Statement | Status |
|---|---|
| `INSERT INTO` (append) | ✅ |
| `INSERT OVERWRITE` | ✅ |
| `UPDATE` | ❌ — not implemented |
| `DELETE` | ❌ — in progress (see issue #63 / #74) |
| Primary key / upsert | ❌ — PK declaration and CDC changelog not yet supported |
| `UPDATE` | ❌ — no dedicated statement; use an upsert table with a primary key |
| `DELETE` | ❌ — no dedicated statement; use an upsert table with a primary key |

> The sink currently declares insert-only changelog mode. CDC `UPDATE` / `DELETE`
> support is tracked in the connector roadmap.
> The sink declares insert-only changelog mode unless the table declares a primary key.
12 changes: 12 additions & 0 deletions docs/src/operations/dql/select.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,3 +46,15 @@ CREATE TABLE lance_table (
| `read.columns` | Comma-separated columns to read |
| `read.filter` | SQL `WHERE`-style filter string |
| `read.limit` | Maximum rows to read |

## Aggregate pushdown

`SupportsAggregatePushDown` lets eligible aggregates run natively in Lance instead of pulling every
row into Flink:

```sql
SELECT COUNT(*), MIN(id), MAX(id), SUM(id), AVG(id)
FROM lance_table;
```

Aggregates that cannot be translated fall back to Flink execution transparently.
2 changes: 2 additions & 0 deletions docs/src/operations/dql/time-travel.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,3 +42,5 @@ CREATE TABLE lance_table (
`Instant.parse`, `OffsetDateTime.parse`, or `ZonedDateTime.parse`;
a bare `yyyy-MM-ddTHH:mm:ss` is assumed UTC.
- A timestamp predating the oldest version raises a clear error.
- Time travel is read-only: `write.mode = overwrite` replaces the dataset with a new version, so
earlier versions stay reachable until Lance garbage-collects them.
1 change: 1 addition & 0 deletions docs/src/performance.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ read/search throughput and write amplification in the Lance Flink connector.
|---|---|---|
| `write.batch-size` | 1024 | Rows buffered before a flush; larger = fewer, bigger writes |
| `write.max-rows-per-file` | 1000000 | Rows per data file; larger = fewer files, larger compaction units |
| `write.memory-cap-bytes` | -1 | Buffered-bytes threshold that forces an early fragment materialization. `-1` (unbounded) removes the orphan-file window at the cost of memory growing with the checkpoint interval; set an explicit cap for jobs with long checkpoint intervals |

## Distance metrics

Expand Down
Loading
Loading