Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 3 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,22 +63,15 @@
> 本 connector 在 1.20 和 2.1 上共享大部分源码,仅对版本不兼容部分通过模块隔离
> (`lance-flink-2.1/` 子模块)提供替代实现。

> **⚠️ Flink 2.1 OVERWRITE 并行度局限(R3)**:Flink 2.1 上 `write.mode = overwrite` 的"预清理" 由 `LanceSinkWriter` 的 subtask 0 在 `Fragment.write()` 之前触发;不同 subtask 之间没有 barrier。
> 在真实集群启动时序偏斜的情况下,subtask-N (N>0) 可能在 subtask 0 完成 `Dataset.drop` 之前就已经 `Fragment.write()` —— 它刚物化的 `.lance` 文件会被随后到来的 drop 一并删掉,导致当前 checkpoint 窗口丢数据。
>
> **缓解措施**(二选一):
> 1. **推荐**:对 OVERWRITE 作业显式 `sink.parallelism = 1`(SQL 侧用 `'sink.parallelism' = '1'` 表选项,DataStream 侧用 `dataStream.sinkTo(sink).setParallelism(1)`);
> 2. 作业编排:先用 parallelism=1 跑一个 OVERWRITE truncate,再启动真正的 parallelism=N 的 APPEND 作业。
>
> 该局限只影响 Flink 2.1 + OVERWRITE + 并行度>1 的组合;APPEND 模式与 Flink 1.20 不受影响。tracking issue 见 follow-up。
> **Flink 2.1 OVERWRITE 语义**:`write.mode = overwrite` 的「每个 job 替换一次」判定由串行的 `LanceCommitter` 承担 —— `LanceSinkWriter` 不再执行任何 `Dataset.drop` 预清理。Committer 给它在 OVERWRITE 模式下产生的每个版本都打上 `flink.job-id` transaction property:job 首次提交执行 `Overwrite` 替换旧数据集,后续 checkpoint 提交自动切换为 `Append`,因此 OVERWRITE 作业可以安全地以并行度 > 1 运行。

### Maven Dependency

```xml
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-lance</artifactId>
<version>1.0.0-SNAPSHOT</version>
<artifactId>lance-flink-2.1</artifactId>
<version>0.2.0</version>
</dependency>
```

Expand Down
19 changes: 9 additions & 10 deletions docs/src/install.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,17 +2,16 @@

## Requirements

* JDK 11 or higher
* JDK 11+ (Flink 1.20) or JDK 17+ (Flink 2.1)
* Maven 3.6+
* Apache Flink 1.18 / 1.19 / 1.20
* Apache Flink 1.20 / 2.1

The connector ships one artifact per supported Flink minor version:

| Flink version | Artifact |
|---|---|
| 1.18 | `lance-flink-1.18` |
| 1.19 | `lance-flink-1.19` |
| 1.20 | `lance-flink-1.20` |
| Flink version | Artifact | Required JDK |
|---|---|---|
| 1.20 | `lance-flink-1.20` | JDK 11+ |
| 2.1 | `lance-flink-2.1` | JDK 17+ |

## Dependencies

Expand All @@ -28,8 +27,8 @@ supported platform (linux-x86-64, linux-aarch64, darwin-aarch64).
```xml
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>lance-flink-1.18</artifactId>
<version>0.1.0</version>
<artifactId>lance-flink-2.1</artifactId>
<version>0.2.0</version>
</dependency>
```

Expand All @@ -39,7 +38,7 @@ supported platform (linux-x86-64, linux-aarch64, darwin-aarch64).
mvn clean verify
```

The build produces a fat jar per module (e.g. `lance-flink-1.18/target/lance-flink-1.18-*.jar`).
The build produces a fat jar per module (e.g. `lance-flink-2.1/target/lance-flink-2.1-*.jar`).
Add the jar to your Flink cluster or job classpath, then use the `lance` / `lance-namespace`
catalog types in SQL.

Expand Down
2 changes: 1 addition & 1 deletion docs/src/operations/dql/vector-search.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ temporary function before use:
```sql
CREATE TEMPORARY FUNCTION vector_search AS
'org.apache.flink.connector.lance.table.LanceVectorSearchFunction'
LANGUAGE JAVA USING JAR '/path/to/lance-flink-1.18-0.1.0.jar';
LANGUAGE JAVA USING JAR '/path/to/lance-flink-2.1-0.2.0.jar';
```

The registered function name (here `vector_search`) is user-chosen. Its `eval` overloads are:
Expand Down
2 changes: 1 addition & 1 deletion lance-flink-1.18/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>org.apache.flink</groupId>
<artifactId>lance-flink-root</artifactId>
<version>0.1.0</version>
<version>0.2.0</version>
<relativePath>../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion lance-flink-1.19/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>org.apache.flink</groupId>
<artifactId>lance-flink-root</artifactId>
<version>0.1.0</version>
<version>0.2.0</version>
<relativePath>../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion lance-flink-1.20/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<parent>
<groupId>org.apache.flink</groupId>
<artifactId>lance-flink-root</artifactId>
<version>0.1.0</version>
<version>0.2.0</version>
<relativePath>../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion lance-flink-2.1/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@
<parent>
<groupId>org.apache.flink</groupId>
<artifactId>lance-flink-root</artifactId>
<version>0.1.0</version>
<version>0.2.0</version>
<relativePath>../pom.xml</relativePath>
</parent>

Expand Down
4 changes: 2 additions & 2 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@

<groupId>org.apache.flink</groupId>
<artifactId>lance-flink-root</artifactId>
<version>0.1.0</version>
<version>0.2.0</version>
<packaging>pom</packaging>

<name>Lance Flink Connector</name>
Expand All @@ -28,7 +28,7 @@
<argLine>--add-opens=java.base/java.nio=ALL-UNNAMED</argLine>

<!-- Lance Flink Connector version -->
<lance-flink.version>0.1.0</lance-flink.version>
<lance-flink.version>0.2.0</lance-flink.version>

<!-- Lance dependency version -->
<lance.version>7.0.0</lance.version>
Expand Down
Loading