Skip to content

[VL] Enable assert_not_null expression for Velox backend#11685

Open
yaooqinn wants to merge 1 commit intoapache:mainfrom
yaooqinn:feature/assert-not-null
Open

[VL] Enable assert_not_null expression for Velox backend#11685
yaooqinn wants to merge 1 commit intoapache:mainfrom
yaooqinn:feature/assert-not-null

Conversation

@yaooqinn
Copy link
Member

@yaooqinn yaooqinn commented Mar 3, 2026

What changes were proposed in this pull request?

Map Spark's AssertNotNull expression to Velox's assert_not_null function, enabling native execution for NOT NULL constraint validation during table inserts.

AssertNotNull is used by Spark's TableOutputResolver to enforce NOT NULL column constraints. When a nullable column is inserted into a non-nullable target column, Spark wraps the expression with AssertNotNull to validate at runtime.

The Velox implementation (merged in facebookincubator/velox#16562) validates non-null values at runtime and throws a user error for null inputs.

How was this patch tested?

Existing CI tests. The Velox function has been thoroughly tested with unit tests and fuzzer testing.

@github-actions github-actions bot added the CORE works for Gluten Core label Mar 3, 2026
@github-actions
Copy link

github-actions bot commented Mar 3, 2026

Run Gluten Clickhouse CI on x86

@github-actions
Copy link

github-actions bot commented Mar 3, 2026

Run Gluten Clickhouse CI on x86

@yaooqinn yaooqinn force-pushed the feature/assert-not-null branch from 6552555 to aefddaf Compare March 3, 2026 08:38
@github-actions
Copy link

github-actions bot commented Mar 3, 2026

Run Gluten Clickhouse CI on x86

@jinchengchenghh
Copy link
Contributor

jinchengchenghh commented Mar 5, 2026

Please update, the corresponding PR should have been merged to Gluten Velox branch

@yaooqinn yaooqinn force-pushed the feature/assert-not-null branch from aefddaf to b400747 Compare March 5, 2026 15:40
@github-actions github-actions bot removed the BUILD label Mar 5, 2026
@yaooqinn
Copy link
Member Author

yaooqinn commented Mar 5, 2026

Updated. Rebased onto latest main and reverted to the official Velox branch (IBM/velox dft-2026_03_03-iceberg). The get-velox.sh changes are removed — only the expression mapping and test remain.

@github-actions
Copy link

github-actions bot commented Mar 5, 2026

Run Gluten Clickhouse CI on x86

@yaooqinn yaooqinn force-pushed the feature/assert-not-null branch from b400747 to f90483e Compare March 5, 2026 16:11
@github-actions
Copy link

github-actions bot commented Mar 5, 2026

Run Gluten Clickhouse CI on x86

@yaooqinn yaooqinn force-pushed the feature/assert-not-null branch 3 times, most recently from 749ac80 to f7ff6e4 Compare March 5, 2026 17:37
@github-actions github-actions bot added DATA_LAKE and removed VELOX labels Mar 5, 2026
@github-actions
Copy link

github-actions bot commented Mar 5, 2026

Run Gluten Clickhouse CI on x86

@yaooqinn yaooqinn force-pushed the feature/assert-not-null branch from f7ff6e4 to bd3347c Compare March 6, 2026 03:29
@github-actions
Copy link

github-actions bot commented Mar 6, 2026

Run Gluten Clickhouse CI on x86

@yaooqinn yaooqinn force-pushed the feature/assert-not-null branch from bd3347c to 1a7e9a4 Compare March 6, 2026 09:58
@github-actions
Copy link

github-actions bot commented Mar 6, 2026

Run Gluten Clickhouse CI on x86

1 similar comment
@github-actions
Copy link

github-actions bot commented Mar 6, 2026

Run Gluten Clickhouse CI on x86

@yaooqinn yaooqinn force-pushed the feature/assert-not-null branch from d2ae4c1 to 7b3c2ec Compare March 7, 2026 05:13
@github-actions
Copy link

github-actions bot commented Mar 7, 2026

Run Gluten Clickhouse CI on x86

1 similar comment
@github-actions
Copy link

github-actions bot commented Mar 9, 2026

Run Gluten Clickhouse CI on x86

@yaooqinn yaooqinn force-pushed the feature/assert-not-null branch from a796b5a to 3831650 Compare March 10, 2026 15:07
@github-actions
Copy link

github-actions bot commented Mar 10, 2026

Run Gluten Clickhouse CI on x86

@github-actions
Copy link

github-actions bot commented Mar 11, 2026

Run Gluten Clickhouse CI on x86

baibaichen
baibaichen approved these changes Mar 11, 2026
View reviewed changes
Copy link
Contributor

@baibaichen baibaichen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yaooqinn yaooqinn force-pushed the feature/assert-not-null branch from b79b500 to 643825e Compare March 11, 2026 07:34
@github-actions
Copy link

github-actions bot commented Mar 11, 2026

Run Gluten Clickhouse CI on x86

@yaooqinn yaooqinn force-pushed the feature/assert-not-null branch from 643825e to dc58e35 Compare March 11, 2026 10:18
@github-actions
Copy link

github-actions bot commented Mar 11, 2026

Run Gluten Clickhouse CI on x86

@yaooqinn yaooqinn force-pushed the feature/assert-not-null branch from dc58e35 to c0da9a4 Compare March 11, 2026 13:00
@github-actions
Copy link

github-actions bot commented Mar 11, 2026

Run Gluten Clickhouse CI on x86

@yaooqinn
[VL] Enable assert_not_null expression for Velox backend
9176d36 
Register Spark's AssertNotNull expression mapping for the Velox backend.
The corresponding Velox implementation was merged via
facebookincubator/velox#16562.
@yaooqinn yaooqinn force-pushed the feature/assert-not-null branch from c0da9a4 to 9176d36 Compare March 11, 2026 16:08
@github-actions
Copy link

github-actions bot commented Mar 11, 2026

Run Gluten Clickhouse CI on x86

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@baibaichen baibaichen baibaichen approved these changes

Assignees

No one assigned

Labels

CORE works for Gluten Core DATA_LAKE

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@yaooqinn @jinchengchenghh @baibaichen