[GLUTEN-11683][VL] Add Parquet type widening support#11719
Draft
baibaichen wants to merge 6 commits intoapache:mainfrom
Draft
[GLUTEN-11683][VL] Add Parquet type widening support#11719baibaichen wants to merge 6 commits intoapache:mainfrom
baibaichen wants to merge 6 commits intoapache:mainfrom
Conversation
|
Run Gluten Clickhouse CI on x86 |
1 similar comment
|
Run Gluten Clickhouse CI on x86 |
baibaichen
force-pushed
the
pr3/parquet-type-widening
branch
3 times, most recently
from
abbc057 to
c2d50e1
Compare
…olumns When Gluten creates HiveTableHandle, it was passing all columns (including partition columns) as dataColumns. This caused Velox's convertType() to validate partition column types against the Parquet file's physical types, failing when they differ (e.g., LongType in file vs IntegerType from partition inference). Fix: build dataColumns excluding partition columns (ColumnType::kPartitionKey). Partition column values come from the partition path, not from the file. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
With OAP INT narrowing commit replaced by upstream Velox PR #15173: - Remove 2 excludes now passing: LongType->IntegerType, LongType->DateType - Add 2 excludes for new failures: IntegerType->ShortType (OAP removed) Exclude 63 (net unchanged: -2 +2). Test results: 21 pass / 63 ignored. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
With Velox PR3 type widening (INT->Decimal, INT->Double, Float->Double): - Remove 15 excludes for widening tests now passing Remaining 48 excludes. Test results: 36 pass / 48 ignored. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This suite tests the READ path only. Disable native writer so Spark's writer produces correct V2 encodings (DELTA_BINARY_PACKED/DELTA_BYTE_ARRAY). - Remove 10 excludes for decimal widening tests now passing Remaining 38 excludes: - 34: Velox native reader rejects incompatible decimal conversions regardless of reader config (no parquet-mr fallback) - 4: Velox does not support DELTA_BYTE_ARRAY encoding Test results: 46 pass / 38 ignored. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
baibaichen
force-pushed
the
pr3/parquet-type-widening
branch
from
c2d50e1 to
f926c56
Compare
Velox native reader always behaves like Spark's vectorized reader, so tests that rely on parquet-mr behavior (vectorized=false) fail. Instead of just excluding these 33 tests, add testGluten overrides with expectError=true to verify Velox correctly rejects incompatible conversions. - 16 unsupported INT->Decimal conversions - 6 decimal precision narrowing cases - 11 decimal precision+scale narrowing/mixed cases VeloxTestSettings: 38 excludes (parent tests) + 33 testGluten overrides Test results: 79 pass / 38 ignored (33 excluded parent + 5 truly excluded)
baibaichen
force-pushed
the
pr3/parquet-type-widening
branch
from
f926c56 to
b376d6d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Fix SPARK-18108, parquet-thrift compatibility, and add Parquet type widening support to Velox. Enables 79 of 84 tests in
GlutenParquetTypeWideningSuite.Changes
Fix SPARK-18108 (
SubstraitToVeloxPlan.cc):Exclude partition columns from
HiveTableHandle.dataColumns()to prevent type validation failures when partition column types differ from file column types.Point Velox to type widening branch (
get-velox.sh):Use
baibaichen/pr3/parquet-type-wideningVelox branch which includes:Update VeloxTestSettings (
spark40 + spark41):Remove excludes for widening tests now passing with the updated Velox branch.
Disable native writer (
GlutenParquetTypeWideningSuite.scala):This suite tests the READ path only. Disable native writer so Spark's writer produces correct V2 encodings (DELTA_BINARY_PACKED/DELTA_BYTE_ARRAY).
Override 33 tests with testGluten (
GlutenParquetTypeWideningSuite.scalaspark40/41):Velox native reader always behaves like Spark's vectorized reader (no parquet-mr fallback). The parent tests use
withAllParquetReaderswhich includesvectorized=falsewhere parquet-mr allows conversions the vectorized reader rejects. We override 33 parent tests withtestGlutenandexpectError=trueto verify Velox correctly rejects these incompatible conversions:The copied private methods (
checkAllParquetReaders,readParquetFiles,writeParquetFiles, etc.) mirror the parent's structure but remove thewithAllParquetReaderswrapper.Test Results
38 ignored = 33 parent tests excluded (overridden by testGluten) + 5 truly excluded:
Additionally fixed (not in TypeWideningSuite):
Fixes #11683
How was this patch tested?
Local tests: TypeWideningSuite 79 pass / 38 ignored (spark40 and spark41).
Was this patch authored or co-authored using generative AI tooling?
Yes, co-authored with GitHub Copilot.