Delta Kernel Draft PR #729

vaibhavk1992 · 2025-06-30T10:42:16Z

Important Read

Please ensure the GitHub issue is mentioned at the beginning of the PR

What is the purpose of the pull request

(For example: This pull request implements the sync for delta format.)

Brief change log

(for example:)

Fixed JSON parsing error when persisting state
Added unit tests for schema evolution

Verify this pull request

(Please pick either of the following options)

This pull request is a trivial rework / code cleanup without any test coverage.

(or)

This pull request is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

Added integration tests for end-to-end.
Added TestConversionController to verify the change.
Manually verified the change by running a job locally.

rahil-c · 2025-07-01T04:16:34Z

pom.xml

        <module>xtable-aws</module>
        <module>xtable-hive-metastore</module>
-        <module>xtable-service</module>
+<!--        <module>xtable-service</module>-->


This should be added back, any reason why you had to comment this out?

yes, xtable-service was giving build failures. As xtable-service is independent of the delta kernel changes. Can we just review the delta kernel changes as of now? I just want to get my changes validated once and anyhow the final version of delta kernel changes would have xtable-service module too.

@vaibhavk1992 I think if you rebase with latest main branch you shouldn't see those failures.

rahil-c · 2025-07-01T06:39:07Z

xtable-core/pom.xml

+        <dependency>
+            <groupId>io.delta</groupId>
+            <artifactId>delta-kernel-api</artifactId>
+            <version>4.0.0</version>


Can you add a properly in the root pom called <delta.kernel.version>4.0.0</delta.kernel.version>, instead of using the hardcoded value?

Also curious how you ended up choosing delta kernel version, is there some specific version that needs to align with delta lake version we have in the repo?

rahil-c · 2025-07-01T06:40:42Z

xtable-core/src/main/java/org/apache/xtable/delta/DeltaKernelConversionSourceProvider.java

+public class DeltaKernelConversionSourceProvider extends ConversionSourceProvider<Long> {
+  @Override
+  public DeltaKernelConversionSource getConversionSourceInstance(SourceTable sourceTable) {
+    Configuration hadoopConf = new Configuration();


Is there any reason why you are creating a new hadoopConf, can you instead use the hadoopConf from the parent class similar to what DeltaConversionSourceProvider does.

xtable-core/src/main/java/org/apache/xtable/delta/DeltaKernelSchemaExtractor.java

xtable-core/src/main/java/org/apache/xtable/delta/DeltaKernelTableExtractor.java

rahil-c · 2025-07-01T06:51:38Z

xtable-core/src/main/java/org/apache/xtable/delta/DeltaSchemaExtractor.java

- * Converts between Delta and InternalTable schemas. Some items to be aware of:
- *
- * <ul>
- *   <li>Delta schemas are represented as Spark StructTypes which do not have enums so the enum


I think we can live this file as is right?

@vaibhavk1992 let's remove the changes to this file. They don't seem necessary

@vaibhavk1992 please follow up on this

xtable-core/src/main/java/org/apache/xtable/kernel/DeltaKernelConversionSource.java

xtable-core/src/main/java/org/apache/xtable/delta/DeltaKernelDataFileExtractor.java

vinishjail97 · 2025-06-30T15:13:35Z

pom.xml

        <module>xtable-aws</module>
        <module>xtable-hive-metastore</module>
-        <module>xtable-service</module>
+<!--        <module>xtable-service</module>-->


why comment this?

xtable-core/src/test/resources/junit-platform.properties

vinishjail97

This is great progress @vaibhavk1992, added some comments.

vinishjail97 · 2025-07-28T14:53:01Z

pom.xml

        <module>xtable-aws</module>
        <module>xtable-hive-metastore</module>
-        <module>xtable-service</module>
+<!--        <module>xtable-service</module>-->


@vaibhavk1992 I think if you rebase with latest main branch you shouldn't see those failures.

pom.xml

xtable-utilities/src/test/resources/my_config.yaml

xtable-core/src/main/java/org/apache/xtable/delta/DeltaKernelActionsConverter.java

…itions,big fixes

xtable-core/src/main/java/org/apache/xtable/kernel/DeltaKernelIncrementalChangesState.java

xtable-core/src/main/java/org/apache/xtable/kernel/DeltaKernelConversionSource.java

the-other-tim-brown · 2025-10-05T13:45:49Z

@vaibhavk1992 can you write up a summary of next steps and blockers for this feature?

vaibhavk1992 · 2025-10-05T18:51:54Z

Below is the summary of the difference between two schemes (Delta vs Kernel) also added the what remains the difference between two.

Comparison of Schema Responses: Delta Kernel vs Delta Log

This document outlines the differences in schema responses when using Delta Kernel and Delta Log APIs to retrieve changes in a Delta table. The comparison highlights the structure and format of the responses, providing insights into how the two approaches differ.

Delta Kernel Schema Response

When using the DeltaKernelIncrementalChangesState class to retrieve changes, the response is in the form of a row of columnar batch type. Each row is represented as an object of the io.delta.kernel.data.Row interface, which provides methods to access individual fields. The response is minimalistic and focuses on the raw data representation.

Sample Output

1 row is an object ==> io.delta.kernel.internal.data.ColumnarBatchRow@20c03e47

Key Characteristics

Row Representation: Each row is an instance of ColumnarBatchRow, which provides methods to access fields like getLong, getString, etc.
Minimal Metadata: The response contains only the essential fields (e.g., version, timestamp, commitInfo).
Raw Data: The schema is not enriched with additional metadata or actions; it is a direct representation of the data in the columnar batch.

Use Case

This format is suitable for low-level data processing where the focus is on performance and accessing raw data.

Delta Log Schema Response

When using the DeltaLog.getChanges method, the response is a tuple containing the version number and a list of actions. The actions include detailed metadata about the changes, such as CommitInfo and AddFile.
Sample Output from delta table changes
(2,
Vector(
CommitInfo(None, 2025-08-15 15:00:46.05, None, None, WRITE, Map(mode -> Append, partitionBy -> []), None, None, None, Some(1), Some(Serializable), Some(true), Some(Map(numFiles -> 1, numOutputRows -> 50, numOutputBytes -> 10226)), None, None, Some(Apache-Spark/3.4.2 Delta-Lake/2.4.0), Some(cf7b1472-4c68-4f89-aa97-c8f16512ecfc)),
AddFile(part-00000-e8eeadc8-4e26-46a7-8c61-0bf60e5e7ada-c000.snappy.parquet, Map(), 10226, 1755250246045, true, {"numRecords":50,"minValues":{"id":51,"firstName":"0WI98","lastName":"08VkW","gender":"Female","birthDate":"2013-02-16T21:18:43.000+05:30","level":"ERROR","date_field":"2025-08-15","timestamp_field":"2025-08-15T15:00:45.884+05:30","double_field":0.018425752795049544,"float_field":0.109567106,"long_field":-8844008067348082419,"record_field":{"nested_int":-2060061976}},"maxValues":{"id":100,"firstName":"xZnER","lastName":"ymLQw","gender":"Male","birthDate":"2023-08-07T15:06:55.000+05:30","level":"WARN","date_field":"2025-08-15","timestamp_field":"2025-08-15T15:00:45.885+05:30","double_field":0.9914942463945434,"float_field":0.9841615,"long_field":8775924211265194460,"record_field":{"nested_int":1923869027}},"nullCount":{"id":0,"firstName":25,"lastName":23,"gender":0,"birthDate":0,"level":0,"boolean_field":28,"date_field":24,"timestamp_field":26,"double_field":25,"float_field":28,"long_field":25,"binary_field":32,"primitive_map":22,"record_map":25,"primitive_list":28,"record_list":29,"record_field":{"nested_int":28}}}, null, null)
))
Key Characteristics

Rich Metadata: The response includes detailed metadata such as CommitInfo (e.g., operation type, timestamp, and user metadata) and AddFile (e.g., file path, size, and statistics).
Structured Actions: Each action is represented as a specific object (e.g., CommitInfo, AddFile), making it easier to interpret the changes.
Verbose Output: The response is more verbose, providing a comprehensive view of the changes.

This issue is currently in blocked state. I raised it with delta team quite a few time but no response over it.
https://delta-users.slack.com/archives/C04TRPG3LHZ/p1758730559515289

the-other-tim-brown · 2025-10-06T00:37:35Z

@vaibhavk1992 I pushed some changes in the latest commit to extract the add and remove files per version.

xtable-core/src/main/java/org/apache/xtable/kernel/DeltaKernelSchemaExtractor.java

xtable-core/src/main/java/org/apache/xtable/kernel/DeltaKernelPartitionExtractor.java

xtable-core/src/main/java/org/apache/xtable/kernel/DeltaKernelStatsExtractor.java

xtable-core/src/main/java/org/apache/xtable/kernel/DeltaKernelActionsConverter.java

the-other-tim-brown · 2025-10-07T14:10:05Z

xtable-core/src/main/java/org/apache/xtable/delta/DeltaSchemaExtractor.java

- * Converts between Delta and InternalTable schemas. Some items to be aware of:
- *
- * <ul>
- *   <li>Delta schemas are represented as Spark StructTypes which do not have enums so the enum


@vaibhavk1992 let's remove the changes to this file. They don't seem necessary

the-other-tim-brown · 2025-10-07T14:10:47Z

xtable-core/src/main/java/org/apache/xtable/kernel/DeltaKernelDataFileExtractor.java

+      this.fields = schema.getFields();
+
+      StructType fullSchema = snapshot.getSchema(); // The full table schema
+      List<String> partitionColumns = snapshot.getPartitionColumnNames(); // List<String>


Remove the comment // List<String>?

xtable-core/src/main/java/org/apache/xtable/kernel/DeltaKernelDataFileExtractor.java

xtable-core/src/main/java/org/apache/xtable/kernel/DeltaKernelConversionSource.java

xtable-core/src/test/java/org/apache/xtable/kernel/ITDeltaKernelConversionSource.java

vaibhavk1992 added 6 commits May 14, 2025 10:43

single commit

dfea6a4

adding delta kernel

b75bc7c

adding the test file

16134b3

adding workable code for iteration over data

3929e95

adding Kernel 4.0 code

c6379b5

adding the working code with xtable that check getcurrenttable

6deb5f7

vaibhavk1992 marked this pull request as draft June 30, 2025 10:42

vaibhavk1992 marked this pull request as ready for review June 30, 2025 15:40

vaibhavk1992 added 2 commits June 30, 2025 21:13

Merge branch 'main' into test-4

05d9984

adding the dependecies

c7ba4b9

rahil-c reviewed Jul 1, 2025

View reviewed changes

xtable-core/src/main/java/org/apache/xtable/delta/DeltaKernelSchemaExtractor.java Outdated Show resolved Hide resolved

rahil-c reviewed Jul 1, 2025

View reviewed changes

xtable-core/src/main/java/org/apache/xtable/delta/DeltaKernelTableExtractor.java Outdated Show resolved Hide resolved

rahil-c reviewed Jul 1, 2025

View reviewed changes

xtable-core/src/main/java/org/apache/xtable/kernel/DeltaKernelConversionSource.java Show resolved Hide resolved

vaibhavk1992 added 4 commits July 19, 2025 22:15

adding getcurrentsnapshot code

0ff36a5

spotless fix

18ab9d6

spotless fix 2

e906091

spotless fix 2

e00241c

the-other-tim-brown reviewed Jul 20, 2025

View reviewed changes

xtable-core/src/main/java/org/apache/xtable/delta/DeltaKernelDataFileExtractor.java Outdated Show resolved Hide resolved

xtable-core/src/main/java/org/apache/xtable/delta/DeltaKernelDataFileExtractor.java Show resolved Hide resolved

fixed partitioned test case

3fdfd31

vinishjail97 reviewed Jul 28, 2025

View reviewed changes

setting junit parallel execution to true

e0102e3

vinishjail97 reviewed Jul 28, 2025

View reviewed changes

vaibhavk1992 added 4 commits August 5, 2025 10:08

testInsertsUpsertsAndDeletes test case addition,internal datatype add…

381722a

…itions,big fixes

added the fix for table basepath listing wrong paths

809bfe8

added the fix for table basepath listing wrong paths

40172f2

adding all tests

e0b7829

vaibhavk1992 added 2 commits August 27, 2025 22:45

adding refactored code

9ac022a

spotless fix

73f33b6

vaibhavk1992 commented Sep 8, 2025

View reviewed changes

xtable-core/src/main/java/org/apache/xtable/kernel/DeltaKernelIncrementalChangesState.java Show resolved Hide resolved

vaibhavk1992 commented Sep 8, 2025

View reviewed changes

xtable-core/src/main/java/org/apache/xtable/kernel/DeltaKernelConversionSource.java Outdated Show resolved Hide resolved

fix change extraction

bee3e8a

the-other-tim-brown reviewed Oct 6, 2025

View reviewed changes

vaibhavk1992 added 2 commits October 7, 2025 18:04

adding the commitbacklog test cases changes

e75bb55

Merge branch 'apache:main' into test-4

21044af

the-other-tim-brown reviewed Oct 7, 2025

View reviewed changes

vaibhavk1992 added 2 commits October 13, 2025 22:43

adding a test case testConvertFromDeltaPartitionFormat

e212f52

adding a test case testConvertFromDeltaPartitionFormat

988cda1

vaibhavk1992 commented Oct 13, 2025

View reviewed changes

xtable-core/src/test/java/org/apache/xtable/kernel/ITDeltaKernelConversionSource.java Show resolved Hide resolved

vaibhavk1992 added 3 commits October 24, 2025 01:19

adding the KernelPartitionExtractor test under kernel

1705ce4

commiting schema extractor and stats extrator

8f81109

adding unit test cases with the request changes on the PR

49ebf21

Delta Kernel Draft PR #729

Are you sure you want to change the base?

Delta Kernel Draft PR #729

Conversation

vaibhavk1992 commented Jun 30, 2025

Important Read

What is the purpose of the pull request

Brief change log

Verify this pull request

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vinishjail97 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

the-other-tim-brown commented Oct 5, 2025

Uh oh!

vaibhavk1992 commented Oct 5, 2025

Comparison of Schema Responses: Delta Kernel vs Delta Log

Delta Kernel Schema Response

Sample Output

Key Characteristics

Use Case

Delta Log Schema Response

Uh oh!

the-other-tim-brown commented Oct 6, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants