Sqlite types fix by AndreiKingsley · Pull Request #2001 · Kotlin/dataframe

AndreiKingsley · 2026-07-27T18:41:34Z

Fixes #1013.
Fixes #1935.

Significantly improved different types support for SQLite DbType.

Add out-of-box support for Boolean and date-time types. They don't have a their own storage classes inside SQLite, but they have an official specification, so we can detect them from column metadata name and try to extract from possible SQLite storage classes (primitives like Int, Long, String, etc., see full list in code).
Add a special DSL for specifying any other custom SQLite type , by providing converter from storage classes to expected Kotlin types. Column KTypes are infered automatically!

…s and add tests

Copilot

Pull request overview

This PR improves SQLite type handling in the dataframe-jdbc module by introducing built-in conversions for SQLite’s “declared type vs storage class” mismatch (notably booleans and temporal types) and adding a DSL to register per-type / per-column custom converters, addressing issues like #1013/#1935.

Changes:

Added Sqlite.withCustomConverters { ... } DSL for mapping declared SQLite types / columns to Kotlin KTypes and optional value conversion.
Implemented SQLite-specific preprocessing to convert storage classes into idiomatic Kotlin types for BOOLEAN, DATE, TIME, DATETIME, and TIMESTAMP.
Expanded SQLite test coverage to include boolean and temporal column conversions, and updated docs/samples to the new custom-converter approach.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
samples/src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/schemas/DataSchemasTroubleshooting.kt	Updates troubleshooting sample to use the new SQLite custom converter DSL.
docs/StardustDocs/topics/schemas/Data-Schemas-And-Extension-Properties-Troubleshooting.md	Updates documentation section explaining SQLite type affinity and custom converter DSL usage.
dataframe-jdbc/src/test/kotlin/org/jetbrains/kotlinx/dataframe/io/sqliteTest.kt	Adds regression tests for SQLite boolean + date/time/timestamp conversions.
dataframe-jdbc/src/test/kotlin/org/jetbrains/kotlinx/dataframe/io/sqliteCustomTypesTest.kt	Updates and expands tests demonstrating custom converter DSL behavior and precedence rules.
dataframe-jdbc/src/test/kotlin/org/jetbrains/kotlinx/dataframe/io/db/jdbcTypesTest.kt	Updates SQLite type tests to assert nullability behavior of identity `forType<T>(...)` mappings.
dataframe-jdbc/src/main/kotlin/org/jetbrains/kotlinx/dataframe/io/db/Sqlite.kt	Implements new converter DSL, custom mapping precedence, and storage-class→Kotlin conversions for SQLite.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+/**
+ * A user-provided converter from an SQLite declared column type name to
+ * the [DataFrame] column type and a lambda thatн converts each stored value.
+ *


+        // Column-name override wins over type-name override. Column nullability from the
+        // schema is always applied on top of the KType the user declared.


+    private fun conversionError(problem: String, meta: TableColumnMetadata): Nothing =
+        error(
+            "SQLite: $problem from column '${meta.name}' (declared '${meta.sqlTypeName}'). " +
+                "Register a custom converter for this type or column via " +
+                "`Sqlite.withCustomConverters { } to override the built-in mapping.",
+        )


+    private fun julianDayToInstant(julianDay: Double): Instant {
+        val epochSeconds = ((julianDay - JULIAN_DAY_UNIX_EPOCH) * SECONDS_PER_DAY).toLong()
+        return Instant.fromEpochSeconds(epochSeconds)
+    }


        public val default: Sqlite = Sqlite()

-        public fun withCustomTypes(customTypesMap: Map<String, KType>): Sqlite = Sqlite(customTypesMap)
+        /**
+         * Builds a [Sqlite] with custom type converters registered via a [SqliteCustomConvertersBuilder] DSL block.


+import kotlinx.datetime.LocalDateTime
+import kotlinx.datetime.format.DateTimeFormat
 import org.jetbrains.kotlinx.dataframe.DataFrame


+        val customFormat = LocalDateTime.Format { year() }
+
        // SampleStart
-        val sqliteCustom = Sqlite.withCustomTypes(
-            mapOf(
-                "LONGVARCHAR" to typeOf<String>(),
-                "LONGINT" to typeOf<Long>(),
-            ),
-        )
+        val sqliteCustom = Sqlite.withCustomConverters {
+            // SQLite assigns `NUMERIC` affinity to the custom `LONGVARCHAR` type,


+    // Convert values from the "time_stamp" column regardless of its SQL type.
+    // The raw values are stored as strings and parsed into LocalDateTime values;
+    //  the resulting column has LocalDateTime type as well.
+    forColumn("time_stamp") { raw: String ->
+        LocalDateTime.parse(raw, customFormat)
+    }


Jolanrensen · 2026-07-28T13:46:02Z

+        }
+        val target = expectedPreprocessedValueType.classifier
+        @Suppress("UNCHECKED_CAST")
+        return when (target) {


best to stick to KTypes, they're cheaper than KClasses AFAIK

Jolanrensen · 2026-07-28T13:59:08Z

+         * @param R the target Kotlin type for the resulting DataFrame column.
+         */
+        public inline fun <T, reified R> forColumn(columnName: String, crossinline convert: (T) -> R) {
+            val mapping: SqliteCustomTypeConverter<T> = typeOf<R>() to { raw -> convert(raw) }


You only store type R, making getExpectedJdbcType return the wrong type. getExpectedJdbcType should return the type getObject returns. Then, if preprocessing is done, getPreprocessedValueType should return the preprocessed type, aka R here.

Same for forType(""") {}

Jolanrensen · 2026-07-28T13:59:50Z

+    // For DECIMAL/NUMERIC we already resolved the DataFrame type from the storage class in
+    // getExpectedJdbcType, so we keep that as-is. For other types we let the base decide
+    // (base maps TIMESTAMP → Instant, BINARY(UUID) → Uuid, etc.).
+    override fun getPreprocessedValueType(tableColumnMetadata: TableColumnMetadata, expectedJdbcType: KType): KType =


This function should return the type the values have after they are preprocessed. However, it doesn't seem to do anything with the type mappings

Jolanrensen · 2026-07-28T14:08:41Z

+ *     chars(" UTC")
+ * }
+ *
+ * val sqliteCustom = Sqlite.withCustomConverters {


that's a nice looking dsl btw :)

zaleslaw · 2026-07-29T11:10:55Z

@AndreiKingsley please fix Copilot and Jolan review comments
After that I'll do a last round

Jolanrensen · 2026-07-29T11:24:42Z

Seems to have a lot of duplication with AdvancedDbType. Maybe the necessary changes (like converters based on column name) could be added there instead. No need to reinvent the wheel :)

AndreiKingsley added 4 commits July 23, 2026 15:36

Handle SQLite BOOLEAN columns correctly and add tests

bf8ce52

Support idiomatic Kotlin type conversions for SQLite date/time column…

71ee483

…s and add tests

Add custom type and column mappings for SQLite

e4365b2

formatting

84d45d7

AndreiKingsley requested review from Jolanrensen and zaleslaw July 27, 2026 18:41

zaleslaw requested a review from Copilot July 28, 2026 09:57

Copilot started reviewing on behalf of zaleslaw July 28, 2026 09:58 View session

Copilot AI reviewed Jul 28, 2026

View reviewed changes

api dump

36e07e8

Jolanrensen reviewed Jul 28, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Sqlite types fix - #2001

Sqlite types fix#2001
AndreiKingsley wants to merge 5 commits into
masterfrom
sqlite_types_fix

AndreiKingsley commented Jul 27, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Jolanrensen Jul 28, 2026

Uh oh!

Jolanrensen Jul 28, 2026

Uh oh!

Jolanrensen Jul 28, 2026

Uh oh!

Jolanrensen Jul 28, 2026

Uh oh!

Jolanrensen Jul 28, 2026

Uh oh!

zaleslaw commented Jul 29, 2026 •

edited

Loading

Uh oh!

Jolanrensen commented Jul 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		// Column-name override wins over type-name override. Column nullability from the
		// schema is always applied on top of the KType the user declared.

Uh oh!

Conversation

AndreiKingsley commented Jul 27, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Jolanrensen Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

Jolanrensen Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

Jolanrensen Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

Jolanrensen Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

Jolanrensen Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

zaleslaw commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Jolanrensen commented Jul 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

zaleslaw commented Jul 29, 2026 •

edited

Loading