Problem
Kotlin DataFrame already has a Parquet documentation page and a realistic example project:
docs/StardustDocs/topics/dataSources/Parquet.md
samples/src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/io/Parquet.kt
examples/projects/spark-parquet-dataframe
However, the current docs are closer to an API reference for DataFrame.readParquet(...). We should add or expand a user-facing guide that explains how to read Parquet data in practical workflows, especially when Parquet files are produced by tools such as Apache Spark.
Goal
Create a user guide for reading Parquet files with Kotlin DataFrame, using the existing Spark + Parquet example as the main realistic scenario.
The guide should help users understand:
- when Parquet is useful;
- how to set up the required dependency/module;
- how to read one or multiple Parquet files with
DataFrame.readParquet(...);
- how to read Spark-generated Parquet output, including
part-*.parquet files;
- current limitations and platform constraints.
Suggested Scope
The guide should cover:
-
Setup
- Mention that Parquet reading is provided through the
dataframe-arrow module.
- Clarify that it is included in the general
dataframe artifact and in %use dataframe.
-
Basic reading examples
- Read from a file path/string.
- Read from
Path.
- Read from
File.
- Read from
URL.
- Read multiple Parquet files:
val df = DataFrame.readParquet(file1, file2, file3)
-
Spark-produced Parquet
- Explain that Spark commonly writes a Parquet dataset as a directory containing
part-*.parquet files.
- Show that Kotlin DataFrame currently reads concrete Parquet files, so users may need to pass the actual part files rather than the directory.
- Reuse the existing example project:
examples/projects/spark-parquet-dataframe.
-
Realistic workflow
- Spark reads/prepares data.
- Spark writes Parquet.
- Kotlin DataFrame reads the exported Parquet files.
- Kotlin DataFrame explores/transforms/prints the result.
-
Limitations
- Kotlin DataFrame currently supports reading Parquet, not writing Parquet.
- Android is not supported because Apache Arrow is not supported there.
- Structured/nested Arrow types such as
Struct are not supported yet.
- Predicate pushdown / column pruning are not exposed through
readParquet; users should select / filter after reading.
Acceptance Criteria
- The guide references or reuses validated snippets from
samples/src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/io/Parquet.kt.
- The guide links to the existing Spark example project.
- The guide explains how to handle Spark output directories with
part-*.parquet files.
- The limitations are clearly documented.
- The page is added to the docs navigation if a new guide page is created.
- Code snippets are validated through the existing docs sample mechanism where possible.
Problem
Kotlin DataFrame already has a Parquet documentation page and a realistic example project:
docs/StardustDocs/topics/dataSources/Parquet.mdsamples/src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/io/Parquet.ktexamples/projects/spark-parquet-dataframeHowever, the current docs are closer to an API reference for
DataFrame.readParquet(...). We should add or expand a user-facing guide that explains how to read Parquet data in practical workflows, especially when Parquet files are produced by tools such as Apache Spark.Goal
Create a user guide for reading Parquet files with Kotlin DataFrame, using the existing Spark + Parquet example as the main realistic scenario.
The guide should help users understand:
DataFrame.readParquet(...);part-*.parquetfiles;Suggested Scope
The guide should cover:
Setup
dataframe-arrowmodule.dataframeartifact and in%use dataframe.Basic reading examples
Path.File.URL.Spark-produced Parquet
part-*.parquetfiles.examples/projects/spark-parquet-dataframe.Realistic workflow
Limitations
Structare not supported yet.readParquet; users shouldselect/filterafter reading.Acceptance Criteria
samples/src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/io/Parquet.kt.part-*.parquetfiles.