Skip to content

Add a User Guide for the Parquet file reading #1452

Description

@zaleslaw

Problem

Kotlin DataFrame already has a Parquet documentation page and a realistic example project:

  • docs/StardustDocs/topics/dataSources/Parquet.md
  • samples/src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/io/Parquet.kt
  • examples/projects/spark-parquet-dataframe

However, the current docs are closer to an API reference for DataFrame.readParquet(...). We should add or expand a user-facing guide that explains how to read Parquet data in practical workflows, especially when Parquet files are produced by tools such as Apache Spark.

Goal

Create a user guide for reading Parquet files with Kotlin DataFrame, using the existing Spark + Parquet example as the main realistic scenario.

The guide should help users understand:

  • when Parquet is useful;
  • how to set up the required dependency/module;
  • how to read one or multiple Parquet files with DataFrame.readParquet(...);
  • how to read Spark-generated Parquet output, including part-*.parquet files;
  • current limitations and platform constraints.

Suggested Scope

The guide should cover:

  1. Setup

    • Mention that Parquet reading is provided through the dataframe-arrow module.
    • Clarify that it is included in the general dataframe artifact and in %use dataframe.
  2. Basic reading examples

    • Read from a file path/string.
    • Read from Path.
    • Read from File.
    • Read from URL.
    • Read multiple Parquet files:
      val df = DataFrame.readParquet(file1, file2, file3)
  3. Spark-produced Parquet

    • Explain that Spark commonly writes a Parquet dataset as a directory containing part-*.parquet files.
    • Show that Kotlin DataFrame currently reads concrete Parquet files, so users may need to pass the actual part files rather than the directory.
    • Reuse the existing example project:
      examples/projects/spark-parquet-dataframe.
  4. Realistic workflow

    • Spark reads/prepares data.
    • Spark writes Parquet.
    • Kotlin DataFrame reads the exported Parquet files.
    • Kotlin DataFrame explores/transforms/prints the result.
  5. Limitations

    • Kotlin DataFrame currently supports reading Parquet, not writing Parquet.
    • Android is not supported because Apache Arrow is not supported there.
    • Structured/nested Arrow types such as Struct are not supported yet.
    • Predicate pushdown / column pruning are not exposed through readParquet; users should select / filter after reading.

Acceptance Criteria

  • The guide references or reuses validated snippets from samples/src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/io/Parquet.kt.
  • The guide links to the existing Spark example project.
  • The guide explains how to handle Spark output directories with part-*.parquet files.
  • The limitations are clearly documented.
  • The page is added to the docs navigation if a new guide page is created.
  • Code snippets are validated through the existing docs sample mechanism where possible.

Metadata

Metadata

Assignees

Labels

documentationImprovements or additions to documentation (not KDocs)

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions