Skip to content

Add SessionContext::read_columns and deprecate DataFrame::from_column… - #26934

Open
cj-zhukov wants to merge 6 commits into
apache:mainfrom
cj-zhukov:cj-zhukov/sessioncontext-read-columns
Open

cj-zhukov wants to merge 6 commits into
apache:mainfrom
cj-zhukov:cj-zhukov/sessioncontext-read-columns

Conversation

@cj-zhukov

@cj-zhukov cj-zhukov commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

DataFrame::from_columns creates its own default SessionContext, so the returned DataFrame cannot see UDFs, tables, configuration or runtime settings registered on the session the caller already has:

let ctx = SessionContext::new();
ctx.register_udf(add_one);

let df = DataFrame::from_columns([("x", x)])?;
df.parse_sql_expr("add_one(x)")?; // Error: Invalid function 'add_one'

Every other way to build a DataFrame from a data source is a read_* method on SessionContext. This PR adds read_columns next to them so a DataFrame built from arrays runs in the caller's session, and deprecates from_columns so there is one way to do this rather than two:

let df = ctx.read_columns([("x", x)])?;
df.parse_sql_expr("add_one(x)")?; // Ok

What changes are included in this PR?

  • New SessionContext::read_columns, implemented on top of read_batch.
  • DataFrame::from_columns is now a #[deprecated] wrapper that calls
    SessionContext::new().read_columns(..). Signature unchanged.
  • The dataframe! macro calls read_columns internally; its API is unchanged.
  • Field nullability is derived from each array via Array::is_nullable
    instead of always true, so columns without nulls get a non-nullable schema.
  • Docs: read_columns example in the DataFrame user guide and an entry in
    the 56.0.0 upgrade guide.

What is the testing strategy for this PR?

Existing from_columns tests in datafusion/core/tests/dataframe/mod.rs are reused for read_columns, plus new tests:

  • test_dataframe_read_columns – types, Vec/array/iterator input, results.
  • test_dataframe_read_columns_empty – empty input is an error.
  • test_dataframe_read_columns_uses_session – the DataFrame shares the
    caller's session id and config, and a UDF registered on the context
    resolves and runs.
  • test_dataframe_read_columns_nullability_follows_arrays – nullability from
    the arrays, including dictionary arrays with logical nulls, and the resulting
    IS NULL simplification.
  • test_dataframe_from_columns_deprecated – the wrapper still works.

The doc examples on read_columns and from_columns are compiled doctests.
test_dataframe_macro covers the macro change.

Are there any user-facing changes?

Yes:

  • New API SessionContext::read_columns.
  • DataFrame::from_columns is deprecated (still works, emits a warning).
  • Schemas from read_columns, from_columns and dataframe! may now contain
    non-nullable fields. Code that compares them against a schema with
    nullable: true everywhere will need updating.

Documented in docs/source/library-user-guide/upgrading/56.0.0.md.

@github-actions github-actions Bot added documentation Improvements or additions to documentation core Core DataFusion crate auto detected api change Auto detected API change labels Oct 30, 2026
@cj-zhukov
cj-zhukov marked this pull request as ready for review October 30, 2026 11:31

@kosiew kosiew left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cj-zhukov, thanks for the update. Moving array construction to SessionContext follows the existing reader API and fixes session binding without breaking the old constructor.

Requesting changes because the existing entry points now produce different schema nullability. Please preserve or explicitly document this compatibility change and add regression coverage.

.into_iter()
.map(|(name, array)| {
(
Field::new(name, array.data_type().clone(), array.is_nullable()),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both existing entry points now infer nullability instead of marking all fields nullable, potentially breaking exact schema comparisons. Please preserve the old behavior or document the change for both APIs, and test their nullability contracts.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this point. I'll keep nullability derived from the arrays and document the change for from_columns and dataframe! and added nullability assertions to both tests.

Comment thread datafusion/core/src/dataframe/mod.rs Outdated

($($name:expr => $data:expr),+ $(,)?) => {{
use datafusion::prelude::DataFrame;
use datafusion::prelude::SessionContext;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The block-scoped SessionContext import can shadow caller types used in macro arguments. Consider using $crate-qualified paths without local imports in both macro arms.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a good point, I agree.

fn test_dataframe_read_columns_empty() {
let ctx = SessionContext::new();
assert!(ctx.read_columns(vec![]).is_err());
assert!(ctx.read_columns([]).is_err());

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider testing columns with mismatched lengths to verify the documented error behavior. RecordBatch::try_new already validates lengths, but the current negative test covers only empty input.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed

///
/// The arrays are wrapped in a single [`RecordBatch`] and read through
/// [`Self::read_batch`], so the returned `DataFrame` runs in this session
/// and sees its registered functions, tables, configuration and runtime,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please clarify that read_batch snapshots SessionState, so later UDF registrations are not visible through the existing DataFrame, while shared catalog and runtime state can reflect later changes.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with you

@github-actions github-actions Bot removed the auto detected api change Auto detected API change label Nov 19, 2026
@cj-zhukov

Copy link
Copy Markdown
Contributor Author

@kosiew thanks for the thorough review! I’ve addressed all the comments: kept array-derived nullability and documented/tested it for both from_columns and dataframe!, switched the macro to $crate paths, clarified the session-state snapshot in the read_columns docs, and added the mismatched-length test. Ready for another look.

@kosiew kosiew left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cj-zhukov, thanks for addressing the earlier feedback. The nullability change is now documented and tested, macro hygiene is fixed, and session behavior and input validation are covered.

The implementation looks good. I have one non-blocking documentation suggestion.


Code that compares the resulting schema against a hand-written one with
`nullable: true` for every field needs to be updated. Query results are
unchanged.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we remove the RecordBatch::try_from_iter equivalence? It uses null_count() > 0, unlike Array::is_nullable(). Also clarify that only the legacy APIs previously marked all fields nullable, and replace the query-results claim with a note that row values remain unchanged but schema inspection can differ.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree, I'll do it

@cj-zhukov

Copy link
Copy Markdown
Contributor Author

@kosiew I removed the try_from_iter comparison, scoped the previous behaviour to from_columns and dataframe!, and reworded to “row values unchanged, schema inspection can differ”.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants