Repository navigation
Conversation
kosiew
left a comment
There was a problem hiding this comment.
@cj-zhukov, thanks for the update. Moving array construction to SessionContext follows the existing reader API and fixes session binding without breaking the old constructor.
Requesting changes because the existing entry points now produce different schema nullability. Please preserve or explicitly document this compatibility change and add regression coverage.
| .into_iter() | ||
| .map(|(name, array)| { | ||
| ( | ||
| Field::new(name, array.data_type().clone(), array.is_nullable()), |
There was a problem hiding this comment.
Both existing entry points now infer nullability instead of marking all fields nullable, potentially breaking exact schema comparisons. Please preserve the old behavior or document the change for both APIs, and test their nullability contracts.
There was a problem hiding this comment.
Thanks for this point. I'll keep nullability derived from the arrays and document the change for from_columns and dataframe! and added nullability assertions to both tests.
|
|
||
| ($($name:expr => $data:expr),+ $(,)?) => {{ | ||
| use datafusion::prelude::DataFrame; | ||
| use datafusion::prelude::SessionContext; |
There was a problem hiding this comment.
The block-scoped SessionContext import can shadow caller types used in macro arguments. Consider using $crate-qualified paths without local imports in both macro arms.
There was a problem hiding this comment.
This is a good point, I agree.
| fn test_dataframe_read_columns_empty() { | ||
| let ctx = SessionContext::new(); | ||
| assert!(ctx.read_columns(vec![]).is_err()); | ||
| assert!(ctx.read_columns([]).is_err()); |
There was a problem hiding this comment.
Consider testing columns with mismatched lengths to verify the documented error behavior. RecordBatch::try_new already validates lengths, but the current negative test covers only empty input.
| /// | ||
| /// The arrays are wrapped in a single [`RecordBatch`] and read through | ||
| /// [`Self::read_batch`], so the returned `DataFrame` runs in this session | ||
| /// and sees its registered functions, tables, configuration and runtime, |
There was a problem hiding this comment.
Please clarify that read_batch snapshots SessionState, so later UDF registrations are not visible through the existing DataFrame, while shared catalog and runtime state can reflect later changes.
|
@kosiew thanks for the thorough review! I’ve addressed all the comments: kept array-derived nullability and documented/tested it for both |
kosiew
left a comment
There was a problem hiding this comment.
@cj-zhukov, thanks for addressing the earlier feedback. The nullability change is now documented and tested, macro hygiene is fixed, and session behavior and input validation are covered.
The implementation looks good. I have one non-blocking documentation suggestion.
|
|
||
| Code that compares the resulting schema against a hand-written one with | ||
| `nullable: true` for every field needs to be updated. Query results are | ||
| unchanged. |
There was a problem hiding this comment.
Could we remove the RecordBatch::try_from_iter equivalence? It uses null_count() > 0, unlike Array::is_nullable(). Also clarify that only the legacy APIs previously marked all fields nullable, and replace the query-results claim with a note that row values remain unchanged but schema inspection can differ.
There was a problem hiding this comment.
I agree, I'll do it
|
@kosiew I removed the |
Which issue does this PR close?
Rationale for this change
DataFrame::from_columnscreates its own defaultSessionContext, so the returnedDataFramecannot see UDFs, tables, configuration or runtime settings registered on the session the caller already has:Every other way to build a
DataFramefrom a data source is aread_*method onSessionContext. This PR addsread_columnsnext to them so aDataFramebuilt from arrays runs in the caller's session, and deprecatesfrom_columnsso there is one way to do this rather than two:What changes are included in this PR?
SessionContext::read_columns, implemented on top ofread_batch.DataFrame::from_columnsis now a#[deprecated]wrapper that callsSessionContext::new().read_columns(..). Signature unchanged.dataframe!macro callsread_columnsinternally; its API is unchanged.Array::is_nullableinstead of always
true, so columns without nulls get a non-nullable schema.read_columnsexample in the DataFrame user guide and an entry inthe 56.0.0 upgrade guide.
What is the testing strategy for this PR?
Existing
from_columnstests indatafusion/core/tests/dataframe/mod.rsare reused forread_columns, plus new tests:test_dataframe_read_columns– types,Vec/array/iterator input, results.test_dataframe_read_columns_empty– empty input is an error.test_dataframe_read_columns_uses_session– theDataFrameshares thecaller's session id and config, and a UDF registered on the context
resolves and runs.
test_dataframe_read_columns_nullability_follows_arrays– nullability fromthe arrays, including dictionary arrays with logical nulls, and the resulting
IS NULLsimplification.test_dataframe_from_columns_deprecated– the wrapper still works.The doc examples on
read_columnsandfrom_columnsare compiled doctests.test_dataframe_macrocovers the macro change.Are there any user-facing changes?
Yes:
SessionContext::read_columns.DataFrame::from_columnsis deprecated (still works, emits a warning).read_columns,from_columnsanddataframe!may now containnon-nullable fields. Code that compares them against a schema with
nullable: trueeverywhere will need updating.Documented in
docs/source/library-user-guide/upgrading/56.0.0.md.