Skip to content

Make config_namespace! structs #[non_exhaustive] again - #27610

Draft
kamilk-pal wants to merge 1 commit into
apache:mainfrom
kamilk-pal:config-namespace-non-exhaustive
Draft

kamilk-pal wants to merge 1 commit into
apache:mainfrom
kamilk-pal:config-namespace-non-exhaustive

Conversation

@kamilk-pal

@kamilk-pal kamilk-pal commented Nov 20, 2026 •

Copy link
Copy Markdown

Rationale for this change

The structs generated by config_namespace!, such as ExecutionOptions, OptimizerOptions and ParquetOptions, are plain structs with all-pub fields. Downstream code can therefore build or match them exhaustively, so every new config option is technically a breaking change. That includes options backported to release branches: #23296 added execution.enable_file_stream_work_stealing in 54.1.0, and this downstream code compiles against 54.0.0 but not 54.1.0:

let ExecutionOptions { batch_size: _, /* ...every other 54.0.0 field... */ } = options;
// error[E0027]: pattern does not mention field `enable_file_stream_work_stealing`

#[non_exhaustive] makes adding fields non-breaking. The macro used to emit it:

What changes are included in this PR?

config_namespace_with_hashmap! (ParquetColumnOptions, ColumnEncryptionProperties, ColumnDecryptionProperties) never had the attribute and is left unchanged; it could be a follow-up.

One trade-off for reviewers: the exhaustive struct literals in the protobuf conversions acted as a compile-time reminder to map new file-format options to protobuf. With this change, a new option that isn't added to the conversion falls back to its default after a round trip. If reviewers would rather keep that check, the file-format structs could opt out of #[non_exhaustive] while the session options (ExecutionOptions, OptimizerOptions, ...) stay non-exhaustive.

What is the testing strategy for this PR?

  • New doctests on config_namespace!: a compile_fail,E0639 struct literal, and the default-then-assign replacement. Doctests compile as an external crate, so the compile_fail test fails if #[non_exhaustive] is dropped again. I checked this by removing the attribute locally.
  • The converted code is covered by existing tests, including the protobuf round-trip tests in datafusion-proto-common, datafusion-proto-models and datafusion-proto.

Are there any user-facing changes?

Yes, a breaking API change, so this needs the api change label. Outside the defining crate, config_namespace! structs can no longer be built with struct literals, including ..Default::default(), or matched without ... Users start from Default::default() and set fields instead. This also applies to third-party structs defined with config_namespace!, when they are used from another crate.

🤖 Generated with Claude Code

@github-actions github-actions Bot added documentation Improvements or additions to documentation sql SQL Planner core Core DataFusion crate common Related to common crate proto Related to proto crate datasource Changes to the datasource crate labels Nov 20, 2026
Adding a field to a struct generated by `config_namespace!` (such as
`ExecutionOptions`) is a breaking change, because downstream code can
build or match these structs exhaustively. `#[non_exhaustive]` makes
new config options non-breaking.

The macro emitted `#[non_exhaustive]` when it was introduced in apache#4771.
apache#9382 replaced it with `PartialEq` in the derive, in the same change
that added protobuf conversions building the file-format options with
struct literals from another crate.

Convert those conversions, and the other struct literals in the
workspace, to start from `Default::default()` and assign fields. Add
doctests that guard the attribute, and tests that fail when the fields
of the protobuf-converted structs change, since the conversions no
longer get a compile error for an unmapped field.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kamilk-pal
kamilk-pal force-pushed the config-namespace-non-exhaustive branch from e57b8a7 to 70179fd Compare November 20, 2026 11:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

common Related to common crate core Core DataFusion crate datasource Changes to the datasource crate documentation Improvements or additions to documentation proto Related to proto crate sql SQL Planner

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant