Repository navigation
Make config_namespace! structs #[non_exhaustive] again - #27610
Draft
kamilk-pal wants to merge 1 commit into
Draft
kamilk-pal wants to merge 1 commit into
kamilk-pal wants to merge 1 commit into
Conversation
Adding a field to a struct generated by `config_namespace!` (such as `ExecutionOptions`) is a breaking change, because downstream code can build or match these structs exhaustively. `#[non_exhaustive]` makes new config options non-breaking. The macro emitted `#[non_exhaustive]` when it was introduced in apache#4771. apache#9382 replaced it with `PartialEq` in the derive, in the same change that added protobuf conversions building the file-format options with struct literals from another crate. Convert those conversions, and the other struct literals in the workspace, to start from `Default::default()` and assign fields. Add doctests that guard the attribute, and tests that fail when the fields of the protobuf-converted structs change, since the conversions no longer get a compile error for an unmapped field. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kamilk-pal
force-pushed
the
config-namespace-non-exhaustive
branch
from
November 20, 2026 11:28
e57b8a7 to
70179fd
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
The structs generated by
config_namespace!, such asExecutionOptions,OptimizerOptionsandParquetOptions, are plain structs with all-pubfields. Downstream code can therefore build or match them exhaustively, so every new config option is technically a breaking change. That includes options backported to release branches: #23296 addedexecution.enable_file_stream_work_stealingin 54.1.0, and this downstream code compiles against 54.0.0 but not 54.1.0:#[non_exhaustive]makes adding fields non-breaking. The macro used to emit it:config_namespace!with#[non_exhaustive]on the generated struct.PartialEqin the derive. The same PR added protobuf conversions that buildCsvOptions,ParquetOptionsandJsonOptionswith struct literals from another crate, which#[non_exhaustive]forbids. The macro's doc comment still shows#[non_exhaustive], but the generated struct doesn't have it.What changes are included in this PR?
config_namespace!emits#[non_exhaustive]again. Its doc example now matches the generated derive, and new doctests cover the attribute.Default::default()and assign fields:datafusion-proto-common:CsvOptions,ParquetOptions,ParquetCdcOptions,JsonOptionsdatafusion-proto-models:CsvOptions,JsonOptions,ParquetCdcOptions,ParquetOptionsDFParserindatafusion-sql, theCsvSourcedoc example and one example.config_namespace_with_hashmap!(ParquetColumnOptions,ColumnEncryptionProperties,ColumnDecryptionProperties) never had the attribute and is left unchanged; it could be a follow-up.One trade-off for reviewers: the exhaustive struct literals in the protobuf conversions acted as a compile-time reminder to map new file-format options to protobuf. With this change, a new option that isn't added to the conversion falls back to its default after a round trip. If reviewers would rather keep that check, the file-format structs could opt out of
#[non_exhaustive]while the session options (ExecutionOptions,OptimizerOptions, ...) stay non-exhaustive.What is the testing strategy for this PR?
config_namespace!: acompile_fail,E0639struct literal, and the default-then-assign replacement. Doctests compile as an external crate, so thecompile_failtest fails if#[non_exhaustive]is dropped again. I checked this by removing the attribute locally.datafusion-proto-common,datafusion-proto-modelsanddatafusion-proto.Are there any user-facing changes?
Yes, a breaking API change, so this needs the
api changelabel. Outside the defining crate,config_namespace!structs can no longer be built with struct literals, including..Default::default(), or matched without... Users start fromDefault::default()and set fields instead. This also applies to third-party structs defined withconfig_namespace!, when they are used from another crate.🤖 Generated with Claude Code