Repository navigation
Cherry-pick apache/datafusion#27547: preserve date_trunc ordering for timezone-aware timestamps at minute and finer granularities - #94
Merged
xudong963 merged 1 commit intoNov 19, 2026
Conversation
…inute and finer granularities For minute and finer granularities date_trunc rounds the epoch value down without consulting the timezone, so it cannot reorder timestamps across a timezone transition. Propagate the input ordering in that case, using the predicate that selects the arithmetic kernel. (cherry picked from commit 5a30a39)
zhuqi-lucas
approved these changes
Nov 19, 2026
xudong963
deleted the
xudong963/cherry-pick-27547-date-trunc-ordering
branch
November 19, 2026 02:44
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
date_truncordering for timezone-aware timestamps at minute and finer granularities apache/datafusion#27547 (perf: preserve date_trunc ordering for timezone-aware timestamps at minute and finer granularities, part of Restore date_trunc ordering propagation for safe timezone-aware timestamps apache/datafusion#25464) tobranch-56. The upstream PR is still open, so this copy may need to be updated if review changes it.America/New_York). The aggregate only streams perdate, so a full day of(ticker, bucket)groups sits in the hash table until the date changes.Rationale for this change
apache#25409 stopped
date_truncfrom propagating input ordering for every timezone-aware timestamp, because truncating in local time can go backwards across a timezone transition. Forminuteand finer granularities,date_truncnever consults the timezone. It rounds the epoch value down, so it cannot reorder values.What changes are included in this PR?
git cherry-pick -x 5a30a391applied cleanly with no adaptations.DateTruncFunc::output_orderingnow keeps the input ordering for a timezone-aware timestamp when the precision is a known constant andtruncates_in_input_unitholds, which is the same predicate that selects the arithmetic kernel.hourand coarser granularities in a timezone stay unordered.For Atlas, the 22 minute/second aggregate MVs (crypto, forex, indices, stocks sip/otc, futures) should get the time-bucket key back. The 10 day-granularity ones stay as they are; their leading
datekey already provides most of the streaming.Are these changes tested?
cargo test -p datafusion-functions --lib date_trunc: 16 passed, including the newoutput_ordering_with_timezone_depends_on_granularity.timestamps.sltcases pass, along with every other.sltfile that mentionsdate_trunc(14 files:group_by,group,order,sort_pushdown,range_sorted_time_bin_agg,information_schema,clickbench,datetime/*,pg_compat_timestamptz,spark/datetime/*).cargo clippy -p datafusion-functions --all-targets --all-features -- -D warningsandcargo fmt --checkare clean.Are there any user-facing changes?
Queries that sort or group timezone-aware timestamps by
date_truncatminuteor finer granularity, over input ordered on the timestamp, no longer add a sort and use clustered aggregation. Results are unchanged.