Skip to content

perf: avoid Arc<str> refcount contention in grouped first_value/last_value ordered by a time zone aware timestamp - #27047

Open
akupchinskiy wants to merge 2 commits into
apache:mainfrom
akupchinskiy:first-last-tz-ordering-contention
Open

akupchinskiy wants to merge 2 commits into
apache:mainfrom
akupchinskiy:first-last-tz-ordering-contention

Conversation

@akupchinskiy

@akupchinskiy akupchinskiy commented Nov 3, 2026 •

Copy link
Copy Markdown

Which issue does this PR close?

Rationale for this change

A grouped first_value(x ORDER BY ts) / last_value(x ORDER BY ts) with ts a
Timestamp(_, Some(tz)) gets slower as target_partitions grows, and so does every
DISTINCT ON with a time zone aware key (it is planned as first_value per column). At 10
partitions it is 3.5-14x slower than the same query ordered by a naive timestamp. I found it with
a benchmark of my own where a DISTINCT ON (session_id, event_name, client_ts) dedup was 25x
slower than DuckDB.

The cause is in FirstLastGroupsAccumulator: it keeps each group's winning ordering values as
Vec<Vec<ScalarValue>>. A time zone aware ScalarValue::Timestamp* holds its time zone as an
Arc<str>, and that Arc comes from the column's DataType, so it is shared by all partitions.
Extracting a winner row, replacing a group's orderings (clear() + extend_from_slice()), and
growing the groups (resize(n, default_orderings.clone())) each clone or drop it, i.e. several
atomic operations on one cache line per updated group, from every thread at once.

The time zone does not take part in the comparison (the values are UTC instants), so the
accumulator does not need it.

What changes are included in this PR?

In FirstLastGroupsAccumulator (datafusion/functions-aggregate/src/first_last.rs):

  • try_new records the time zone of each ordering column (ordering_timezones) and builds
    default_orderings without it.
  • update_batch and merge_batch relabel the time zone aware ordering columns of their input as
    time-zone-free before anything else (PrimitiveArray::with_timezone_opt(None): no copy, no value
    change, unlike a cast). From there on the stored ScalarValues carry None and cloning them is a
    plain copy. When no ordering column has a time zone the input is passed through unchanged
    (Cow::Borrowed).
  • state puts the original time zone back on the ordering columns it emits, so the intermediate
    state schema is unchanged.

Results, the state schema, plans and the non-timestamp paths are unchanged. Size accounting now
reports slightly less per stored ordering value, because a time-zone-free ScalarValue::size()
no longer counts the (shared) time zone string.

Not changed: the non-grouped FirstValueAccumulator / LastValueAccumulator extract their
ordering row once per accumulator and batch rather than once per updated group, so they are much
less exposed; nth_value keeps ScalarValue ordering rows too and may have a milder form of the
same contention; ordered array_agg already stores encoded rows. The general fix, keeping
per-group orderings in the arrow::row format or as typed columns instead of ScalarValues
(which would also remove the remaining per-group ScalarValue work for every ordering type, and
cover time zone aware timestamps nested in structs), is a larger refactor and is left as a
follow-up.

What is the testing strategy for this PR?

  • Unit test test_group_acc_timezone_ordering (first and last): the stored orderings carry no time
    zone, size_of_orderings stays consistent, the state() ordering column has the original
    Timestamp(µs, "+02:00") type also when the state is emitted in parts (EmitTo::First, then
    EmitTo::All), and merging those states with another partial state gives the right values.
  • aggregate.slt: first_value / last_value ordered by a Timestamp(µs, "+05:00") column over a
    UNION ALL of two tables (multi-partition, partial state merged in the final stage), with NULL
    ordering values (NULLS FIRST / NULLS LAST), ties broken by a second key, the time zone aware
    column as the second ordering key, DESC, the time zone aware value itself as the result
    (arrow_typeof keeps the time zone, displayed values are not shifted), DISTINCT ON with a
    time zone aware key, and second, millisecond and nanosecond timestamps with three different time
    zones as ordering keys, with and without IGNORE NULLS.
  • Existing first_last unit tests and the aggregate, first_last_*, distinct_on and
    ordered_aggregate_* sqllogictests pass.

Benchmark (datafusion-cli --release, 2M rows in 1M groups of 2 rows whose order in ts is
scrambled, data generated in SQL; queries in the issue). Median of 5 hot runs after a warm-up, ms,
two interleaved rounds (main, branch, main, branch) shown as round 1 / round 2, Apple M2 Pro,
10 cores, otherwise idle:

query (ordered by Timestamp(µs, "UTC")) target_partitions main this PR
first_value(v ORDER BY ts) GROUP BY k1, k2 1 193 / 194 189 / 194
4 324 / 329 104 / 104
10 491 / 505 97 / 99
last_value(v ORDER BY ts) GROUP BY k1, k2 1 191 / 191 192 / 191
4 337 / 318 106 / 104
10 485 / 502 98 / 96
DISTINCT ON (k1) ... ORDER BY k1, ts, o 1 266 / 267 267 / 266
4 443 / 440 165 / 165
10 558 / 573 157 / 153

With this PR the time zone aware queries run at the speed of the same queries ordered by a naive
timestamp (10 partitions: 97-99 ms for first_value / last_value, 153-161 ms for DISTINCT ON,
on both builds), i.e. about 5x faster for first_value / last_value and 3.6x for DISTINCT ON at
10 partitions and unchanged at 1. A latest-state-per-order query (10M events, 2M orders,
DISTINCT ON (order_id) ... ORDER BY order_id, ingest_ts DESC, default partitions) goes from
6.77-6.92 s to 0.47-0.52 s (about 14x), the same as with a naive ingest_ts (0.50-0.54 s). Query
results are identical between main and the branch (and, for the latest-state query, with DuckDB).

Are there any user-facing changes?

No, only performance.

AI disclosure

I found this while comparing query engines on a benchmark of mine (a DISTINCT ON dedup ran 25x slower than in DuckDB). Claude Code helped write the fix, the tests and this description, and Codex did a review pass. I reviewed the code and verified it myself: unit tests, the sqllogictests, and the benchmarks above.

@github-actions github-actions Bot added sqllogictest SQL Logic Tests (.slt) functions Changes to functions implementation labels Nov 3, 2026
@akupchinskiy
akupchinskiy marked this pull request as ready for review November 3, 2026 09:18

@kosiew kosiew left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@akupchinskiy, thanks for addressing the timezone reference-count contention. The approach preserves ordering semantics and intermediate-state types while removing shared Arc operations from group updates.

Please add the SQL reproducer in #27046 into a runnable benchmark under benchmarks/sql_benchmarks as part of this PR in a commit, then rebase it so that benchmark commit is the first commit in this PR. Then I can run benchmark like this.

In addition I have two non-blocking suggestions.

return Cow::Borrowed(columns);
}
let mut columns = columns.to_vec();
for (column, tz) in columns[1..].iter_mut().zip(&self.ordering_timezones) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we use columns.iter_mut().skip(1) instead of columns[1..].iter_mut()? Empty merge state currently panics instead of returning the existing Empty row in FIRST_VALUE error.

let first = partial.state(EmitTo::First(1))?;
let rest = partial.state(EmitTo::All)?;
assert_eq!(first[1].data_type(), &ts_type);
assert_eq!(rest[1].data_type(), &ts_type);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we assert the emitted timestamp ticks in both state chunks? Expected values are [10] and [5] for FIRST_VALUE, and [30] and [15] for LAST_VALUE. This verifies timezone restoration preserves timestamp values.

akupchinskiy and others added 2 commits November 20, 2026 16:06
… time zone aware timestamp

Adds the reproducer from apache#27046 as a data-free suite under
benchmarks/sql_benchmarks: grouped first_value (Q01) and DISTINCT ON (Q03)
ordered by a Timestamp(Microsecond, "UTC") column, and the same queries over
the same instants without a time zone as controls (Q02, Q04). Both orderings
have a second key, so they run on FirstLastGroupsAccumulator; a single
primitive ORDER BY key takes the specialized accumulator from apache#27303.

Run with `./bench.sh run first_last_tz` or
`cargo run --release --bin benchmark_runner -- first_last_tz --partitions 12`.
…value ordered by a time zone aware timestamp

FirstLastGroupsAccumulator stores each group's winning ordering values as
ScalarValues. A time zone aware ScalarValue::Timestamp* holds the time zone
as an Arc<str> shared by all partitions, so cloning and dropping it per
updated group made the threads contend on its reference count: the query
got slower with more partitions (5x slower than a naive timestamp at 10).

The time zone does not take part in the comparison, so strip it from the
ordering columns on input (a relabel, not a cast) and put it back on the
arrays returned by state().

Closes apache#27046.
@akupchinskiy
akupchinskiy force-pushed the first-last-tz-ordering-contention branch from 0ede219 to 97d4414 Compare November 20, 2026 12:13
@akupchinskiy

Copy link
Copy Markdown
Author

Thanks for the review! I rebased onto main and added the reproducer as a new SQL suite, first_last_tz, as the first commit (a07bd3e), so it gives a benchmark-only baseline.

Since #27303, the original single-key ORDER BY ts already takes the new primitive-key accumulator, so the suite orders by ts, o and uses DISTINCT ON, which still go through FirstLastGroupsAccumulator, with matching controls over a naive timestamp. Locally at 10 partitions Q01/Q03 go from 956/942 ms to 49/68 ms; the controls are 54/66 ms vs 50/67 ms. At 1 partition there is no material change.

Both suggestions are applied: an empty merge state now returns the Empty row in FIRST_VALUE error instead of panicking (confirmed it panicked before), and the test asserts the emitted timestamp ticks in both state chunks for FIRST_VALUE and LAST_VALUE.

The suite isn't in bench.sh on main yet, so it should run through the Criterion sql target:

run benchmark sql
```yaml
env:
  BENCH_NAME: "first_last_tz"
  BENCH_FILTER: "first_last_tz"
baseline:
  ref: "a07bd3e430588842f4c602f6b9a80b2a9509ee0c"
```

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

functions Changes to functions implementation sqllogictest SQL Logic Tests (.slt)

Projects

None yet

2 participants