DataFusion support for `TimestampWithOffset`

[LiaCastaneda]

Is your feature request related to a problem or challenge?

DataFusion has no support for the arrow.timestamp_with_offset which is an Arrow extension type introduced in arrow-rs 58.0.0 (apache/arrow-rs#8743). This type represents SQL TIMESTAMP WITH TIME ZONE by storing a UTC timestamp alongside a per row offset in minutes as a Struct { timestamp: Timestamp(unit, "UTC"), offset_minutes: Int16 }.

Describe the solution you'd like

Support for this type in the existing date functions: date_part, date_trunc, to_char, to_unixtime

Describe alternatives you've considered

The main concern I have on implementing this is about efficiency. TimestampWithOffset stores a per-row offset, therefore existing datetime functions cannot be applied to the array as a whole, since they require a uniform timezone for the entire array.
One approach is to just loop row by row applying each function individually, which is simple but does not take advantage of arrow's kernels. A more efficient alternative is to group rows by their offset_minutes value (which in theory should be low cardinality), process each group as a sub-array using the existing kernels, then reassemble the results in original row order. However, this grouping approach adds implementation complexity.

Additional context

For substrait queries we will need substrait support to specify the offset and a different timezone to UTC -- substrait-io/substrait#841

[coderfender]

IMHO there is no right way to represent timestamps with various offsets and effectively leverage vector (SIMD) advantages. We could probably go with a hybrid approach and fork implementation based on CARDINALITY_FACTOR kind of parameter (say 16 / 32 etc) and decide between grouping and row-by-row process. Let me raise a draft PR in the following week to provide a POC with some benchmarks

[LiaCastaneda]

Thanks @coderfender! Yeah, having benchmarks on this would be great. My rule of thumb is to always prefer SIMD and avoid going row-by-row, but honestly I'm not sure it would make a huge difference here. Internally we have a few UDFs that work row-by-row just fine since the core operation isn't that expensive, I'm not sure if optimizing these date functions would be micro optimizing or not.

[coderfender]

take

[coderfender]

Ill take a stab at it sometime unless you are @LiaCastaneda ?

[LiaCastaneda]

Yeah, feel free to take this on. I probably won’t have time to work on it myself until next week anyway.