Draft: Add dual-path Parquet level-prepass rollout

[vyasr]

Summary

Rebuilds the Parquet level-prepass experiment as a linear, opt-in dual-path series.

• Adds the internal LIBCUDF_PARQUET_LEVEL_PREPASS selector, snapshotted per read.
• Keeps legacy and prepass consumers separately compiled and selected host-side.
• Migrates generic, GENERAL/BSS, and delta decoders across flat, nested, and list pages.
• Enables the prepass by default (0x1ff) while retaining 0 as a rollback mode.
• Includes the nested page-local rank correction.

Validation

• PARQUET_TEST passed in default-prepass and legacy-selector modes (544 passing; one expected skip).
• Targeted nested specialized-path test passed under compute-sanitizer memcheck.
• Matched Parquet NVBench coverage was collected.
• PDS-H SF1000 Q1-Q22 was run in both modes. Q21 is allocation-limited in both modes and is excluded from timing comparisons.

Follow-up

This draft intentionally retains the selector and legacy paths. Their removal is a later PR after default-on has landed and completed a release-cycle soak.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[vyasr]

By default on this branch the environment variable has the value LIBCUDF_PARQUET_LEVEL_PREPASS=0x1ff. If you export LIBCUDF_PARQUET_LEVEL_PREPASS=0 you will recover the legacy kernels. We should be able to compare those for performance on any benchmark/system we need.

On a V100 the TPC benchmarks are somewhat noisy but here's what I get at SF1k

Query	Prepass n	Legacy n	Prepass mean (s)	Legacy mean (s)	Mean delta	Prepass median (s)	Legacy median (s)	Median delta	Result / qualification
Q1	5	5	14.6356	15.9567	-8.28%	14.6721	15.9109	-7.79%	Faster
Q2	5	5	1.2778	1.5825	-19.26%	1.2639	1.7242	-26.70%	Faster; legacy variable
Q3	5	5	46.8743	50.5278	-7.23%	46.8930	50.4846	-7.11%	Faster
Q4	5	5	13.9694	10.3703	+34.71%	10.8592	10.1575	+6.91%	Prepass slower; large prepass outliers
Q5	5	5	40.2255	43.9078	-8.39%	40.2377	43.7161	-7.96%	Faster
Q6	5	5	9.4212	9.6089	-1.95%	9.5120	9.6019	-0.94%	Near parity
Q7	5	5	395.6586	335.6325	+17.88%	312.7653	250.0035	+25.10%	High variance; report separately
Q8	5	5	40.3593	35.5610	+13.49%	40.4379	35.4423	+14.10%	Prepass slower
Q9	5	5	119.1780	119.5558	-0.32%	119.3049	119.2502	+0.05%	Parity
Q10	5	5	27.3076	27.0293	+1.03%	28.2160	27.5640	+2.37%	Slight regression; noisy samples
Q11	5	5	1.8245	1.8219	+0.14%	1.8373	1.8200	+0.95%	Parity
Q12	5	5	8.4302	8.2945	+1.64%	8.2617	8.1905	+0.87%	Parity
Q13	5	4	13.9601	13.8574	+0.74%	14.0754	13.8639	+1.53%	Incomplete legacy sample; do not gate
Q14	5	5	24.4790	24.4747	+0.02%	24.4190	24.5804	-0.66%	Parity
Q15	5	5	10.9899	11.0991	-0.98%	10.9916	11.0697	-0.71%	Parity
Q16	5	5	8.6639	8.6447	+0.22%	8.6750	8.6604	+0.17%	Parity
Q17	5	5	11.2132	11.2998	-0.77%	11.2205	11.2968	-0.68%	Parity
Q18	5	5	608.1900	580.5300	+4.76%	603.8800	587.5700	+2.78%	High variance; near parity
Q19	5	5	17.1650	17.1380	+0.16%	17.1990	17.1050	+0.54%	Parity
Q20	5	5	17.6060	17.7060	-0.56%	17.5840	17.7880	-1.15%	Parity; slight prepass advantage
Q21	0	0	n/a	n/a	n/a	n/a	n/a	n/a	Allocation-limited in both modes; not timing evidence
Q22	5	5	3.5520	3.8740	-8.30%	3.5270	3.5590	-0.92%	Mean advantage; conditional-join fallback, use median

I don't trust the huge performance differences, but I'm planning to run on a B200 next for data on more modern systems.

[vyasr]

Here are the results of V100 libcudf microbenchmarks on this branch. I'll see if I can find any further room for optimization.

Benchmark / case	Prepass vs legacy
parquet_read_decode, integral	+19.61% slower
parquet_read_decode, string	+7.70% slower
parquet_read_decode, list	+6.46% slower
parquet_read_delta_binary, flat	+10.08% slower
parquet_read_delta_binary, nested	−30.25% faster
parquet_read_delta_string, nested delta-length	−6.42% faster
parquet_read_delta_string, nested delta-byte-array	−5.87% faster