Skip to content

Add native prepared Parquet batches - #40

Merged
platypii merged 3 commits into
masterfrom
feat/native-prepared-batches
Aug 18, 2026
Merged

Add native prepared Parquet batches#40
platypii merged 3 commits into
masterfrom
feat/native-prepared-batches

Conversation

@platypii

@platypii platypii commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Summary

  • expose Iceberg field-id schemas and Squirreling 0.16 prepareScan() sources
  • yield one native batch per Parquet row group with independently deferred, selection-aware columns
  • preserve decoded arrays as native vectors, use offset indexes for narrowed ranges, and reuse decoded covering ranges when a field participates in both filtering and output
  • retain scan() and scanColumn() compatibility; snapshots with delete files use the existing delete-aware row scanner through the bounded row-to-batch adapter

End-to-end benchmarks

Measured on Node v26.5.1 with squirreling@0.16.1 against two current local Hypaware Iceberg tables containing 339,953 rows. The seven-query suite is derived from Hypaware overview, retrieval, and top-K workloads and filters to the most recent seven days. Before timing each query, the released baseline, same-engine legacy source, and prepared source were exact-compared. Medians are from three alternating runs with explicit garbage collection.

The prepared source is 1.31× faster geometrically than the same Squirreling 0.16.1 engine using Icebird's legacy row source, with every query faster. It is 1.35× faster geometrically than the released Icebird 0.8.22 / Squirreling 0.15.3 pair.

Query Legacy source Prepared source Source speedup Legacy peak heap Prepared peak heap
Overview date counts 203.34 ms 139.09 ms 1.46× 57.22 MiB 43.37 MiB
Tool calls and sessions 95.82 ms 86.66 ms 1.11× 49.84 MiB 49.50 MiB
Provider/model JSON tokens 804.94 ms 411.57 ms 1.96× 244.98 MiB 65.71 MiB
Daily sessions and JSON tokens 590.32 ms 389.92 ms 1.51× 238.96 MiB 65.38 MiB
Selective tool-result payload 7.38 ms 6.27 ms 1.18× 9.69 MiB 7.45 MiB
Recent attributes preview 6.55 ms 6.30 ms 1.04× 10.57 MiB 8.64 MiB
Largest recent system prompts 385.34 ms 346.80 ms 1.11× 228.85 MiB 172.47 MiB

The provider/model rollup requested the same 77.49 MiB through both source implementations, attributing its 1.96× improvement to native batching rather than reduced I/O. Peak heap measurements are sampled every 5 ms and are directional.

Validation

  • npm test — 59 files, 686 tests passed
  • npm run lint
  • npx tsc
  • npm run build:types — no tracked declaration diff

This depends only on the already-published squirreling@0.16.1 release.

@platypii
platypii merged commit 795ab90 into master Aug 18, 2026
6 checks passed
@platypii
platypii deleted the feat/native-prepared-batches branch August 18, 2026 22:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant