[POC] Add TPC-H rewrite into Hive partitions - #408
Draft
GregoryKimball wants to merge 10 commits into
Draft
Conversation
Generate resumable, key-sorted local or S3 datasets with measured file-size calibration so large scale factors remain bounded and benchmark-ready.
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
Use the executable rewrite module directly to keep a single maintained CLI entry point.
Model partitioned, flat-split, and copied tables explicitly so identity and completion logic reflect each processing path.
Keep the benchmark-critical partitioning and flat-table sizing while removing production lifecycle machinery, using bounded date ranges, one-write calibration, and final-writer codec overrides.
Treat pylibcudf as a required runtime dependency and keep direct operations visible at their call sites.
Prevent S3 bucket-wide deletion and enforce source/staging separation for every destination type.
Allow CPU-only test collection while keeping the GPU dependency scoped to rewrite execution without adding wrapper helpers.
Add bounded work assignments and measured row-level file sizing so large partitioned datasets can be generated concurrently without oversized outputs.
Cover worker assignment validation and measured within-day splitting to prevent regressions in parallel SF-scale rewrites.
Feed sorted date ranges to measured output sizing in file-scale chunks so large dates do not require a second date-sized GPU allocation.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Target file size is 512 MiB per parquet, snappy compressed.
orders: Hive-partitioned monthly byo_orderdateaso_ordermonth=YYYY-MM; sorted within each partition by(o_orderdate, o_orderkey).o_ordermonthis exposed by Hive as a virtual partition column.lineitem: Hive-partitioned monthly byl_shipdateasl_shipmonth=YYYY-MM; sorted within each partition by(l_shipdate, l_orderkey, l_linenumber).customer: Unpartitioned; split into files sorted byc_custkey.part: Unpartitioned; split into files sorted byp_partkey.partsupp: Unpartitioned; split into files sorted by(ps_partkey, ps_suppkey). File cuts align tops_partkeyboundaries to preserve global ordering.supplier: Unpartitioned; split into files sorted bys_suppkey.nation,region: Small tables copied unchanged.