[POC] Add EC2 orchestration for Presto benchmarks - #404
Draft
GregoryKimball wants to merge 12 commits into
Draft
Conversation
Provide an SSM-managed cluster harness so distributed CPU baselines, cache studies, and controlled tuning runs can share one archived configuration and lifecycle.
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
Reuse the existing GPU config and launch path so G7e fleets get NVIDIA preflight checks, pinned Kvikio settings, GPU telemetry, and the same safe lifecycle as CPU fleets.
Keep the reusable branch independent of local benchmark series by using generic examples, tags, run IDs, and timing terminology.
Avoid replacing the DLAMI Docker installation so its NVIDIA runtime integration remains intact during worker bootstrap.
Capture protocol logs, socket state, and exchange counters so distributed GPU qualification proves that cuDF exchange transferred data instead of merely accepting its configuration.
Keep GPU-native connector properties out of the Java coordinator catalog so mixed-role fleets start with strict configuration validation enabled.
Make GPU input and batching behavior reproducible while supporting DLAMI NVMe layouts, instance-role S3 access, and valid single-worker execution.
Keep benchmark repetition semantics aligned with velox-testing by using query-major --iterations for both cache configurations.
Expose reproducible UCX controls, prove active peer traffic, and tolerate package-manager startup races found during multi-node qualification.
Run one pinned worker per local GPU while preserving distinct ports, cache paths, and artifacts, and allow GPU fleets to attach EFA devices for inter-node exchange testing.
Configure Prestissimo's native spill gate and worker address so memory-heavy queries can use instance-store NVMe independently of async cache.
Count unique worker endpoints so stale node records left by service restarts do not inflate reported cluster size.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Provide an SSM-managed cluster harness so distributed CPU baselines, cache studies, and controlled tuning runs can share one archived configuration and lifecycle.