Skip to content

Add direct EC2 orchestration for distributed Presto benchmarks - #403

Closed
GregoryKimball wants to merge 15 commits into
mainfrom
aws-102-ec2-cpu-cluster
Closed

Add direct EC2 orchestration for distributed Presto benchmarks#403
GregoryKimball wants to merge 15 commits into
mainfrom
aws-102-ec2-cpu-cluster

Conversation

@GregoryKimball

Copy link
Copy Markdown

Summary

  • add an SSM-managed EC2 lifecycle for dedicated coordinators and distributed native workers
  • support cache-off cold suites, AsyncDataCache warm suites, telemetry, artifact collection, and controlled CPU tuning
  • support role-specific AMIs and images for mixed x86/ARM fleets

Depends on #376. Until that PR merges, this branch also contains its S3 commits; the final diff should be reviewed after rebasing onto main.

Test plan

  • python3 -m unittest presto.aws.ec2.test_aws_cluster
  • bash -n presto/aws/ec2/remote/configure_and_start.sh presto/aws/ec2/remote/collect_node.sh
  • 8/16/32-worker M7a and M9gd cold SF1000 qualification
  • 8-worker M9gd cache-off, memory-cache, and memory+NVMe qualification
  • 16/32-worker M9gd memory-cache qualification
  • G7e GPU worker qualification at 1/2/3/4 workers

@copy-pr-bot

copy-pr-bot Bot commented Aug 25, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@GregoryKimball

Copy link
Copy Markdown
Author

Closing at the author's request; development will continue on a fork branch before a PR is opened.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants