This is the official repository for EnerTune at SOSP'26. EnerTune is an inference serving system that reduces energy consumption while meeting performance SLOs. EnerTune introduces analytical models to estimate per-model performance and power, and the power draw of colocated models on shared GPUs, and uses them in an energy-aware bin-packing algorithm to jointly determine model placement and configuration. EnerTune meets performance SLOs while reducing energy consumption by 1.4-2.3× and power draw by 1.3-2.6× over state-of-the-art baselines, including Usher, FGD, GPUlets, and ParvaGPU.
Our contributions include:
- We develop analytical models that capture the impact of GPU allocation size, frequency, and batch size on permodel power. Using these models, our energy-conscious profiling reduces pro!ling time by 7.3× on average compared to prior systems while retaining high accuracy.
- We introduce a novel analytical model that accurately estimates the power draw of spatially multiplexed GPUs from individual model power profiles. Our methodology eliminates tens of hours of joint pro!ling overhead and enables EnerTune to reduce power and energy consumption by 1.3× over prior work.
- We develop a energy-aware bin-packing algorithm and build EnerTune on top of PyTorch. Compared to four state-of-the-art performance-driven baselines, EnerTune reduces energy and power draw by 1.4-2.3× and 1.3-2.6×, respectively, without violating SLOs, across load.
We conduct our experiments on an A100 80GB GPU. CUDA Version is 12.9 and Driver Version is 575.57.08. Ensure CUDA is installed:
python -m pip uninstall -y cuda || true
python -m pip install --no-cache-dir "cuda-python>=12,<13"
Install all python requirements in a python virtual environment:
sudo apt install python3-venv
python3 -m venv <venv-path>
source <venv-path>/bin/activate
pip3 install -r requirements.txt
Compile the C++ per-GPU monitoring daemone
cd profiler
make all
We require SUDO to set MIG slices, MPS, and adjust GPU frequency.
export USE_SUDO=1
Indicate to our scripts where the python virtual environment lives
export VENV=<venv-path>
Run scripts for each baseline to reproduce E2E results. For easier reproducability efforts, we have adapted our scripts to use just a single GPU, running each set of models that would be placed on a GPU one at a time. This reduces the number of GPUs required for reproducability efforts.
Run each system at 5 load levels (25% to 125% load). In systems/, there is a directory per system: EnerTune, FGD, GPULets, ParvaGPU, and Usher.
For each baseline (other than EnerTune), run the following:
cd systems/{baseline} # either fgd, gpulets, parva, usher
cd load-{load_level} # either 25, 50, 75, 100, or 125
load-{load_level}-{baseline}.sh
For EnerTune, there a few different scripts to run to get E2E results depending on the metric we are optimizing for: power, energy, or carbon. For each, we have different scripts to run.
cd systems/ener-tune/{optimization_metric} # power, energy, or carbon
cd load-{load_level} # either 25, 50, 75, 100, or 125
./gpu0.sh
./gpu1.sh
./gpu2.sh
./gpu3.sh
Please note, you should open each bash script and uncomment the job mix you want to get data for. Moreover, view the comment next to each job mix in the mixes() bash array, and udpate the frequency variable freq in the bash script depending on the job mix you are going to run.
If running smoothly, you will see the following
System: {system} | Running mix: {job_mix} | Device: {device_id}
Logs at /tmp/print_outs-{random-id}.txt
Running point experiment for mig with round #0
Made it before while loop for duration arg
Duration is 0
Run success: results are stored in results/a100/{job_mix_path}
Exiting with error_code=0 (0 is clean exit)
Examine /tmp/print_outs-{random-id}.txt for logs
Completed {job_mix} on GPU {device_id} with {frequency} MHz for system {system}.