AMD Agentic AI Profiling Workshop

Profile an autonomous AI agent, find its bottleneck, and optimize it on AMD Instinct™ GPUs.

---

Overview

This is a hands-on, beginner-friendly workshop on observability-driven optimization of AI agents. You run a real Hermes Agent, capture its telemetry (execution traces in MLflow and CPU/GPU metrics in Grafana otel-lgtm), read a purpose-built dashboard to spot the slowest step, optimize that one tool, and prove the speed-up with hardware metrics from an AMD Instinct™ MI300X GPU.

Text-to-speech (TTS) is only the example. The real subject is a repeatable profiling loop you can point at any agent task.

What you will learn

• Run a Hermes agent and capture its telemetry end to end
• Read a telemetry dashboard to identify a slow tool
• Swap in an optimized implementation of that tool
• Compare before and after, and see how GPU utilization changes

The workflow you will follow

Run the agent, Fetch the run in the dashboard, inspect the spans and GPU usage, optimize the slow tool, and measure again. Repeat until the bottleneck is gone.

The optimization at a glance

The default TTS uses Edge TTS, which has some limitations. We therefore use a local TTS model. However, it processes one sentence at a time, leaving the MI300X mostly idle. The workshop introduces a batched mode that processes multiple sentences together in a single GPU pass, improving GPU utilization and performance.

---

Repository contents

Path	What it is
agentic_ai_profiling.ipynb	The workshop notebook. Start here.
agentic_ai_profiling_executed.ipynb	The same notebook with all cells already executed, so you can read the expected outputs without a GPU.
utils/helper.sh	One-shot launcher for the full backend (agent, telemetry, dashboard).
utils/kokoro_server.py	The local Kokoro TTS server (FastAPI + Uvicorn), including the batched inference path.
utils/start_kokoro_server.sh	Launches the Kokoro TTS server and waits until it answers /health. The notebook triggers it for the local Kokoro runs the optimization builds on.
utils/hermes_profiler.py	The Streamlit telemetry dashboard.
utils/requirements.txt	Python dependencies for the notebook and dashboard.
utils/clear_cache.sh	Clears the GPU kernel cache for cold-run benchmarks.
custom_tools/kokoro_tts_tool.py	The custom kokoro_tts tool added to Hermes.
utils/Dockerfile, utils/docker-entrypoint.sh	Build and run the all-in-one workshop container. See utils/DOCKER.md.
assets/	Diagrams, dashboard screenshots, and reference outputs.
scripts/	Generator that rebuilds the concept diagrams.

---

Prerequisites

Requirement	Version tested
Operating system	Ubuntu 24.04
GPU	AMD Instinct™ MI300X (192 GB VRAM) with ROCm support
ROCm	7.14.1 (use rocm-smi instead of amd-smi on 6.4 and earlier)
Python	3.12 with venv and pip

Verify your GPUs are visible before you start:

amd-smi

Model note. The agent is powered by Muse-Glimmer-30B, served with vLLM on the MI300X. The image builds vLLM 0.29.0 from source on a ROCm 7.14.1 base (rocm/dev-ubuntu-24.04:7.14.1-full, Python 3.12). Muse-Glimmer-30B support is native in vLLM since v0.28.1 (upstream PR #51655), so no source patch is required for the model itself. The from-source build is what lets the workshop run on the latest ROCm 7.14.1 line; AMD's prebuilt vllm/vllm-openai-rocm release images still ship on ROCm 7.2.3.

---

Quick start

The fastest path is the prebuilt Docker image, which bundles every service so there is nothing to install. If you prefer to run on the host directly, skip to Manual setup.

Option A: Docker (recommended)

docker run -d --name amd-agentic-ai-profiling \
  --device=/dev/kfd --device=/dev/dri \
  --security-opt seccomp=unconfined --group-add video \
  --ipc=host --shm-size 16G \
  -p 8888:8888 -p 8501:8501 -p 5004:5004 \
  -e HERMES_PROXY_BASE="" \
  -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
  shailensobhee1/amd-agentic-ai-profiling:mi300x

Set HERMES_PROXY_BASE. It controls the base of the notebook's "detailed view" links (the telemetry dashboard and MLflow). The container inherits the value you pass with -e, so set it to match where you open the notebook from:

• -e HERMES_PROXY_BASE="" → direct http://127.0.0.1:<port>/ links (use with the -p maps above; shown in the command)
• -e HERMES_PROXY_BASE="https://<your-proxy>" → your own reverse proxy, as https://<your-proxy>/<hostname>/proxy/<port>/
• omit it → the AMD hosted-notebook proxy https://notebooks.amd.com/<hostname>/proxy/<port>/

Watch it start with docker logs -f amd-agentic-ai-profiling. When it prints All services are ready, open http://<host>:8888/lab/tree/agentic_ai_profiling.ipynb.

The first start downloads about 60 GB of model weights, so mount the Hugging Face cache as shown to pay that cost only once. Full details, flags and troubleshooting are in utils/DOCKER.md.

Manual setup

1. Clone the repository

git clone https://github.com/shailensobhee/amd-agentic-ai-profiling-workshop.git
cd amd-agentic-ai-profiling-workshop

2. Create and activate a virtual environment

sudo apt install -y python3-venv
python3 -m venv env
source env/bin/activate

3. Install the notebook dependencies

python -m pip install --upgrade pip
python -m pip install -r utils/requirements.txt

4. Start the backend (leave this terminal open)

In a separate terminal, set HERMES_PROXY_BASE and launch the full stack. This one script starts the agent model, the hermes-otel telemetry plugin, MLflow, Grafana otel-lgtm, and the dashboard, and prepares the GPU environment the local TTS server needs:

export HERMES_PROXY_BASE=""
bash utils/helper.sh

Set HERMES_PROXY_BASE. It controls the base of the service links helper.sh prints and the notebook's "detailed view" links (the telemetry dashboard and MLflow):

• HERMES_PROXY_BASE="" → direct http://127.0.0.1:<port>/ links (shown above)
• HERMES_PROXY_BASE="https://<your-proxy>" → your own reverse proxy, as https://<your-proxy>/<hostname>/proxy/<port>/
• unset → the AMD hosted-notebook proxy https://notebooks.amd.com/<hostname>/proxy/<port>/

Leave that terminal running for the whole workshop. It keeps the services alive; closing it shuts the backend down. When it finishes starting, it prints the service URLs you will use in the notebook.

5. Launch JupyterLab and open the notebook

jupyter lab --ip=0.0.0.0 --port=8888 --no-browser

Open agentic_ai_profiling.ipynb and work through it top to bottom. Everything from here on happens inside the notebook. Pick the link base from the dropdown in the notebook's Step 2 cell (defaults to the value you set in step 4, if this is the same terminal).

Optional: force JupyterLab dark theme

The Hermes environment is designed with a dark theme. To match it:

mkdir -p ./env/share/jupyter/lab/settings
echo '{"@jupyterlab/apputils-extension:themes": {"theme": "JupyterLab Dark"}}' > ./env/share/jupyter/lab/settings/overrides.json

---

The workshop backend

Service	Port	Role
Hermes backend (vLLM · Muse-Glimmer-30B)	8001	The agent's model that plans and picks tools.
Hermes OTel	n/a	Sends execution traces to MLflow and CPU/GPU metrics to Grafana otel-lgtm. helper.sh sets a 100 ms sampling interval (psutil for CPU, amdsmi for GPU).
MLflow tracking server	5004	Stores the execution traces the dashboard visualizes.
Grafana otel-lgtm	4318 / 9090	Receives the CPU/GPU metrics over OTLP and stores them (Prometheus), which the dashboard queries.
Telemetry dashboard (Streamlit)	8501	A clean overview of each run: spans, CPU/GPU timeline, tool breakdown.
Kokoro TTS server	8092	The local, self-hosted TTS engine used in the optimization step. The notebook starts this one (see below).

The local TTS server is started from the notebook. helper.sh prepares everything its launch depends on (the shared env/ venv, the ROCm PyTorch wheels, the MIOpen JIT headers) and leaves the server itself to the notebook, which runs bash utils/start_kokoro_server.sh once the cloud (Edge TTS) baseline has been profiled and the workshop moves to local Kokoro. The script is safe to re-run: a healthy server already on the port is left alone. The Docker path behaves the same way; utils/docker-entrypoint.sh leaves the server to that same notebook cell.

---

Service reference

After utils/helper.sh is running, these are reachable on the host (replace <server-ip> with your machine's address):

Interface	URL
Telemetry dashboard	http://<server-ip>:8501
MLflow UI (advanced)	http://<server-ip>:5004
JupyterLab	http://<server-ip>:8888

---

Troubleshooting

Symptom	Fix
which hermes prints nothing in the notebook	utils/helper.sh has not finished starting, or the notebook was launched from a different environment. Wait for the backend, then relaunch Jupyter from the same shell.
Dashboard shows no runs	Click Fetch. Runs appear newest first; select the top one.
First Kokoro run is slow	Cold-run GPU kernel compilation. Subsequent runs reuse the cached kernels and are much faster.
kokoro_tts cannot connect on port 8092	The TTS server is not running. Re-run the notebook cell that starts it (bash utils/start_kokoro_server.sh) and check kokoro_server.log in the repository root.
A port is already in use	utils/helper.sh frees its ports on start, but a stale process may linger. Stop it, then rerun.
vLLM cannot reserve enough VRAM	Another process is holding GPU memory. vLLM takes an 0.80 share by default; lower it with GPU_MEMORY_UTILIZATION=0.70 bash utils/helper.sh.

---

Regenerating the assets

The concept diagrams are generated from a script so they stay reproducible and reviewable:

python scripts/build_diagrams.py       # rebuild the AMD-branded concept diagrams

---

Credits

Authors: Shailen Sobhee, Sabira Shaik, Jereshea John Mary

Built for AMD developer enablement on AMD Instinct™ GPUs. Powered by Hermes Agent (Nous Research), MLflow, hermes-otel, docker-otel-lgtm and Kokoro TTS