fix: recover the local stack after a port clash

Docker Engine 29.2 leaves a container whose port could not be bound
without its network. Every later start of that container has loopback
only, even once the port is free. Only a new container recovers.

For the Local course that meant:

- Lab 0 step 2: after `Bind for 0.0.0.0:8080 failed`, running
  local/engine.sh again (what --check says to do) failed with
  `registry-1 exited (1)`, and stopping the other program did not help.
- Lab 0 step 1: after stopping the other program, `up` exited 0 and the
  step's check listed six services, but the container had no published
  port.

local/engine.sh now removes the engine's containers that are not
running before `ork local start`. The engine's data is in volumes. A
rerun gives the same bind error while the port is taken, and a working
engine once it is free.

Troubleshooting gives a recovery that works for both stacks, covers a
doctor that cannot reach a container that lost its network, and has a
row for a missing python/.venv on the CLI path. Lab 0 steps 1 and 2
point to it.

Author: sijie

labs/local/00-set-up.md

@@ -41,6 +41,9 @@ The first run downloads about 4.4 GB of images. It starts six services
 - `risingwave`: streaming SQL.
 - `risingwave-mcp`: RisingWave's MCP server, the agent's SQL tools.
 
+If it fails with a port already in use, see
+[Troubleshooting](troubleshooting.md) before you run it again.
+
 ### Check
 
 The six services are running. Before the step this prints nothing.
@@ -84,6 +87,9 @@ header:
 Run `local/engine.sh` again whenever the engine has been restarted. It is safe
 to run at any time.
 
+If it stops with `port is already allocated`, another program has a port the
+engine needs, usually 8080: see [Troubleshooting](troubleshooting.md).
+
 ### Check
 
 The engine is up, has a provider key, and can reach the MCP server. The script

labs/local/troubleshooting.md

@@ -13,14 +13,15 @@ Each failed line prints its fix.
 
 | Symptom | Fix |
 |---|---|
-| `docker compose ... up` fails with a port already in use | Another program has one of the ports the stack publishes on `127.0.0.1`: 29092 (Kafka), 18081 (schema registry), 4566 and 5691 (RisingWave), 8000 (MCP). Stop that program. The ports are fixed: the broker tells its clients to come back to `127.0.0.1:29092`, and `local/write-env.sh` writes these ports into `.env`. |
-| `local/engine.sh` fails because port 8080 is taken (`Bind for 0.0.0.0:8080 failed: port is already allocated`) | Pick another port for the registry: `export ORCA_LOCAL_REGISTRY_PORT=18080`, run `local/engine.sh` again, then `local/write-env.sh` so `.env` has the new address. Export it in every terminal you run `local/engine.sh` from: a run without it goes back to 8080. |
+| `docker compose ... up` fails with a port already in use | Another program has one of the ports the stack publishes on `127.0.0.1`: 29092 (Kafka), 18081 (schema registry), 4566 and 5691 (RisingWave), 8000 (MCP). Stop that program, run `local/down.sh`, then run the `up` command again. Running it again without `local/down.sh` is not enough: Docker brings the container whose port was taken back without its network. The ports are fixed: the broker tells its clients to come back to `127.0.0.1:29092`, and `local/write-env.sh` writes these ports into `.env`. |
+| `local/engine.sh` fails because a port is taken (`Bind for 0.0.0.0:8080 failed: port is already allocated`) | Another program has a port the engine publishes on `127.0.0.1`: 8080 (the registry) or 18082. If it is a container, `docker ps` shows which: look for `:8080->` under PORTS. Stop that program and run `local/engine.sh` again. If the port is 8080 and you want to keep that program running, move the registry instead: `export ORCA_LOCAL_REGISTRY_PORT=18080`, run `local/engine.sh` again, then `local/write-env.sh` so `.env` has the new address. Export it in every terminal you run `local/engine.sh` from: a run without it goes back to 8080. |
 | `local/engine.sh`: `ANTHROPIC_API_KEY is not set in this shell` | `export ANTHROPIC_API_KEY=<your key>` in the terminal where you run the script. The engine reads the key only when it starts. |
 | `local/engine.sh`: `bootstrap refused: an organization already exists` | The engine's volumes exist but its keys in `.lab/ork` are gone. Start over: `local/down.sh --reset`, then Lab 0. |
+| CLI path: `.venv/bin/python: No such file or directory` | The Python path is not installed. If you installed the TypeScript path, use the `npm` command the lab gives beside the Python one. Otherwise install one of the two: Lab 0, "Before you start". |
 | Doctor: `Agent Engine HTTP 401` | The key in `.env` is not the running engine's key. Run `local/write-env.sh`. If it still fails, the engine's volumes and keys are out of step: `local/down.sh --reset`, then Lab 0. |
 | Doctor: `Kafka ... not found` | The topic does not exist yet: Lab 0, step 4. |
 | Doctor: `Schema Registry ... not found` | The schema is registered when you seed the topic: `python seed.py` or `npm run seed`. |
-| Doctor: `Kafka`, `Schema Registry`, or `MCP server` cannot be reached, or `RisingWave through MCP` fails | The streaming stack is not up: `docker compose -f local/compose.yaml up -d --wait`. |
+| Doctor: `Kafka`, `Schema Registry`, or `MCP server` cannot be reached, or `RisingWave through MCP` fails | The streaming stack is not up: `docker compose -f local/compose.yaml up -d --wait`. If it is up and the doctor still says so, a container lost its network when its port was taken: run `local/down.sh`, then start both stacks again. |
 | Doctor: `Agent Engine   Connection error` | The engine is not up: `local/engine.sh`. Start the streaming stack first. |
 | `seed`: `already holds 246 events` | The topic is seeded. Nothing to do. |
 | `table or source not found: security.login_events` | Create the source: Lab 2, step 1. |

local/engine.sh

@@ -11,7 +11,9 @@
 #   1. ork local --data-dir <this checkout>/.lab/ork start --with-gateway
 #      The Agent Engine, with the AI Gateway. The gateway makes every MCP call on
 #      your agent's behalf, so MCP tools need it. The data directory is given as a
-#      full path: ork v0.6.0 does not resolve a relative one.
+#      full path: ork v0.6.0 does not resolve a relative one. First, the engine's
+#      containers that are not running are removed, so that it starts from fresh
+#      ones. Its data is in volumes.
 #
 #   2. The link. The gateway refuses private MCP hosts unless they are on its
 #      allowlist, and `ork local start` writes that allowlist empty every time.
@@ -44,6 +46,14 @@ start_engine() {
   [ -n "$(mcp_container)" ] ||
     die "The streaming stack is not running. Start it first:
   docker compose -f local/compose.yaml up -d --wait"
+  # Replace what is not running. A container whose port could not be bound
+  # (another program had it) stays cut off from its network: Docker starts it
+  # with loopback only from then on, even once the port is free (seen with
+  # Docker Engine 29.2). The engine's data is in volumes, so nothing is lost.
+  local name
+  while IFS= read -r name; do
+    [ -z "$name" ] || docker rm "$name" >/dev/null
+  done < <(engine_stopped_containers)
   ork local --data-dir "$ORK_DIR" start --with-gateway
 }
 

local/lib.sh

@@ -44,6 +44,12 @@ engine_container() {  # engine_container <service>
     --filter "label=com.docker.compose.service=$1" --format '{{.Names}}' | head -n 1
 }
 
+# The Agent Engine's containers that exist but are not running.
+engine_stopped_containers() {
+  docker ps -a --filter "label=com.docker.compose.project.working_dir=$ORK_DIR" \
+    --filter status=created --filter status=exited --filter status=dead --format '{{.Names}}'
+}
+
 mcp_container() {
   docker ps --filter "label=com.docker.compose.project=$STREAMING_PROJECT" \
     --filter "label=com.docker.compose.service=risingwave-mcp" --format '{{.Names}}' | head -n 1

local/tests/run.sh

@@ -10,26 +10,53 @@ set -euo pipefail
 TESTS=$(cd "$(dirname "$0")" && pwd)
 LOCAL=$(dirname "$TESTS")
 WORK=$(mktemp -d "${TMPDIR:-/tmp}/hello-local-tests.XXXXXX")
+WORK=$(cd "$WORK" && pwd)   # as the scripts see it: a TMPDIR ending in "/" leaves a "//"
 trap 'rm -rf "$WORK"' EXIT
 
-# A `docker` that knows which host port the registry is published on, and
-# remembers what it was asked.
+# A `docker` that knows which host port the registry is published on, which
+# containers exist, and remembers what it was asked.
 mkdir -p "$WORK/bin"
 cat >"$WORK/bin/docker" <<'EOF'
 #!/usr/bin/env bash
 [ -n "${FAKE_DOCKER_DIR:-}" ] || exit 1
 printf '%s\n' "$*" >>"$FAKE_DOCKER_DIR/calls"
 case "$1" in
-  ps) [ -f "$FAKE_DOCKER_DIR/registry-port" ] && echo registry-1 ;;
+  ps)
+    case " $* " in
+      *" -a "*)
+        # Every container, running or not: the "<status> <name>" lines of the
+        # fixture, when the question is narrowed to the engine's project. A
+        # status filter keeps the lines with that status.
+        if [[ " $* " != *" label=com.docker.compose.project.working_dir=${FAKE_DOCKER_DIR%/*}/.lab/ork "* ]]; then
+          echo someone-elses-container
+        elif [ -f "$FAKE_DOCKER_DIR/containers" ]; then
+          while read -r status name; do
+            [[ " $* " == *" status="* && " $* " != *" status=$status "* ]] || echo "$name"
+          done <"$FAKE_DOCKER_DIR/containers"
+        fi
+        ;;
+      *) [ -f "$FAKE_DOCKER_DIR/registry-port" ] && echo registry-1 ;;
+    esac
+    ;;
   port)
     [ ! -f "$FAKE_DOCKER_DIR/port-fails" ] || exit 1
     [ -f "$FAKE_DOCKER_DIR/registry-port" ] && echo "127.0.0.1:$(cat "$FAKE_DOCKER_DIR/registry-port")"
     ;;
+  rm) [ $# -ge 2 ] || exit 1 ;;   # like docker, it wants at least one container
   compose) ;;
   *) exit 1 ;;
 esac
 EOF
 chmod +x "$WORK/bin/docker"
+# An `ork` that remembers the call and stops the script there: what comes after
+# a start needs a running stack.
+cat >"$WORK/bin/ork" <<'EOF'
+#!/usr/bin/env bash
+[ -n "${FAKE_DOCKER_DIR:-}" ] || exit 1
+printf 'ork %s\n' "$*" >>"$FAKE_DOCKER_DIR/calls"
+exit 1
+EOF
+chmod +x "$WORK/bin/ork"
 export PATH="$WORK/bin:$PATH"
 # Nothing from the developer's own shell may leak into the tests.
 unset ORCA_LOCAL_REGISTRY_PORT TUTORIAL_STACK PARTICIPANT ORCA_MODEL ORCA_API_KEY
@@ -63,6 +90,7 @@ run() {  # run <script> [args...]
 env_is() { [ "$(sed -n "s/^$1=//p" "$R/.env")" = "$2" ]; }
 out_has() { grep -qF -- "$1" "$R/out"; }
 err_has() { grep -qF -- "$1" "$R/err"; }
+not() { ! "$@"; }
 
 check() {  # check <description> <command...>
   if "${@:2}"; then
@@ -166,6 +194,41 @@ test_write_env_refreshes_a_local_env_and_keeps_your_choices() {
   check "keeps your model" env_is ORCA_MODEL claude-haiku-4-5
 }
 
+# ----------------------------------------------------- starting the engine --
+
+start_engine() {  # local/engine.sh, as far as `ork local start`
+  ANTHROPIC_API_KEY=not-a-real-key run engine.sh
+}
+
+started() { grep -qE '^ork local .* start --with-gateway$' "$FAKE_DOCKER_DIR/calls"; }
+removed() { grep -qE "^rm( .*)? $1( |\$)" "$FAKE_DOCKER_DIR/calls"; }
+removed_before_the_start() {  # removed_before_the_start <container>
+  sed '/^ork /,$d' "$FAKE_DOCKER_DIR/calls" | grep -qE "^rm( .*)? $1( |\$)"
+}
+
+test_engine_replaces_its_containers_that_are_not_running() {
+  # A container whose port could not be bound comes back without its network,
+  # even once the port is free (Docker Engine 29.2). A start must not reuse it.
+  fresh_repo engine-stopped
+  engine_started
+  printf '%s\n' 'created ork-registry-1' 'exited ork-migrate-1' 'running ork-harness-1' >"$FAKE_DOCKER_DIR/containers"
+  start_engine
+  check "removes a container that never started, before the engine starts" removed_before_the_start ork-registry-1
+  check "removes a container that has stopped, before the engine starts" removed_before_the_start ork-migrate-1
+  check "leaves a running container alone" not removed ork-harness-1
+  check "leaves other projects' containers alone" not removed someone-elses-container
+  check "then starts the engine" started
+}
+
+test_engine_starts_when_nothing_has_stopped() {
+  fresh_repo engine-running
+  engine_started
+  printf '%s\n' 'running ork-registry-1' >"$FAKE_DOCKER_DIR/containers"
+  start_engine
+  check "asks docker to remove nothing" not grep -q '^rm' "$FAKE_DOCKER_DIR/calls"
+  check "and starts the engine" started
+}
+
 # ------------------------------------------------------- the gateway patch --
 
 gateway_yaml() {  # the two lines of `ork local`'s gateway.yaml that matter here