Skip to content

[Bug] Worker registers with the server as RUNNING as soon as constructed, before being run() #1428

Description

@anatolec

What are you really trying to do?

I have a shared builder that constructs several Worker objects (one per task queue), and depending on a configuration (an env proprerty) the process actually runs only a subset of them (by calling .run() on them). The Worker objects that aren't selected are constructed but never run. This way I deploy the same code in many processes and decide which process listens to which task queue with an env var.

I expected a Worker I never run() to have no effect on the server — to be an inert local object until started. Instead, merely constructing it registers it with the server as a live, running worker.

Describe the bug

Constructing a temporalio.worker.Worker (Python SDK 1.30.0) registers it with the server and reports it as WORKER_STATUS_RUNNING — before run() (or async with worker:) is ever called. A worker that is only constructed, and never run, is advertised to the server as a live running worker on its task queue (visible via the DescribeWorker / temporal worker list API).

If registering/heartbeating at construction is intended, it would help to document it in a very visible way; if not, registration should presumably start in run().

Below is a snippet proving that those workers are heartbeating. I also suspect that, in certain conditions, they might also poll some tasks, which is much more problematic. Though I haven't been able to reproduce it consistently.

Minimal Reproduction

Start a throwaway dev server:

temporal server start-dev --port 7255 --ui-port 8255 --headless --log-level error &

Repro 1 — construction registers the worker as RUNNING without run().
Constructs 3 workers, runs only 1, then queries worker registration:

import asyncio, json, os, subprocess
from temporalio import activity
from temporalio.client import Client
from temporalio.worker import Worker

ADDR, NS = os.environ.get("ADDR", "localhost:7255"), "default"
RUN_Q = "constructed-and-run"
NORUN_Q = ["constructed-not-run-a", "constructed-not-run-b"]

@activity.defn
async def _noop() -> None:
    return None

def registered_workers():
    out = subprocess.run(
        ["temporal", "worker", "list", "--address", ADDR, "--namespace", NS, "-o", "json"],
        capture_output=True, text=True, timeout=15,
    )
    rows = json.loads(out.stdout or "[]")
    return [(r["workerHeartbeat"].get("taskQueue"), r["workerHeartbeat"].get("status")) for r in rows]

async def main():
    client = await Client.connect(ADDR, namespace=NS)
    print("baseline:", registered_workers())                  # []

    run_worker = Worker(client, task_queue=RUN_Q, activities=[_noop])
    _norun = [Worker(client, task_queue=q, activities=[_noop]) for q in NORUN_Q]  # constructed, never run
    run_task = asyncio.create_task(run_worker.run())           # run ONLY this one

    await asyncio.sleep(5)
    print("after construct + partial run:")
    for tq, status in registered_workers():
        print(f"  {tq}: {status}")
    run_task.cancel()

asyncio.run(main())

Actual output — all three, including the two never run, are registered RUNNING:

baseline: []
after construct + partial run:
  constructed-not-run-a: WORKER_STATUS_RUNNING
  constructed-not-run-b: WORKER_STATUS_RUNNING
  constructed-and-run:   WORKER_STATUS_RUNNING

Expected: only constructed-and-run is registered.

Environment/Versions

  • OS and processor: M1/ARM Mac (Apple Silicon, arm64), macOS 15.7.7
  • Temporal Version: temporal CLI 1.6.1 / Server 1.30.1 (dev server); Python SDK temporalio 1.30.0; Python 3.14.2
  • Are you using Docker or Kubernetes or building Temporal from source? No — temporal server start-dev (in-memory dev server) reproduces it; originally observed on Temporal Cloud.

Additional context

Now that I've understood this, I obviously don't create workers I don't want to run, but it's still worth fixing as it could bite other users of the lib.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions