Loud fail for missing build tools; fix worker payload serialization on editable installs; mode-independent run-stats block counts

[rhoadesScholar]

Three things: a loud, actionable failure when build tooling is missing; a fix for subprocess-worker payload serialization breaking on editable/source-tree installs; and correct blocks_processed run stats across all worker execution modes.

1. Loud fail for missing install tools (357d040)

Building daisy from source requires a C linker and the Rust toolchain, and the stock maturin error when they're missing is cryptic. build_wrapper.py now wraps maturin's PEP 517 hooks (build_wheel / build_editable) and checks for cc/gcc/clang and cargo up front, failing with per-platform install instructions instead. pyproject.toml also gains dev, examples, and docs extras so pip users get the same dependency groups uv users had.

2. Fix: subprocess-worker payloads no longer break on editable installs (ce3f488)

Symptom: with daisy installed editable (or from a source tree), run_blockwise failed at startup with TypeError: cannot pickle '_thread._local' object for any 1-arg block function that references the daisy module as a global (e.g. block.status = daisy.BlockStatus.SUCCESS).

Root cause: subprocess workers (the default execution mode) serialize the block function with dill.dumps(..., recurse=True), which ships the globals the function references. dill pickles any module outside sys.prefix/site-packages by value — its entire __dict__ — so one unpicklable module-level object anywhere in the namespace fails the whole payload. daisy itself contains two (daisy.logging._active, a threading.local, and daisy._worker_processes._LEN, a struct.Struct). Wheel installs never trip this, which is why CI stayed green; editable installs always do.

Fix: _serialize now uses a dill.Pickler subclass that pickles every importable module by reference (_make_modules_by_ref_pickler). This is sound because the child replicates the parent's sys.path before deserializing (read_payload), so anything importable in the parent is importable in the child. Only __main__ (and __mp_main__) keep dill's by-value path — the child's __main__ is the worker shim, so script/notebook globals must ride in the payload, and they still do (recurse=True is load-bearing and untouched). This also fixes the same failure for user packages installed editable with unpicklable module-level state.

Also:

• daisy.logging._active is now a _PicklableLocal (pickles as a fresh, empty local) as defense-in-depth for other transports.
• A payload that genuinely cannot be serialized now raises eagerly with guidance (restructure imports, or Task(worker_processes=False)) instead of a bare dill traceback.

3. Fix: blocks_processed counted server-side, correct in every mode (ce3f488)

Symptom: the Resource Utilization report showed blocks 0 per task (and empty per-worker counts) for subprocess-mode runs — the default — and tests/test_run_stats.py had to pin tests to thread mode. This was the documented gap in RUN_STATS.md.

Root cause: WorkerStats.blocks_processed was incremented only inside the in-process worker thread's block loop. Subprocess shim workers and external cluster workers process blocks where the stats layer never looked.

Fix: counting moves to the server, where every worker's block returns already arrive over TCP:

• New protocol message Register { task_id, worker_id }, sent by Client::connect right after the connection opens (appended as the last enum variant, so existing message discriminants are unchanged on the wire).
• The bookkeeper records each connection's registered identity; every valid block return (ReleaseBlock or BlockFailed) increments a RunTally counter per task and per worker.
• build_run_stats merges the per-worker counts into the exit-channel WorkerStats by (task_id, worker_id); registered workers with no thread in the server process (fully external workers) get synthetic entries carrying their block counts.

Semantics notes:

• Counts are now identical across thread, subprocess-shim, and external workers (the shim child registers with the same worker id as its babysitter thread).
• Reclaimed attempts (block timeout, dead client) were never valid returns and are no longer counted — slightly more honest than the old thread-loop counter.
• Failed-but-returned blocks still count, matching the old accounting.
• A client running an older daisy that never registers still yields correct per-task counts; only its per-worker attribution is absent. (The reverse — this branch's client against an older server — was already a non-goal: server and workers ship together.)
• Registration doubles as a liveness signal: registering clears any stale closed-connection marker left on an OS-recycled ephemeral port, and disconnecting drops the registration, so recycled ports can't inherit stale identities or get their in-flight blocks falsely reclaimed.

Internal API change: Client::connect and _rs.SyncClient now take a worker_id (both internal; daisy.Client reads it from DAISY_CONTEXT as before). docs/source/design/RUN_STATS.md is updated — the "known gap" section is replaced by the Register design, with daisy.profile_block kept as the future path for per-block CPU/RSS.

Tests (9f6bf9f)

• tests/test_worker_serialization.py (new): forces the by-value failure mode deterministically with a synthetic module carrying a threading.local + struct.Struct (so coverage doesn't depend on install layout), the daisy-module-global shape, __main__-defined function end-to-end through real subprocess workers, eager-failure guidance, and _PicklableLocal round-trip.
• tests/test_run_stats.py: the previously pinned tests now run the default subprocess mode and their blocks_processed assertions pass; a new pinned test keeps thread-mode counting covered.
• Rust: unit tests for RunTally counting and the build_run_stats merge (including synthetic external-worker entries and the unregistered-client fallback); bookkeeper registration lifecycle + recycled-port hygiene; the TCP integration test now asserts externally-connected registered workers appear in run stats.
• Full suites: cargo test -p daisy-core 46 passed; pytest tests/ 204 passed, 1 xfailed (the intentional daisy.messages documentation xfail).

🤖 Generated with Claude Code

https://claude.ai/code/session_01KxYd8bCjgZLoESSe3smbwF