Skip to content

TESTING.md

Latest commit

 

History

History
164 lines (129 loc) · 9.58 KB

TESTING.md

File metadata and controls

164 lines (129 loc) · 9.58 KB
Raw

Actor testing and reliability campaigns

Run cargo test -p spawned-concurrency --test actor_scenarios.

The capability examples demonstrate useful application behavior separately from these test oracles. CI runs their finite programs and assertions; future APIs have an explicitly proposed example catalog.

Controlled failures, histories and simulations

Suite Evidence provided Limit
failure_windows Nine deliberately reached lifecycle/request windows across async, blocking, dedicated-thread and synchronous-thread actors; optional SPAWNED_FAILURE_REPORT records coverage A bounded demonitor observation is not exhaustive proof over schedules
request_histories Independent bounded search for a legal sequential history, with wrong-reply/duplicate/lost-effect negative controls At most 16 operations and 100,000 search states; exhausted search is inconclusive and fails, not a pass
delivery_simulation Enumerate 125 states of a finite cooperative-stop model and replay 500 witnesses across four backends, plus seeded scripts and an audit digest Controls handler-progress gates, not arbitrary OS/Tokio scheduling, network, clocks or all lifecycle signals
payload_stress Unique owned payloads from 128 concurrent producers; verify FIFO delivery or complete destruction on stop Explicitly bounded payload volume; not a whole-process memory proof
process_failures (Unix) Abort or kill an owned subprocess after an actor effect; check process failure does not promise cleanup callbacks or rollback Temporary marker is not a storage durability acknowledgement
benchmark_startup Startup rendezvous opens for all participants, aborts after setup failure and has a finite timeout Tests the research driver, not production actor supervision 
cargo test --locked -p spawned-concurrency --test failure_windows --test request_histories --test delivery_simulation --test process_failures --test benchmark_startup
SPAWNED_HISTORY_CASES=128 SPAWNED_HISTORY_SEED=20261106 cargo test --locked -p spawned-concurrency --test request_histories -- --nocapture
SPAWNED_SIM_CASES=256 SPAWNED_SIM_SEED=20261106 cargo test --locked -p spawned-concurrency --test delivery_simulation -- --nocapture
SPAWNED_STRESS_PRODUCERS=128 SPAWNED_STRESS_MESSAGES=256 SPAWNED_STRESS_PAYLOAD=1024 cargo test --locked -p spawned-concurrency --test payload_stress -- --ignored --nocapture

A timeout, dropped caller or closed reply after dispatch has an unknown effect outcome. The history oracle explores both effect/no-effect possibilities without pretending the caller timeout ended the operation. A proven enqueue rejection permits no effect. Normal successful replies require their matching effect; the checker never treats ambiguous operations as acknowledged success. No automatic retry or exactly-once guarantee is introduced.

The Linux campaign driver runs isolated owned process groups with CPU affinity, nice level, wall-time, sampled RSS, thread and descriptor budgets. Builds have a separate memory allowance from stress executables. It kills only its own process group on failure and writes commands, source revision, environment and resource peaks. The default 768-thread ceiling accommodates 256 offered-load producers and their collectors; they still share the declared CPU mask. Shared-page RSS can be counted more than once and short peaks can escape sampling.

python3 benchmarks/run_reliability.py --source . --out /tmp/spawned-reliability --cpus 2,3,4,5 --phase all --seconds 30 --replicates 3

Use an available CPU set appropriate to the machine. The benchmark phase records closed-loop requests, actor-to-actor graphs, concentrated offered load and actor churn/idle soaks. Benchmarks describe their generator as well as actor work; high producer counts do not imply that many actor workers. See the recorded campaign for results, excluded preflights and reproducibility details. No throughput threshold is enforced in shared CI. Nightly CI retains failure coverage and stress logs.

The suite generates 2–8 worker actors and an observer, with sequences of sends, requests, and dropped reply receivers. Every generated scenario runs on Tokio async, Tokio blocking-pool, dedicated-thread, and synchronous-thread backends. It then races monitor registration and requests against clean shutdown or a handler panic. The fixed regression case exercises every termination mode.

Contract checked

  • A request fence observes every earlier message from the same sender in order, exactly once. Dropping a reply receiver does not cancel queued work.
  • A request racing shutdown either returns its matching reply or ActorStopped. Accepted sends alone do not guarantee execution across termination.
  • Termination resolves within a five-second test watchdog. Post-exit requests return ActorStopped.
  • Each uncancelled monitor observes the matching exit reason, including when registration races termination. Unknown or duplicate observed notifications fail the test. Clean shutdown and injected handler panic remain distinct.

The reference model is the per-actor sequence of submitted message IDs. During shutdown it permits the final racing message to execute or be discarded; a successful reply requires execution. It does not impose cross-sender ordering.

Replay and shrinking

Proptest prints and shrinks failing inputs and persists failure seeds alongside this integration test (actor_scenarios.proptest-regressions). Retain that file and the test output; commit useful regressions. A minimized input can also be added as a fixed scenario(...) invocation in lifecycle_race_regression.

Run a larger campaign:

PROPTEST_CASES=1000 cargo test -p spawned-concurrency --test actor_scenarios

Set PROPTEST_RNG_SEED to a reported seed to regenerate a campaign. A seed reproduces input generation, not the operating system or Tokio schedule. Repeat a minimized case to investigate scheduling-dependent failures. Nightly CI retains regression files on failure and has an outer job timeout to catch hangs that prevent an in-process watchdog from making progress.

Scope

This is generated, model-checked scenario testing on real runtimes, not an exhaustive scheduler or coverage-guided fuzzer. It does not cover bounded mailboxes or supervisor restart policies. Demonitor races permit an already-queued notification; tests use bounded observation windows and do not claim exhaustive delivery verification. Do not interpret a passing campaign as proof of race freedom.

Additional checks

  • Concurrent producers verify per-producer FIFO, exactly-once processing during normal operation, and reply identity across all four backends.
  • Monitor cleanup unit tests verify waiter-owned references are released while the target remains alive, after demonitor or owner termination.
  • Resource cleanup tests verify timer payloads and pending async streams are dropped after shutdown. Blocking iterator next() calls are not preemptible; the listener checks cancellation before invoking the next call.
  • Compile tests accept a working actor/protocol and snapshot diagnostics for unsupported async methods, associated types, and missing receivers.
  • Early-stop regressions ensure a cleanup panic remains an abnormal exit.

The Loom tests instrument the actual thread CancellationToken implementation:

RUSTFLAGS="--cfg spawned_loom" cargo test -p spawned-rt --test cancellation_loom

The custom cfg isolates instrumentation from Tokio's own Loom configuration. These models cover registration/cancellation races and reentrant callbacks, not the entire actor runtime. Do not run ordinary actor tests with this cfg.

Run sustained actor creation, traffic, monitoring, and shutdown:

SPAWNED_STRESS_ROUNDS=100 cargo test -p spawned-concurrency --test actor_scenarios sustained_actor_churn -- --ignored --nocapture

This is a correctness/churn campaign with elapsed-time logging, not a comparative performance benchmark. For process memory measurement, wrap the command with /usr/bin/time -v on Linux or /usr/bin/time -l on macOS. No portable RSS threshold is asserted; allocator retention can differ between platforms.

Bugs reproduced by these regressions

  1. Demonitor and owner shutdown suppressed delivery but retained the monitor waiter until target exit, in both execution modes. Waiters are now explicitly cancelled and outgoing monitors are drained on every actor exit path.
  2. Thread listeners called next() even when their actor was already stopped. The listener now checks cancellation before polling the iterator.
  3. Stopping inside started() swallowed a subsequent stopped() panic and reported Normal. Both execution modes now preserve the panic exit reason.

Thread actor completion now uses a watch notification, so async exit waits can be cancelled without leaving a blocking-pool worker waiting for target exit. A regression with a one-worker blocking pool checks that property.

Toolchain

The repository's .tool-versions selects Rust 1.93.0. The workspace tests, including generated cases and compile diagnostics, have been validated with that compiler. Proptest and trybuild are dev-dependencies; proptest's unused default features are disabled. Cargo generates the lockfile.

Execution guarantees

Protocols continue to separate application logic from message dispatch. Each actor handles one message at a time, with both async and synchronous thread execution supported. No restart or queue-capacity policies are added by these tests. See CHANGELOG.md for the observable lifecycle corrections.