Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,19 @@ Versions before 0.8.0 are in the [git tags](https://github.com/WireCatLabs/cli-b
Every entry says what changed as a caller sees it, why, and what to watch for — the rules are
[`docs/dev/CONVENTIONS.md`](docs/dev/CONVENTIONS.md#the-changelog).

## Unreleased

### Added

- **The shared test setup waits for a machine-wide test slot.** Several agents ran suites on one machine at once, each
with its own workers, and together they took every core and all the memory until the computer froze. Now a test
file from `@wirecat/cli-base/testing/setup` starts only when it holds one of 8 slots for the whole machine
(files in `~/.cache/wirecat-test-slots`) and at least 4 GB of memory is available. It gives the slot back when it
ends; a run killed before that loses its slot to the next one that looks. After 10 minutes of waiting a file runs
without a slot, and a wait over 30 seconds prints who holds the slots. CI skips this. Settings:
`WIRECAT_TEST_SLOTS` (`0` turns it off), `WIRECAT_TEST_MIN_FREE_GB`, `WIRECAT_TEST_MAX_WAIT_S`,
`WIRECAT_TEST_SLOT_DIR`. A CLI gets it with its next cli-base bump; `maxWorkers` still caps each run.

## 0.125.3 — 21.11.2026

### Fixed
Expand Down
11 changes: 11 additions & 0 deletions docs/dev/TESTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,17 @@ from one line in its `vitest.config.ts`: `setupFiles: ["@wirecat/cli-base/testin
([WC-TEST-3](../standards/testing.md#wc-test-3--no-test-contacts-a-provider-or-the-network)). A test about the
network itself calls `onTestFinished(allowNetwork())`. `src/testing/network.test.ts` covers each path.

- **A test file waits for one of the machine's test slots.** Several agents running suites at once took every core
and all the memory. [`src/testing/machine-slot.ts`](../../src/testing/machine-slot.ts) lets a file start when it
holds one of `WIRECAT_TEST_SLOTS` (8) slot files in `~/.cache/wirecat-test-slots` and `os.freemem()` is at least
`WIRECAT_TEST_MIN_FREE_GB` (4). On Linux `os.freemem()` is MemAvailable, not MemFree. A slot whose process is dead
is taken back. After `WIRECAT_TEST_MAX_WAIT_S` (600) the file runs without one. `CI` skips it, `0` slots turns it
off, and a child vitest started by a test inherits `WIRECAT_TEST_SLOT_HELD` and does not wait on its parent's
slot. A slot file appears with its owner already in it (written to a temporary file, then linked), so an empty
file is never taken for a dead run's, and a run gives back only a slot that still names it. Checked on 2026-11-21:
four suites of 25 files each, run at once with 2 slots, never held more than 2 and left none behind; a file whose
only slot was held waited 90 s, ran without one and passed.

- **The owner's home refuses a test even without this setup.** A run that skips `setupFiles` — `vitest run --root`
at a folder outside the repository reads no `vitest.config.ts` — keeps the real `HOME`. Under vitest (`VITEST` in
the `env` passed), `resolvePaths` and the skill commands throw while `HOME` or `USERPROFILE` is the account's own
Expand Down
101 changes: 101 additions & 0 deletions src/testing/machine-slot.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
import { existsSync, mkdtempSync, readdirSync, readFileSync, rmSync, writeFileSync } from "node:fs"
import { tmpdir } from "node:os"
import { join } from "node:path"
import { afterEach, describe, expect, it } from "vitest"
import { acquireSlot, SLOT_HELD, type SlotOptions, slotOptions } from "./machine-slot.js"

const GB = 1024 ** 3
const dirs: string[] = []
afterEach(() => {
for (const dir of dirs.splice(0)) rmSync(dir, { recursive: true, force: true })
})

const optionsWith = (overrides: Partial<SlotOptions> = {}) => {
const dir = mkdtempSync(join(tmpdir(), "slots-"))
dirs.push(dir)
let clock = 0
const lines: string[] = []
const options: SlotOptions = {
dir,
slots: 2,
minFreeBytes: 4 * GB,
maxWaitMs: 10_000,
pollMs: 500,
freeBytes: () => 8 * GB,
alive: () => true,
now: () => clock,
sleep: async (ms) => {
clock += ms
},
log: (line) => lines.push(line),
...overrides,
}
return { options, lines, dir }
}

describe("machine test slots", () => {
it("lets as many files run as there are slots, and the next one in when one ends", async () => {
const { options, lines } = optionsWith({ maxWaitMs: 2_000 })
const first = await acquireSlot(options)
const second = await acquireSlot(options)
const third = await acquireSlot(options)
expect(lines.at(-1)).toMatch(/running without a slot/)
third()
first()
const again = await acquireSlot(options)
expect(readdirSync(options.dir)).toHaveLength(2)
again()
second()
expect(readdirSync(options.dir)).toHaveLength(0)
})

it("waits while available memory is below the floor, and says why", async () => {
let free = 1 * GB
let clock = 0
const { options, lines } = optionsWith({
maxWaitMs: 600_000,
freeBytes: () => free,
now: () => clock,
sleep: async (ms) => {
clock += ms
if (clock >= 40_000) free = 6 * GB
},
})
const release = await acquireSlot(options)
expect(lines).toHaveLength(1)
expect(lines[0]).toMatch(/1\.0 GB available of 4 GB wanted/)
release()
})

it("takes back the slot of a run that died before giving it back", async () => {
const { options } = optionsWith({ slots: 1, alive: (pid) => pid !== 999_999 })
writeFileSync(join(options.dir, "slot-0"), "999999 0\n")
const release = await acquireSlot(options)
expect(existsSync(join(options.dir, "slot-0"))).toBe(true)
release()
})

it("leaves a slot alone at release once another run holds it", async () => {
const { options } = optionsWith({ slots: 1 })
const release = await acquireSlot(options)
writeFileSync(join(options.dir, "slot-0"), "424242 0\n")
release()
expect(readFileSync(join(options.dir, "slot-0"), "utf8")).toBe("424242 0\n")
expect(readdirSync(options.dir)).toEqual(["slot-0"])
})

it("marks the run while it holds a slot, so a vitest it starts does not wait on it", async () => {
const { options } = optionsWith()
const release = await acquireSlot(options)
expect(process.env[SLOT_HELD]).toBe("1")
expect(slotOptions({ [SLOT_HELD]: "1" })).toBeUndefined()
release()
expect(process.env[SLOT_HELD]).toBeUndefined()
})

it("stays out of CI and of a run that turns it off", () => {
expect(slotOptions({ CI: "true" })).toBeUndefined()
expect(slotOptions({ WIRECAT_TEST_SLOTS: "0" })).toBeUndefined()
expect(slotOptions({ WIRECAT_TEST_SLOTS: "3", WIRECAT_TEST_SLOT_DIR: "/x" })).toMatchObject({ slots: 3, dir: "/x" })
})
})
140 changes: 140 additions & 0 deletions src/testing/machine-slot.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,140 @@
import { linkSync, mkdirSync, readdirSync, readFileSync, rmSync, writeFileSync } from "node:fs"
import { freemem, homedir } from "node:os"
import { join } from "node:path"
import { threadId } from "node:worker_threads"

const GB = 1024 ** 3
/** Set in a run that holds a slot, so a vitest it starts does not wait on its parent's slots. */
export const SLOT_HELD = "WIRECAT_TEST_SLOT_HELD"

export interface SlotOptions {
dir: string
slots: number
minFreeBytes: number
maxWaitMs: number
pollMs: number
freeBytes: () => number
alive: (pid: number) => boolean
now: () => number
sleep: (ms: number) => Promise<void>
log: (line: string) => void
}

const alive = (pid: number): boolean => {
try {
process.kill(pid, 0)
return true
} catch (error) {
return (error as NodeJS.ErrnoException).code === "EPERM"
}
}

const number = (value: string | undefined, fallback: number): number => {
const parsed = Number(value)
return value !== undefined && value !== "" && Number.isFinite(parsed) && parsed >= 0 ? parsed : fallback
}

/**
* Several agents run test suites on one machine at once, each with its own workers, and together they took every core
* and all the memory. A slot is a file in one folder for the whole machine: a test file starts once it holds one and
* enough memory is available, and gives it back when it ends. CI runs one suite per machine and skips this.
*/
export const slotOptions = (env: NodeJS.ProcessEnv = process.env): SlotOptions | undefined => {
if (env.CI || env[SLOT_HELD]) return undefined
const slots = number(env.WIRECAT_TEST_SLOTS, 8)
if (slots === 0) return undefined
return {
dir: env.WIRECAT_TEST_SLOT_DIR ?? join(homedir(), ".cache", "wirecat-test-slots"),
slots,
minFreeBytes: number(env.WIRECAT_TEST_MIN_FREE_GB, 4) * GB,
maxWaitMs: number(env.WIRECAT_TEST_MAX_WAIT_S, 600) * 1000,
pollMs: 500,
freeBytes: freemem,
alive,
now: Date.now,
sleep: (ms) => new Promise((resolve) => setTimeout(resolve, ms)),
log: (line) => process.stderr.write(`${line}\n`),
}
}

const holders = (options: SlotOptions): number[] => {
const pids: number[] = []
for (const name of readdirSync(options.dir)) {
if (!name.startsWith("slot-") || name.endsWith(".tmp")) continue
const path = join(options.dir, name)
let pid: number
try {
pid = Number.parseInt(readFileSync(path, "utf8"), 10)
} catch {
continue
}
// A run killed before its afterAll, by Ctrl-C or the OOM killer, leaves its slot behind.
if (Number.isFinite(pid) && options.alive(pid)) pids.push(pid)
else rmSync(path, { force: true })
}
return pids
}

const OWNER = `${process.pid} ${threadId}\n`

const take = (options: SlotOptions): string | undefined => {
// A slot appears with its owner already written: an empty file read in between would look dead and be taken twice.
const draft = join(options.dir, `slot-${process.pid}-${threadId}.tmp`)
writeFileSync(draft, OWNER)
try {
for (let index = 0; index < options.slots; index++) {
const path = join(options.dir, `slot-${index}`)
try {
linkSync(draft, path)
return path
} catch {}
}
return undefined
} finally {
rmSync(draft, { force: true })
}
}

const giveBack = (path: string) => {
let owner: string | undefined
try {
owner = readFileSync(path, "utf8")
} catch {}
if (owner === OWNER) rmSync(path, { force: true })
}

/** Waits for a slot and enough free memory; the result gives the slot back. Past maxWaitMs it goes ahead without one. */
export const acquireSlot = async (options: SlotOptions): Promise<() => void> => {
mkdirSync(options.dir, { recursive: true })
const started = options.now()
let told = started
for (;;) {
const held = holders(options)
const free = options.freeBytes()
if (free >= options.minFreeBytes) {
const path = take(options)
if (path) {
// The worker runs its next test file in the same process, which must ask for a slot again.
process.env[SLOT_HELD] = "1"
return () => {
delete process.env[SLOT_HELD]
giveBack(path)
}
}
}
const waited = options.now() - started
if (waited >= options.maxWaitMs) {
options.log(`test slots: waited ${Math.round(waited / 1000)} s, running without a slot`)
return () => {}
}
if (options.now() - told >= 30_000) {
told = options.now()
options.log(
`test slots: waiting, ${held.length} of ${options.slots} held (pids ${[...new Set(held)].join(", ") || "none"}), ` +
`${(free / GB).toFixed(1)} GB available of ${(options.minFreeBytes / GB).toFixed(0)} GB wanted; ` +
"WIRECAT_TEST_SLOTS=0 turns this off",
)
}
await options.sleep(options.pollMs)
}
}
14 changes: 12 additions & 2 deletions src/testing/setup.ts
Original file line number Diff line number Diff line change
@@ -1,12 +1,16 @@
import { afterAll, afterEach } from "vitest"
import { acquireSlot, slotOptions } from "./machine-slot.js"
import { installNetworkGuard } from "./network.js"
import { enterSandbox, installThrowingKeyring, takeKeyringReach } from "./sandbox.js"

/**
* One vitest `setupFiles` entry for every WireCat CLI: `setupFiles: ["@wirecat/cli-base/testing/setup"]`.
* Each test file gets its own sandbox directory, removed when the file is done; the real keyring and
* the network stay out of reach for the whole worker.
* the network stay out of reach for the whole worker. Off CI, a file first waits for one of the machine's test slots
* (machine-slot.ts), so suites that several agents start at once share the cores and memory.
*/
const slots = slotOptions()
const release = slots ? await acquireSlot(slots) : () => {}
const sandbox = enterSandbox()
installThrowingKeyring()
installNetworkGuard()
Expand All @@ -16,4 +20,10 @@ afterEach(() => {
if (stack) throw new Error(`the test passed only because the keyring fell back to a file:\n${stack}`)
})

afterAll(() => sandbox.restore())
afterAll(() => {
try {
sandbox.restore()
} finally {
release()
}
})