Skip to content

feat(benchmarks): phase 6 - replace pending advanced metric panels #183

Description

@zackees

Parent: #177
Design authority: #176
Phase: 6 of 6
Blocker: #182 must be fully complete, including PR 5B and live digest audit
Branch: none - coordinator issue
PR count: no umbrella PR; one PR per nested metric issue

Context

The first public release may show explicit pending panels for memory, honest latency, scaling, and pprof tax. This coordinator replaces those placeholders with validated real data through four nested work packages.

Do not implement all metrics in one PR. They have different measurement semantics and failure risks. Each nested issue owns a schema extension, RED -> GREEN controls, workflow budget, rendering change, and live verification.

Required order

To minimize schema/render/workflow conflicts, merge in this order:

#184 6A memory/RSS
  -> #185 6B honest latency
    -> #186 6C scaling
      -> #187 6D pprof tax

A later child branches only after the prior child merges and posts its handoff.

Shared rules

Every metric child must:

  • keep the four allocator identities/pins from the live core dashboard unless the issue explicitly defines a fork-only configuration matrix;
  • use isolated child processes and exact operation/checksum oracles;
  • version metric semantics and units;
  • store raw samples, not only summaries;
  • define higher/lower-is-better direction;
  • reject invalid/missing/mixed-host data;
  • use paired complete blocks and the Phase 3 statistical implementation where comparisons apply;
  • add comparison-key inputs for every new compatibility field;
  • preserve existing history lineages rather than rewriting old rows;
  • keep the panel pending until the first complete validated live run;
  • update workflow runtime projections and stay within hard limits or use a separate bounded scheduled/manual workflow;
  • add no shipped C dependency or C-core path change;
  • make hosted data informational;
  • update renderer/HTML/methodology/manifest/site tests;
  • publish only through the existing sealed-artifact/eligibility pipeline.

Schema extension policy

Do not mutate benchmark-latest-v1 semantics in place.

Each child must either:

  • add a backward-compatible optional metric object with its own metric_schema_version and explicit pending|complete|invalid status, or
  • bump the containing schema when backward compatibility is impossible.

Old history rows remain valid and display not collected under this metric version, not zero.

Comparison-key rules:

  • core throughput lineage is not broken by adding an optional metric object;
  • the new metric receives its own metric comparison key including protocol-specific fields;
  • charts connect only identical metric keys.

Publication transition

For each child:

  1. merge code with panel still pending by default;
  2. run complete scheduled/manual collection on default branch;
  3. validate and publish through existing pipeline;
  4. verify branch/Pages digest and raw artifacts;
  5. only then mark the panel complete for that run/lineage;
  6. record live evidence on the child issue.

Never hand-edit generated branch assets.

Coordinator acceptance criteria

  • Four metric children are attached as native sub-issues.
  • Each child is merged in the required order with its own branch/PR.
  • Memory panel reports validated OS-specific memory metrics and fragmentation proxy.
  • Latency panel reports an honest measured latency distribution with quantified instrumentation overhead.
  • Scaling panel reports declared thread points with topology/affinity metadata.
  • Pprof-tax panel separates upstream, fork OFF, compiled/stopped, active rates, and frame-pointer controls.
  • Every panel has raw samples, metric version/key, complete/pending/invalid status, units/direction, and informational label.
  • Old history remains readable and never gains fabricated zeros.
  • Final live branch and Pages digest match after all four metrics.
  • Workflow budgets and all repository gates pass.
  • Coordinator handoff closes chore(benchmarks): implementation guide for the public benchmark dashboard #177 only after a final perf(benchmarks): publish a statistically valid scheduled dashboard #176 acceptance audit.

Decisions

  • Phase 6 is a coordinator because one umbrella implementation would be too large and statistically incoherent.
  • Merge metric children serially to keep schema/workflow/render changes reviewable.
  • A metric remains pending indefinitely rather than shipping an approximation under a stronger label.
  • Optional metric versions preserve core throughput history compatibility.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions