Skip to content

v0.1.6: implement and qualify mixed-development benchmark families, 10/100-commit histories and concurrent workspaces #122

Description

@yifanxuaaa

Objective

Implement the v0.1.6 benchmark families and shared infrastructure, complete successful performance and independent verification runs for every declared case, and report the actual measurements and retained failures honestly in this issue. Fix measured infrastructure/product defects through their shared causes; do not weaken workloads or verification to obtain a pass.

This is the tracking issue for all six new/changed families below. It binds their roadmap scope; separate family issues are not required unless useful for execution. Website publication, tagging and releasing v0.1.6 are outside this issue.

Source of truth and initial status

The owner-reviewed roadmap is in docs/roadmap/0.1/0.1.6/:

  • README.md
  • benchmark-families.md
  • fixtures.md
  • workloads.md
  • execution-and-verification.md
  • review-decisions.md
  • cases.json — all 36 expanded cases
  • check_plan.py — product-free arithmetic/membership checks

At issue creation, these are local planning artifacts in the main working checkout and are not yet committed/pushed. They describe planned, unimplemented, unqualified benchmarks. The plan checker passed; no v0.1.6 product/performance/verification result is claimed. Preserve and commit the exact roadmap before benchmark implementation or sampling, then add immutable commit links here. Read benchmark/AGENTS.md, docs/general/benchmark_rules.md and current benchmark/fs-bench-pro/QUICKSTART.md first. Coordinate with existing release work and the measurement lock.

Regular scope: 33 cases in six families

Family Action New regular cases Required coverage
dedup_branch_history Extend 6 Large-file hotset, compact namespace/inode history, boundary-crossing history; each K10/K100. Reuse existing small-file history cases instead of duplicating them.
file_size_transition New 7 4 KiB, 131071 B, 131072 B, 131073 B, 1 MiB controls; size roundtrip; alias-aware roundtrip.
mixed_load_bearing Extend 4 Sequential mixed development: L100/L500 × 10/100 commits.
multi_workspace_development New 4 L100/L500 × 10/100 commits per branch, two concurrently live workspaces, one discard/reopen while peer is dirty.
branch_development New 6 Four mixed historical-fork cases plus compact convergent-content and descendant-fork controls.
historical_access Extend 6 Exact before/after size transition and inode replacement states, historical fork point and divergent head.

Every regular case needs a separate perf result and verify result. Twelve are mixed load-bearing cases: three workloads × two sizes × two depths. Existing benchmark identities/evidence stay intact.

Exact load-bearing environment

Setting L100 L500
Maximum non-directory paths per view 5,000 30,000
Maximum logical path bytes per view 100,000,000 500,000,000
Initial regular paths/inodes 4,984 29,984
Initial logical bytes 99,934,464 499,934,464
Reserved temporary/link capacity 16 names / 65,536 B 16 names / 65,536 B
Empty files 34 284
Tiny files 3,950 × 4,096 B 23,700 × 4,096 B
Medium files 706 × 28,146 B + 244 × 28,147 B 3,736 × 22,566 B + 2,064 × 22,567 B
Threshold files 131071 / 131072 / 131073 B Same
Moderate large files 46 × 1 MiB 195 × 1 MiB
Anchors 1 × 8 MiB 2 × 32 MiB
Initial/maximum directories excluding root 62 312

The path cap includes hardlink aliases, symlinks and temporary files. Logical path bytes charge each hardlink name its referent length and symlinks their target-string bytes. Report distinct-inode bytes, physical Store growth and open-unlinked/spool allocation separately. These are per-view limits, not a claim that retained history or all branch views fit one physical-size cap.

  • macOS host owns the one shared SQLite Store, SDK/coordinator, canonical publication and spool.
  • One Linux Docker container owns daemon/FUSE/workload helper only: 2 CPUs, 2 GiB RAM, no swap, 256 PIDs total.
  • All concurrent workspaces share that container/resource budget, with distinct roots /workspace/a, /workspace/b (extension adds c/d).
  • Record host resources separately; container quotas do not cap the host.
  • Reuse compatible sealed builds and pristine inputs; independent writable byte copy per mutation invocation. No cached post-edit histories or live workspaces.
  • Seeds 1/2/3; deterministic structured/random content, exact/local/unique reuse cohorts, hot/rotating target selection. K10 is the exact prefix of K100 under the same topology/seed.

Mixed workload: five commits per cycle

  1. Bulk-refresh 64 files: delete then create, with 32 recurrent, 16 local variants and 16 unique contents.
  2. Touch 16 files spanning tiny/medium/1 MiB/large-anchor/threshold sizes: POSIX overwrites, append/truncate and two single-file SDK range edits for the large pair.
  3. Move two populated 32-file directories; replace eight empty directory names.
  4. Create eight hardlinks and four symlinks; four alias writes; chmod 16 files/four directories; set 16 mtimes.
  5. Delete/recreate a 32-file subtree; test open-unlinked descriptor lifetime; four atomic saves over linked destinations; remove aliases/symlinks.

Every stage must produce a real Created commit. Track UpToDate, Busy, HeadMoved, presentation failure and errors distinctly; do not count attempted/no-op commits toward depth. SDK batches are same-file only: cross-file pairs require two ordered calls and are not atomic batches. Close stage helpers/writable handles before Commit. Use supported chmod/mtime behavior; do not claim persistent xattrs/chown/ACLs.

Sequential: K10/K100 new commits. Concurrent: two branches with K10/K100 each, 20/200 total. Historical-fork: trunk10, children fork from trunk5 and each add K10/K100, 30/210 total, longest ancestry15/105. Follow exact schedules/operation counters in the roadmap.

Deadlines and extensions

Every regular perf invocation and its separate verify invocation must complete in 15 seconds total, including acquisition, authentication, independent copying, runtime readiness, all scheduled operations, receipts and cleanup. Worker deadline12s reserves3s for cleanup. No hidden setup, duration-filling loops, timed soak, selective retries, shortening K100, or moving a valid miss to extended.

This issue explicitly requests successful runs of all three declared extensions as well; they remain suppressed in regular defaults:

  • v016-mixed-exhaustive-100mb-5000-k100-v1: verify every payload/state across101 roots, 120s complete watchdog.
  • v016-mixed-exhaustive-500mb-30000-k100-v1: same at L500, 300s complete watchdog.
  • v016-workspace-four-100mb-5000-k100-v1: four concurrent workspaces,100 commits each,400 total; perf and verify separately,60s each.

These are fixed-work maximum budgets, not durations to fill. No new compaction or automatic optional repository-history replay.

Implementation and completion checklist

  • Freeze/commit the roadmap; bind this issue and exact case/version membership; link immutable specification commit.
  • Implement shared registry/selection, fixture preparation, resource/deadline supervision, multi-session coordination, operation receipts and independent oracles using existing infrastructure.
  • Implement all six families, preserving inherited semantics and versioning changed contracts.
  • Product-free tests cover fixture/path/byte algebra, exact cardinality, Created counts, ancestry, operation routes, regular/extended selection and perf/verify separation.
  • Meaningful runtime tests cover inode lifetime, same-file SDK limitations, branch isolation, discard/reopen, deterministic recurrence, timeout and cleanup.
  • Complete at least one successful selected perf+verify for every one of33 regular cases, then the declared seeds1/2/3 qualification and matched source-arm campaign where semantically comparable.
  • Complete both verify-only exhaustive extensions and perf+verify for the four-workspace extension. Verify-only modes have perf marked N/A, not zero or PASS.
  • Correctly produce/seal historical-access and exhaustive-proof inputs; missing input is NOT_READY, never fabricated verification success.
  • Resolve valid failures through root-cause fixes; rerun affected evidence on the exact final candidate; run affected inherited regressions.
  • Publish per-case measurements and overall matrices here with raw evidence manifests and exact source/binary/image identity.
  • Keep issue open if required rows fail, time out, remain unsupported or are not run. No release/tag is requested.

Required honest reporting

Post progress and final evidence comments to this issue. Retain initial failures and fixes; do not overwrite raw results or select only favorable attempts. For each case/seed/arm/mode report:

  • Exact case/version/configuration, requested/observed paths/bytes, branch/workspace count, local commits, total commits and ancestry.
  • PASS/FAIL/TIMEOUT/NOT_READY/NOT_RUN/N/A with separate correctness, timing, resource, cleanup and custody status.
  • Complete command wall and separate preparation/workload/cleanup; edit/Commit/fork timings in raw units, sample count, median and min–max; max Commit latency and actual concurrency overlap.
  • Logical and physical bytes, Store growth, dedup/reuse and spool; host and container CPU/memory/I/O separately, with availability/sampling caveats.
  • Verification coverage and omissions; full-payload versus affected-set verification explicitly distinguished.
  • Source/product/harness/fixture/oracle/image seals, unique artifact paths, manifests, reproducible commands and before/after evidence for fixes.

Publish a machine-readable full matrix plus readable family/case tables. Selected diagnostic passes are not full admission. Report unpaired/incomparable baselines honestly; no speedup without matched evidence. A failed15s gate remains a failure even if the inner Commit is fast. Success requires all applicable required rows and gates, not a success-shaped summary.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions