Skip to content

[Follow-up to #49] Restore upstream fuser behavior and minimize vendored patches #51

Description

@yifanxuaaa

Latest explicit user instruction — active, supersedes conflicting guidance below: completely withdraw ALL local fuser patches and return to an unmodified upstream dependency. Drop the buffer/thread changes and cache-retrieval additions; migrate LayerFS callers using supported upstream APIs, then remove the vendor directory/Cargo override and obsolete references. Do not relocate patches into a fork or custom protocol implementation. Preserve the shared Workspace architecture and required coherence/concurrency/error behavior. Take the fastest correct route to finish: no new frameworks, speculative extraction or optional optimization campaign; focused changed-seam checks, existing real create/delete qualification and final selected proofs. If a reproduced upstream limitation prevents correctness, report exact evidence promptly rather than retain patches or weaken behavior. Earlier permission to retain a minimal local extension is superseded. Existing targets and evidence requirements remain.

Objective and scheduling

Active requirement for #49: completely withdraw every local fuser patch and return to an unmodified upstream dependency. Finish this through the shortest correct implementation route as part of the current architecture migration. Preserve the shared-live-Workspace architecture; no new optimization gate.

Latest coordination instruction: the user explicitly requested notifying implementation task 01a07516-9163-7da3-82ef-84bd986e6630 to drop all patches and take the fast lane to finish; it has been notified. This supersedes the earlier deferred/no-notification direction.

User rationale

  • Approximately 1.6 GiB of receive-buffer capacity for 100 mounted workspaces is acceptable to the user. This is source-derived allocation arithmetic, not measured resident memory or the total deployment footprint.
  • One thread does not mean one CPU core. The extra upstream supervising thread mostly waits; removing it has not demonstrated a material create-100 latency benefit.
  • Many workspaces may be distributed across many containers. The benchmark's 256-PID/task limit applies per container, not as a shared limit across all containers. Host/VM resources and any ancestor limits still apply; do not assume all 100 mounts share one benchmark container.
  • Prefer standard dependency behavior over maintaining resource patches whose benefits are unnecessary for the intended deployment.

Current dependency change to reassess

Before 09d9325b6d5a39a2f3567978a61c1f4c497a7f18, LayerFS already used crates.io fuser = 0.18.0. That commit copied the released package into crates/vendor/fuser and selected it through root [patch.crates-io].

The original maintained changes were:

  1. src/session.rs: run the single configured receive loop directly rather than retain a supervising thread waiting for a separately spawned receive worker.
  2. src/read_buf.rs and negotiation handling in session.rs: derive Linux receive capacity from negotiated supported request limits instead of the upstream approximately 16 MiB + 4 KiB allocation. The current LayerFS configuration derives approximately 1 MiB per loop.

Later in-progress edits expose Notifier::retrieve, Filesystem::retrieve_reply, notification encoding and validated reply decoding for cache-coherence work. Their final disposition must be inspected on completed #49; they cannot be removed merely because the buffer/thread patches are reverted.

Relevant paths: root Cargo.toml / Cargo.lock, crates/layerfs-fuse/Cargo.toml, crates/vendor/fuser/LAYERFS-PATCH.md, crates/vendor/fuser/CHANGELOG.md, vendored src/session.rs, src/read_buf.rs, src/notify.rs, src/lib.rs, src/request.rs, src/ll/notify.rs, src/ll/request.rs, src/ll/fuse_abi.rs; LayerFS callers in crates/layerfs-fuse/src/filesystem.rs and live_owner.rs. Resolve callers again on the final source.

Proposed scope

  1. Inventory the final vendor diff against its exact released upstream version. Separate receive-buffer sizing, thread topology and required cache-coherence API changes. Use a reproducible comparison; do not count copied upstream source as new LayerFS implementation.
  2. Restore upstream receive-buffer and single-session thread behavior together as one reviewable dependency-simplification candidate, unless a concrete supported-behavior issue requires a smaller sequence. Revert corresponding patch-only tests and documentation when no longer applicable; preserve useful behavior checks.
  3. Determine whether the required cache-retrieval/coherence functionality is available in a suitable upstream release or can use an existing supported upstream API. Do not assume it is available. Prefer a narrowly scoped compatible upstream update if validated; avoid an unrelated upgrade campaign.
  4. If all required behavior is supplied upstream, remove the vendored directory and root Cargo patch, resolve the registry dependency/lockfile through Cargo, and remove obsolete patch documentation/build references. Preserve licensing for any third-party source that remains.
  5. If cache retrieval still needs a local extension, retain only that documented necessary delta and explicitly report partial restoration. Do not replace it with a custom FUSE decoder, new kernel protocol implementation, weaker mapping semantics or duplicated live authority merely to delete the vendor folder. Full de-vendoring remains a conditional outcome, not permission to break correctness.

Validation and stop rules

  • Reuse the existing [v0.1.3] Redesign FUSE Exec around one shared live Workspace core #49 final source and evidence; do not run a broad benchmark campaign or physical 100-workspace experiment.
  • Check dependency resolution and focused affected session lifecycle, mount/unmount, request capacity, exactly-once destruction and cache/mapping coherence behavior. Include concurrent shell/SDK edits and Commit checks where the changed dependency boundary can affect them. Reuse existing checks rather than build a new harness.
  • Use the existing host SQLite + Docker Linux + real FUSE topology, no data mounts, existing container resource limits and matched host/image source seals. Keep shared-lock execution serial and reuse setup through --setup clone.
  • Collect one unchanged create-100 sample for the substantive candidate and a same-source delete control if retained. Inspect complete lifecycle, cleanup, thread count and memory with their actual scopes. Do not search for favorable repeats or claim a speedup is required from removing the patches.
  • Final selected independent proofs stay final-only, 45 s work / 59 s hard per proof, preparation/builds separate. Existing performance acceptance applies; no tighter target or new headroom requirement. A material regression must be reported and investigated, not hidden as simplification.
  • Keep CAS/CDC/COW/rope/extent construction, live-owner architecture, stable nodes/handles, concurrency, kernel-cache coherence, recovery and all existing resource accounting intact. Accepting larger receive buffers does not authorize unbounded queues or payload caches.

Done when

  • Upstream receive-buffer and thread behavior is restored, or a concrete reason for retaining either is documented for a separate decision.
  • Vendoring is removed if upstream can satisfy all required APIs; otherwise the remaining minimal patch and its removal condition are documented honestly.
  • Dependency/caller documentation and checks match the resulting code, with no stale claim that only two files differ if a cache-retrieval extension remains.
  • Correctness and scoped product validation pass with explicit resource/performance evidence. No change to [v0.1.3] Redesign FUSE Exec around one shared live Workspace core #49's architecture or expansion of its active work.

Related: #50 tracks later Exec overhead recovery. Coordinate against the final baseline when these deferred tasks are eventually undertaken; do not attribute unrelated performance changes to dependency restoration.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions