Conversation
x86_64 and aarch64 now lower fetching bitwise arena atomics (OR/AND/XOR with BPF_FETCH), the one arena shape both backends previously left to the interpreter. x86 has no single-instruction atomic fetch-and/or/xor, so it runs a `cmpxchg` retry loop. That loop needs `rax` (which holds R0) plus a scratch register, but the arena fault ABI already claims `rcx` (the recovery handle) and `r11` (the effective address), and the loop must not move `rsp` (the fault epilogue reads the parked arena base and register saves relative to it). The fix reserves one frame word — `ATOMIC_SPILL_SLOT`, widening the prologue to `ATOMIC_SPILL_RESERVE` only for programs that contain the shape, keeping the residue ≡ 8 (mod 16) so the SysV call-alignment argument is unchanged — to preserve R0, and reuses `rcx` as the loop's candidate register: `rcx` is needed only up to the comparand load, the sole faulting access, after which the line is proven mapped and the handle is spent. aarch64 already lowered the shape correctly through `emit_atomic_mem` — the fetch OR/XOR land the old value in a scratch that is not the recovery handle, and fetch AND writes the handle register only after the faulting `LDCLR` (a faulting LSE atomic takes the exception before writeback) — so the over-conservative guard is simply removed. Tests: host goldens for the x86 spill-slot lowering (including the src == R0 operand-from-spill path) and the aarch64 LSE forms; `generate_arena` now emits the full atomic repertoire so the native-vs-interpreter differential covers it (48/48 verified programs compiled, 100%); and the former unlowered-shape probe becomes a pinned positive differential. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
An arena access in a program that also makes BPF-to-BPF calls previously ran interpreted: the arena base was parked at a fixed offset from a stack pointer a call moves, so a callee's access read the wrong slot, and an arena fault or out-of-fuel exit reached from a callee resumed at the single entry-frame teardown with the stack several frames too deep. x86_64 closes this by claiming its one free host register (`r10`) as an anchor to the entry frame's `rsp`. An arena access reaches its base through `[r10 + slot]` at any call depth, the fetching-bitwise atomic's R0 spill moves to the same anchor, and the arena-fault / out-of-fuel epilogues do `mov rsp, r10` before unwinding so a fault inside a subprogram recovers correctly. `r10` is caller-saved, so kfunc calls wrap it with a balanced save/restore; BPF-to-BPF calls need no wrapper because the callee is this crate's own code and only reads it. aarch64 has no equally free register without growing every arena frame, so it composes arena with calls for accesses in the main program and leaves an access inside a subprogram interpreted — correct, just not native. The gate that refused every arena+subprog program is removed; both backends are now sound for the whole class. Tests: the x86 goldens move to the anchored addressing and the rsp-resetting fault epilogue; a new kernel test runs an arena store+load inside a subprogram both in bounds and out of bounds, requiring the native run to match the interpreter on value and on trap discriminant. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Implement the XDP_TX (reflect) and XDP_REDIRECT (forward) actions as
retransmission of the unmodified RX frame, without widening RxHandler or
any driver RX path to &mut. This is a real bump-in-the-wire capability
reachable with the frame kept read-only.
- XdpAction gains Tx and Redirect{ifindex}; BpfXdp::run maps Linux return
codes 3->Tx and 4->Redirect, other codes stay Aborted.
- XDP_REDIRECT conveys its target through a new bpf_redirect(ifindex,flags)
kfunc that stashes the ifindex in a per-CPU slot (sound because XDP runs
with IRQs masked under XDP_PROGS, so the CPU cannot change between the
call and the classifier's read). BpfXdp::run clears the slot before each
run and reads it on a return of 4; a bare return-4 with no prior
bpf_redirect is Aborted.
- classifier Verdict gains Transmit and Redirect{ifindex}; classify maps
the new actions and run_xdp does not transmit under the lock. New
XDP_TXS/XDP_TX_DROPS counters + xdp_tx_stats()/count_xdp_tx_drop().
- iface gains send_on(name) and send_on_ifindex(ifindex); tcp_stack::rx
transmits after classify returns, counting send failures as drops.
- In-place packet mutation (writable data, bpf_xdp_adjust_head) and
devmap/cpumap fan-out remain a follow-on; docs updated accordingly.
Tests (kernel-side smokes): XDP_TX maps to XdpAction::Tx; bpf_redirect(N)
then return 4 maps to Redirect{ifindex:N} and a bare return-4 aborts; an
end-to-end link.rs smoke drives the real classify() and asserts
Verdict::Transmit / Verdict::Redirect.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Admit a bare BPF_LDX load through a schema-tracked trace-object pointer directly, without routing through the narf_probe_read mediator, while preserving the mediator's exact safety property: the load is admitted only when the verifier proves the base is a PtrClass::TraceObject pointer at offset zero and the (offset, size) names one exact declared field of its schema. A non-field offset, a truncated width, a variable offset, a store, or an untracked/opaque object pointer is rejected (VerifyError::TypedFieldMismatch, or OpaqueDeref for an ordinary Object) -- never lowered to a raw dereference. The base register holds the tracing wrapper, not the object, so the field lives one indirection past it and the certified load is not a bare host dereference. It is recorded as a TypedLoadSite (disjoint from BareAccessSite), so jit_glue gate 5 refuses the program and it runs interpreted; the interpreter reads the field through the live TypedProbeRef and copy_field repeats the field and whole-object bound checks at runtime, keeping the mediated path's defense-in-depth. Runtime path is interpreted, not JITted: a bare native deref of the wrapper pointer would read the wrapper struct rather than the object field, so a JIT lowering would be wrong, not merely complex. Tests: verifier host tests for a proven field load (verifies, recorded as a typed load and not a bare access), an in-object non-field (rejected), and an ordinary object pointer (rejected); in-kernel smoke_bpf_typed_probe_direct_field_load_reads_bytes proving the direct load reads the exact field bytes, runs interpreted, and declines a forged raw context; and the existing direct-deref rejection test updated to the new field-gated behavior. Docs: domain-confinement.md and bpf/README.md updated to reflect that direct typed-field loads now land. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
BPF hardware confinement was PKS-only: `bpf/src/domain.rs::enter()` gated on `pks::is_active()` and left execution unconfined on every other platform. It now enters the BPF domain through the unified enforcer on each backend. x86_64 calls `Pks::enter_domain(FRAME, BPF)`, which dispatches to PKS on Intel SPR+ or, when PKS is absent, to **PCID** on AMD / pre-SPR Intel — a `CR3` swap into the BPF domain's PML4. That PML4 is a byte-clone of the bootstrap tables, so BPF's own kernel-VA regions (per-CPU stack, arena, JIT text) stay mapped while the domain's private VA range is its own. The gate widens to `pks::is_active() || pcid::is_active()`, so an unconfined platform still constructs an inert guard and pays nothing. aarch64 calls `Mte::enter_domain(FRAME, BPF)`, a structural `SCTLR_EL1`/`GCR_EL1` save today; real tag-fault enforcement pairs with the Stage-3 MTE-tag-aware allocator. As with PKS, the mechanism is complete on each backend and the isolation strength grows as subsystems move state into private domains. Validated under KVM: the boot reports `domain enforcer: pcid (CR4.PCIDE=1, 16 PML4 clones, ...)` and the full bpf kernel suite runs green (208 pass, 0 fail) with every `run_atomic` — arena JITs, fetching-bitwise atomics, subprogram arena accesses, XDP, typed-field loads — executing correctly behind a real CR3 swap into the BPF PML4. Both arches lint clean on the bare-metal target with `-D warnings`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
XDP exposed the packet frame read-only: `XDP_TX`/`XDP_REDIRECT` retransmitted the *unmodified* frame and a store through `data` was rejected. A program can now rewrite header bytes in place, and the modified frame is what `XDP_TX` reflects and `XDP_REDIRECT` forwards. Verifier: drop `ArgFlags::READONLY` from `XDP_CTX[0]` (the `data` pointer), keep it on `[1]` (`data_end`). This is the whole verifier change — a `PtrClass::Mem` write is already bounded against `data_end` by the same `p.size` interval check `access()` applies to a read (`fixpoint.rs`), gated only by `p.readonly`. `data_end` stays read-only and non-dereferenceable. Interpreter: the hook region is now `&mut [u8]` (`packet_region`). The store path services a bounded packet write with the read path's exact bound — `checked_add` on both sides, a write only inside `[data, data_end)`, one past either edge traps (`BadAccess`) and never writes. Fail-closed. JIT: no change needed. The verifier publishes the bounded store as a `BareAccessSite`, so `scan_program` admits it and the existing non-arena `Decoded::Store` lowering emits a native `mov [base+disp], src` against the proven-in-bounds `data` address — symmetric to how bounded packet reads already JIT. A packet-writing program compiles rather than falling back. net + drivers: thread `&mut [u8]` through `RxHandler`, `on_rx_frame_from`, `tcp_stack::rx_handler`, `classifier::classify`/`run_xdp`/`XdpProgram::run`, and `BpfXdp::run`. The e1000 and virtio-net RX call sites hand over a `&mut` borrow of the buffer they own and recycle (e1000: a stack scratch copy; virtio: the owned `DmaBuffer` before it is recycled) — the device is not reading it back, so an in-place write before the stack copies the payload out is sound. `XDP_TX`/`XDP_REDIRECT` re-borrow immutably after `classify` returns, so they retransmit the possibly-modified bytes. `BPF_PROG_TEST_RUN` now copies the post-program frame back to `data_out`, matching Linux. Tests: verifier host-tests prove a bounded packet store verifies (and is a native bare access), an unchecked store / a store past the proven bound / a store through `data_end` are rejected, and a store through a still-READONLY `data` is rejected. A kernel smoke rewrites `frame[12]` and asserts the buffer reflects the write after `run_xdp_interpreted` and `BpfXdp::run`. All existing XDP read/TX/REDIRECT tests stay green. Scope: writable frames only. `bpf_xdp_adjust_head`/`_tail` (head/tailroom), devmap, and cpumap remain documented follow-ons. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The "Hardware confinement (design)" narrative still called the fence a no-op on AMD PCID / aarch64 MTE and listed those backends as remaining; both are wired now (PCID does a CR3 swap into the BPF PML4, MTE a structural save). What remains is subsystem private-domain tagging and the Stage-3 tag-aware allocator. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
XDP frame resizing. `bpf_xdp_adjust_head`/`_tail` move the packet's `data`/`data_end` to trim or grow the frame, matching Linux's helpers: adjust_head with delta>0 shrinks from the front, delta<0 prepends headroom; adjust_tail with delta>0 appends tailroom, delta<0 trims. Both return 0 on success or -ENOMEM (fail-closed, pointer unmoved) when there is no room. Grow is supported via a per-CPU staging buffer laid out `[headroom | packet | tailroom]` (256B headroom, matching Linux's XDP_PACKET_HEADROOM). A bare RX frame has no slack — the driver hands a slice sized to exactly the packet — so a resizing program (known from a load-time flag off the verifier's kfunc-call sites) runs against the staged buffer instead, with data/data_end pointing at the packet sub-range. The resulting `[data, data_end)` window is copied back into the caller's frame and its length threaded through XdpProgram::run → classifier → rx_handler, so XDP_PASS/TX/REDIRECT and BPF_PROG_TEST_RUN all act on the resized packet. A non-resizing program pays no staging copy: the common RX path is unchanged. The two kfuncs are interpreter intrinsics (like the ring-buffer ones): in the interpreter the context is synthetic, so a plain shim cannot write ctx[0]/ctx[1] — the interpreter intercepts the ids and mutates the VM's context words and packet window directly. The JIT refuses any program that calls one, so it always runs interpreted; the program re-reads the moved pointers from the ctx on its next load. Verifier: a new PtrKind::Ctx kfunc argument (Rust type XdpCtx) types the mutable context pointer and is the structural marker for packet-bound invalidation — after a call to a kfunc taking one, every proven packet extent is struck off every PtrClass::Mem packet register, so a subsequent packet access without a fresh `data < data_end` comparison is rejected (Linux's clear_all_pkt_pointers). Tests: verifier host tests prove access-after-adjust-without-recheck is rejected and re-check re-certifies; kernel smokes cover adjust_head(+14) shift-and-read, adjust_tail(-4) trim, a grow within headroom, and a grow past headroom returning -ENOMEM. All existing XDP tests preserved. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The adjust_head/tail smokes could not run from the worktree they were authored in; two only-at-boot bugs surfaced on integration. adjust_tail: the short-frame branch offset jumped to the main-body EXIT, where R0 is uninitialised on that path, so the verifier rejected the load. Target the short-frame PASS (which sets R0) instead. grow-within-headroom: delivery is bounded by the caller frame's capacity, so a 64-byte frame cannot report a 96-byte window. Assert the honest observable — the kfunc succeeds (PASS, not -ENOMEM) and the grown head shifts the original byte 0 to offset 32 behind zeroed prepended headroom. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add BPF_MAP_TYPE_DEVMAP (14): a dense u32-keyed table of 4-byte interface indexes, backed by the array storage verbatim (one value per key, value_size == 4 enforced at creation). bpf_redirect_map(map, key, flags) looks the slot up and, on a hit, arms the ifindex as this CPU's XDP redirect target — the same per-CPU slot bpf_redirect writes, now factored into set_xdp_redirect_target — and returns XDP_REDIRECT. A miss (not a devmap, key past max_entries, or an empty slot) returns the program's flags fallback action; a flags value that is not a bare XDP action aborts, matching Linux's devmap helper. Unlike the ring-buffer and frame-resize kfuncs it is an ordinary shim, not an interpreter intrinsic: the map handle is a real kernel address on both backends (LD_IMM64's map pseudo-form resolves to it), so the JIT lowers the call natively and the interpreter dispatches through the registry. A runtime devmap-kind check stands in for Linux's load-time map-type/func-id compatibility check, which NARF's one kfunc ABI omits. Tests: devmap shape/lookup at the map layer, and a JIT-vs-interpreter differential proving a hit arms ifindex + returns REDIRECT and a miss returns the fallback with no target armed (both paths agree, program JITs). BPF_F_BROADCAST fan-out and cpumap remain a follow-on. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add BPF_MAP_TYPE_CPUMAP (16): a dense u32-keyed table whose key is a
target CPU and whose 4-byte value marks that CPU a live redirect target
(standing in for Linux's struct bpf_cpumap_val queue config). Backed by
the array storage, like devmap.
bpf_redirect_map now serves both redirect map kinds, dispatched on the
map kind at the (runtime) call: a devmap arms the looked-up ifindex, a
cpumap arms the key CPU. The per-CPU redirect slot is now kind-tagged
(RedirectTarget::{Iface,Cpu}), widened to u64 with the kind in the high
bits so a target value of 0 stays distinct from the no-request sentinel;
set_xdp_redirect_target split into set_xdp_redirect_iface/_cpu.
BpfXdp::run decodes the tag: a devmap redirect stays XdpAction::Redirect
{ifindex}, a cpumap redirect becomes XdpAction::RedirectCpu{cpu}, which
the classifier delivers to the local stack (the running CPU is NARF's
only RX-processing context) and counts under XDP_CPU_REDIRECTS. Linux's
cross-CPU steering is the documented degradation.
Tests: cpumap shape/target at the map layer; a JIT-vs-interpreter
differential proving a hit arms the CPU + returns REDIRECT and a miss
returns the fallback; and an attach-layer test proving devmap and cpumap
redirects reach Redirect{ifindex} vs RedirectCpu{cpu} distinctly.
BPF_F_BROADCAST fan-out remains the sole follow-on.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
bpf_redirect_map(devmap, key, BPF_F_BROADCAST) now fans a frame out to every live port of the devmap (the key ignored), with BPF_F_EXCLUDE_INGRESS skipping the iface it arrived on — the last follow-on from the redirect-map work. The redirect slot gains a third kind, RedirectTarget::Broadcast, and a per-CPU port buffer the shim stages the devmap's live ifindexes into (BpfMap::devmap_ports enumerates them, capped at MAX_BROADCAST_PORTS=16 to stay allocation-free on the run path). attach_xdp bridges the staged ports into a net-side per-CPU buffer and returns XdpAction::Broadcast; the classifier maps that to Verdict::Broadcast, and tcp_stack drains the ports after the XDP_PROGS lock releases and sends to each, skipping the ingress ifindex (resolved via the new iface::ifindex_of) when asked. The staging buffers are safe across the lock release because NARF drains RX one frame at a time in a cooperative poll loop, not by nested rx_handler re-entry — the same invariant the single-target redirect slot documents. Since NARF cannot deliver a grown packet beyond a caller frame anyway, fan-out is the honest maximal semantics here; cross-CPU/queue steering stays the documented cpumap degradation. Flag validation: BPF_F_BROADCAST is devmap-only, rejects a stray flag bit or an empty devmap as XDP_ABORTED, and a single-target call still takes only a bare fallback action. Tests: devmap_ports enumeration (skip-empty, cap-at-buffer, devmap-only); a JIT-vs-interpreter broadcast differential (arms N ports, records exclude-ingress, stages the live ifindexes); bad-shape rejection (cpumap, empty devmap, stray flag); and an attach-to-net-staging round-trip proving XdpAction::Broadcast + the sender drains the staged ports. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.