fix(hot-reload): make it work on Linux, and pin down the one-flecs invariant - #1079
Merged
Conversation
--export-dynamic cannot undo what rustc does to a dylib. rustc links one with its own anonymous version script ending in local: *, which demotes every symbol it did not generate, so flecs C symbols sit in the object with DEFAULT visibility and LOCAL binding -- present and unreachable. Measured on x86_64-linux: 9001 exported symbols, zero of them ecs_*, ecs_init reading FUNC LOCAL DEFAULT. --export-dynamic-symbol=ecs_* is equally powerless against a version-script demotion; both were tried. The crate was therefore not merely untested on Linux, it did not work there at all. The demo host fails to link: rust-lld: error: undefined symbol: ecs_progress rust-lld: error: undefined symbol: ecs_ensure_id ld merges multiple version scripts and an explicit pattern beats a * wildcard, so a second script naming these globs promotes exactly them and leaves rustcs own exports alone. After: 10717 exported, 666 of them ecs_*, ecs_init GLOBAL, and demo.sh runs all three cases on x86_64-linux with the same output as aarch64-darwin -- code-only change accepted with state intact, layout change refused with the world still ticking on the old build, migration accepted rewriting 3 instances. Darwin re-verified unchanged.
…dylib boundary Everything the reload gate does assumes the host and a module dylib draw component indices from one pool. Nothing in the reload path checks it, and both sides are internally consistent when they do not, so the world is simply indexed two different ways and no error is raised. The probe is behavioural rather than an address comparison. Comparing ecs_init as usize across the boundary reports a mismatch even when the copy is shared, because an executable taking the address of a dynamically linked function gets its own PLT stub; measured that trap directly, with separate and shared copies both printing different addresses. Allocation order cannot be faked: with one pool, an index taken in the module is strictly greater than every index the host took first. It also catches a false positive worth recording. Before the module referenced the runtime crate at all it linked its own static copy of everything, and two independent pools each starting at 1 made hyperion::simulation::Position read index 1 on both sides. That looks exactly like success. The measurement that distinguishes them is allocation order, not equality. Result with hyperion as a plain rlib -- separate pools: host indices: [1, 2, 3, 4] (max 4) module own type index: 1 Position index: host 4, module 2 SHARED_POOL=false The host is what creates the second copy: it drags flecs_ecs in through hyperions rlib while the module resolves it from the runtime dylib. Result with flecs_ecs and hyperion both dylibs -- one pool: host indices: [1, 2, 3, 4] (max 4) module own type index: 5 Position index: host 4, module 4 SHARED_POOL=true SHARED_HYPERION_INDEX=true PROBE_OK Verified on x86_64-linux (dev-compute-6).
…eploy shape The document claimed the crate was merely untested on Linux. It did not work there, in two independent ways, and both are quiet: exports demoted to LOCAL by rustcs own version script, and a second flecs copy that leaves host and module indexing one world two different ways with no error from anything. Records the two measurements that give the wrong answer (comparing ecs_init addresses across a PLT boundary, and comparing one types index for equality when two pools both start at 1), because both look like success. Also writes down the deploy mechanism: NixOS reloads rather than restarts a unit when only X-Reload-Triggers changed, which is the whole basis for a game-logic change reaching a running server without disturbing a player. States plainly that it is not built.
andrewgazelka/Flecs-Rust f09dc53 gives flecs_ecs crate-type ["dylib", "rlib"] and a build script re-exporting flecs C symbols from the dylib. Hot reloading needs exactly one copy of that crate in the process, because it owns the process-global pool handing out each component types index into a worlds component array. Verified against the pin with no local patch, on x86_64-linux: host took component indices 1..4, the module then took 5, and hyperion::simulation::Position read index 4 on both sides. PROBE_OK. Darwin re-verified: demo.sh exits 0 with the same three verdicts.
Benchmark Results for generalComparing to dd8311f |
Allows print_stdout in the probe host, matching the demo host: a binary whose whole purpose is reporting a measurement. Drops the hyperion-hot-reload dependency from both probe crates. It was unused by cargo-machete's reckoning and, more to the point, the comment claiming that reference was what made the two sides share one flecs was wrong once flecs_ecs became a dylib. Removing it entirely leaves the probe passing: host indices: [1, 2, 3, 4] (max 4) module own type index: 5 Position index: host 4, module 4 PROBE_OK The doc comment now says what actually makes the pool shared, and records that a single index reading equal on both sides is not evidence: two separate pools both start at 1.
Benchmark Results for generalComparing to dd8311f |
The repin commit left `crate-type = ["dylib", "rlib"]` on `hyperion`. That is required for the shared component-index pool, but only together with the rest of the recipe -- `-C prefer-dynamic`, `-Wl,--undefined-version`, `-Wl,--allow-shlib-undefined` -- which a plain `cargo test` does not use. Without them the dylib fails to link: ld: Undefined symbols for architecture arm64 ld: symbol(s) not found for architecture arm64 so the whole workspace stopped building. `cargo test --workspace` now passes again. Making `hyperion` a dylib is a packaging decision affecting every consumer, and it does not belong in a fix PR. The recipe and the measurements that justify it are in docs/hot-reload.md. The probe consequently cannot pass on a default build, which is the honest state rather than a regression: it exists to detect exactly the configuration the repo currently has. Its failure message now says so and points at the recipe instead of just reporting two numbers.
Benchmark Results for generalComparing to dd8311f |
A rules-only change is roughly 2 s of compile and tens of milliseconds of reload; touching the engine is 4.42 s of compile and costs the process, the world and every connected player. That contrast is the argument for the whole design, and it was not written down anywhere. States what the numbers are not: debug rather than release, a whole binary relink rather than a rules dylib, and a 165 KB probe module rather than smash.
Benchmark Results for generalComparing to dd8311f |
Benchmark Results for generalComparing to dd8311f |
Codecov Report❌ Patch coverage is
@@ Coverage Diff @@
## main #1079 +/- ##
==========================================
- Coverage 54.65% 54.59% -0.07%
==========================================
Files 361 363 +2
Lines 33257 33298 +41
Branches 1259 1259
==========================================
Hits 18178 18178
- Misses 14793 14834 +41
Partials 286 286
🚀 New features to boost your workflow:
|
The repin changed the Cargo.lock source string, and outputHashes is keyed by that exact string, so every nix check failed to evaluate: error: outputHashes is missing hashes for git source strings in Cargo.lock: git+https://github.com/andrewgazelka/Flecs-Rust?rev=f09dc53... sha256-DlMOSY7NyoPoR8w4yswm3O97BegcNWcLl/fW3wOAmRs= from nix-prefetch-git --fetch-submodules at that rev.
Benchmark Results for generalComparing to dd8311f |
…iour-only argument Three gaps a stranger would have hit. A behaviour-only module does not avoid the shared-pool requirement. The registration/behaviour split in CLAUDE.md makes it look like it should: a library that registers nothing cannot collide with anything. But registering a component and looking one up are different operations and only the first is avoided -- a system's query still resolves T through T::index(). The probe module registers nothing at all and still read Position as index 2 where the host had it at 4. The split is worth keeping for the hazard it does address, which is component layout, and that is now said explicitly along with the boundary it implies: changing what a system does is a reload, changing a component type is a rebuild and a restart. What makes the pool shared is the dependency being a dylib, not a consumer referencing it. An earlier probe carried a call to AbiToken::current() with a comment claiming otherwise; removing the dependency entirely leaves the probe passing. And the four remaining steps are now written down in order, with the packaging step named as the risky one and the reason why: every measurement here came from a cargo build, not a nix one. Also records that adoption needs no scheduled restart -- the fleet already restarts for version bumps, so the split host takes effect on the next apply happening for its own reasons -- and that the build tree on dev-compute-6 was deleted when the node was released.
Benchmark Results for generalComparing to dd8311f |
Making flecs_ecs a dylib means it emits two artifacts on every build. I tried to measure whether that costs build time and could not confirm the comparison: 8875 ms with both against 8817 ms with the rlib alone is within noise, but a leftover dylib in the target directory means the rlib-only configuration may never have taken effect. Recording it as unmeasured rather than as zero, because the difference between those two claims is what decides whether the next person needs to look.
Benchmark Results for generalComparing to dd8311f |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes
crates/hyperion-hot-reloadwork on Linux, and establishes the linking invariant the whole design rests on. Groundwork for hot reloading a game module onix applywithout disconnecting players; the deployment half is designed and written up but deliberately not in this PR.It did not work on Linux, in two quiet ways
docs/hot-reload.mdsaid "only tested on aarch64-darwin". That undersold it.Exports. rustc links a
dylibwith its own anonymous version script ending inlocal: *, which demotes every symbol it did not generate. flecs's C symbols arrived withDEFAULTvisibility andLOCALbinding — present and unreachable — so the demo host would not even link:--export-dynamicand--export-dynamic-symbol=ecs_*both leave the exported count at exactly 0; a version-script demotion is not something either can reverse. A second version script works, because ld merges them and an explicit pattern beats a*wildcard. Before: 9001 exported, 0 of themecs_*. After: 10717 exported, 666 of themecs_*,ecs_initGLOBAL.One flecs, or the world is indexed two ways.
flecs_ecs's derive emits astatic INDEXper component type, initialised from a process-global pool, and that index is a slot in the world's component array. Theflecs_manual_registrationnote in the doc is about the id being per-world; the index is not. Two copies offlecs_ecsis two pools — and nothing detects it.AbiTokenpasses, no error is raised, and both sides stay internally consistent while disagreeing about which slot is which.What this PR contains
crates/hyperion-hot-reload/build.rs— the ELF version script (ENG-11272).crates/hyperion-hot-reload/demo/index-probe-{host,module}— a probe for the shared-pool invariant.flecs_ecsonto andrewgazelka/Flecs-Rust@f09dc53, which gives that cratecrate-type = ["dylib", "rlib"]and moves the flecs export script to where the dylib is actually produced.docs/hot-reload.md— the findings, the traps, the build recipe, and the deploy design.The probe is behavioural, because the two obvious tests both lie
Recorded in the code and the doc, because both look like success:
ecs_init as usizeacross the boundary reports a mismatch even when the copy is shared. An executable taking the address of a dynamically-linked function gets its own PLT stub. Measured shared and separate copies; both printed different addresses.1and1. This is exactly what the probe reported before the module referenced the runtime crate at all.Allocation order cannot coincide. With one pool an index taken in the module is strictly greater than every index the host took first.
Measurements
hyperionas a plain rlib — separate pools. The host creates the second copy, pullingflecs_ecsin through hyperion's rlib while the module resolves it from the runtime dylib; a module that touches no hyperion component does not avoid this:With
flecs_ecsandhyperionboth dylibs, against the real pin with no local patch:demo.shon x86_64-linux now matches darwin case for case: code-only change accepted with state intact, layout change refused with the world still ticking on the previous build, migration accepted rewriting 3 instances (21u32becoming22.0f32, converted rather than reinterpreted). Exit 0.Negative control: reverting
build.rsreproduces the link failure, so the fix is load-bearing rather than incidental.Verified on dev-compute-6 (x86_64-linux, rustc 1.99.0-nightly
dc3f85158) and re-verified on aarch64-darwin.What is NOT in this PR, stated plainly
hyperionis not yetcrate-type = ["dylib", "rlib"]. The measurement above required it, but it is not part of this change, because making it a dylib needs-C prefer-dynamic -C link-arg=-Wl,--undefined-version -C link-arg=-Wl,--allow-shlib-undefinedacross the build and that is a packaging decision, not a one-line edit. Recipe and rationale are in the doc.export_module!, no host/rules split ofsmash, no reload loop.app.run()is still flecs's own main loop with no per-tick hook.reloadTriggers, the/etcindirection and theExecReloadclient are written up in the doc and not built.hyperionis a dylib; adding it before then would just be red.--allow-shlib-undefinedrequirement has a tidier fix I did not take.simulation/metadata/mod.rs:212hand-writesimpl PartialOrd for $name where $type: PartialOrd, unsatisfiable for 7 metadata types because glam'sQuatandVec3have noPartialOrd. rustc never codegens thosepartial_cmpbodies but still exports them. They can never be called, so allowing them undefined is sound — but removing the blanket impl would remove the need for the flag.Related: ENG-11272 (this), ENG-11279 (devShell cannot build
smashon Linux; jemalloc 5.3.1 vs GCC 15 — pre-existing, unrelated, does not affect the nix build path).🤖 Generated with Claude Code