Tenant API: the add-hosts prepare and onboard endpoints - #381
Conversation
Canton runs this same validator when a participant uploads a DAR, so a broken lineage failed at deploy time on a real network and the PR that caused it got no signal. #320 bumped governance-utility-onboarding-v1 to 0.3.0 and a reviewer ran the check by hand. The command in the issue, `dpm upgrade-check --both`, does not run as written: the tool requires DAR arguments. Passing ../releases/v1/*.dar checks every committed version together, so a package with three releases has its whole chain validated rather than only the newest pair. It reads the DARs dpm build --all already produced, so it costs seconds. Verified it fails, not only that it passes: built governance-action-v1 as 0.2.0 with a non-Optional field added to GovernableActionView, and the check exited 1 with NOT_VALID_UPGRADE_PACKAGE naming the field. Reverted after. Closes #357
The step's comment claimed it read what dpm build --all produced. It did not: it read ../releases/v1/*.dar, the committed history. Copilot caught the wording, and checking the code showed the wording was describing the better behaviour. A version bumped in daml.yaml has no committed DAR until someone commits one, so the committed-only check could not see it. That is exactly when a reviewer wants to hear about a broken upgrade. Now passes both sets. A package in both appears once, since the validator keys on package id and the DAR-verification step above already pins those identical. Lineages checked went from 57 to 69. Re-proved the gate still fails with the shipping invocation: built governance-action-v1 as 0.2.0 with a non-Optional field and no committed DAR, and it exited 1 naming the field. That case would have passed under the previous command. Review finding from Copilot on #357.
The tenant API could only ever write serial 1. validate_onboarding_topology rejects any higher serial and rejects the onboarding marker outright, so an external party could never gain a host after it was created — which both plans for decentralizing an existing party need. add_hosts_mapping reads the current PartyToParticipant, carries every current host over untouched, and appends the new ones at Confirmation with Canton's Onboarding marker. The threshold does not move: a new host does not count toward it until its marker clears, so bundling the raise would let the party's active hosts fall below its own threshold mid-flight. The wallet pins the base serial because it compares what every host prepared byte-for-byte, and two hosts reading head state a moment apart would otherwise disagree for a reason that is not an attack. validate_add_hosts_topology is the sibling of the onboarding validator and exists for the same reason: the host co-signs the caller's bytes with its own topology key. The stakes are higher here because the party already holds contracts, so a forged serial N+1 could evict its current hosts or drop its threshold. Every field is checked against this node's own head-state read. No endpoint yet — that is the tenant API v2 work.
The serial-N+1 builder had no caller. These are it: POST /v0/tenant/add-hosts/prepare returns the unsigned topology plus the hashes to sign, and POST /v0/tenant/add-hosts/onboard co-signs the wallet-signed bundle with this node's topology key and submits it. Prepare reads head state before it builds, so a wallet whose pinned base serial has gone stale gets a 409 it can act on — re-read and retry — rather than a 500 that reads as a broken host. Onboard returns 400 rather than 500 when validation refuses the bundle, because that is the caller's bytes failing against this host's own view, not this host malfunctioning. Both report only this host's view. The wallet calls every host itself and aggregates, exactly as onboarding already works.
There was a problem hiding this comment.
Pull request overview
This PR wires the previously-added “add hosts to an existing external party” topology builder/validator into the Tenant API by adding prepare/onboard HTTP endpoints, associated request/response DTOs, and the workflow submission function that co-signs and submits the wallet-provided topology bundle.
Changes:
- Added
POST /v0/tenant/add-hosts/prepareandPOST /v0/tenant/add-hosts/onboardhandlers and registered them in the server. - Introduced wallet-facing DTOs for add-hosts prepare/onboard requests and responses (and re-exported them via server types).
- Added
submit_add_hostsworkflow function to validate, co-sign, and submit the wallet-signed add-hosts topology bundle.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| crates/decman/src/workflow/external_party/add_hosts.rs | Adds submit_add_hosts for co-signing + submission of wallet-signed add-hosts topology. |
| crates/decman/src/server/types.rs | Re-exports new Tenant add-hosts DTOs for handler use. |
| crates/decman/src/server/mod.rs | Registers the new tenant add-hosts endpoints on the Actix server. |
| crates/decman/src/server/handlers/tenant.rs | Implements the new /v0/tenant/add-hosts/* endpoints plus shared base64 decode helper/tests. |
| crates/decman/src/server/handlers/mod.rs | Re-exports the new tenant add-hosts handlers. |
| crates/common/src/api.rs | Defines new API DTOs for add-hosts prepare/onboard request/response payloads. |
Suppressed comments (1)
crates/decman/src/server/handlers/tenant.rs:443
- All
submit_add_hosts(...)errors are mapped to HTTP 400, butsubmit_add_hostscan also fail due to internal RPC issues (signing/submission/topology reads) that should be 500 per the endpoint’s documented responses. This makes client retry/diagnostics harder.
Err(e) => {
tracing::error!("tenant add-hosts onboard: submission failed: {e:#}");
return HttpResponse::BadRequest().json(ErrorResponse {
error: format!("Failed to submit the add-hosts topology on this host: {e}"),
});
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
The wire-shape test used a truncated namespace, which CantonId refuses, so the unit-test job failed. Fixed with a full fingerprint. The rest is Copilot's review, which was right. submit_add_hosts and prepare_add_hosts both returned anyhow, so the handlers could not tell a caller error from a Canton failure and answered 400 to everything — including the stale-serial race that the docs promised a 409 for, and RPC failures that are not the caller's fault at all. AddHostsError names the four cases and the handlers map them: 404 unknown party, 409 stale pin, 400 refused bundle, 500 Canton failure. Each is a different remedy for the wallet, so collapsing them sent it to the wrong one. Dropped the pre-read in the prepare handler: prepare_add_hosts reads head state itself and now reports why it refused, so the second read only widened the window in which the serial could move between check and build. Also documented the 404 and 409 the handlers actually return.
The add-hosts endpoints had no integration coverage: everything was unit tests over pure functions, so no Canton interaction was exercised at all. The phase onboards a party on P1+P2 only, so P3 is a genuine new host, then prepares on all three and asserts they produce byte-identical transactions. That is the property the whole design rests on — the wallet's only defence against a lying host is comparing what the others built — and a phase that prepared on one host would not test it. Assertions stop at the mapping reaching three hosts, because the flow stops at the topology write: nothing replicates the ACS and nothing clears the onboarding marker yet, so P3 ends up named but unusable. The module documents that, plus the two gaps it walks past: the base serial is pinned from knowledge rather than read from an endpoint, and host_onboarding_status would call P3 hosted while the marker is still set.
The comment claimed the threshold could not safely move in the same serial bump as an add. That is wrong: the decparty add-party flow writes a marked new member and a new threshold together and Canton accepts it. The real rule is that the threshold must not exceed the hosts that can confirm, and a marked host cannot. Splitting the writes is about rollback, not safety: a raise that lands with the add and is then followed by a failed ACS replication leaves the party at a threshold its live hosts may not meet.
Review fixes. The stale-pin check broke the idempotence the endpoint promises. Canton treats a re-submitted identical transaction as a no-op, but this host read the advanced serial first and answered 409 before Canton saw it — so a wallet whose successful call lost its HTTP response could never learn it succeeded. A retry is now treated as success, but only when head state is exactly one past the pin AND already carries the mapping this bundle submitted. Another write could have moved the party, and calling that success would tell the wallet its change is live when it is not. Signatures go to Canton labelled as concatenated Ed25519, always 64 bytes, but base64 decoding accepts any length. An empty or truncated signature passed validation and failed inside a Canton RPC, surfacing as a 500 for plainly malformed caller input. Now refused at validation with the length named.
…dd-hosts-topology
…pology' into feat/api/tenant-add-hosts-endpoints # Conflicts: # crates/decman/src/server/handlers/mod.rs # crates/decman/src/server/types.rs
4da729c to
d5529c2
Compare
scolear
left a comment
There was a problem hiding this comment.
Approved. The host validates the wallet's bytes against its own head-state read before its key signs anything, and the #379 validator already refuses threshold changes, key changes, evictions and transactions that do not name this participant. The retry path reports success only when head state is exactly one serial past the pin and carries the submitted mapping. The integration phase checks the property the design rests on: three hosts produce byte-identical transactions.
Three nits inline, none blocking. The pinned BASE_SERIAL in the test needs no comment here, since /v0/tenant/{party}/state arrives later in the stack.
…the error chain Review from scolear, all three real. signed_by was never checked against the party's namespace. Canton only accepts a signature attributed to it, so a wrong fingerprint failed inside SignTransactions and came back as a 500 — telling the wallet this host is unhealthy when the request was simply wrong, which is the failure mode this PR argues against everywhere else. It is one comparison, next to the 64-byte one. The 404 said 'This host does not host this party'. UnknownParty fires when the synchronizer store holds no authorized mapping at all; a joiner that does not host the party gets 200, and the integration phase relies on that. Now says what it means. The 500 body returned the full error chain. A tonic transport error in it names the admin endpoint address, and this response crosses a tenant API boundary to a wallet provider — the same leak omnibus#46 fixed. The chain goes to the log and the body carries a fixed message. Nothing in it was actionable to the caller anyway: a Canton failure is ours to fix.
scolear raised this on #381 and it applied to seven more sites the later handlers added. A tonic transport error in the chain names the admin endpoint address, and these responses cross a tenant API boundary to a wallet provider — the class omnibus#46 fixed. Every one already logged the chain, so only the body changed: a fixed message plus a pointer to the host's logs. Nothing in the chain was actionable to a wallet anyway, since a Canton failure is ours to fix. The 400s keep their detail. Those describe the caller's own input — bad base64, a malformed party id, a wrong signature length — which is exactly what the caller needs and reveals nothing about this host.
The ranged relay added seven more 500s that interpolated the error, same class scolear raised on #381: a chain carrying an admin endpoint address, returned across the tenant boundary to a wallet provider. Fixed message plus a pointer to the logs, which already carry the chain.
…osts-endpoints # Conflicts: # crates/decman/src/server/types.rs # crates/decman/src/workflow/external_party/add_hosts.rs
The 11 September merge recorded main with -s ours after I proved the #381 squash was an ancestor of this branch. Ancestry is not containment: later commits here had already reverted parts of add_hosts.rs, so the merge kept those reversions and this PR removed three fixes that are live on main — the 64-byte Ed25519 signature check, the signed_by namespace check and its test, and the submitted_mapping_matches retry path. add_hosts.rs is now main's file plus this PR's own additions: the tenant artifact keys, the replication instance/target helpers, the pre-submit offset capture, and the active-hosts-versus-threshold guard with its two tests.
* ci(daml): verified the upgrade lineage of every committed package Canton runs this same validator when a participant uploads a DAR, so a broken lineage failed at deploy time on a real network and the PR that caused it got no signal. #320 bumped governance-utility-onboarding-v1 to 0.3.0 and a reviewer ran the check by hand. The command in the issue, `dpm upgrade-check --both`, does not run as written: the tool requires DAR arguments. Passing ../releases/v1/*.dar checks every committed version together, so a package with three releases has its whole chain validated rather than only the newest pair. It reads the DARs dpm build --all already produced, so it costs seconds. Verified it fails, not only that it passes: built governance-action-v1 as 0.2.0 with a non-Optional field added to GovernableActionView, and the check exited 1 with NOT_VALID_UPGRADE_PACKAGE naming the field. Reverted after. Closes #357 * ci(daml): checked freshly built DARs too, not only committed ones The step's comment claimed it read what dpm build --all produced. It did not: it read ../releases/v1/*.dar, the committed history. Copilot caught the wording, and checking the code showed the wording was describing the better behaviour. A version bumped in daml.yaml has no committed DAR until someone commits one, so the committed-only check could not see it. That is exactly when a reviewer wants to hear about a broken upgrade. Now passes both sets. A package in both appears once, since the validator keys on package id and the DAR-verification step above already pins those identical. Lineages checked went from 57 to 69. Re-proved the gate still fails with the shipping invocation: built governance-action-v1 as 0.2.0 with a non-Optional field and no committed DAR, and it exited 1 naming the field. That case would have passed under the previous command. Review finding from Copilot on #357. * feat(external-party): added the serial-N+1 add-hosts topology builder The tenant API could only ever write serial 1. validate_onboarding_topology rejects any higher serial and rejects the onboarding marker outright, so an external party could never gain a host after it was created — which both plans for decentralizing an existing party need. add_hosts_mapping reads the current PartyToParticipant, carries every current host over untouched, and appends the new ones at Confirmation with Canton's Onboarding marker. The threshold does not move: a new host does not count toward it until its marker clears, so bundling the raise would let the party's active hosts fall below its own threshold mid-flight. The wallet pins the base serial because it compares what every host prepared byte-for-byte, and two hosts reading head state a moment apart would otherwise disagree for a reason that is not an attack. validate_add_hosts_topology is the sibling of the onboarding validator and exists for the same reason: the host co-signs the caller's bytes with its own topology key. The stakes are higher here because the party already holds contracts, so a forged serial N+1 could evict its current hosts or drop its threshold. Every field is checked against this node's own head-state read. No endpoint yet — that is the tenant API v2 work. * refact(add-party): extracted a party-type-agnostic replication core Moving a party onto a participant that does not hold it yet is the same sequence of Canton calls whoever owns the party: capture an offset before the topology moves, export the ACS scoped to the target, import it across a synchronizer disconnect, clear the Onboarding marker. None of it needs a DecentralizedNamespaceDefinition, but all of it took an AddPartyConfig, which tied it to decparties for no reason other than where it happened to live. The core now takes a ReplicationTarget — a party, a participant, and the artifact keys one run's durable markers live under. add_party supplies its own keys via ADD_PARTY_REPLICATION_ARTIFACTS, whose values are the strings it has always used, so a run interrupted before this still resumes after it. What stayed behind is what is genuinely party-type specific: deciding the topology and getting it authorized. A decparty needs owner-threshold signatures over a DNS; that lives with the workflow, not the core. No behavior change. Only artifact-missing error text differs, and it now names the key it looked for. * feat(api): added the tenant add-hosts endpoints The serial-N+1 builder had no caller. These are it: POST /v0/tenant/add-hosts/prepare returns the unsigned topology plus the hashes to sign, and POST /v0/tenant/add-hosts/onboard co-signs the wallet-signed bundle with this node's topology key and submits it. Prepare reads head state before it builds, so a wallet whose pinned base serial has gone stale gets a 409 it can act on — re-read and retry — rather than a 500 that reads as a broken host. Onboard returns 400 rather than 500 when validation refuses the bundle, because that is the caller's bytes failing against this host's own view, not this host malfunctioning. Both report only this host's view. The wallet calls every host itself and aggregates, exactly as onboarding already works. * fix(api): mapped add-hosts failures onto distinct status codes The wire-shape test used a truncated namespace, which CantonId refuses, so the unit-test job failed. Fixed with a full fingerprint. The rest is Copilot's review, which was right. submit_add_hosts and prepare_add_hosts both returned anyhow, so the handlers could not tell a caller error from a Canton failure and answered 400 to everything — including the stale-serial race that the docs promised a 409 for, and RPC failures that are not the caller's fault at all. AddHostsError names the four cases and the handlers map them: 404 unknown party, 409 stale pin, 400 refused bundle, 500 Canton failure. Each is a different remedy for the wallet, so collapsing them sent it to the wrong one. Dropped the pre-read in the prepare handler: prepare_add_hosts reads head state itself and now reports why it refused, so the second read only widened the window in which the serial could move between check and build. Also documented the 404 and 409 the handlers actually return. * test(api): covered add-hosts in the integration suite The add-hosts endpoints had no integration coverage: everything was unit tests over pure functions, so no Canton interaction was exercised at all. The phase onboards a party on P1+P2 only, so P3 is a genuine new host, then prepares on all three and asserts they produce byte-identical transactions. That is the property the whole design rests on — the wallet's only defence against a lying host is comparing what the others built — and a phase that prepared on one host would not test it. Assertions stop at the mapping reaching three hosts, because the flow stops at the topology write: nothing replicates the ACS and nothing clears the onboarding marker yet, so P3 ends up named but unusable. The module documents that, plus the two gaps it walks past: the base serial is pinned from knowledge rather than read from an endpoint, and host_onboarding_status would call P3 hosted while the marker is still set. * feat(external-party): replicated the ACS onto an added host The add-hosts flow stopped at the topology write, so a host it added sat named in the mapping, holding none of the party's contracts, marked forever. This is the other half. The wallet carries the snapshot. The tenant API has no inter-node channel and a partner's host is generally not in this node's Noise mesh, but the wallet already talks to every host — GET /v0/tenant/{party}/acs/{target} exports it, POST /v0/tenant/add-hosts/import takes it on the joiner, imports it and clears the marker. Import and clear are one endpoint on purpose: a host that imported but stayed marked is not usable, and one that cleared without importing would start confirming transactions it cannot validate. The replication offsets are captured at prepare time, before the topology moves, because both ExportPartyAcs and ClearPartyOnboardingFlag search forward from an offset that must predate the party's activation on the joiner. Every host captures both keys since none knows yet which role it plays. host_onboarding_status decided hosted from the uid and permission alone, so it called a marked host live — reachable the moment add-hosts existed. It now reports Onboarding for that state. Guardrail: an add is refused if it would leave fewer unmarked hosts than the threshold. Marked hosts cannot confirm, so that add would stop the party transacting the moment it landed. * test(api): drove the full add-hosts replication in the suite The phase stopped at the topology write, matching what the code did. It now relays the ACS from P1 to P3 and asserts P3 reports the party with the marker cleared — the only state in which it can confirm for the party — plus that P1 never stopped reporting it live, since the import disconnects only the joiner. That final assertion doubles as the empirical answer to the scoping study's open question: whether a single-key party's onboarding participant can clear its own flag, or whether the party key must sign a second round. A timeout there means the latter. * feat(external-party): added the confirmation-threshold change The last write Plan A needs, and the reason add-hosts deliberately froze the threshold: a host still carrying the onboarding marker cannot confirm, so a write that added hosts and raised the threshold together would leave the party needing more confirmations than it has hosts able to give. Add, replicate, then raise. /v0/tenant/threshold/{prepare,onboard}. Canton's rules make this the simplest of the topology writes — a threshold change needs the party namespace alone, so no host co-signs and the party's key is the whole authorization. Each host still validates what it submits to its own store: the validator refuses any bundle that moves the host set or the signing keys, so an eviction cannot ride along on a routine-looking change. The bound is the number of hosts that can actually confirm, not the number listed. Marked hosts do not count until their import completes. * fix(db): gave tenant replication its own artifact table The add-hosts prepare call died with FOREIGN KEY constraint failed on every request. workflow_artifacts.instance_name references workflow_runs, which is right for the Noise workflows — every artefact belongs to a run the coordinator persisted — but the tenant API has no run, so its offsets had nothing to point at and could not be written at all. Migration 000016 adds tenant_replication_artifacts: same shape, no foreign key. ReplicationTarget now carries which store backs it and dispatches its own reads and writes, so the replication core stops assuming a workflow run exists — which is the assumption that made it decparty-shaped in the first place. Three sqlx tests cover it, including the write that previously failed. Unit tests could not have caught this: the constraint only exists in the schema. Migration 000016 is new, never applied anywhere, so it carries none of the checksum risk that editing an applied migration would. * fix(api): made the package preflight best-effort, and review fixes CI: the ACS export died on collect_party_package_ids, which reads the party's contracts over the Ledger API and needs a credential for that party. A node hosting an external party has none — the key is the wallet's — so the scan cannot run there at all. Failing the export over a preflight this deployment cannot perform would block replication entirely, so the response now carries package_preflight: false with an empty list. That is not unsafe: Canton re-validates every contract during ImportPartyAcs, so a missing package still fails the import, just after the joiner disconnects rather than before. Review fixes: The active-host guardrail counted any unmarked participant while claiming 'able to confirm'. An Observation host is unmarked and confirms nothing, so it could wave through an add that leaves the party short. Now Confirmation only. The add-party resume guard checked only the DNS artefact. DNS and P2P are written one after the other, so a coordinator that died between the two left the DNS present and the P2P missing, and the guard waved it through to fail later on a mismatched pair. Both kinds are checked now. An empty ACS is skipped rather than imported, so a clear-flag failure no longer reports 'Imported the ACS'. README lists the six new tenant endpoints. * docs(external-party): corrected why the threshold write is separate The comment claimed the threshold could not safely move in the same serial bump as an add. That is wrong: the decparty add-party flow writes a marked new member and a new threshold together and Canton accepts it. The real rule is that the threshold must not exceed the hosts that can confirm, and a marked host cannot. Splitting the writes is about rollback, not safety: a raise that lands with the add and is then followed by a failed ACS replication leaves the party at a threshold its live hosts may not meet. * fix(api): kept add-hosts idempotent, and rejected malformed signatures Review fixes. The stale-pin check broke the idempotence the endpoint promises. Canton treats a re-submitted identical transaction as a no-op, but this host read the advanced serial first and answered 409 before Canton saw it — so a wallet whose successful call lost its HTTP response could never learn it succeeded. A retry is now treated as success, but only when head state is exactly one past the pin AND already carries the mapping this bundle submitted. Another write could have moved the party, and calling that success would tell the wallet its change is live when it is not. Signatures go to Canton labelled as concatenated Ed25519, always 64 bytes, but base64 decoding accepts any length. An empty or truncated signature passed validation and failed inside a Canton RPC, surfacing as a 500 for plainly malformed caller input. Now refused at validation with the length named. * fix(api): closed the review findings on the replication path Six real ones. threshold.rs had its own copy of the active-host count and it counted Observation hosts, which confirm nothing. It could authorize a threshold the party cannot meet. Same Confirmation filter the add-hosts guard already uses. The replication artifact key was (party, target). Tenant artefacts are never deleted and the offset capture is once-only, so a target removed and later re-added inherited the first attempt's offsets — and an offset predating the earlier activation makes ExportPartyAcs find the stale flag-less one and abort. The base serial is now part of the key, and the ACS endpoints take it explicitly rather than guessing. First-capture-wins was a read-then-write, which cannot enforce it: two concurrent prepares both see nothing and the slower overwrites the first, possibly after activation. write_artifact_if_absent lets the database decide, and losing that race is success. The onboarding status asserted the ACS had not been replicated. The marker does not say that: the ACS may be present with the clearing transaction proposed and not yet authorized. It now reports both possibilities. ARCHITECTURE still said an external party cannot be decentralized in place, which these endpoints contradict. Reworded to the limitation that actually holds, which is about local parties and their permanent namespace. The IT claimed P1 stayed live 'throughout' the import from a Then, but steps run in sequence so it only looked afterwards and would pass even if P1 had dropped out during it. Moved inside the step that does the import. * feat(api): exposed a party's serial so the writes are usable Every write in this API needs base_serial pinned, and nothing reported it: a wallet has no Canton Admin API access, and neither the status endpoint nor /external-parties returned the serial. The writes were only usable by a caller that already knew something it had no way to learn. GET /v0/tenant/{party}/state returns the authorized mapping's serial, its threshold, its host count, and how many hosts still carry the onboarding marker — the last because a threshold above the unmarked count is one the party cannot currently meet. The integration phase reads the serial from it rather than assuming a freshly onboarded party sits at 1, so the endpoint is exercised and the test no longer depends on knowledge a real wallet lacks. * fix(test): passed the base serial the ACS relay now requires The integration phase is a caller of these endpoints too, and I changed their contract without updating it. It read the ACS without base_serial and got a 400 for a missing query field. It takes the base from head state minus one rather than a constant: the staged replication is keyed by the serial the add was pinned to, not the current one, and the add advanced exactly one. * fix(api): checked signed_by, corrected the 404 text, stopped leaking the error chain Review from scolear, all three real. signed_by was never checked against the party's namespace. Canton only accepts a signature attributed to it, so a wrong fingerprint failed inside SignTransactions and came back as a 500 — telling the wallet this host is unhealthy when the request was simply wrong, which is the failure mode this PR argues against everywhere else. It is one comparison, next to the 64-byte one. The 404 said 'This host does not host this party'. UnknownParty fires when the synchronizer store holds no authorized mapping at all; a joiner that does not host the party gets 200, and the integration phase relies on that. Now says what it means. The 500 body returned the full error chain. A tonic transport error in it names the admin endpoint address, and this response crosses a tenant API boundary to a wallet provider — the same leak omnibus#46 fixed. The chain goes to the log and the body carries a fixed message. Nothing in it was actionable to the caller anyway: a Canton failure is ours to fix. * fix(api): stopped the remaining 500s leaking the error chain scolear raised this on #381 and it applied to seven more sites the later handlers added. A tonic transport error in the chain names the admin endpoint address, and these responses cross a tenant API boundary to a wallet provider — the class omnibus#46 fixed. Every one already logged the chain, so only the body changed: a fixed message plus a pointer to the host's logs. Nothing in the chain was actionable to a wallet anyway, since a Canton failure is ours to fix. The 400s keep their detail. Those describe the caller's own input — bad base64, a malformed party id, a wrong signature length — which is exactly what the caller needs and reveals nothing about this host. * fix(tenant): 1-based ACS blocks, guard the import, stop holding the request The pipe refuses anything but served_seq + 1 and starts at 0, so asking for block 0 aborted every snapshot. import_party_acs disconnects the participant before Canton rejects a party it does not host, so check the onboarding marker names this node first. Clearing the marker now only proposes the clear instead of waiting out Canton's safe time and polling; /status reports the rest. * fix(external-party): restored the add_hosts fixes main already had The 11 September merge recorded main with -s ours after I proved the #381 squash was an ancestor of this branch. Ancestry is not containment: later commits here had already reverted parts of add_hosts.rs, so the merge kept those reversions and this PR removed three fixes that are live on main — the 64-byte Ed25519 signature check, the signed_by namespace check and its test, and the submitted_mapping_matches retry path. add_hosts.rs is now main's file plus this PR's own additions: the tenant artifact keys, the replication instance/target helpers, the pre-submit offset capture, and the active-hosts-versus-threshold guard with its two tests.
* ci(daml): verified the upgrade lineage of every committed package Canton runs this same validator when a participant uploads a DAR, so a broken lineage failed at deploy time on a real network and the PR that caused it got no signal. #320 bumped governance-utility-onboarding-v1 to 0.3.0 and a reviewer ran the check by hand. The command in the issue, `dpm upgrade-check --both`, does not run as written: the tool requires DAR arguments. Passing ../releases/v1/*.dar checks every committed version together, so a package with three releases has its whole chain validated rather than only the newest pair. It reads the DARs dpm build --all already produced, so it costs seconds. Verified it fails, not only that it passes: built governance-action-v1 as 0.2.0 with a non-Optional field added to GovernableActionView, and the check exited 1 with NOT_VALID_UPGRADE_PACKAGE naming the field. Reverted after. Closes #357 * ci(daml): checked freshly built DARs too, not only committed ones The step's comment claimed it read what dpm build --all produced. It did not: it read ../releases/v1/*.dar, the committed history. Copilot caught the wording, and checking the code showed the wording was describing the better behaviour. A version bumped in daml.yaml has no committed DAR until someone commits one, so the committed-only check could not see it. That is exactly when a reviewer wants to hear about a broken upgrade. Now passes both sets. A package in both appears once, since the validator keys on package id and the DAR-verification step above already pins those identical. Lineages checked went from 57 to 69. Re-proved the gate still fails with the shipping invocation: built governance-action-v1 as 0.2.0 with a non-Optional field and no committed DAR, and it exited 1 naming the field. That case would have passed under the previous command. Review finding from Copilot on #357. * feat(external-party): added the serial-N+1 add-hosts topology builder The tenant API could only ever write serial 1. validate_onboarding_topology rejects any higher serial and rejects the onboarding marker outright, so an external party could never gain a host after it was created — which both plans for decentralizing an existing party need. add_hosts_mapping reads the current PartyToParticipant, carries every current host over untouched, and appends the new ones at Confirmation with Canton's Onboarding marker. The threshold does not move: a new host does not count toward it until its marker clears, so bundling the raise would let the party's active hosts fall below its own threshold mid-flight. The wallet pins the base serial because it compares what every host prepared byte-for-byte, and two hosts reading head state a moment apart would otherwise disagree for a reason that is not an attack. validate_add_hosts_topology is the sibling of the onboarding validator and exists for the same reason: the host co-signs the caller's bytes with its own topology key. The stakes are higher here because the party already holds contracts, so a forged serial N+1 could evict its current hosts or drop its threshold. Every field is checked against this node's own head-state read. No endpoint yet — that is the tenant API v2 work. * refact(add-party): extracted a party-type-agnostic replication core Moving a party onto a participant that does not hold it yet is the same sequence of Canton calls whoever owns the party: capture an offset before the topology moves, export the ACS scoped to the target, import it across a synchronizer disconnect, clear the Onboarding marker. None of it needs a DecentralizedNamespaceDefinition, but all of it took an AddPartyConfig, which tied it to decparties for no reason other than where it happened to live. The core now takes a ReplicationTarget — a party, a participant, and the artifact keys one run's durable markers live under. add_party supplies its own keys via ADD_PARTY_REPLICATION_ARTIFACTS, whose values are the strings it has always used, so a run interrupted before this still resumes after it. What stayed behind is what is genuinely party-type specific: deciding the topology and getting it authorized. A decparty needs owner-threshold signatures over a DNS; that lives with the workflow, not the core. No behavior change. Only artifact-missing error text differs, and it now names the key it looked for. * feat(api): added the tenant add-hosts endpoints The serial-N+1 builder had no caller. These are it: POST /v0/tenant/add-hosts/prepare returns the unsigned topology plus the hashes to sign, and POST /v0/tenant/add-hosts/onboard co-signs the wallet-signed bundle with this node's topology key and submits it. Prepare reads head state before it builds, so a wallet whose pinned base serial has gone stale gets a 409 it can act on — re-read and retry — rather than a 500 that reads as a broken host. Onboard returns 400 rather than 500 when validation refuses the bundle, because that is the caller's bytes failing against this host's own view, not this host malfunctioning. Both report only this host's view. The wallet calls every host itself and aggregates, exactly as onboarding already works. * fix(api): mapped add-hosts failures onto distinct status codes The wire-shape test used a truncated namespace, which CantonId refuses, so the unit-test job failed. Fixed with a full fingerprint. The rest is Copilot's review, which was right. submit_add_hosts and prepare_add_hosts both returned anyhow, so the handlers could not tell a caller error from a Canton failure and answered 400 to everything — including the stale-serial race that the docs promised a 409 for, and RPC failures that are not the caller's fault at all. AddHostsError names the four cases and the handlers map them: 404 unknown party, 409 stale pin, 400 refused bundle, 500 Canton failure. Each is a different remedy for the wallet, so collapsing them sent it to the wrong one. Dropped the pre-read in the prepare handler: prepare_add_hosts reads head state itself and now reports why it refused, so the second read only widened the window in which the serial could move between check and build. Also documented the 404 and 409 the handlers actually return. * test(api): covered add-hosts in the integration suite The add-hosts endpoints had no integration coverage: everything was unit tests over pure functions, so no Canton interaction was exercised at all. The phase onboards a party on P1+P2 only, so P3 is a genuine new host, then prepares on all three and asserts they produce byte-identical transactions. That is the property the whole design rests on — the wallet's only defence against a lying host is comparing what the others built — and a phase that prepared on one host would not test it. Assertions stop at the mapping reaching three hosts, because the flow stops at the topology write: nothing replicates the ACS and nothing clears the onboarding marker yet, so P3 ends up named but unusable. The module documents that, plus the two gaps it walks past: the base serial is pinned from knowledge rather than read from an endpoint, and host_onboarding_status would call P3 hosted while the marker is still set. * feat(external-party): replicated the ACS onto an added host The add-hosts flow stopped at the topology write, so a host it added sat named in the mapping, holding none of the party's contracts, marked forever. This is the other half. The wallet carries the snapshot. The tenant API has no inter-node channel and a partner's host is generally not in this node's Noise mesh, but the wallet already talks to every host — GET /v0/tenant/{party}/acs/{target} exports it, POST /v0/tenant/add-hosts/import takes it on the joiner, imports it and clears the marker. Import and clear are one endpoint on purpose: a host that imported but stayed marked is not usable, and one that cleared without importing would start confirming transactions it cannot validate. The replication offsets are captured at prepare time, before the topology moves, because both ExportPartyAcs and ClearPartyOnboardingFlag search forward from an offset that must predate the party's activation on the joiner. Every host captures both keys since none knows yet which role it plays. host_onboarding_status decided hosted from the uid and permission alone, so it called a marked host live — reachable the moment add-hosts existed. It now reports Onboarding for that state. Guardrail: an add is refused if it would leave fewer unmarked hosts than the threshold. Marked hosts cannot confirm, so that add would stop the party transacting the moment it landed. * test(api): drove the full add-hosts replication in the suite The phase stopped at the topology write, matching what the code did. It now relays the ACS from P1 to P3 and asserts P3 reports the party with the marker cleared — the only state in which it can confirm for the party — plus that P1 never stopped reporting it live, since the import disconnects only the joiner. That final assertion doubles as the empirical answer to the scoping study's open question: whether a single-key party's onboarding participant can clear its own flag, or whether the party key must sign a second round. A timeout there means the latter. * feat(external-party): added the confirmation-threshold change The last write Plan A needs, and the reason add-hosts deliberately froze the threshold: a host still carrying the onboarding marker cannot confirm, so a write that added hosts and raised the threshold together would leave the party needing more confirmations than it has hosts able to give. Add, replicate, then raise. /v0/tenant/threshold/{prepare,onboard}. Canton's rules make this the simplest of the topology writes — a threshold change needs the party namespace alone, so no host co-signs and the party's key is the whole authorization. Each host still validates what it submits to its own store: the validator refuses any bundle that moves the host set or the signing keys, so an eviction cannot ride along on a routine-looking change. The bound is the number of hosts that can actually confirm, not the number listed. Marked hosts do not count until their import completes. * fix(db): gave tenant replication its own artifact table The add-hosts prepare call died with FOREIGN KEY constraint failed on every request. workflow_artifacts.instance_name references workflow_runs, which is right for the Noise workflows — every artefact belongs to a run the coordinator persisted — but the tenant API has no run, so its offsets had nothing to point at and could not be written at all. Migration 000016 adds tenant_replication_artifacts: same shape, no foreign key. ReplicationTarget now carries which store backs it and dispatches its own reads and writes, so the replication core stops assuming a workflow run exists — which is the assumption that made it decparty-shaped in the first place. Three sqlx tests cover it, including the write that previously failed. Unit tests could not have caught this: the constraint only exists in the schema. Migration 000016 is new, never applied anywhere, so it carries none of the checksum risk that editing an applied migration would. * fix(api): made the package preflight best-effort, and review fixes CI: the ACS export died on collect_party_package_ids, which reads the party's contracts over the Ledger API and needs a credential for that party. A node hosting an external party has none — the key is the wallet's — so the scan cannot run there at all. Failing the export over a preflight this deployment cannot perform would block replication entirely, so the response now carries package_preflight: false with an empty list. That is not unsafe: Canton re-validates every contract during ImportPartyAcs, so a missing package still fails the import, just after the joiner disconnects rather than before. Review fixes: The active-host guardrail counted any unmarked participant while claiming 'able to confirm'. An Observation host is unmarked and confirms nothing, so it could wave through an add that leaves the party short. Now Confirmation only. The add-party resume guard checked only the DNS artefact. DNS and P2P are written one after the other, so a coordinator that died between the two left the DNS present and the P2P missing, and the guard waved it through to fail later on a mismatched pair. Both kinds are checked now. An empty ACS is skipped rather than imported, so a clear-flag failure no longer reports 'Imported the ACS'. README lists the six new tenant endpoints. * docs(external-party): corrected why the threshold write is separate The comment claimed the threshold could not safely move in the same serial bump as an add. That is wrong: the decparty add-party flow writes a marked new member and a new threshold together and Canton accepts it. The real rule is that the threshold must not exceed the hosts that can confirm, and a marked host cannot. Splitting the writes is about rollback, not safety: a raise that lands with the add and is then followed by a failed ACS replication leaves the party at a threshold its live hosts may not meet. * fix(api): kept add-hosts idempotent, and rejected malformed signatures Review fixes. The stale-pin check broke the idempotence the endpoint promises. Canton treats a re-submitted identical transaction as a no-op, but this host read the advanced serial first and answered 409 before Canton saw it — so a wallet whose successful call lost its HTTP response could never learn it succeeded. A retry is now treated as success, but only when head state is exactly one past the pin AND already carries the mapping this bundle submitted. Another write could have moved the party, and calling that success would tell the wallet its change is live when it is not. Signatures go to Canton labelled as concatenated Ed25519, always 64 bytes, but base64 decoding accepts any length. An empty or truncated signature passed validation and failed inside a Canton RPC, surfacing as a 500 for plainly malformed caller input. Now refused at validation with the length named. * fix(api): closed the review findings on the replication path Six real ones. threshold.rs had its own copy of the active-host count and it counted Observation hosts, which confirm nothing. It could authorize a threshold the party cannot meet. Same Confirmation filter the add-hosts guard already uses. The replication artifact key was (party, target). Tenant artefacts are never deleted and the offset capture is once-only, so a target removed and later re-added inherited the first attempt's offsets — and an offset predating the earlier activation makes ExportPartyAcs find the stale flag-less one and abort. The base serial is now part of the key, and the ACS endpoints take it explicitly rather than guessing. First-capture-wins was a read-then-write, which cannot enforce it: two concurrent prepares both see nothing and the slower overwrites the first, possibly after activation. write_artifact_if_absent lets the database decide, and losing that race is success. The onboarding status asserted the ACS had not been replicated. The marker does not say that: the ACS may be present with the clearing transaction proposed and not yet authorized. It now reports both possibilities. ARCHITECTURE still said an external party cannot be decentralized in place, which these endpoints contradict. Reworded to the limitation that actually holds, which is about local parties and their permanent namespace. The IT claimed P1 stayed live 'throughout' the import from a Then, but steps run in sequence so it only looked afterwards and would pass even if P1 had dropped out during it. Moved inside the step that does the import. * feat(api): exposed a party's serial so the writes are usable Every write in this API needs base_serial pinned, and nothing reported it: a wallet has no Canton Admin API access, and neither the status endpoint nor /external-parties returned the serial. The writes were only usable by a caller that already knew something it had no way to learn. GET /v0/tenant/{party}/state returns the authorized mapping's serial, its threshold, its host count, and how many hosts still carry the onboarding marker — the last because a threshold above the unmarked count is one the party cannot currently meet. The integration phase reads the serial from it rather than assuming a freshly onboarded party sits at 1, so the endpoint is exercised and the test no longer depends on knowledge a real wallet lacks. * fix(test): passed the base serial the ACS relay now requires The integration phase is a caller of these endpoints too, and I changed their contract without updating it. It read the ACS without base_serial and got a 400 for a missing query field. It takes the base from head state minus one rather than a constant: the staged replication is keyed by the serial the add was pinned to, not the current one, and the add advanced exactly one. * fix(api): checked signed_by, corrected the 404 text, stopped leaking the error chain Review from scolear, all three real. signed_by was never checked against the party's namespace. Canton only accepts a signature attributed to it, so a wrong fingerprint failed inside SignTransactions and came back as a 500 — telling the wallet this host is unhealthy when the request was simply wrong, which is the failure mode this PR argues against everywhere else. It is one comparison, next to the 64-byte one. The 404 said 'This host does not host this party'. UnknownParty fires when the synchronizer store holds no authorized mapping at all; a joiner that does not host the party gets 200, and the integration phase relies on that. Now says what it means. The 500 body returned the full error chain. A tonic transport error in it names the admin endpoint address, and this response crosses a tenant API boundary to a wallet provider — the same leak omnibus#46 fixed. The chain goes to the log and the body carries a fixed message. Nothing in it was actionable to the caller anyway: a Canton failure is ours to fix. * fix(api): stopped the remaining 500s leaking the error chain scolear raised this on #381 and it applied to seven more sites the later handlers added. A tonic transport error in the chain names the admin endpoint address, and these responses cross a tenant API boundary to a wallet provider — the class omnibus#46 fixed. Every one already logged the chain, so only the body changed: a fixed message plus a pointer to the host's logs. Nothing in the chain was actionable to a wallet anyway, since a Canton failure is ours to fix. The 400s keep their detail. Those describe the caller's own input — bad base64, a malformed party id, a wrong signature length — which is exactly what the caller needs and reveals nothing about this host. * fix(tenant): 1-based ACS blocks, guard the import, stop holding the request The pipe refuses anything but served_seq + 1 and starts at 0, so asking for block 0 aborted every snapshot. import_party_acs disconnects the participant before Canton rejects a party it does not host, so check the onboarding marker names this node first. Clearing the marker now only proposes the clear instead of waiting out Canton's safe time and polling; /status reports the rest. * test(spike): asked Canton whether a local party can adopt a wallet key The scoping study's deciding question for Plan B1 — converting a partner's existing local party into a co-validated, externally-signed one — and it left it open because nobody had tried it. The authorization rules permit adding signing keys to an existing party; whether Canton's runtime accepts the flip from participant-signed to externally-signed mid-life was unproven. It is a question about Canton's behaviour, so it is answered by running it rather than by reasoning about it. The phase allocates a genuine local party on P1 — namespace is P1's own participant namespace, which is what makes a party local — then writes serial 2 adding a wallet-held key as its party_signing_keys, and asserts head state carries it. A failure here is a result, not a flake: if Canton refuses, B1 does not exist and the product answer for a local party is B2, threshold-1 failover hosting with no application change. The key is built by the product's own party_signing_key helper, so this proves Canton accepts the key DecMan would really write rather than a lookalike. Not proven yet: that the party can then submit externally-signed transactions with that key. Adding the key and honouring it at submission are different runtime paths, and the second needs a ledger credential this harness does not wire up for an arbitrary party. This is the necessary condition. * fix(spike): authored the spike topology with Authorize, not SignTransactions The phase died at its first step with PROTO_DESERIALIZATION_FAILURE, before the question it exists to ask was even put to Canton. Cause: it built the write the way the wallet path does — GenerateTransactions, then SignTransactions to attach signatures, then AddTransactions. That shape is for attaching signatures to bytes someone else produced. A node writing its own topology uses Authorize, which is what the decparty proposal builder does. For a local party the participant's namespace key IS the party's namespace, so must_fully_authorize: true is correct: it can authorize the write alone, which is the whole reason a local party can be changed without asking anyone. Serial is left at 0 so Canton picks rather than the test guessing, and AllowUnvalidatedSigningKeys is passed for the same reason create.rs passes it — the wallet key has no NamespaceDelegation behind it, which is precisely the situation under test. * fix(spike): signed the key-adoption with the new key as well The previous run got past allocating the local party and was refused with INVALID_ARGUMENT on the step that matters. Canton's reason is not in the client response, but topology.proto's authorization table has it: adding a signing key needs 'party namespace + all the new signing key'. Only the participant signed. The namespace half was there, the proof that the caller holds the key it wants Canton to trust was not. So the step now uses the same two-signature shape as external-party onboarding: Canton builds the transaction, the wallet key signs the returned hash, and the participant co-signs with its namespace key. Also switched the converted mapping to Confirmation. A party that signs its own transactions is not submitted for by its host, and the study notes Canton forbids Submission once the party is externally signed. The assertion no longer accepts any key. It compares the key material against the wallet's and pins the signing threshold at 1, since presence alone would pass for a key we never sent. Usage is left out of the comparison because Canton normalizes it. * fix(spike): waited for the allocation before naming the next serial TOPOLOGY_SERIAL_MISMATCH: provided 2, expected 1. Canton still saw no mapping at all when the second write was built, so the next serial it wanted was 1. Authorize returning does not mean the write is effective; the transaction has to reach the synchronizer and come back. The phase now polls head state until the party appears and takes the serial from what it observes, so the adoption write is one past whatever is really there rather than a hardcoded 2. * fix(external-party): restored the add_hosts fixes main already had The 11 September merge recorded main with -s ours after I proved the #381 squash was an ancestor of this branch. Ancestry is not containment: later commits here had already reverted parts of add_hosts.rs, so the merge kept those reversions and this PR removed three fixes that are live on main — the 64-byte Ed25519 signature check, the signed_by namespace check and its test, and the submitted_mapping_matches retry path. add_hosts.rs is now main's file plus this PR's own additions: the tenant artifact keys, the replication instance/target helpers, the pre-submit offset capture, and the active-hosts-versus-threshold guard with its two tests.
Stacked on #379. Base is
feat/external-party/add-hosts-topology, so review that first — the diff here is only this commit.Item 3 of the scoping study's §06. #379 built the serial-N+1 topology write and its validator; nothing called them. These are the callers.
Endpoints
POST /v0/tenant/add-hosts/preparePOST /v0/tenant/add-hosts/onboardSame shape as onboarding: every host prepares independently, the wallet compares the bytes before it signs, and each host submits only its own authorization. Canton accumulates the rest and promotes the mapping when the last one lands. No host relays to another, and a retry converges because re-submitting an identical transaction is a no-op.
Status codes worth arguing about
409 on a stale pin. Prepare reads head state before it builds. If the wallet's
base_serialno longer matches, that is not a malfunction — the party moved between the wallet's read and this call — so it returns 409 with both serials and the wallet can re-read and retry. A 500 would read as "this host is broken" and send the wallet to the wrong remedy.400 when validation refuses the bundle.
validate_add_hosts_topologyrejecting a submission means the caller's bytes failed against this host's own head-state read. That is the caller's problem, not the host's.Both endpoints report only this host's view, matching
/v0/tenant/onboard.Completedmeans this host's authorized mapping has already advanced past the base serial;InProgressmeans the change is still a proposal here.New in
add_hosts.rssubmit_add_hosts— the add-hosts counterpart toallocate_party. It re-reads head state, runs the validator, co-signs with an emptysigned_byso the node picks its own key, and submits. Validation happens before this node's key touches the bytes, which is the whole point: the party already exists and already holds contracts, so a forged serial N+1 could evict its current hosts rather than merely create something unwanted.Tests
Five unit tests:
decode_all(including that one bad entry fails the whole batch, since a silently dropped entry would break the index alignment the signatures depend on), and wire-shape round-trips for both request DTOs —decman-walletmirrors these types, so a renamed field would fail to deserialize on the other end.The endpoint behavior itself needs a live Canton, so an integration-test phase is the honest coverage. That plus the wallet-side flow (§06 item 4) is the natural next PR.
cargo clippy --workspace --all-targets --all-features -- -D warningsis clean. Per the usual rule I have not runcargo testlocally.