Skip to content

dev-guide.md

Latest commit

 

History

History
770 lines (662 loc) · 40.8 KB

dev-guide.md

File metadata and controls

770 lines (662 loc) · 40.8 KB
Raw

Developer Guide

The repository contains two main components located in cmd/protoc-gen-gorums:

  • dev: Contains static code and code generated by templates.
  • gengorums: Contains the code for the protoc-gen-gorums compiler, as well as templates.

Workflow and File Endings

  1. Template Changes: Changes to generated code in the dev/zorums_*_gorums.pb.go files should be done by editing the various templates in the corresponding gengorums/template_*.go file. Making changes to the zorums* files will be overwritten. Note that there is not a one-to-one correspondence for all files.

  2. Static Code File Changes: Any .go file in dev/ whose name does not start with zorums and does not end in _test.go is treated as static code and bundled into template_static.go. Currently, aliases.go holds the only static declarations, but additional files can be added to the dev/ directory and they will be included automatically.

Generated Static Code and Reserved Identifiers

The generator writes the static code of the dev/ package verbatim into one file per Go package, named gorums.pb.go. Every protoc run that generates Gorums code for a Go package writes this file. The generator reports an error if protoc-gen-go writes a file with the same path for a .proto file of the package, such as gorums.proto in the same directory. It goes in the output directory of the first file of the package that the run generates Gorums code for, in protoc's file order. The path is path.Dir(prefix) + "/gorums.pb.go", where prefix is that file's GeneratedFilenamePrefix, so the output options of protoc, such as paths=source_relative and module=, apply to it. The content depends only on the Go package name and the versions of protoc-gen-gorums and protoc. Thus, separate runs for the files of one Go package write the same file. The _gorums.pb.go files hold no static code. In dev mode, the generator writes no gorums.pb.go file, because dev/aliases.go declares the static code. The static code currently consists of four type aliases from dev/aliases.go:

type (
    Config                = gorums.Config
    ID                    = gorums.ID
    Node                  = gorums.Node
    Correctable[Resp any] = gorums.Correctable[Resp]
)

These aliases make the most commonly used Gorums types available as pb.Config, pb.ID, *pb.Node, and *pb.Correctable[*pb.X]. Application code and the generated clients use them. The generated client constructors take a Config or a *Node, and the client structs hold these values. Thus, a _gorums.pb.go file depends on the aliases in the gorums.pb.go file of its package. The other generated code refers to the gorums types directly, such as *gorums.Call and *gorums.OnewayCall.

Because the static code appears once per Go package, it may declare package-level names. Each exported package-level name of the static code becomes a reserved identifier. The bundler (gengorums/bundle.go) discovers these names by inspecting TypesInfo.Defs for all exported, package-scope declarations in the dev package. Thus, declaring an exported top-level name in aliases.go is sufficient to reserve that name. The Gorums generator rejects a proto file whose messages, enums, enum values, or oneof wrappers use a reserved identifier as their Go name. Such a name collides with the generated alias. The check includes nested messages and enums, and every file of the Go package. For a oneof wrapper, the check tests both the name of the open API and the name of the opaque API, which starts with a lowercase letter. goPackage.checkReservedIdents in gengorums/package.go runs the check, and shouldGenerate in gengorums/generate.go calls it for each file with Gorums methods. RPC methods are methods of the generated client interfaces, so they do not collide with the aliases; Generated Client Interfaces describes the name checks for the generated client and server names.

The .proto files of a Go package may be compiled in one protoc run or in separate runs. Separate runs write the same gorums.pb.go, so the static code is declared once. The _gorums.pb.go files declare only the names of their own services.

The TestBundleReservedIdentifiers test in gengorums/bundle_test.go pins the expected set. If you add or remove a static declaration, update that test to match.

The $(static_file) rule of the Makefile runs the bundler when a static file changes. The rule first copies template_static.go to template_static.go.bak. It then runs protoc-gen-gorums --bundle=<file>, which calls GenerateBundleFile in gengorums/bundle.go. The bundler loads the static files of dev/ and writes three variables into template_static.go:

  • staticCode holds the text of the static declarations, without the package and import clauses.
  • reservedIdents holds the reserved identifiers.
  • pkgIdentMap maps the import path of each package that the static code uses to one exported identifier of that package.

generatePackageFile in gengorums/generate.go writes staticCode into gorums.pb.go. It then qualifies each identifier of pkgIdentMap, so the file gets the imports that the static code needs. If the bundle changes, the bundler prints the diff for inspection and a command to revert the change.

Generated Client Interfaces

A Go package may hold several services, in one .proto file or in several. For each service S, the generator emits up to two client interfaces, as gRPC does. The call type of each RPC method decides which client the method belongs to:

Client Constructor Implementing struct Call types
SClient NewSClient(config Config) SClient sClient quorum call, multicast
SNodeClient NewSNodeClient(node *Node) SNodeClient sNodeClient remote call, unicast

A service with only one kind of call gets only that client. The struct name is an unexported variant of the interface name. sClient holds a Config, and sNodeClient holds a *Node. Config and Node are the package's own aliases from gorums.pb.go. The structs have value receivers. The clients have no helper methods.

Every method has the same shape (ctx, in), and calls one runtime constructor:

Call type Result Runtime constructor
quorum call *gorums.Call gorumsimpl.QuorumCall(ctx, c.config, in, method)
quorum call with a stream response *gorums.Call gorumsimpl.QuorumCallStream(ctx, c.config, in, method)
multicast *gorums.OnewayCall gorumsimpl.Multicast(ctx, c.config, in, method)
remote call (*Out, error) gorumsimpl.RemoteCall(ctx, c.node, in, method)
unicast *gorums.OnewayCall gorumsimpl.Unicast(ctx, c.node, in, method) 
flowchart LR
    gen["generated method<br/>c.ReadQC(ctx, in)"] --> rt["runtime/gorumsimpl<br/>QuorumCall(ctx, c.config, in, method)"]
    rt --> impl["internal/impl<br/>QuorumCall(ctx, config, in, method)"]
Loading

A method signature is written once, for both the interface and the struct. godoc shows the doc comments of the interface methods.

The server interface SServer and the function RegisterSServer come from gengorums/template_server.go.

In dev mode, the generator writes one dev/zorums_{name}_gorums.pb.go file for the client interfaces, for the server interface, and for each entry of gorumsCallTypesInfo in gengorums/calltype.go:

Dev file Template Contents
zorums_client_gorums.pb.go template_client.go client interfaces, constructors, and structs
zorums_quorumcall_gorums.pb.go template_quorumcall.go quorum call and streaming quorum call methods
zorums_multicast_gorums.pb.go template_multicast.go multicast methods
zorums_remotecall_gorums.pb.go template_remotecall.go remote call methods
zorums_unicast_gorums.pb.go template_unicast.go unicast methods
zorums_server_gorums.pb.go template_server.go server interface and handler registration

The client file holds the interface declarations of all call types. Each call type file holds the implementing methods of that call type.

Outside dev mode, all of this code goes into a single _gorums.pb.go file, and the static code goes into the package's gorums.pb.go file.

Running the Generator

Any changes to templates or static code requires the invocation of make in order to:

  • Bundle the static code into template_static.go.
  • Rebuild the protoc-gen-gorums compiler.
  • Update the zorums_*_gorums.pb.go files.

A compile error in a generated zorums* file does not stop the bundler. The bundler reads the static files only. make dev can regenerate the generated files after such an error. A compile error in a static file still stops the bundle. The bundle is the text of those files, so they have to compile.

See the Makefile for more details. To force compile, e.g. following a protoc update, you can use:

make -B

Testing

Testing the protoc-gen-gorums compiler itself can be done by running the command below. These tests verify that the code generated by the Gorums compiler is correct and stable.

Default Mode (bufconn)

By default, tests use in-memory bufconn connections:

make test

Integration Mode (real network)

For end-to-end validation with real TCP connections:

make integrationtest

Or directly:

go test -tags=integration ./...

Generator Tests

The tests in internal/testprotos run protoc-gen-go and protoc-gen-gorums in process, through the internal/gentest harness. They test the generator in this tree, not a protoc-gen-gorums binary installed on PATH, so they need no make installgorums. Only parsing uses protoc. The tests write nothing into the tree: gentest.Compile type-checks the generated code from memory.

Test Checks
TestFailingProtoFiles Proto files written inline in the test fail with the expected generator error.
TestFailingProtoFilesPackageFile A proto file whose .pb.go path is the package's gorums.pb.go path fails.
TestGenerateProtoFiles The code generated for each fixture under internal/testprotos compiles.
TestGenerateImportedPackageNames Generated code compiles with message packages named c, ctx, in, or node.
TestGoldenProtoFiles The Gorums code for each fixture equals its golden file in testdata.
TestGeneratePackageFiles Several files of one Go package compile, with one gorums.pb.go per package.
TestGeneratePackageFilesSeparateRuns Separate runs for one Go package write the same gorums.pb.go and compile.
TestStabilityZorums Two runs on dev/zorums.proto produce identical output.
TestStabilityDevFilesUpToDate The committed dev/zorums*.pb.go files equal the output of the generator.

To add a failing case, add a table entry to TestFailingProtoFiles. To add a fixture, add a .proto file in a new directory under internal/testprotos, with a go_package that matches the directory, and create its golden files as shown below. Each fixture directory with Gorums output has two golden files: X_gorums.pb.go.golden and gorums.pb.go.golden.

After an intended change to a template, refresh the golden files and the dev files, and review both diffs:

make golden
make dev

Golden files replace the tool versions in the header with VERSION, so a release or a new protoc does not change them.

Error Strings

An error string describes the operation that was attempted, not the failure. Wrapped errors then read like prose: dialing node 3: creating stream: connection refused.

  • Do not start an error with failed to, could not, unable to, or error. Write creating node %d: %w, not failed to create node %d: %w.
  • An error that wraps nothing names the condition: no nodes in configuration.
  • A precondition has the form cannot X without Y.
  • Do not add an annotation that only says something failed; return the error unchanged.
  • Use a lowercase first letter, no trailing punctuation, and no newlines. Sentinel errors (ErrXxx) name the condition.
  • Only the layer that creates a Gorums error adds the gorums: prefix. A caller that wraps such an error does not add it again.
  • Messages that an application end user reads directly, such as protoc-gen-gorums and command-line diagnostics, may keep explicit failure wording.

TestErrorStringsDescribeOperation in internal/lint rejects the banned prefixes in errors.New and fmt.Errorf calls.

Benchmarking

See benchmarking.md

Makefile

Below is a description of the current Makefile targets. The Makefile itself also serves as documentation; inspect it for details.

Target Description
all Builds dev, benchmark, and compiles tests (default target).
dev Updates template_static.go and regenerates generated files from templates.
genproto Force-regenerates all protobuf and Gorums files across the repo (dev, benchmark, tests, examples).
benchmark Compiles the benchmark tool.
benchkit Regenerates the protobuf and Gorums files of the benchkit module.
sweep Builds the sweep command in benchkit/cmd/sweep.
compiletests Compiles test protos in internal/tests.
tools Installs required tools (protoc-gen-go, protoc-gen-go-grpc, stress, etc.) via go install tool.
installgorums Reinstalls the protoc-gen-gorums plugin.
bootstrapgorums Bootstraps the plugin when it is not yet installed.
test Runs all tests (using bufconn by default).
integrationtest Runs integration tests with real TCP connections.
testrace Runs tests with the race detector enabled.
benchtest Validates that benchmarks compile and run without error (short runs).
bench Runs benchmarks with proper measurement for performance analysis.
golden Rewrites the generator golden files in internal/testprotos/testdata after a template change.
stresstest Runs the ordering tests for a fixed duration (test flag: -stress) for thorough testing.
stressdev Runs the tests of the dev package repeatedly with the stress tool.
modernize Applies go fix and x/tools/modernize@latest across all workspace modules.
goplscheck Fails on gopls diagnostics, including hint-level suggestions, in non-generated Go source.
lint Runs golangci-lint with .golangci.yml on all workspace modules, as CI does.
deadcode Lists functions unreachable from any main or test; advisory, and run by lint.

Releasing

Releases are made with the gorums-release program in internal/cmd/gorums-release. The release guide describes how to install and use it.

Runtime Architecture

The runtime is split into layers, each in its own package. The diagrams below show the main types of each layer and how they connect. They omit helper types and most fields and methods; the package doc comments and the source are authoritative.

Package Layering

Generated code reaches the call engine through runtime/gorumsimpl, and application code uses the types that the root gorums package re-exports as aliases. Each internal package depends only on the packages below it.

flowchart TD
    app["Application code"]
    gen["Generated code<br/>(service clients, *_gorums.pb.go)"]
    gorums["gorums<br/>(Server, type aliases)"]
    gorumsimpl["runtime/gorumsimpl<br/>(call constructors)"]
    impl["internal/impl<br/>(call engine)"]
    conn["internal/conn<br/>(nodes, configurations, connectivity)"]
    stream["internal/stream<br/>(channels, sessions, wire messages)"]
    grpc["gRPC NodeStream<br/>(bidirectional stream)"]

    app --> gen
    app --> gorums
    gen --> gorumsimpl
    gen --> gorums
    gorumsimpl --> impl
    gorums -. "type aliases" .-> impl
    gorums -. "type aliases" .-> conn
    gorums --> stream
    impl --> conn
    impl --> stream
    conn --> stream
    stream --> grpc
Loading

Stream Layer (internal/stream)

A Transport is everything a call needs to reach one node. It holds a channelRef, an atomically replaceable reference to the node's current Channel. There are three channel kinds:

  • OutboundChannel runs over streams this node dials; it opens a new session for each stream and requeues pending calls when a stream is lost, as far as the send queue has space. A stream that the server refuses is not lost: it ends with a refusal code (Unauthenticated, PermissionDenied, InvalidArgument, or Unimplemented) before the server accepted it. Its pending two-way and streaming calls fail with the refusal, as does a one-way call whose send fails on it. The channel then backs off before it opens the next stream, failing requests at once with the refusal until then. A stream that cannot be opened because of a refusal code is handled the same way, for example when the connection failed the dial-side identity check. A stream opened after a refusal is suspect for health reporting: its sends do not clear LastErr until the server accepts it. The channel sends on a new stream at once, but the stream counts as up only once the server accepts it: when the gorums-accepted response header or the first frame arrives. The receive goroutine of each session waits for the response headers before it starts receiving. A refused stream ends without the gorums-accepted header, although a server interceptor may have sent other headers. Only the channel's live session can bring the stream up, so a late acceptance of a session that a GOAWAY replaced has no effect.
  • InboundChannel runs over one InboundStream that the server accepted; its single session ends with the stream. An InboundStream is a BidiStream that can also send response headers, as a server stream can and a client stream cannot. Its send loop first sends the gorums-accepted header, before any message. A stream sends its response headers only once, so the header fails if a gRPC stream interceptor on the server has already sent them. The send loop then sends an empty acceptance frame instead. Message IDs start at 1, so the dialer drops that frame after it counts it as the server's first frame.
  • LocalChannel serves requests in-process through the handler, with no stream.

The two stream channels embed an endpoint with the send queue, the request dispatcher, the optional RequestHandler, and the Latency estimate. A session sends queued requests on its BidiStream, records two-way calls in pendingCalls, and routes received frames either to a pending call or to the dispatcher. The dispatcher runs handlers one at a time, in arrival order. A handler that returns without calling Release hands its goroutine to the next queued handler, so a busy stream keeps one goroutine and the stack it has grown; an early Release starts the next handler on a new goroutine. On the server side, Server.NodeStream asks a PeerAcceptor for the InboundChannel of each accepted stream. The stream layer does not check identities; the connection layer below does.

classDiagram
    direction LR
    class Channel {
        <<interface>>
        Enqueue(Request)
        StreamUp() bool
        PendingCount() int
        Close() error
    }
    class RequestHandler {
        <<interface>>
        HandleRequest(ctx, senderID, msg, release, send)
    }
    class BidiStream {
        <<interface>>
        Send(msg) error
        Recv() msg, error
    }
    class PeerAcceptor {
        <<interface>>
        AcceptPeer(ctx, stream) InboundChannel
    }
    class Transport {
        id ID
        shared bool
        Enqueue(Request)
        StoreChannel(Channel)
        NextMsgID() uint64
    }
    class session {
        sendLoop()
        receive()
        handle(msg)
    }
    class Request {
        Ctx context.Context
        Msg Message
        ResponseChan
    }

    Transport --> channelRef : channel
    Transport --> Latency
    channelRef --> Channel : current
    Channel <|.. OutboundChannel
    Channel <|.. InboundChannel
    Channel <|.. LocalChannel
    OutboundChannel *-- endpoint : embeds
    InboundChannel *-- endpoint : embeds
    OutboundChannel o-- "0..*" session : one per dialed stream
    InboundChannel *-- "1" session
    LocalChannel --> RequestHandler
    LocalChannel *-- dispatcher
    session --> endpoint
    session --> BidiStream
    session *-- pendingCalls
    endpoint *-- sendQueue
    endpoint *-- dispatcher
    endpoint --> RequestHandler
    endpoint --> Latency
    sendQueue o-- Request
    pendingCalls o-- Request
    Request --> Message
    Server --> PeerAcceptor
    PeerAcceptor ..> InboundChannel : returns
Loading

Only the receive loop of an outbound session learns whether the server refused its stream: when the server ends a stream, a concurrent Send returns io.EOF, and only Recv returns the server's status. So a send that fails with io.EOF before the server accepted the stream only cancels the stream, and the receive loop disposes of the pending calls. The session's handoff state ensures that exactly one goroutine disposes of them. The session's verdict channel closes when the server accepts the stream or receiving ends; the run loop waits for it after a session ends, and then reads whether the server refused the stream. A one-way send that fails this way also waits for the verdict, so that it fails with the refusal too. If the acceptance header or a frame arrives after the failed send, the stream was accepted: receiving delivers the frames that the stream buffered, and retries the calls still pending when Recv fails.

stateDiagram-v2
    [*] --> Fresh: stream opened
    Fresh --> Accepted: acceptance header or first frame arrives
    Fresh --> SendFailed: a send fails with io.EOF
    Fresh --> Closed: Recv fails
    SendFailed --> Accepted: a buffered header or frame arrives, so deliver the buffered frames
    SendFailed --> Closed: Recv fails
    Accepted --> Closed: Recv fails
    Closed --> [*]
    note right of Closed
        From Fresh or SendFailed with a refusal code:
        fail the pending calls with the refusal.
        Otherwise: retry them.
    end note
Loading

Connectivity Layer (internal/conn)

A Config is a slice of *Node, and each Node has a fixed stream.Transport. An outboundManager creates and owns the nodes of the configurations built by NewConfig and Config.Extend. NewConfig returns the manager's Close method as the close function; Config has no Close method, so only the creator of a pool can close it. The manager's Close closes all its nodes and logs any node's close error, since a caller cannot act on it. Config.Extend fails on a closed manager before it builds any node. A closed manager also rejects new nodes under the same lock that guards its node list, so a node that races with Close cannot outlive it. A server's InboundManager owns the nodes for its known peers and for the peer-capable clients that connect to it, and maintains the connected-peer and connected-client configurations. It implements stream.PeerAcceptor, attaching each accepted stream to the matching node. If a PeerVerifier is set, InboundManager checks each claimed peer ID. It refuses unknown IDs. For a known ID, it attaches the stream only after verification succeeds. The outbound manager uses the same verifier for the configured peers that it dials. It wraps the node's transport credentials in verifiedCredentials, which run the verifier right after each TLS handshake, with the handshake's AuthInfo in the context. A failed check closes the connection before it carries any stream. The handshake error carries codes.Unauthenticated, and gRPC passes that code on to the stream that the OutboundChannel tries to open. The channel treats it as a refusal and waits before it opens the next stream. gRPC's own connection backoff, which WithBackoff sets, decides when the next handshake runs. A node with a verifier needs transport credentials for verifiedCredentials to wrap. WithTransportCredentials sets them in DialOptions.Credentials. Both managers implement nodeRegistry, which a NodeSource uses to build a configuration. WithNodes captures each node's address and optional TLSIdentity together. When a server uses that source through WithPeers, conn.NewPeerVerifier validates the identity set and creates the internal TLS verifier. nodeMap.newConfig also builds a verifier and passes it to nodeRegistry.newNode, so NewConfig and Config.Extend check the servers they dial. The InboundManager ignores that verifier and uses its own for its known peers. Address-only sources yield no verifier, while mixed, empty, invalid, or duplicate identities are configuration errors.

classDiagram
    direction LR
    class Config {
        <<slice>>
        NodeIDs() []ID
        Size() int
    }
    class Node {
        id ID
        addr string
        IsShared() bool
    }
    class NodeSource {
        <<interface>>
    }
    class nodeRegistry {
        <<interface>>
        Nodes() []*Node
    }
    class InboundManager {
        AcceptPeer(ctx, stream)
        ConnectedPeers() Config
        ConnectedClients() Config
    }
    class StreamTransport["stream.Transport"]
    class StreamInboundChannel["stream.InboundChannel"]
    class StreamPeerAcceptor["stream.PeerAcceptor"] {
        <<interface>>
    }
    class StreamRequestHandler["stream.RequestHandler"] {
        <<interface>>
    }

    Config o-- "*" Node
    Node --> StreamTransport : transport
    Node --> outboundManager : mgr
    Node o-- "0..*" StreamInboundChannel : liveChannels
    outboundManager o-- "*" Node : nodes
    outboundManager *-- DialOptions : opts
    DialOptions --> InboundManager : InboundManager
    DialOptions --> StreamRequestHandler : Handler
    InboundManager o-- "*" Node : knownNodes, clientNodes
    InboundManager --> Config : peer, connected, client configs
    InboundManager --> StreamRequestHandler : handler
    StreamPeerAcceptor <|.. InboundManager
    nodeRegistry <|.. outboundManager
    nodeRegistry <|.. InboundManager
    NodeSource ..> nodeRegistry : builds Config with
Loading

Node Kinds

A node's kind is set by the transport and channel it is created with. Calls only ever see the node's transport; the channel behind the transport's channelRef may change as streams come and go.

  • An outbound node dials its peer and owns an OutboundChannel.
  • A local node is the server's own node in its peer configuration; it owns a LocalChannel that calls the server's handler in-process.
  • An inbound node is a known peer or a client of the server; InboundManager.AcceptPeer stores an InboundChannel in its transport when the peer's stream arrives, and clears it when the last live stream ends.
  • A shared node is the higher-ID peer's outbound node under stream deduplication; its shared transport borrows the inbound peer node's channelRef, Latency, and message-ID generator.
flowchart LR
    subgraph outbound["Outbound node"]
        N1[Node] --> T1[Transport] --> R1[channelRef] --> C1[OutboundChannel]
    end
    subgraph local["Local node"]
        N2[Node] --> T2[Transport] --> R2[channelRef] --> C2[LocalChannel]
    end
    subgraph inbound["Inbound node"]
        N3[Node] --> T3[Transport] --> R3[channelRef] --> C3[InboundChannel]
        N3 -. liveChannels .-> C3
    end
    subgraph shared["Shared node (stream dedup)"]
        N4[Node] --> T4["Transport<br/>(shared)"]
    end
    T4 -- borrows --> R3
Loading

Call Engine (internal/impl)

A call constructor takes the call's context.Context and its target directly: a Config for quorum calls and multicasts, and a *Node for remote calls and unicasts. On an empty Config, the CallContext holds ErrEmptyConfig: a quorum call sends nothing, its terminal methods return the error, and its response iterator yields one response that holds the error; Multicast returns a OnewayCall that sends nothing and returns the error from Send and Wait. On a nil *Node, RemoteCall returns ErrNoNode, and Unicast returns a OnewayCall whose CallContext holds ErrNoNode and that sends nothing. A quorum call constructor builds a CallContext that holds the request, the target Config, the response channel, and the response iterator (ResponseSeq). Multicast and Unicast build a CallContext that holds the request and the target Config, but no response iterator. A unicast's CallContext holds a one-node Config. A one-way CallContext gets its response channel when Send or Async dispatches the call; the channel carries the local send confirmations. RemoteCall builds no CallContext; it enqueues one request on the node's transport and waits for the response. A quorum call returns a Call, which embeds Responses and its terminal methods (First, Majority, All, Threshold, Aggregate, Reduce, Matching), as well as the Correctable future. Call.Async sends the requests at once and returns the dispatched Responses, so any terminal method collects the replies later. A one-way call (Multicast, Unicast) returns a OnewayCall, whose Send waits for local send confirmations and whose Async returns a OnewayAsync. A ClientInterceptor wraps the response iterator and may register per-node request transformations before the call is dispatched. CallContext reaches each node's send path through conn.NodeTransport.

classDiagram
    direction LR
    class CallContext~Req, Resp~ {
        request Req
        method string
        responseChan
        sendNow()
    }
    class Call~Req, Resp~ {
        Intercept(interceptors) Call
        Async() Responses
    }
    class Responses~Resp~ {
        First() Resp, error
        Majority() Resp, error
        All() Resp, error
        Threshold(n) Resp, error
        Aggregate(fn) Out, error
        Reduce(n, initial, add) Acc, error
        Matching(n, key) Agreement, error
        Results() ResponseSeq
    }
    class OnewayCall~Req~ {
        Send() error
        Async() OnewayAsync
    }
    class OnewayAsync {
        Wait() error
    }
    class ResponseSeq~Resp~ {
        <<iterator>>
    }
    class ClientInterceptor~Req, Resp~ {
        <<func>>
    }
    class Correctable~Resp~ {
        Get() Resp, int, error
        Watch(level)
    }
    class starter {
        <<interface>>
        sendNow()
        markDispatched()
    }
    class ConnConfig["conn.Config"]
    class StreamTransport["stream.Transport"]

    Call *-- Responses : embeds
    Call --> CallContext : ctx
    OnewayCall --> CallContext : ctx
    OnewayCall ..> OnewayAsync : Async()
    Responses --> ResponseSeq : seq
    Responses --> starter : start
    starter <|.. CallContext
    Call ..> Responses : Async()
    Responses ..> Correctable : Correctable(n)
    CallContext --> ConnConfig : config
    CallContext --> ResponseSeq : responseSeq
    ClientInterceptor ..> CallContext : wraps ResponseSeq
    CallContext ..> StreamTransport : Enqueue via conn.NodeTransport
Loading

Quorum Call Path

A quorum call is dispatched lazily, on the first iteration of its response iterator, so that interceptors can be registered first. Each node's channel sends on its own goroutine, and responses are matched to pending calls by message ID. The fan-out starts at a random node of the configuration and wraps around, so no node is always enqueued last. With a fixed order, the last node would wait longest on every call, and its measured latency would be higher than that of the other nodes. Each node still receives a caller's requests in call order.

sequenceDiagram
    participant Gen as Generated method
    participant Call as impl.Call
    participant CC as impl.CallContext
    participant T as stream.Transport
    participant Ch as stream.OutboundChannel
    participant S as stream.session
    participant G as gRPC stream

    Gen->>Call: QuorumCall(ctx, config, req, method)
    Call->>CC: newQuorumCallContext
    Gen->>Call: Majority()
    Call->>CC: iterate ResponseSeq (sendNow)
    loop each node in Config, from a random start
        CC->>T: conn.NodeTransport(n).Enqueue(Request)
        T->>Ch: Channel.Enqueue: push on sendQueue
    end
    Note over Ch,S: channel goroutine
    S->>Ch: sendLoop takes Request from sendQueue
    S->>S: pendingCalls.add(msgID, Request)
    S->>G: Send(Message)
    G-->>S: Recv() response Message
    S->>S: handle: pendingCalls.take(msgID)
    S-->>CC: Request.deliver to responseChan
    CC-->>Call: ResponseSeq yields NodeResponse
    Call-->>Gen: majority response or QuorumCallError
Loading

Server Side (gorums)

The root gorums.Server ties the layers together on the server side. It registers a stream.Server with its gRPC server, uses its conn.InboundManager as the server's stream.PeerAcceptor, and serves as the stream.RequestHandler for inbound streams, for its local node, and for back-channel requests on its outbound peer configuration. Server.HandleRequest creates a ServerContext per request, records the sender ID that the channel passes in, and runs the registered Handler, wrapped by any ServerInterceptors. A gorums.Message wraps a request's stream.Message together with its decoded proto payload. How a handler delivers its responses depends on the call type:

  • A unary handler returns its response as a proto.Message.
  • A streaming handler calls its send func(proto.Message) callback once for each response, and returns nil, nil.
  • A one-way handler returns nil, nil, and the server sends no response.

Server.HandleRequest builds the response envelope for the returned response and for each sent response, so handlers never touch envelopes.

classDiagram
    direction LR
    class Server {
        handlers map[string]Handler
        RegisterHandler(method, Handler)
        HandleRequest(ctx, senderID, msg, release, send)
        ConnectedPeers() Config
        ConnectedClients() Config
    }
    class ServerContext {
        SenderID() ID
        Release()
        PeerConfig() Config
        ConnectedClients() Config
    }
    class Handler {
        <<func(ServerContext, *Message, func(proto.Message)) (proto.Message, error)>>
    }
    class ServerInterceptor {
        <<func>>
    }
    class Message {
        Proto proto.Message
    }
    class GRPCServer["grpc.Server"]
    class StreamServer["stream.Server"]
    class StreamMessage["stream.Message"]
    class InboundManager["conn.InboundManager"]
    class ConnConfig["conn.Config"]
    class StreamRequestHandler["stream.RequestHandler"] {
        <<interface>>
    }

    Server --> GRPCServer : grpcServer
    Server --> StreamServer : srv
    Server --> InboundManager : im
    Server --> ConnConfig : outbound, with WithPeers
    Server o-- "*" Handler : handlers
    Server o-- "*" ServerInterceptor : interceptors
    StreamRequestHandler <|.. Server
    StreamServer --> InboundManager : acceptor
    InboundManager --> Server : handler
    Server ..> ServerContext : creates per request
    ServerContext --> Server : srv
    Message *-- StreamMessage : embeds
Loading

Inbound Request Path

Requests that arrive on one stream run through the handler one at a time until the handler calls ServerContext.Release or returns. A handler's reply is queued on the stream's own channel and sent by its send loop.

sequenceDiagram
    participant P as Remote peer
    participant SS as stream.Server
    participant IM as conn.InboundManager
    participant N as conn.Node
    participant IC as stream.InboundChannel
    participant D as dispatcher
    participant H as gorums.Server

    P->>SS: open NodeStream
    SS->>IM: AcceptPeer(ctx, stream)
    IM->>IM: verify(ctx, claimed ID) if a PeerVerifier is set
    IM->>N: attachStream: NewInboundChannel
    N->>N: Transport.StoreChannel(ch)
    IM-->>SS: InboundChannel, cleanup
    SS->>IC: Serve(): receive loop
    P->>IC: request Message
    IC->>D: push handler job
    D->>H: HandleRequest(ctx, senderID, msg, release, send)
    H->>H: Handler(ServerContext, Message, send)
    H->>IC: send(reply): push on sendQueue
    IC->>P: Send(reply) on send loop
Loading

Stream Deduplication Internals

Symmetric stream deduplication makes the lower-ID peer of each pair the only dialer. The higher-ID peer's outbound node is born shared: it borrows the matching inbound transport when the configuration is created, before the peer has connected. The borrower cannot dial through a shared transport, so its calls fail with gorums.ErrStreamDown while the dialer's stream is unavailable; callers match it with errors.Is. The dialer re-establishes a lost stream eagerly with capped backoff from its channel's goroutine, independently of its own sends.

Client-initiated message IDs use the low 63 bits, while server-initiated and shared-call IDs have bit 63 set. A session uses this bit to route each received frame. A frame in the peer's ID space is a request for the handler. A frame in the session's own ID space is a response to one of its pending calls. A shared call travels on the stream that the lower-ID peer dialed, as a server-initiated request. So it takes its ID from the generator of the inbound peer node, which sets bit 63, and the dialer's session passes it to the handler. The dialer's own calls on that stream keep their client-initiated IDs, so the calls of the two peers never collide on one stream. These IDs encode as ten-byte protobuf varints because the high bit is set; the small wire-cost increase is intentional and avoids a second stream.

Each session owns the pendingCalls of its stream, so ending a session affects only the calls sent on that stream. When an outbound session ends, it requeues its pending two-way calls on the channel's send queue for the next session. A call fails instead if the server refused the stream, if the stream ended with ResourceExhausted, if the send queue is full, or if the channel is closed. A pending streaming call is not requeued; it fails with ErrStreamDown, or with ErrNodeClosed if the channel is closed. An inbound session has no next session, so its pending calls fail with ErrStreamDown when it ends. These include the calls of a shared node that borrows the inbound stream. Closing a channel ends its sessions and fails the requests in its send queue.

Node identity and its transport are fixed at construction; only the channel behind the transport's channelRef changes as streams come and go. A call therefore reads the node's message-ID generator and current channel directly from that fixed transport, with no per-call binding step.

Every session handles a received request the same way, on inbound and outbound streams alike. It appends the message's metadata to the incoming context and pushes the handler call to the endpoint's dispatcher, which runs handlers one at a time, in arrival order. So a back-channel request on a stream that this node dialed is served like a request on an inbound stream.