diff --git a/docs/.nojekyll b/docs/.nojekyll new file mode 100644 index 0000000..e69de29 diff --git a/docs/_sidebar.md b/docs/_sidebar.md new file mode 100644 index 0000000..206d17b --- /dev/null +++ b/docs/_sidebar.md @@ -0,0 +1,10 @@ +- [Overview](overview.md) +- [Getting Started](getting-started.md) +- [Architecture](architecture.md) +- [Usage Guide](usage-guide.md) + - [MultiNode](usage-guide.md#multinode) + - [Transaction Manager](usage-guide.md#transaction-manager) + - [Head Tracker](usage-guide.md#head-tracker) + - [Write Target](usage-guide.md#write-target) +- [Contributing](contributing.md) +- [Changelog](changelog.md) diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 0000000..2ff2ffe --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,412 @@ +# Architecture + +This document describes the internal architecture of each Chainlink Framework module and how they interact. + +## System Overview + +```mermaid +flowchart TB + subgraph "Chainlink Node" + subgraph "chainlink-framework" + MN[MultiNode] + TXM[TxManager] + HT[HeadTracker] + WT[WriteTarget] + end + + subgraph "Chain-Specific" + RPC1[RPC Client 1] + RPC2[RPC Client 2] + RPCN[RPC Client N] + end + end + + subgraph "Blockchain Network" + N1[Node 1] + N2[Node 2] + NN[Node N] + end + + MN --> RPC1 + MN --> RPC2 + MN --> RPCN + TXM --> MN + HT --> MN + WT --> TXM + + RPC1 --> N1 + RPC2 --> N2 + RPCN --> NN +``` + +## MultiNode Architecture + +MultiNode is the core component that manages connections to multiple RPC endpoints, providing health monitoring, load balancing, and automatic failover. + +### Component Diagram + +```mermaid +flowchart TB + subgraph MultiNode + NS[NodeSelector] + NM[Node Manager] + SO[SendOnly Nodes] + TS[Transaction Sender] + + subgraph "Node Pool" + N1[Node 1] + N2[Node 2] + N3[Node N] + end + end + + subgraph "Per Node" + subgraph "Node" + FSM[State Machine] + RPC[RPC Client] + LC[Lifecycle Manager] + HC[Health Checks] + end + end + + NS --> N1 + NS --> N2 + NS --> N3 + TS --> N1 + TS --> N2 + TS --> N3 + TS --> SO + + N1 --> FSM + FSM --> RPC + LC --> FSM + HC --> FSM +``` + +### Node State Machine + +Each node maintains a finite state machine tracking its health status: + +```mermaid +stateDiagram-v2 + [*] --> Undialed: Created + Undialed --> Dialed: Dial success + Undialed --> Unreachable: Dial failed + + Dialed --> Alive: Verification passed + Dialed --> InvalidChainID: Chain ID mismatch + Dialed --> Syncing: Node is syncing + Dialed --> Unreachable: Connection error + + Alive --> OutOfSync: Head too far behind + Alive --> Unreachable: Connection lost + + OutOfSync --> Alive: Caught up + OutOfSync --> Unreachable: Connection lost + + InvalidChainID --> Dialed: Retry verification + Syncing --> Dialed: Retry verification + Unreachable --> Undialed: Redial + + Alive --> Closed: Shutdown + OutOfSync --> Closed: Shutdown + Unreachable --> Closed: Shutdown + Closed --> [*] +``` + +### Node Selection Flow + +```mermaid +sequenceDiagram + participant Client + participant MultiNode + participant NodeSelector + participant Node1 + participant Node2 + + Client->>MultiNode: SelectRPC() + MultiNode->>MultiNode: Check activeNode + alt Active node alive + MultiNode-->>Client: Return activeNode.RPC() + else Need new node + MultiNode->>NodeSelector: Select() + NodeSelector->>Node1: State() + Node1-->>NodeSelector: Alive + NodeSelector->>Node2: State() + Node2-->>NodeSelector: OutOfSync + NodeSelector-->>MultiNode: Node1 (best) + MultiNode->>MultiNode: Set activeNode = Node1 + MultiNode-->>Client: Return Node1.RPC() + end +``` + +## Transaction Manager (TxManager) + +The TxManager handles transaction lifecycle from creation through finalization. + +### Component Structure + +```mermaid +flowchart TB + subgraph TxManager + BC[Broadcaster] + CF[Confirmer] + TK[Tracker] + FN[Finalizer] + RS[Resender] + RP[Reaper] + TS[TxStore] + end + + subgraph External + HT[HeadTracker] + KS[KeyStore] + MN[MultiNode] + end + + HT -->|New heads| BC + HT -->|New heads| CF + HT -->|New heads| FN + + BC --> TS + CF --> TS + TK --> TS + FN --> TS + + BC --> MN + CF --> MN + RS --> MN + + KS --> BC + KS --> TK +``` + +### Transaction Lifecycle + +```mermaid +stateDiagram-v2 + [*] --> Unstarted: CreateTransaction() + + Unstarted --> InProgress: Broadcaster picks up + InProgress --> Unconfirmed: Broadcast success + InProgress --> FatalError: Broadcast fatal error + + Unconfirmed --> Confirmed: Receipt found + Unconfirmed --> Unconfirmed: Bump gas (rebroadcast) + + Confirmed --> ConfirmedMissingReceipt: Receipt lost + Confirmed --> Finalized: Block finalized + + ConfirmedMissingReceipt --> Confirmed: Receipt refound + ConfirmedMissingReceipt --> FatalError: Abandoned + + Finalized --> [*] + FatalError --> [*] +``` + +### Transaction Flow + +```mermaid +sequenceDiagram + participant Client + participant TxManager + participant Broadcaster + participant Confirmer + participant Finalizer + participant TxStore + participant MultiNode + + Client->>TxManager: CreateTransaction(request) + TxManager->>TxStore: Store transaction + TxManager->>Broadcaster: Trigger(address) + + Broadcaster->>TxStore: Get unstarted txs + Broadcaster->>Broadcaster: Build attempt + Broadcaster->>MultiNode: SendTransaction + Broadcaster->>TxStore: Update state -> Unconfirmed + + Note over Confirmer: On new block head + Confirmer->>TxStore: Get unconfirmed txs + Confirmer->>MultiNode: GetReceipt + Confirmer->>TxStore: Update state -> Confirmed + + Note over Finalizer: On finalized block + Finalizer->>TxStore: Get confirmed txs + Finalizer->>TxStore: Update state -> Finalized + Finalizer-->>Client: Resume callback +``` + +## Head Tracker + +The HeadTracker monitors blockchain heads and maintains a local chain cache. + +### Component Structure + +```mermaid +flowchart TB + subgraph HeadTracker + HL[HeadListener] + HS[HeadSaver] + HB[HeadBroadcaster] + BF[Backfiller] + end + + subgraph External + MN[MultiNode/RPC] + DB[(Database)] + TXM[TxManager] + LP[LogPoller] + end + + MN -->|Subscribe| HL + HL -->|New head| HS + HS --> DB + HS --> HB + HS --> BF + BF --> MN + HB --> TXM + HB --> LP +``` + +### Head Processing Flow + +```mermaid +sequenceDiagram + participant RPC + participant Listener + participant Saver + participant Backfiller + participant Broadcaster + participant Subscribers + + RPC->>Listener: New head (height N) + Listener->>Saver: Save(head) + Saver->>Saver: Update chain cache + + alt Head is new highest + Saver->>Backfiller: Backfill(head, prevHead) + Backfiller->>RPC: Fetch missing blocks + Backfiller->>Saver: Save missing heads + Saver->>Broadcaster: BroadcastNewLongestChain + Broadcaster->>Subscribers: OnNewLongestChain(head) + else Head is duplicate/old + Note over Saver: Skip broadcast + end +``` + +## Write Target (Capabilities) + +The WriteTarget capability enables chain-agnostic transaction submission for Chainlink workflows. + +### Component Structure + +```mermaid +flowchart TB + subgraph WriteTarget + WT[WriteTarget] + TS[TargetStrategy] + MB[MessageBuilder] + RT[Retry Logic] + end + + subgraph External + WE[Workflow Engine] + CR[ChainReader] + CW[ChainWriter] + BH[Beholder/Telemetry] + end + + WE -->|Execute| WT + WT --> TS + TS --> CR + TS --> CW + WT --> MB + MB --> BH + WT --> RT +``` + +### Write Execution Flow + +```mermaid +sequenceDiagram + participant Workflow + participant WriteTarget + participant Strategy + participant ChainReader + participant ChainWriter + participant Beholder + + Workflow->>WriteTarget: Execute(request) + WriteTarget->>WriteTarget: Validate config + WriteTarget->>WriteTarget: Parse signed report + WriteTarget->>Beholder: Emit WriteInitiated + + alt Empty report + WriteTarget->>Beholder: Emit WriteSkipped + WriteTarget-->>Workflow: Success (no-op) + else Valid report + WriteTarget->>Strategy: QueryTransmissionState + Strategy->>ChainReader: Check if already transmitted + + alt Already transmitted + WriteTarget->>Beholder: Emit WriteConfirmed + WriteTarget-->>Workflow: Success + else Not transmitted + WriteTarget->>Strategy: TransmitReport + Strategy->>ChainWriter: Submit transaction + WriteTarget->>Beholder: Emit WriteSent + + loop Until confirmed or timeout + WriteTarget->>Strategy: GetTransactionStatus + WriteTarget->>Strategy: QueryTransmissionState + end + + WriteTarget->>Beholder: Emit WriteConfirmed + WriteTarget-->>Workflow: Success + fee metering + end + end +``` + +## Data Flow Summary + +```mermaid +flowchart LR + subgraph Input + WF[Workflows] + TX[Transactions] + end + + subgraph Processing + WT[WriteTarget] + TXM[TxManager] + HT[HeadTracker] + end + + subgraph Infrastructure + MN[MultiNode] + end + + subgraph Output + BC[Blockchain] + MT[Metrics] + TL[Telemetry] + end + + WF --> WT + TX --> TXM + WT --> TXM + TXM --> MN + HT --> MN + MN --> BC + MN --> MT + WT --> TL + TXM --> MT +``` + +## Key Design Principles + +1. **Generic Types**: All components use Go generics to remain chain-agnostic +2. **Interface-Based**: Chain-specific behavior is injected via interfaces +3. **Service Pattern**: Components implement `services.Service` for lifecycle management +4. **Observability**: Built-in metrics and structured logging throughout +5. **Resilience**: Automatic retries, reconnection, and graceful degradation diff --git a/docs/index.html b/docs/index.html new file mode 100644 index 0000000..ca26397 --- /dev/null +++ b/docs/index.html @@ -0,0 +1,104 @@ + + + + + Chainlink Framework + + + + + + + +
+ + + + + + + + + + + + + + + + + diff --git a/docs/overview.md b/docs/overview.md new file mode 100644 index 0000000..751fe86 --- /dev/null +++ b/docs/overview.md @@ -0,0 +1,65 @@ +# Chainlink Framework + +> Common components for Chainlink blockchain integrations across EVM and non-EVM chains. + +## Purpose + +Chainlink Framework provides the foundational building blocks used by Chainlink's Blockchain Integrations team to support multi-chain operations. These components abstract away chain-specific complexities, enabling consistent behavior across diverse blockchain ecosystems. + +## Value Proposition + +- **Multi-RPC Resilience**: Connect to multiple RPC endpoints simultaneously with automatic failover and health checks +- **Chain-Agnostic Design**: Generic interfaces that work with any blockchain via type parameters +- **Production-Ready**: Battle-tested components used across Chainlink's chain integrations +- **Observable**: Built-in metrics, tracing, and monitoring via Prometheus and OpenTelemetry + +## Target Audience + +- **Chain Integration Developers**: Teams integrating new blockchains with Chainlink +- **Node Operators**: Operators running Chainlink infrastructure +- **Contributors**: Developers extending Chainlink's multi-chain capabilities + +## Prerequisites + +- **Go 1.21+** +- Familiarity with Go generics +- Understanding of blockchain concepts (RPC, transactions, finality) + +## Modules + +The framework is organized as a Go workspace with four main modules: + +| Module | Path | Description | +| ---------------- | --------------- | ----------------------------------------------------------------------- | +| **MultiNode** | `multinode/` | Multi-RPC client with health checks, load balancing, and node selection | +| **Chains** | `chains/` | Core chain abstractions including TxManager and HeadTracker | +| **Capabilities** | `capabilities/` | Chainlink capability implementations (WriteTarget) | +| **Metrics** | `metrics/` | Prometheus metrics for chain observability | + +## Module Dependencies + +```mermaid +flowchart TD + subgraph chainlink-framework + capabilities[capabilities/writetarget] + chains[chains] + metrics[metrics] + multinode[multinode] + end + + subgraph external + common[chainlink-common] + end + + capabilities --> common + chains --> multinode + metrics --> common + multinode --> metrics +``` + +## Quick Links + +- [Getting Started](getting-started.md) — Installation and first steps +- [Architecture](architecture.md) — Deep dive into component design +- [Usage Guide](usage-guide.md) — Integration patterns and examples +- [Contributing](contributing.md) — Development setup and workflow diff --git a/docs/usage-guide.md b/docs/usage-guide.md new file mode 100644 index 0000000..9169b6f --- /dev/null +++ b/docs/usage-guide.md @@ -0,0 +1,499 @@ +# Usage Guide + +This guide provides detailed integration patterns and examples for each Chainlink Framework component. + +## MultiNode + +MultiNode enables resilient connections to multiple RPC endpoints with automatic failover. + +### Implementing RPCClient + +The core interface you must implement for your chain: + +```go +// RPCClient wraps chain-specific RPC functionality +type RPCClient[CHAIN_ID ID, HEAD Head] interface { + // Connection lifecycle + Dial(ctx context.Context) error + Close() + + // Chain information + ChainID(ctx context.Context) (CHAIN_ID, error) + IsSyncing(ctx context.Context) (bool, error) + + // Head tracking + SubscribeToHeads(ctx context.Context) (<-chan HEAD, Subscription, error) + SubscribeToFinalizedHeads(ctx context.Context) (<-chan HEAD, Subscription, error) + + // Subscription management + UnsubscribeAllExcept(subs ...Subscription) + + // Observations (for node selection) + GetInterceptedChainInfo() (latest, highest ChainInfo) +} +``` + +### Node Configuration + +Configure node behavior via the `NodeConfig` interface: + +```go +type NodeConfig interface { + PollFailureThreshold() uint32 // Failures before marking unreachable + PollInterval() time.Duration // Health check interval + SelectionMode() string // Node selection strategy + SyncThreshold() uint32 // Blocks behind before out-of-sync + NodeIsSyncingEnabled() bool // Check if node is syncing + FinalizedBlockPollInterval() time.Duration + EnforceRepeatableRead() bool + DeathDeclarationDelay() time.Duration + NewHeadsPollInterval() time.Duration + VerifyChainID() bool // Verify chain ID on connect +} +``` + +### Node Selection Strategies + +#### HighestHead + +Selects the node with the highest observed block number: + +```go +mn := multinode.NewMultiNode( + lggr, + metrics, + multinode.NodeSelectionModeHighestHead, + // ... +) +``` + +#### RoundRobin + +Cycles through healthy nodes evenly: + +```go +mn := multinode.NewMultiNode( + lggr, + metrics, + multinode.NodeSelectionModeRoundRobin, + // ... +) +``` + +#### PriorityLevel + +Selects based on configured node priority: + +```go +// When creating nodes, set priority via nodeOrder parameter +node := multinode.NewNode( + nodeCfg, chainCfg, lggr, metrics, + wsURL, httpURL, "primary-node", + 1, // node ID + chainID, + 0, // priority: lower = higher priority + rpc, + "EVM", + false, +) +``` + +### Broadcasting to All Nodes + +Use `DoAll` to execute operations across all healthy nodes: + +```go +err := mn.DoAll(ctx, func(ctx context.Context, rpc *MyRPCClient, isSendOnly bool) { + if isSendOnly { + // Handle send-only nodes differently if needed + return + } + // Execute operation on each node + rpc.DoSomething(ctx) +}) +``` + +### Monitoring Node Health + +Check current node states: + +```go +states := mn.NodeStates() +for nodeName, state := range states { + fmt.Printf("Node %s: %s\n", nodeName, state) +} + +// Get chain info from live nodes +nLive, chainInfo := mn.LatestChainInfo() +fmt.Printf("Live nodes: %d, Highest block: %d\n", nLive, chainInfo.BlockNumber) +``` + +--- + +## Transaction Manager + +The TxManager handles transaction lifecycle with automatic retry and gas bumping. + +### Creating Transactions + +```go +import ( + "github.com/smartcontractkit/chainlink-framework/chains/txmgr" + txmgrtypes "github.com/smartcontractkit/chainlink-framework/chains/txmgr/types" +) + +// Create a transaction request +request := txmgrtypes.TxRequest[common.Address, common.Hash]{ + FromAddress: fromAddr, + ToAddress: toAddr, + EncodedPayload: calldata, + Value: big.NewInt(0), + FeeLimit: gasLimit, + IdempotencyKey: &idempotencyKey, // Prevents duplicate sends + Strategy: txmgr.NewSendEveryStrategy(), +} + +// Submit transaction +tx, err := txm.CreateTransaction(ctx, request) +if err != nil { + return err +} +``` + +### Transaction Strategies + +Control transaction queueing behavior: + +```go +// Send every transaction (default) +request.Strategy = txmgr.NewSendEveryStrategy() + +// Drop old transactions for same key +request.Strategy = txmgr.NewDropOldestStrategy(subjectKey, maxQueued) + +// Queue limit per subject +request.Strategy = txmgr.NewQueueingTxStrategy(subjectKey, maxQueued) +``` + +### Checking Transaction Status + +```go +status, err := txm.GetTransactionStatus(ctx, idempotencyKey) +switch status { +case commontypes.Pending: + // Transaction submitted, awaiting confirmation +case commontypes.Unconfirmed: + // Transaction confirmed but not yet finalized +case commontypes.Finalized: + // Transaction finalized +case commontypes.Failed: + // Transaction failed (retryable) +case commontypes.Fatal: + // Transaction failed (not retryable) +} +``` + +### Retrieving Transaction Fee + +```go +fee, err := txm.GetTransactionFee(ctx, idempotencyKey) +if err != nil { + return err +} +fmt.Printf("Transaction fee: %s wei\n", fee.TransactionFee.String()) +``` + +### Using Forwarders + +Enable meta-transactions via forwarder contracts: + +```go +// Enable in config +// Transactions.ForwardersEnabled = true + +// Get forwarder for an EOA +forwarder, err := txm.GetForwarderForEOA(ctx, eoaAddress) + +// Use in transaction request +request := txmgrtypes.TxRequest{ + FromAddress: eoaAddress, + ForwarderAddress: forwarder, + // ... +} +``` + +--- + +## Head Tracker + +The HeadTracker monitors blockchain heads and maintains finality information. + +### Subscribing to New Heads + +Implement the `Trackable` interface: + +```go +type MyService struct { + // ... +} + +func (s *MyService) OnNewLongestChain(ctx context.Context, head Head) { + // React to new chain head + fmt.Printf("New head: %d\n", head.BlockNumber()) +} +``` + +Register with the HeadBroadcaster: + +```go +unsubscribe := headBroadcaster.Subscribe(&myService) +defer unsubscribe() +``` + +### Getting Latest Chain Info + +```go +// Get the latest chain head +latestHead := tracker.LatestChain() +if latestHead.IsValid() { + fmt.Printf("Latest: %d\n", latestHead.BlockNumber()) +} + +// Get latest and finalized blocks +latest, finalized, err := tracker.LatestAndFinalizedBlock(ctx) +if err != nil { + return err +} +fmt.Printf("Latest: %d, Finalized: %d\n", + latest.BlockNumber(), + finalized.BlockNumber()) +``` + +### Safe Block Access + +Get the latest "safe" block (useful for reorg-sensitive operations): + +```go +safeHead, err := tracker.LatestSafeBlock(ctx) +if err != nil { + return err +} +// Use safeHead for operations that shouldn't be affected by reorgs +``` + +--- + +## Write Target + +The WriteTarget capability enables workflow-based transaction submission. + +### Implementing TargetStrategy + +Chain-specific implementations must implement the `TargetStrategy` interface: + +```go +type TargetStrategy interface { + // Check if report was already transmitted + QueryTransmissionState( + ctx context.Context, + reportID uint16, + request capabilities.CapabilityRequest, + ) (*TransmissionState, error) + + // Submit the report transaction + TransmitReport( + ctx context.Context, + report []byte, + reportContext []byte, + signatures [][]byte, + request capabilities.CapabilityRequest, + ) (string, error) + + // Get transaction status + GetTransactionStatus( + ctx context.Context, + transactionID string, + ) (commontypes.TransactionStatus, error) + + // Get fee estimate + GetEstimateFee( + ctx context.Context, + report []byte, + reportContext []byte, + signatures [][]byte, + request capabilities.CapabilityRequest, + ) (commontypes.EstimateFee, error) + + // Get actual transaction fee + GetTransactionFee( + ctx context.Context, + transactionID string, + ) (decimal.Decimal, error) +} +``` + +### Creating a WriteTarget + +```go +import ( + "github.com/smartcontractkit/chainlink-framework/capabilities/writetarget" +) + +// Generate capability ID +capID, err := writetarget.NewWriteTargetID( + "evm", // chain family + "mainnet", // network name + "1", // chain ID + "1.0.0", // version +) + +// Create WriteTarget +wt := writetarget.NewWriteTarget(writetarget.WriteTargetOpts{ + ID: capID, + Config: writeTargetConfig, + ChainInfo: chainInfo, + Logger: lggr, + Beholder: beholderClient, + ChainService: chainService, + ConfigValidateFn: validateConfig, + NodeAddress: nodeAddr, + ForwarderAddress: forwarderAddr, + TargetStrategy: myStrategy, +}) +``` + +### Reference Implementations + +- **EVM**: [chainlink/core/services/relay/evm/target_strategy.go](https://github.com/smartcontractkit/chainlink) +- **Aptos**: [chainlink-aptos/relayer/write_target/strategy.go](https://github.com/smartcontractkit/chainlink-aptos) + +--- + +## Metrics + +The metrics module provides Prometheus instrumentation for chain observability. + +### Using Balance Metrics + +```go +import "github.com/smartcontractkit/chainlink-framework/metrics" + +balanceMetrics := metrics.NewBalance(lggr) + +// Record balance +balanceMetrics.Set( + ctx, + balance, // *big.Int + address, // string + chainID, // string + chainFamily, // e.g., "EVM" +) +``` + +### Using TxManager Metrics + +```go +txmMetrics := metrics.NewTxm(lggr) + +// Record transaction states +txmMetrics.RecordTxState(ctx, txState, chainID) + +// Record gas bump +txmMetrics.RecordGasBump(ctx, amount, chainID) +``` + +### Using MultiNode Metrics + +```go +multiNodeMetrics := metrics.NewMultiNode(lggr) + +// Record node states +multiNodeMetrics.RecordNodeStates(ctx, "Alive", 3) +multiNodeMetrics.RecordNodeStates(ctx, "Unreachable", 1) +``` + +--- + +## Performance Tips + +### Connection Pooling + +Configure appropriate lease duration to balance load distribution and connection stability: + +```go +mn := multinode.NewMultiNode( + lggr, + metrics, + multinode.NodeSelectionModeHighestHead, + 5*time.Minute, // Lease duration: higher = more stable, lower = better load distribution + // ... +) +``` + +### Transaction Batching + +For high-throughput scenarios, configure queue capacity: + +```go +// In TxConfig +MaxQueued: 1000 // Max pending transactions per address +ResendAfterThreshold: 30s // Resend stuck transactions +``` + +### Head Sampling + +For chains with fast block times, enable head sampling to reduce processing load: + +```go +// In TrackerConfig +SamplingInterval: 100ms // Process heads at most every 100ms +``` + +--- + +## Troubleshooting + +### All Nodes Unreachable + +Check node health and logs: + +```go +states := mn.NodeStates() +for name, state := range states { + if state == "Unreachable" { + log.Warnf("Node %s is unreachable", name) + } +} +``` + +Common causes: + +- Network connectivity issues +- RPC endpoint rate limiting +- Chain ID mismatch + +### Transactions Stuck + +Check transaction status and consider manual intervention: + +```go +status, err := txm.GetTransactionStatus(ctx, txID) +if status == commontypes.Pending { + // Transaction may need gas bump or is stuck in mempool +} + +// Reset and abandon stuck transactions for an address +err = txm.Reset(address, true) +``` + +### Head Tracker Falling Behind + +Monitor finality violations: + +```go +// Check if tracker detected finality issues +report := tracker.HealthReport() +if err, ok := report["HeadTracker"]; ok && err != nil { + log.Errorf("HeadTracker unhealthy: %v", err) +} +```