From 21d5e0175dcab714f59c2a22543de13970954e09 Mon Sep 17 00:00:00 2001 From: Bruno Galvao Date: Tue, 18 Nov 2025 12:45:31 +0700 Subject: [PATCH 01/21] Add system parachain RPC guide and enhance documentation - Add comprehensive guide for running system parachain RPC nodes - Covers People Chain, Bridge Hub, and Coretime Chain - Provides both Docker and systemd deployment options - Includes snapshot setup, monitoring, and security best practices - Add conclusion sections to RPC and collator guides - polkadot-hub-rpc.md: summarize benefits and next steps - system-parachain-rpc.md: highlight adaptability across system parachains - collator.md: emphasize operator role and responsibilities - Restructure documentation from nodes-and-validators to node-infrastructure --- ...-infrastructure-run-a-collator-collator.md | 560 ++++++++++++++ ...rastructure-run-a-node-polkadot-hub-rpc.md | 622 +++++++++++++++ ...ructure-run-a-node-relay-chain-bootnode.md | 130 ++++ ...ucture-run-a-node-relay-chain-full-node.md | 338 ++++++++ ...cture-run-a-node-relay-chain-secure-wss.md | 125 +++ ...ructure-run-a-node-system-parachain-rpc.md | 652 ++++++++++++++++ ...boarding-and-offboarding-key-management.md | 155 ++++ ...arding-and-offboarding-set-up-validator.md | 213 ++++++ ...arding-and-offboarding-start-validating.md | 255 +++++++ ...oarding-and-offboarding-stop-validating.md | 45 ++ ...or-operational-tasks-general-management.md | 722 ++++++++++++++++++ ...ator-operational-tasks-pause-validating.md | 46 ++ ...tor-operational-tasks-upgrade-your-node.md | 75 ++ ...astructure-run-a-validator-requirements.md | 85 +++ ...-staking-mechanics-offenses-and-slashes.md | 171 +++++ ...n-a-validator-staking-mechanics-rewards.md | 204 +++++ .nav.yml | 2 +- .../.nav.yml | 1 + node-infrastructure/run-a-collator/.nav.yml | 2 + .../run-a-collator/collator.md | 557 ++++++++++++++ node-infrastructure/run-a-node/.nav.yml | 4 + .../run-a-node/polkadot-hub-rpc.md | 619 +++++++++++++++ .../run-a-node/relay-chain}/.nav.yml | 0 .../run-a-node/relay-chain}/bootnode.md | 0 .../run-a-node/relay-chain}/full-node.md | 0 .../run-a-node/relay-chain}/secure-wss.md | 0 .../run-a-node/system-parachain-rpc.md | 649 ++++++++++++++++ .../run-a-validator/.nav.yml | 0 .../onboarding-and-offboarding/.nav.yml | 0 .../key-management.md | 0 .../set-up-validator.md | 0 .../start-validating.md | 0 .../stop-validating.md | 0 .../operational-tasks/.nav.yml | 0 .../operational-tasks/general-management.md | 0 .../operational-tasks/pause-validating.md | 0 .../operational-tasks/upgrade-your-node.md | 0 .../run-a-validator/requirements.md | 0 .../staking-mechanics/.nav.yml | 0 .../staking-mechanics/offenses-and-slashes.md | 0 .../staking-mechanics/rewards.md | 0 41 files changed, 6231 insertions(+), 1 deletion(-) create mode 100644 .ai/pages/node-infrastructure-run-a-collator-collator.md create mode 100644 .ai/pages/node-infrastructure-run-a-node-polkadot-hub-rpc.md create mode 100644 .ai/pages/node-infrastructure-run-a-node-relay-chain-bootnode.md create mode 100644 .ai/pages/node-infrastructure-run-a-node-relay-chain-full-node.md create mode 100644 .ai/pages/node-infrastructure-run-a-node-relay-chain-secure-wss.md create mode 100644 .ai/pages/node-infrastructure-run-a-node-system-parachain-rpc.md create mode 100644 .ai/pages/node-infrastructure-run-a-validator-onboarding-and-offboarding-key-management.md create mode 100644 .ai/pages/node-infrastructure-run-a-validator-onboarding-and-offboarding-set-up-validator.md create mode 100644 .ai/pages/node-infrastructure-run-a-validator-onboarding-and-offboarding-start-validating.md create mode 100644 .ai/pages/node-infrastructure-run-a-validator-onboarding-and-offboarding-stop-validating.md create mode 100644 .ai/pages/node-infrastructure-run-a-validator-operational-tasks-general-management.md create mode 100644 .ai/pages/node-infrastructure-run-a-validator-operational-tasks-pause-validating.md create mode 100644 .ai/pages/node-infrastructure-run-a-validator-operational-tasks-upgrade-your-node.md create mode 100644 .ai/pages/node-infrastructure-run-a-validator-requirements.md create mode 100644 .ai/pages/node-infrastructure-run-a-validator-staking-mechanics-offenses-and-slashes.md create mode 100644 .ai/pages/node-infrastructure-run-a-validator-staking-mechanics-rewards.md rename {nodes-and-validators => node-infrastructure}/.nav.yml (66%) create mode 100644 node-infrastructure/run-a-collator/.nav.yml create mode 100644 node-infrastructure/run-a-collator/collator.md create mode 100644 node-infrastructure/run-a-node/.nav.yml create mode 100644 node-infrastructure/run-a-node/polkadot-hub-rpc.md rename {nodes-and-validators/run-a-node => node-infrastructure/run-a-node/relay-chain}/.nav.yml (100%) rename {nodes-and-validators/run-a-node => node-infrastructure/run-a-node/relay-chain}/bootnode.md (100%) rename {nodes-and-validators/run-a-node => node-infrastructure/run-a-node/relay-chain}/full-node.md (100%) rename {nodes-and-validators/run-a-node => node-infrastructure/run-a-node/relay-chain}/secure-wss.md (100%) create mode 100644 node-infrastructure/run-a-node/system-parachain-rpc.md rename {nodes-and-validators => node-infrastructure}/run-a-validator/.nav.yml (100%) rename {nodes-and-validators => node-infrastructure}/run-a-validator/onboarding-and-offboarding/.nav.yml (100%) rename {nodes-and-validators => node-infrastructure}/run-a-validator/onboarding-and-offboarding/key-management.md (100%) rename {nodes-and-validators => node-infrastructure}/run-a-validator/onboarding-and-offboarding/set-up-validator.md (100%) rename {nodes-and-validators => node-infrastructure}/run-a-validator/onboarding-and-offboarding/start-validating.md (100%) rename {nodes-and-validators => node-infrastructure}/run-a-validator/onboarding-and-offboarding/stop-validating.md (100%) rename {nodes-and-validators => node-infrastructure}/run-a-validator/operational-tasks/.nav.yml (100%) rename {nodes-and-validators => node-infrastructure}/run-a-validator/operational-tasks/general-management.md (100%) rename {nodes-and-validators => node-infrastructure}/run-a-validator/operational-tasks/pause-validating.md (100%) rename {nodes-and-validators => node-infrastructure}/run-a-validator/operational-tasks/upgrade-your-node.md (100%) rename {nodes-and-validators => node-infrastructure}/run-a-validator/requirements.md (100%) rename {nodes-and-validators => node-infrastructure}/run-a-validator/staking-mechanics/.nav.yml (100%) rename {nodes-and-validators => node-infrastructure}/run-a-validator/staking-mechanics/offenses-and-slashes.md (100%) rename {nodes-and-validators => node-infrastructure}/run-a-validator/staking-mechanics/rewards.md (100%) diff --git a/.ai/pages/node-infrastructure-run-a-collator-collator.md b/.ai/pages/node-infrastructure-run-a-collator-collator.md new file mode 100644 index 000000000..18babcfa8 --- /dev/null +++ b/.ai/pages/node-infrastructure-run-a-collator-collator.md @@ -0,0 +1,560 @@ +--- +title: Run a Block-Producing Collator +description: Learn how to set up and run a block-producing collator for Polkadot system parachains, including registration and session key management. +categories: Infrastructure +url: https://docs.polkadot.com/node-infrastructure/run-a-collator/collator/ +--- + +# Run a Block-Producing Collator + +## Overview + +Block-producing collators are the backbone of system parachain operations. Unlike RPC nodes or archive nodes that simply maintain state, collators actively produce blocks and submit them to relay chain validators for inclusion. + +This guide covers setting up a **block-producing collator** for Polkadot system parachains. Running a collator requires: + +- Meeting hardware requirements for reliable block production +- Setting up and registering session keys +- Obtaining governance approval or meeting selection criteria +- Maintaining high uptime and performance + +**Important**: System parachain collators typically require governance approval or being added to the invulnerables list. This is different from non-system parachains where collator selection may be more permissionless. + +## Collator Responsibilities + +Block-producing collators perform critical functions: + +- **Maintain full nodes**: Both relay chain and parachain +- **Collect transactions**: Aggregate user transactions into blocks +- **Produce blocks**: Create parachain block candidates +- **Generate proofs**: Produce state transition proofs (Proof-of-Validity) +- **Submit to validators**: Send block candidates to relay chain validators +- **Facilitate XCM**: Enable cross-chain message passing + +Unlike relay chain validators, collators do not provide security guarantees—that responsibility lies with relay chain validators through the ELVES protocol. However, collators are essential for network liveness and censorship resistance. + +## Prerequisites + +### Hardware Requirements + +Block-producing collators require robust hardware for reliable operation: + +- **CPU**: 4+ cores (8+ cores recommended for optimal performance) +- **Memory**: 32 GB RAM minimum (64 GB recommended) +- **Storage**: + - 500 GB+ NVMe SSD for parachain data + - Additional 200+ GB for relay chain pruned database + - Fast disk I/O is critical for block production performance +- **Network**: + - Public IP address (required) + - 100+ Mbps connection (stable connection critical) + - Open ports: + - 30333 (parachain P2P) + - 30334 (relay chain P2P) + - 9944 (WebSocket RPC - for management) + +**Note**: Uptime is critical. Consider redundancy and monitoring to maintain block production reliability. + +### Software Requirements + +Collators use the **Polkadot Omni Node**, a universal binary that runs any parachain using a chain specification file. + +Required software: + +- **Operating System**: Ubuntu 22.04 LTS (recommended) or similar Linux distribution +- **Docker**: For running subkey utility +- **Rust Toolchain**: Version 1.86 or as specified by the runtime +- **Dependencies**: + ```bash + sudo apt update + sudo apt install -y build-essential git clang curl libssl-dev llvm libudev-dev make protobuf-compiler + ``` + +### Account Requirements + +You'll need: +- **Funded account**: For on-chain transactions and potential bonding +- **Session keys**: For collator identification (generated after node setup) +- **Node key**: For stable P2P peer ID (recommended) + +## Installation + +### Step 1: Install Rust and Required Toolchain + +```bash +# Install Rust +curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh +source $HOME/.cargo/env + +# Install specific Rust version +rustup install 1.86 +rustup default 1.86 +rustup target add wasm32-unknown-unknown --toolchain 1.86 +rustup component add rust-src --toolchain 1.86 +``` + +### Step 2: Install the Polkadot Omni Node + +```bash +# Install polkadot-omni-node +cargo install --locked polkadot-omni-node@0.5.0 + +# Verify installation +polkadot-omni-node --version +``` + +### Step 3: Generate Node Key + +Generate a stable node key for consistent peer ID: + +```bash +# Create directory for node data +sudo mkdir -p /var/lib/polkadot-collator + +# Generate node key using Docker +docker run -it parity/subkey:latest generate-node-key > /var/lib/polkadot-collator/node.key + +# The output displays your peer ID +# Example: 12D3KooWExcVYu7Mvjd4kxPVLwN2ZPnZ5NyLZ5ft477wqzfP2q6E +``` + +Save the peer ID for future reference. + +### Step 4: Generate Account Key + +Generate an account for on-chain transactions: + +```bash +# Generate account key with sr25519 scheme +docker run -it parity/subkey:latest generate --scheme sr25519 +``` + +Save the output containing: +- Secret phrase (seed) - Keep this secure! +- Public key (hex) +- Account ID +- SS58 Address + +**Security**: Store the secret phrase securely. Never share it. Consider using a hardware wallet for production collators. + +### Step 5: Obtain Chain Specification + +Download the chain specification for your target system parachain: + +**Option 1: Download from Chainspec Collection (Recommended)** + +1. Visit the [Chainspec Collection website](https://paritytech.github.io/chainspecs/) +2. Find your target system parachain +3. Download the chain specification JSON file +4. Save it as `chain-spec.json` + +**Option 2: Build from Runtime** + +```bash +# Clone the runtimes repository +git clone https://github.com/polkadot-fellows/runtimes.git +cd runtimes + +# Build the desired runtime (example for Polkadot Hub) +cargo build --release -p asset-hub-polkadot-runtime + +# Install chain-spec-builder +cargo install --locked staging-chain-spec-builder@10.0.0 + +# Generate chain spec +chain-spec-builder create \ + --relay-chain polkadot \ + --para-id 1000 \ + --runtime target/release/wbuild/asset-hub-polkadot-runtime/asset_hub_polkadot_runtime.compact.compressed.wasm \ + named-preset production > chain-spec.json +``` + +**System Parachain Para IDs:** +- Polkadot Hub: 1000 +- Bridge Hub: 1002 +- People Chain: 1004 +- Coretime Chain: 1005 + +### Step 6: Create User and Directory Structure + +```bash +# Create dedicated user +sudo useradd -r -s /bin/bash polkadot + +# Copy chain spec to directory +sudo cp chain-spec.json /var/lib/polkadot-collator/ + +# Set permissions +sudo chown -R polkadot:polkadot /var/lib/polkadot-collator +``` + +## Configuration + +### Create Systemd Service File + +Create a service file for your collator: + +```bash +sudo nano /etc/systemd/system/polkadot-collator.service +``` + +Add the following configuration: + +```ini +[Unit] +Description=Polkadot System Parachain Collator +After=network.target + +[Service] +Type=simple +User=polkadot +Group=polkadot +WorkingDirectory=/var/lib/polkadot-collator + +# Block-Producing Collator Configuration +ExecStart=/usr/local/bin/polkadot-omni-node \ + --collator \ + --chain=/var/lib/polkadot-collator/chain-spec.json \ + --base-path=/var/lib/polkadot-collator \ + --port=30333 \ + --rpc-port=9944 \ + --prometheus-port=9615 \ + --node-key-file=/var/lib/polkadot-collator/node.key \ + --name="YourCollatorName" \ + -- \ + --execution=wasm \ + --chain=polkadot \ + --port=30334 \ + --sync=warp + +Restart=always +RestartSec=10 +LimitNOFILE=65536 + +[Install] +WantedBy=multi-user.target +``` + +**Configuration Notes**: +- `--collator`: Enables block production mode +- `--node-key-file`: Uses the generated node key for stable peer ID +- `--name`: Your collator name (visible in telemetry) +- Relay chain uses `--sync=warp` for faster initial sync + +## Running the Collator + +### Step 1: Start the Service + +```bash +# Reload systemd +sudo systemctl daemon-reload + +# Enable service to start on boot +sudo systemctl enable polkadot-collator + +# Start the service +sudo systemctl start polkadot-collator + +# Check status +sudo systemctl status polkadot-collator + +# View logs +sudo journalctl -u polkadot-collator -f +``` + +### Step 2: Initial Sync + +Your collator must sync both the relay chain and parachain before producing blocks. + +Sync time depends on: +- Network bandwidth +- Disk I/O speed +- Current chain size + +The relay chain uses warp sync for faster synchronization. + +Monitor sync progress: +```bash +# Check logs for sync status +sudo journalctl -u polkadot-collator -f | grep "Syncing" + +# Wait for messages indicating full sync +# Example: "Idle" or "Imported" messages for both chains +``` + +**Important**: Do not proceed with registration until both chains are fully synced. + +### Step 3: Generate Session Keys + +Once your node is fully synced, generate session keys: + +```bash +# Generate session keys via RPC +curl -H "Content-Type: application/json" \ + -d '{"id":1, "jsonrpc":"2.0", "method": "author_rotateKeys", "params":[]}' \ + http://localhost:9944 + +# Returns session keys as a hex string +# Example: "0x1234567890abcdef..." +``` + +**Save the session keys** - you'll need them for on-chain registration. + +**Note**: Session keys are stored in the node's database. If you wipe the database, you'll need to generate new keys. + +## Registration and Governance + +### Understanding Collator Selection + +System parachains use different collator selection mechanisms: + +**Invulnerables List**: +- Fixed list of collators approved through governance +- Most common for system parachains +- Requires governance proposal and approval + +**On-chain Selection**: +- Some parachains use pallet-collator-selection +- May require bonding tokens +- Automatic selection based on criteria + +**Fellowship Decisions**: +- Technical Fellowship may manage some system parachain collators +- Requires Fellowship membership or approval + +### Registration Process + +The registration process varies by system parachain. General steps: + +#### 1. Check Current Collators + +Check the existing collators for your target parachain: + +```bash +# Using Polkadot.js Apps +# Connect to your target system parachain +# Go to Developer > Chain State +# Query: collatorSelection.invulnerables() or similar +``` + +#### 2. Prepare Governance Proposal + +For invulnerables-based selection: + +1. **Draft proposal**: Explain why you should be added as a collator +2. **Technical details**: Provide your session keys and account ID +3. **Infrastructure**: Describe your hardware and monitoring setup +4. **Experience**: Detail your relevant experience + +Submit to: +- Polkadot Forum: https://forum.polkadot.network +- Relevant governance channels + +#### 3. Set Session Keys On-Chain + +Once approved (or if using on-chain selection), set your session keys: + +**Using Polkadot.js Apps:** + +1. Navigate to Polkadot.js Apps and connect to your system parachain +2. Go to **Developer > Extrinsics** +3. Select your account +4. Choose `session.setKeys` extrinsic +5. Enter: + - `keys`: Your session keys (from `author_rotateKeys`) + - `proof`: 0x00 (typically) +6. Submit and sign the transaction + +**Using CLI (alternative):** + +```bash +# This varies by parachain - consult specific documentation +``` + +#### 4. Bond Tokens (if required) + +Some parachains require bonding tokens: + +1. Go to **Developer > Extrinsics** +2. Select `collatorSelection.registerAsCandidate` (if available) +3. Submit with required bond amount +4. Sign transaction + +#### 5. Await Governance Approval + +If using invulnerables: +- Wait for governance vote +- Monitor forum and announcements +- Once approved, you'll be added to the invulnerables list +- Your collator will begin producing blocks in the next session/era + +### Verify Collator Status + +Check if your collator is active: + +```bash +# Monitor logs for block production +sudo journalctl -u polkadot-collator -f | grep -i "imported" + +# Look for messages like: +# "Prepared block for proposing" +# "Imported #123" +``` + +## Monitoring and Maintenance + +### Essential Monitoring + +**Block Production**: +```bash +# Monitor block production +sudo journalctl -u polkadot-collator | grep -i "prepared block" +``` + +**Peer Connections**: +- Maintain 30+ peers for good connectivity +- Check peer count in logs + +**Resource Usage**: +- Monitor CPU, RAM, and disk I/O +- Set up alerts for high usage + +**Sync Status**: +- Ensure both chains stay synced +- Alert on sync issues + +### Prometheus Metrics + +Metrics available at `http://localhost:9615/metrics` + +Example Prometheus configuration: +{% raw %} +```yaml +scrape_configs: + - job_name: 'polkadot-collator' + static_configs: + - targets: ['localhost:9615'] +``` +{% endraw %} + +Key metrics to monitor: +- `substrate_block_height`: Current block height +- `substrate_finalized_height`: Finalized block height +- `substrate_peers_count`: Peer connections +- `substrate_ready_transactions_number`: Transaction queue + +### Setting Up Alerts + +Configure alerts for: +- Service failures +- Sync issues +- Low peer count (< 10 peers) +- Block production gaps +- High resource usage +- Disk space low + +### Log Management + +```bash +# View recent logs +sudo journalctl -u polkadot-collator -n 100 + +# Follow logs in real-time +sudo journalctl -u polkadot-collator -f + +# Filter for errors +sudo journalctl -u polkadot-collator | grep -i error + +# Filter for block production +sudo journalctl -u polkadot-collator | grep -i "imported" +``` + +### Database Maintenance + +Check database size: +```bash +# Check database size +du -sh /var/lib/polkadot-collator +``` + +The node handles pruning automatically. + +### Updates and Upgrades + +**Runtime Upgrades**: +- Automatic via on-chain governance +- No manual action required +- Monitor announcements for breaking changes + +**Client Upgrades**: +- Require manual binary update +- Subscribe to announcements: + - Polkadot Forum + - Fellowship GitHub + - Matrix channels + +**Upgrade Procedure**: + +```bash +# Stop the service +sudo systemctl stop polkadot-collator + +# Backup data (recommended) +sudo cp -r /var/lib/polkadot-collator /var/lib/polkadot-collator.backup + +# Update polkadot-omni-node +cargo install --locked --force polkadot-omni-node@ + +# Verify version +polkadot-omni-node --version + +# Restart service +sudo systemctl start polkadot-collator + +# Monitor logs +sudo journalctl -u polkadot-collator -f +``` + +## Security Best Practices + +### Key Management + +- **Secure storage**: Store session keys and account keys securely +- **Never share**: Never share private keys or secret phrases +- **Hardware wallets**: Consider HSM for production +- **Backup**: Keep encrypted backups of keys +- **Rotation**: Plan for key rotation procedures + +### Network Security + +- **Firewall**: Restrict access to necessary ports only +- **SSH**: Use SSH keys, disable password auth +- **VPN**: Consider VPN for administrative access +- **DDoS protection**: Implement if running in cloud + +### System Security + +- **Updates**: Keep OS and software updated +- **Dedicated user**: Never run as root +- **Fail2ban**: Enable for SSH protection +- **Audits**: Regular security audits +- **Minimal services**: Disable unnecessary services + +### Operational Security + +- **Monitoring**: 24/7 monitoring with alerts +- **Backups**: Regular configuration backups +- **Documentation**: Document procedures +- **Incident response**: Have incident response plan +- **Redundancy**: Consider backup collator (standby) + +## Conclusion + +Running a collator node is essential for parachain operation and network security. By following this guide, you have set up a production-ready collator that: + +- Produces blocks for your parachain and maintains network consensus +- Implements comprehensive security measures to protect keys and operations +- Supports robust monitoring and alerting for reliable performance +- Follows best practices for both Docker and systemd deployments + +As a collator operator, you play a vital role in your parachain's infrastructure. Regular maintenance, security updates, and monitoring will ensure your collator continues to perform reliably. Stay engaged with your parachain community and keep up with updates to maintain optimal performance and security. diff --git a/.ai/pages/node-infrastructure-run-a-node-polkadot-hub-rpc.md b/.ai/pages/node-infrastructure-run-a-node-polkadot-hub-rpc.md new file mode 100644 index 000000000..7282f2ef0 --- /dev/null +++ b/.ai/pages/node-infrastructure-run-a-node-polkadot-hub-rpc.md @@ -0,0 +1,622 @@ +--- +title: Run an RPC Node for Polkadot Hub +description: Complete guide to set up and run an RPC node for Polkadot Hub with Polkadot SDK RPC endpoints. +categories: Infrastructure +url: https://docs.polkadot.com/node-infrastructure/run-a-node/polkadot-hub-rpc/ +--- + +# Run an RPC Node for Polkadot Hub + +## Overview + +[Polkadot Hub](/reference/polkadot-hub/){target=\_blank} is the entry point to Polkadot for all users and application developers. It provides access to essential Web3 services, including: + +- **Asset Management**: Native support for fungible and non-fungible assets +- **Governance, Staking, and Treasury**: Core protocol operations +- **Cross-chain Communication**: XCM message handling + +Running an RPC node for Polkadot Hub enables applications, wallets, and users to interact with the parachain through: +- **Polkadot SDK Node RPC** (Port 9944): Native Polkadot API (WebSocket and HTTP) + +This setup enables block explorer indexing and provides full compatibility with Polkadot SDK development tools. + +**Important Note**: The parameters and configurations in this guide are provided as illustrative examples. You may need to modify them according to your specific environment, hardware capabilities, and network conditions. + +## Prerequisites + +### Hardware Requirements + +RPC nodes serving production traffic require robust hardware: + +- **CPU**: 8+ cores (16+ cores for high traffic) +- **Memory**: 64 GB RAM minimum (128 GB recommended for high traffic) +- **Storage**: + - 500 GB+ NVMe SSD for parachain state (archive nodes require 2-4 TB+) + - Additional 200+ GB for relay chain pruned database + - Fast disk I/O is critical for query performance +- **Network**: + - Public IP address + - 1 Gbps connection (for high traffic scenarios) + - Stable internet connection with sufficient bandwidth + - Open ports: + - 30333 (parachain P2P) + - 30334 (relay chain P2P) + - 9944 (Polkadot SDK WebSocket RPC) + - 9933 (Polkadot SDK HTTP RPC) + - 9615 (Prometheus metrics - optional) + - Consider DDoS protection and rate limiting for production deployments + +**Note**: For development or low-traffic scenarios, you can reduce these requirements proportionally. Consider using a reverse proxy (nginx, Caddy) for production deployments. + +### Software Requirements + +Required software: + +- **Operating System**: Ubuntu 22.04 LTS (recommended) or similar Linux distribution +- **Docker**: Latest version installed and running (for Docker-based setup) +- **rclone**: (Optional but recommended) Command-line program for managing files on cloud storage (https://rclone.org/downloads/) +- **Rust Toolchain**: Version 1.86 or as specified by runtime (for manual build) + +## Setup Options + +This guide provides two deployment options: + +1. **Docker-based Setup**: Simpler to set up and maintain +2. **Manual/Systemd Setup**: For production environments requiring more control + +Choose the option that best fits your needs. + +--- + +## Option 1: Docker-Based Setup + +This option uses Docker containers for the Polkadot SDK node, making it easy to set up and manage. + +### Step 1: Download Chain Specification + +Download the official Polkadot Hub (formerly known as Asset Hub) chain specification: + +```bash +curl -L https://raw.githubusercontent.com/paritytech/polkadot-sdk/master/cumulus/parachains/chain-specs/asset-hub-polkadot.json -o asset-hub-polkadot.json +``` + +**Note**: This chain specification is the official configuration file that defines the network parameters for Polkadot Hub. + +### Step 2: Download Database Snapshots (Optional but Recommended) + +Using pre-synchronized snapshots significantly reduces initial sync time from several days to just a few hours. You need to download both parachain and relay chain data. + +**Snapshot Provider**: https://snapshots.polkadot.io/ + +#### Create Directories + +```bash +mkdir -p my-node-data/chains/asset-hub-polkadot/db +mkdir -p my-node-data/chains/polkadot/db +``` + +#### Download Polkadot Hub Parachain Snapshot + +Choose between archive (complete history) or pruned (recent state) snapshots: + +**Archive Snapshot** (recommended for RPC with historical data): + +```bash +# Check https://snapshots.polkadot.io/ for the latest snapshot URL +export SNAPSHOT_URL_ASSET_HUB="https://snapshots.polkadot.io/polkadot-asset-hub-rocksdb-archive/LATEST" + +rclone copyurl $SNAPSHOT_URL_ASSET_HUB/files.txt files.txt +rclone copy --progress --transfers 20 \ + --http-url $SNAPSHOT_URL_ASSET_HUB \ + --no-traverse --http-no-head --disable-http2 \ + --inplace --no-gzip-encoding --size-only \ + --retries 6 --retries-sleep 10s \ + --files-from files.txt :http: my-node-data/chains/asset-hub-polkadot/db/ + +rm files.txt +``` + +**Parameter Explanation**: +- `--transfers 20`: Uses 20 parallel transfers for faster download +- `--retries 6`: Automatically retries failed transfers up to 6 times +- `--retries-sleep 10s`: Waits 10 seconds between retry attempts +- `--size-only`: Only transfers if sizes differ (prevents unnecessary re-downloads) + +#### Download Polkadot Relay Chain Snapshot + +**Pruned Snapshot** (recommended for RPC nodes): + +```bash +# Check https://snapshots.polkadot.io/ for the latest snapshot URL +export SNAPSHOT_URL_RELAY="https://snapshots.polkadot.io/polkadot-rocksdb-prune/LATEST" + +rclone copyurl $SNAPSHOT_URL_RELAY/files.txt files.txt +rclone copy --progress --transfers 20 \ + --http-url $SNAPSHOT_URL_RELAY \ + --no-traverse --http-no-head --disable-http2 \ + --inplace --no-gzip-encoding --size-only \ + --retries 6 --retries-sleep 10s \ + --files-from files.txt :http: my-node-data/chains/polkadot/db/ + +rm files.txt +``` + +**Alternative Options**: +- Pruned snapshot: `polkadot-rocksdb-prune` (smaller size, recent state) +- Archive snapshot: `polkadot-rocksdb-archive` (complete history, larger size) + +### Step 3: Start Polkadot Hub Node + +Launch the node using the official Parity Docker image: + +**Docker Image**: https://hub.docker.com/r/parity/polkadot-omni-node + +```bash +docker run -d --name polkadot-hub-rpc --restart unless-stopped \ + -p 9944:9944 \ + -p 9933:9933 \ + -p 9615:9615 \ + -p 30334:30334 \ + -p 30333:30333 \ + -v $(pwd)/asset-hub-polkadot.json:/asset-hub-polkadot.json \ + -v $(pwd)/my-node-data:/data \ + parity/polkadot-omni-node:stable2506-4 \ + --name=PolkadotHubRPC \ + --base-path=/data \ + --chain=/asset-hub-polkadot.json \ + --prometheus-external \ + --prometheus-port 9615 \ + --unsafe-rpc-external \ + --rpc-port=9944 \ + --rpc-cors=all \ + --rpc-methods=safe \ + --rpc-max-connections=1000 \ + --state-pruning=archive \ + --blocks-pruning=archive \ + -- \ + --base-path=/data \ + --chain=polkadot \ + --state-pruning=256 \ + --blocks-pruning=archive-canonical +``` + +**Critical Configuration Parameters**: + +**Port Mappings**: +- `9944`: Polkadot SDK RPC endpoint (WebSocket/HTTP) +- `9933`: Polkadot SDK HTTP RPC endpoint +- `9615`: Prometheus metrics endpoint +- `30333/30334`: P2P networking ports + +**Node Parameters**: +- `--unsafe-rpc-external`: Enables external RPC access +- `--rpc-cors=all`: Allows all origins for CORS +- `--rpc-methods=safe`: Only allows safe RPC methods +- `--state-pruning=archive`: Keeps complete state history +- `--blocks-pruning=archive`: Keeps all block data +- `--prometheus-external`: Exposes metrics externally + +**Security Warning**: The `--unsafe-rpc-external` flag should only be used in development or properly secured environments. For production, use a reverse proxy with authentication. + +### Step 4: Monitor Synchronization + +Monitor the node synchronization status: + +```bash +# Check sync status +curl -H "Content-Type: application/json" \ + -d '{"id":1, "jsonrpc":"2.0", "method": "system_syncState", "params":[]}' \ + http://localhost:9944 +``` + +**Expected Response Format**: + +```json +{ + "jsonrpc":"2.0", + "id":1, + "result":{ + "startingBlock":0, + "currentBlock":3394816, + "highestBlock":3394816 + } +} +``` + +**Synchronization Status**: +- **In Progress**: `currentBlock` < `highestBlock` +- **Complete**: `currentBlock` = `highestBlock` + +**Monitor logs**: + +```bash +# View node logs +docker logs -f polkadot-hub-rpc + +# Filter for sync messages +docker logs polkadot-hub-rpc 2>&1 | grep -i "syncing" +``` + +### Step 5: Verify Setup + +Let's verify the Polkadot SDK RPC endpoint is working correctly. + +#### API Endpoint + +**Polkadot SDK RPC (Port 9944)**: +- WebSocket: `ws://your-server:9944` +- HTTP: `http://your-server:9944` +- Purpose: Full Polkadot SDK API access for parachain data +- Use Cases: Polkadot SDK applications, parachain-specific operations + +#### Polkadot SDK RPC Tests + +**Get Chain Information**: + +```bash +curl -H "Content-Type: application/json" \ + -d '{"id":1, "jsonrpc":"2.0", "method": "system_chain", "params":[]}' \ + http://localhost:9944 +``` + +**Get Latest Block**: + +```bash +curl -H "Content-Type: application/json" \ + -d '{"id":1, "jsonrpc":"2.0", "method": "chain_getHeader", "params":[]}' \ + http://localhost:9944 +``` + +**Get Node Health**: + +```bash +curl -H "Content-Type: application/json" \ + -d '{"id":1, "jsonrpc":"2.0", "method": "system_health", "params":[]}' \ + http://localhost:9944 +``` + +### Managing Docker Containers + +**View logs**: + +```bash +# View node logs +docker logs -f polkadot-hub-rpc +``` + +**Stop container**: + +```bash +docker stop polkadot-hub-rpc +``` + +**Start container**: + +```bash +docker start polkadot-hub-rpc +``` + +**Remove container**: + +```bash +docker rm polkadot-hub-rpc +``` + +**Update container**: + +```bash +# Pull latest image +docker pull parity/polkadot-omni-node:stable2506-4 + +# Stop and remove old container +docker stop polkadot-hub-rpc +docker rm polkadot-hub-rpc + +# Start new container with same command as above +``` + +--- + +## Option 2: Manual/Systemd Setup + +This option provides more control and is recommended for production environments requiring custom configurations. + +### Step 1: Install Rust and Required Toolchain + +```bash +# Install Rust +curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh +source $HOME/.cargo/env + +# Install specific Rust version +rustup install 1.86 +rustup default 1.86 +rustup target add wasm32-unknown-unknown --toolchain 1.86 +rustup component add rust-src --toolchain 1.86 +``` + +### Step 2: Install the Polkadot Omni Node + +```bash +# Install polkadot-omni-node +cargo install --locked polkadot-omni-node@0.7.0 + +# Verify installation +polkadot-omni-node --version +``` + +### Step 3: Obtain Chain Specification + +Download the Polkadot Hub chain specification: + +```bash +curl -L https://raw.githubusercontent.com/paritytech/polkadot-sdk/master/cumulus/parachains/chain-specs/asset-hub-polkadot.json -o asset-hub-polkadot.json +``` + +### Step 4: Create User and Directory Structure + +```bash +# Create dedicated user +sudo useradd -r -s /bin/bash polkadot + +# Create data directory +sudo mkdir -p /var/lib/polkadot-hub-rpc + +# Copy chain spec to the directory +sudo cp asset-hub-polkadot.json /var/lib/polkadot-hub-rpc/ + +# Set permissions +sudo chown -R polkadot:polkadot /var/lib/polkadot-hub-rpc +``` + +### Step 5: Create Systemd Service for Polkadot SDK Node + +Create a service file for the Polkadot SDK RPC node: + +```bash +sudo nano /etc/systemd/system/polkadot-hub-rpc.service +``` + +Add the following configuration: + +```ini +[Unit] +Description=Polkadot Hub RPC Node +After=network.target + +[Service] +Type=simple +User=polkadot +Group=polkadot +WorkingDirectory=/var/lib/polkadot-hub-rpc + +ExecStart=/usr/local/bin/polkadot-omni-node \ + --name=PolkadotHubRPC \ + --chain=/var/lib/polkadot-hub-rpc/asset-hub-polkadot.json \ + --base-path=/var/lib/polkadot-hub-rpc \ + --port=30333 \ + --rpc-port=9944 \ + --rpc-external \ + --rpc-cors=all \ + --rpc-methods=safe \ + --rpc-max-connections=1000 \ + --prometheus-port=9615 \ + --prometheus-external \ + --state-pruning=archive \ + --blocks-pruning=archive \ + -- \ + --chain=polkadot \ + --base-path=/var/lib/polkadot-hub-rpc \ + --port=30334 \ + --state-pruning=256 \ + --blocks-pruning=archive-canonical + +Restart=always +RestartSec=10 +LimitNOFILE=65536 + +[Install] +WantedBy=multi-user.target +``` + +### Step 6: Start Service + +```bash +# Reload systemd +sudo systemctl daemon-reload + +# Enable service to start on boot +sudo systemctl enable polkadot-hub-rpc + +# Start the Polkadot SDK node +sudo systemctl start polkadot-hub-rpc + +# Check status and wait for sync +sudo systemctl status polkadot-hub-rpc +sudo journalctl -u polkadot-hub-rpc -f +``` + +### Step 7: Verify Setup + +Use the same verification tests as in the Docker setup (see Step 5 above). + +--- + +## Monitoring and Maintenance + +### Log Management + +**Docker Setup**: + +```bash +# View node logs +docker logs -f polkadot-hub-rpc +``` + +**Systemd Setup**: + +```bash +# View node logs +sudo journalctl -u polkadot-hub-rpc -f + +# View recent logs +sudo journalctl -u polkadot-hub-rpc -n 100 + +# Filter for errors +sudo journalctl -u polkadot-hub-rpc | grep -i error +``` + +### Performance Monitoring + +Monitor key metrics: +- **Sync status**: Ensure node stays fully synced +- **Peer connections**: Maintain 30+ peers for good connectivity +- **Resource usage**: Monitor CPU, RAM, and disk I/O +- **RPC request latency**: Track response times for the Polkadot SDK API +- **Connection count**: Monitor active RPC connections + +**Prometheus Metrics**: + +Metrics are available at `http://localhost:9615/metrics` + +Example Prometheus scrape configuration: + +{% raw %} +```yaml +scrape_configs: + - job_name: 'polkadot-hub-rpc' + static_configs: + - targets: ['localhost:9615'] +``` +{% endraw %} + +**Key Metrics to Monitor**: +- `substrate_block_height`: Current block height +- `substrate_finalized_height`: Finalized block height +- `substrate_peers_count`: Number of connected peers +- `substrate_ready_transactions_number`: Transaction queue size + +### Database Maintenance + +Check database size periodically: + +```bash +# Docker setup +du -sh my-node-data + +# Systemd setup +du -sh /var/lib/polkadot-hub-rpc +``` + +The node handles pruning automatically based on configuration unless running in archive mode. + +### Updates and Upgrades + +**Docker Setup**: + +```bash +# Pull latest image +docker pull parity/polkadot-omni-node:stable2506-4 + +# Restart container +docker stop polkadot-hub-rpc +docker rm polkadot-hub-rpc + +# Start new container (use same command from setup) +``` + +**Systemd Setup**: + +```bash +# Stop service +sudo systemctl stop polkadot-hub-rpc + +# Backup data +sudo cp -r /var/lib/polkadot-hub-rpc /var/lib/polkadot-hub-rpc.backup + +# Update binary +cargo install --locked --force polkadot-omni-node@ + +# Restart service +sudo systemctl start polkadot-hub-rpc +``` + +## Security Best Practices + +### Network Security + +1. **Firewall Configuration**: + - Only expose necessary ports + - Use UFW or iptables to restrict access + - Consider IP whitelisting for RPC endpoints + +2. **Reverse Proxy** (Recommended for Production): + - Use nginx or Caddy as reverse proxy + - Enable SSL/TLS (HTTPS/WSS) + - Implement authentication + - Add rate limiting + +Example nginx configuration: + +```nginx +upstream polkadot_sdk_rpc { + server 127.0.0.1:9944; +} + +server { + listen 443 ssl http2; + server_name your-domain.com; + + ssl_certificate /path/to/cert.pem; + ssl_certificate_key /path/to/key.pem; + + # Polkadot SDK RPC + location /polkadot { + proxy_pass http://polkadot_sdk_rpc; + proxy_http_version 1.1; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection "upgrade"; + + # Rate limiting + limit_req zone=rpc_limit burst=10; + } +} +``` + +### RPC Security + +- **Always use `--rpc-methods=safe`**: Prevents dangerous RPC calls +- **Restrict CORS**: Use specific domains instead of `all` in production +- **Set connection limits**: Prevent resource exhaustion +- **Monitor for abuse**: Track unusual patterns +- **Authentication**: Implement API keys or OAuth for production + +### System Security + +- Keep operating system updated +- Use dedicated user accounts (never root) +- Enable fail2ban for SSH protection +- Regular security audits +- Disable unnecessary services +- Use AppArmor or SELinux for additional isolation + +### Monitoring and Alerting + +Set up alerts for: +- Service failures +- Sync issues +- Low peer count (< 10 peers) +- High resource usage +- Unusual RPC traffic patterns +- Database errors + +## Conclusion + +Running an RPC node for Polkadot Hub provides essential infrastructure for applications and users to interact with the network. By following this guide, you have set up a production-ready RPC node that: + +- Provides reliable access to Polkadot Hub's asset management, governance, and cross-chain communication features +- Supports both Docker and systemd deployment options for flexibility +- Implements proper monitoring, security, and maintenance practices +- Serves as a foundation for building and operating Polkadot SDK applications + +Regular maintenance, security updates, and monitoring will ensure your RPC node continues to serve your users reliably. As the Polkadot network evolves, stay informed about updates and best practices through the official channels and community resources listed in this guide. diff --git a/.ai/pages/node-infrastructure-run-a-node-relay-chain-bootnode.md b/.ai/pages/node-infrastructure-run-a-node-relay-chain-bootnode.md new file mode 100644 index 000000000..8b91741f0 --- /dev/null +++ b/.ai/pages/node-infrastructure-run-a-node-relay-chain-bootnode.md @@ -0,0 +1,130 @@ +--- +title: Set Up a Bootnode +description: Learn how to configure and run a bootnode for Polkadot, including P2P, WS, and secure WSS connections with network key management and proxies. +categories: Infrastructure +url: https://docs.polkadot.com/node-infrastructure/run-a-node/relay-chain/bootnode/ +--- + +# Set Up a Bootnode + +## Introduction + +Bootnodes are essential for helping blockchain nodes discover peers and join the network. When a node starts, it needs to find other nodes, and bootnodes provide an initial point of contact. Once connected, a node can expand its peer connections and play its role in the network, like participating as a validator. + +This guide will walk you through setting up a Polkadot bootnode, configuring P2P, WebSocket (WS), secure WSS connections, and managing network keys. You'll also learn how to test your bootnode to ensure it is running correctly and accessible to other nodes. + +## Prerequisites + +Before you start, you need to have the following prerequisites: + +- Verify a working Polkadot (`polkadot`) binary is available on your machine. +- Ensure you have nginx installed. Please refer to the [Installation Guide](https://nginx.org/en/docs/install.html){target=\_blank} for help with installation if needed. +- A VPS or other dedicated server setup. + +## Accessing the Bootnode + +Bootnodes must be accessible through three key channels to connect with other nodes in the network: + +- **P2P**: A direct peer-to-peer connection, set by. + + ```bash + + --listen-addr /ip4/0.0.0.0/tcp/INSERT_PORT + + ``` + + This is not enabled by default on non-validator nodes like archive RPC nodes. + +- **P2P/WS**: A WebSocket (WS) connection, also configured via `--listen-addr`. +- **P2P/WSS**: A secure WebSocket (WSS) connection using SSL, often required for light clients. An SSL proxy is needed, as the node itself cannot handle certificates. + +## Node Key + +A node key is the ED25519 key used by `libp2p` to assign your node an identity or peer ID. Generating a known node key for a bootnode is crucial, as it gives you a consistent key that can be placed in chain specifications as a known, reliable bootnode. + +Starting a node creates its node key in the `chains/INSERT_CHAIN/network/secret_ed25519` file. + +You can create a node key using: + + ``` bash + polkadot key generate-node-key + ``` + +This key can be used in the startup command line. + +It is imperative that you backup the node key. If it is included in the `polkadot` binary, it is hardcoded into the binary, which must be recompiled to change the key. + +## Running the Bootnode + +A bootnode can be run as follows: + + ``` bash + polkadot --chain polkadot \ + --name dot-bootnode \ + --listen-addr /ip4/0.0.0.0/tcp/30310 \ + --listen-addr /ip4/0.0.0.0/tcp/30311/ws + ``` + +This assigns the p2p to port 30310 and p2p/ws to port 30311. For the p2p/wss port, a proxy must be set up with a DNS name and a corresponding certificate. The following example is for the popular nginx server and enables p2p/wss on port 30312 by adding a proxy to the p2p/ws port 30311: + +``` conf title="/etc/nginx/sites-enabled/dot-bootnode" +server { + listen 30312 ssl http2 default_server; + server_name dot-bootnode.stakeworld.io; + root /var/www/html; + + ssl_certificate "INSERT_YOUR_CERT"; + ssl_certificate_key "INSERT_YOUR_KEY"; + + location / { + proxy_buffers 16 4k; + proxy_buffer_size 2k; + proxy_pass http://localhost:30311; + proxy_http_version 1.1; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection "Upgrade"; + proxy_set_header Host $host; + } + +} +``` + +## Testing Bootnode Connection + +If the preceding node is running with DNS name `dot-bootnode.stakeworld.io`, which contains a proxy with a valid certificate and node-id `12D3KooWAb5MyC1UJiEQJk4Hg4B2Vi3AJdqSUhTGYUqSnEqCFMFg` then the following commands should output `syncing 1 peers`. + +!!!tip + You can add `-lsub-libp2p=trace` on the end to get libp2p trace logging for debugging purposes. + +### P2P + +```bash +polkadot --chain polkadot \ +--base-path /tmp/node \ +--name "Bootnode testnode" \ +--reserved-only \ +--reserved-nodes "/dns/dot-bootnode.stakeworld.io/tcp/30310/p2p/12D3KooWAb5MyC1UJiEQJk4Hg4B2Vi3AJdqSUhTGYUqSnEqCFMFg" \ +--no-hardware-benchmarks +``` + +### P2P/WS + +```bash +polkadot --chain polkadot \ +--base-path /tmp/node \ +--name "Bootnode testnode" \ +--reserved-only \ +--reserved-nodes "/dns/dot-bootnode.stakeworld.io/tcp/30311/ws/p2p/12D3KooWAb5MyC1UJiEQJk4Hg4B2Vi3AJdqSUhTGYUqSnEqCFMFg" \ +--no-hardware-benchmarks +``` + +### P2P/WSS + +```bash +polkadot --chain polkadot \ +--base-path /tmp/node \ +--name "Bootnode testnode" \ +--reserved-only \ +--reserved-nodes "/dns/dot-bootnode.stakeworld.io/tcp/30312/wss/p2p/12D3KooWAb5MyC1UJiEQJk4Hg4B2Vi3AJdqSUhTGYUqSnEqCFMFg" \ +--no-hardware-benchmarks +``` diff --git a/.ai/pages/node-infrastructure-run-a-node-relay-chain-full-node.md b/.ai/pages/node-infrastructure-run-a-node-relay-chain-full-node.md new file mode 100644 index 000000000..17f93a8f1 --- /dev/null +++ b/.ai/pages/node-infrastructure-run-a-node-relay-chain-full-node.md @@ -0,0 +1,338 @@ +--- +title: Set Up a Node +description: Learn how to install, configure, and run Polkadot nodes, including setting up different node types and connecting to the network. +categories: Infrastructure +url: https://docs.polkadot.com/node-infrastructure/run-a-node/relay-chain/full-node/ +--- + +# Set Up a Node + +## Introduction + +Running a node on Polkadot provides direct interaction with the network, enhanced privacy, and full control over RPC requests, transactions, and data queries. As the backbone of the network, nodes ensure decentralized data propagation, transaction validation, and seamless communication across the ecosystem. + +Polkadot supports multiple node types, including pruned, archive, and light nodes, each suited to specific use cases. During setup, you can use configuration flags to choose the node type you wish to run. + +This guide walks you through configuring, securing, and maintaining a node on Polkadot or any Polkadot SDK-based chain. It covers instructions for the different node types and how to safely expose your node's RPC server for external access. Whether you're building a local development environment, powering dApps, or supporting network decentralization, this guide provides all the essentials. + +## Set Up a Node + +Now that you're familiar with the different types of nodes, this section will walk you through configuring, securing, and maintaining a node on Polkadot or any Polkadot SDK-based chain. + +### Prerequisites + +Before getting started, ensure the following prerequisites are met: + +- Ensure [Rust](https://www.rust-lang.org/tools/install){target=\_blank} is installed on your operating system. +- [Install the necessary dependencies for the Polkadot SDK](/parachains/install-polkadot-sdk/){target=\_blank}. + +!!! warning + This setup is not recommended for validators. If you plan to run a validator, refer to the [Running a Validator](/nodes-and-validators/run-a-validator/onboarding-and-offboarding/set-up-validator/){target=\_blank} guide for proper instructions. + +### Install and Build the Polkadot Binary + +This section will walk you through installing and building the Polkadot binary for different operating systems and methods. + +??? interface "macOS" + + To get started, update and configure the Rust toolchain by running the following commands: + + ```bash + source ~/.cargo/env + + rustup default stable + rustup update + + rustup update nightly + rustup target add wasm32-unknown-unknown --toolchain nightly + rustup component add rust-src --toolchain stable-aarch64-apple-darwin + ``` + + You can verify your installation by running: + + ```bash + rustup show + rustup +nightly show + ``` + + You should see output similar to the following: + +
+ rustup show
+ rustup +nightly show + active toolchain + ---------------- + + stable-aarch64-apple-darwin (default) + rustc 1.82.0 (f6e511eec 2024-10-15) + + active toolchain + ---------------- + + nightly-aarch64-apple-darwin (overridden by +toolchain on the command line) + rustc 1.84.0-nightly (03ee48451 2024-11-18) + +
+ + Then, run the following commands to clone and build the Polkadot binary: + + ```bash + git clone https://github.com/paritytech/polkadot-sdk polkadot-sdk + cd polkadot-sdk + cargo build --release + ``` + + Depending upon the specs of your machine, compiling the binary may take an hour or more. After building the Polkadot node from source, the executable binary will be located in the `./target/release/polkadot` directory. + +??? interface "Windows" + + To get started, make sure that you have [WSL and Ubuntu](https://learn.microsoft.com/en-us/windows/wsl/install){target=\_blank} installed on your Windows machine. + + Once installed, you have a couple options for installing the Polkadot binary: + + - If Rust is installed, then `cargo` can be used similar to the macOS instructions. + - Or, the instructions in the Linux section can be used. + +??? interface "Linux (pre-built binary)" + + To grab the [latest release of the Polkadot binary](https://github.com/paritytech/polkadot-sdk/releases){target=\_blank}, you can use `wget`: + + ```bash + wget https://github.com/paritytech/polkadot-sdk/releases/download/polkadot-INSERT_VERSION/polkadot + ``` + + Ensure you note the executable binary's location, as you'll need to use it when running the start-up command. If you prefer, you can specify the output location of the executable binary with the `-O` flag, for example: + + ```bash + wget https://github.com/paritytech/polkadot-sdk/releases/download/polkadot-INSERT_VERSION/polkadot \ + - O /var/lib/polkadot-data/polkadot + ``` + + !!!tip + The nature of pre-built binaries means that they may not work on your particular architecture or Linux distribution. If you see an error like `cannot execute binary file: Exec format error` it likely means the binary is incompatible with your system. You will either need to compile the binary or use [Docker](#use-docker). + + Ensure that you properly configure the permissions to make the Polkadot release binary executable: + + ```bash + sudo chmod +x polkadot + ``` + +??? interface "Linux (compile binary)" + + The most reliable (although perhaps not the fastest) way of launching a full node is to compile the binary yourself. Depending on your machine's specs, this may take an hour or more. + + To get started, run the following commands to configure the Rust toolchain: + + ```bash + rustup default stable + rustup update + rustup update nightly + rustup target add wasm32-unknown-unknown --toolchain nightly + rustup target add wasm32-unknown-unknown --toolchain stable-x86_64-unknown-linux-gnu + rustup component add rust-src --toolchain stable-x86_64-unknown-linux-gnu + ``` + + You can verify your installation by running: + + ```bash + rustup show + ``` + + You should see output similar to the following: + +
+ rustup show
+ rustup +nightly show + active toolchain + ---------------- + + stable-x86_64-unknown-linux-gnu (default) + rustc 1.82.0 (f6e511eec 2024-10-15) +
+ + Once Rust is configured, run the following commands to clone and build Polkadot: + + ```bash + git clone https://github.com/paritytech/polkadot-sdk polkadot-sdk + cd polkadot-sdk + cargo build --release + ``` + + Compiling the binary may take an hour or more, depending on your machine's specs. After building the Polkadot node from the source, the executable binary will be located in the `./target/release/polkadot` directory. + +??? interface "Linux (snap package)" + + Polkadot can be installed as a [snap package](https://snapcraft.io/polkadot){target=\_blank}. If you don't already have Snap installed, take the following steps to install it: + + ```bash + sudo apt update + sudo apt install snapd + ``` + + Install the Polkadot snap package: + + ```bash + sudo snap install polkadot + ``` + + Before continuing on with the following instructions, check out the [Configure and Run Your Node](#configure-and-run-your-node) section to learn more about the configuration options. + + To configure your Polkadot node with your desired options, you'll run a command similar to the following: + + ```bash + sudo snap set polkadot service-args="--name=MyName --chain=polkadot" + ``` + + Then to start the node service, run: + + ```bash + sudo snap start polkadot + ``` + + You can review the logs to check on the status of the node: + + ```bash + snap logs polkadot -f + ``` + + And at any time, you can stop the node service: + + ```bash + sudo snap stop polkadot + ``` + + You can optionally prevent the service from stopping when snap is updated with the following command: + + ```bash + sudo snap set polkadot endure=true + ``` + +### Use Docker + +As an additional option, you can use Docker to run your node in a container. Doing this is more advanced, so it's best left up to those already familiar with Docker or who have completed the other set-up instructions in this guide. You can review the latest versions on [DockerHub](https://hub.docker.com/r/parity/polkadot/tags){target=\_blank}. + +Be aware that when you run Polkadot in Docker, the process only listens on `localhost` by default. If you would like to connect to your node's services (RPC and Prometheus) you need to ensure that you run the node with the `--rpc-external`, and `--prometheus-external` commands. + +```bash +docker run -p 9944:9944 -p 9615:9615 parity/polkadot:v1.16.2 --name "my-polkadot-node-calling-home" --rpc-external --prometheus-external +``` + +If you're running Docker on an Apple Silicon machine (e.g. M4), you'll need to adapt the command slightly: + +```bash +docker run --platform linux/amd64 -p 9944:9944 -p 9615:9615 parity/polkadot:v1.16.2 --name "kearsarge-calling-home" --rpc-external --prometheus-external +``` + +## Configure and Run Your Node + +Now that you've installed and built the Polkadot binary, the next step is to configure the start-up command depending on the type of node that you want to run. You'll need to modify the start-up command accordingly based on the location of the binary. In some cases, it may be located within the `./target/release/` folder, so you'll need to replace polkadot with `./target/release/polkadot` in the following commands. + +Also, note that you can use the same binary for Polkadot as you would for Kusama or any other relay chain. You'll need to use the `--chain` flag to differentiate between chains. + +The base commands for running a Polkadot node are as follows: + +=== "Default pruned node" + + This uses the default pruning value of the last 256 blocks: + + ```bash + polkadot --chain polkadot \ + --name "INSERT_NODE_NAME" + ``` + +=== "Custom pruned node" + + You can customize the pruning value, for example, to the last 1000 finalized blocks: + + ```bash + polkadot --chain polkadot \ + --name INSERT_YOUR_NODE_NAME \ + --state-pruning 1000 \ + --blocks-pruning archive \ + --rpc-cors all \ + --rpc-methods safe + ``` + +=== "Archive node" + + To support the full state, use the `archive` option: + + ```bash + polkadot --chain polkadot \ + --name INSERT_YOUR_NODE_NAME \ + --state-pruning archive \ + --blocks-pruning archive \ + ``` + +If you want to run an RPC node, please refer to the following [RPC Configurations](#rpc-configurations) section. + +To review a complete list of the available commands, flags, and options, you can use the `--help` flag: + +```bash +polkadot --help +``` + +Once you've fully configured your start-up command, you can execute it in your terminal and your node will start [syncing](#sync-your-node). + +### RPC Configurations + +The node startup settings allow you to choose what to expose, how many connections to expose, and which systems should be granted access through the RPC server. + +- You can limit the methods to use with `--rpc-methods`; an easy way to set this to a safe mode is `--rpc-methods safe`. +- You can set your maximum connections through `--rpc-max-connections`, for example, `--rpc-max-connections 200`. +- By default, localhost and Polkadot.js can access the RPC server. You can change this by setting `--rpc-cors`. To allow access from everywhere, you can use `--rpc-cors all`. + +For a list of important flags when running RPC nodes, refer to the Parity DevOps documentation: [Important Flags for Running an RPC Node](https://paritytech.github.io/devops-guide/guides/rpc_index.html?#important-flags-for-running-an-rpc-node){target=\_blank}. + +## Sync Your Node + +The syncing process will take a while, depending on your capacity, processing power, disk speed, and RAM. The process may be completed on a $10 DigitalOcean droplet in about ~36 hours. While syncing, your node name should be visible in gray on Polkadot Telemetry, and once it is fully synced, your node name will appear in white on [Polkadot Telemetry](https://telemetry.polkadot.io/#list/Polkadot){target=_blank}. + +A healthy node syncing blocks will output logs like the following: + +
+ 2024-11-19 23:49:57 Parity Polkadot + 2024-11-19 23:49:57 ✌️ version 1.14.1-7c4cd60da6d + 2024-11-19 23:49:57 ❤️ by Parity Technologies <admin@parity.io>, 2017-2024 + 2024-11-19 23:49:57 📋 Chain specification: Polkadot + 2024-11-19 23:49:57 🏷 Node name: myPolkadotNode + 2024-11-19 23:49:57 👤 Role: FULL + 2024-11-19 23:49:57 💾 Database: RocksDb at /home/ubuntu/.local/share/polkadot/chains/polkadot/db/full + 2024-11-19 23:50:00 🏷 Local node identity is: 12D3KooWDmhHEgPRJUJnUpJ4TFWn28EENqvKWH4dZGCN9TS51y9h + 2024-11-19 23:50:00 Running libp2p network backend + 2024-11-19 23:50:00 💻 Operating system: linux + 2024-11-19 23:50:00 💻 CPU architecture: x86_64 + 2024-11-19 23:50:00 💻 Target environment: gnu + 2024-11-19 23:50:00 💻 CPU: Intel(R) Xeon(R) CPU E3-1245 V2 @ 3.40GHz + 2024-11-19 23:50:00 💻 CPU cores: 4 + 2024-11-19 23:50:00 💻 Memory: 32001MB + 2024-11-19 23:50:00 💻 Kernel: 5.15.0-113-generic + 2024-11-19 23:50:00 💻 Linux distribution: Ubuntu 22.04.5 LTS + 2024-11-19 23:50:00 💻 Virtual machine: no + 2024-11-19 23:50:00 📦 Highest known block at #9319 + 2024-11-19 23:50:00 〽️ Prometheus exporter started at 127.0.0.1:9615 + 2024-11-19 23:50:00 Running JSON-RPC server: addr=127.0.0.1:9944, allowed origins=["http://localhost:*", "http://127.0.0.1:*", "https://localhost:*", "https://127.0.0.1:*", "https://polkadot.js.org"] + 2024-11-19 23:50:00 🏁 CPU score: 671.67 MiBs + 2024-11-19 23:50:00 🏁 Memory score: 7.96 GiBs + 2024-11-19 23:50:00 🏁 Disk score (seq. writes): 377.87 MiBs + 2024-11-19 23:50:00 🏁 Disk score (rand. writes): 147.92 MiBs + 2024-11-19 23:50:00 🥩 BEEFY gadget waiting for BEEFY pallet to become available... + 2024-11-19 23:50:00 🔍 Discovered new external address for our node: /ip4/37.187.93.17/tcp/30333/ws/p2p/12D3KooWDmhHEgPRJUJnUpJ4TFWn28EENqvKWH4dZGCN9TS51y9h + 2024-11-19 23:50:01 🔍 Discovered new external address for our node: /ip6/2001:41d0:a:3511::1/tcp/30333/ws/p2p/12D3KooWDmhHEgPRJUJnUpJ4TFWn28EENqvKWH4dZGCN9TS51y9h + 2024-11-19 23:50:05 ⚙️ Syncing, target=#23486325 (5 peers), best: #12262 (0x8fb5…f310), finalized #11776 (0x9de1…32fb), ⬇ 430.5kiB/s ⬆ 17.8kiB/s + 2024-11-19 23:50:10 ⚙️ Syncing 628.8 bps, target=#23486326 (6 peers), best: #15406 (0x9ce1…2d76), finalized #15360 (0x0e41…a064), ⬇ 255.0kiB/s ⬆ 1.8kiB/s +
+ +Congratulations, you're now syncing a Polkadot full node! Remember that the process is identical when using any other Polkadot SDK-based chain, although individual chains may have chain-specific flag requirements. + +### Connect to Your Node + +Open [Polkadot.js Apps](https://polkadot.js.org/apps/?rpc=ws%3A%2F%2F127.0.0.1%3A9944#/explorer){target=\_blank} and click the logo in the top left to switch the node. Activate the **Development** toggle and input your node's domain or IP address. The default WSS endpoint for a local node is: + +```bash +ws://127.0.0.1:9944 +``` diff --git a/.ai/pages/node-infrastructure-run-a-node-relay-chain-secure-wss.md b/.ai/pages/node-infrastructure-run-a-node-relay-chain-secure-wss.md new file mode 100644 index 000000000..c39c29993 --- /dev/null +++ b/.ai/pages/node-infrastructure-run-a-node-relay-chain-secure-wss.md @@ -0,0 +1,125 @@ +--- +title: Set Up Secure WebSocket +description: Instructions on enabling SSL for your node and setting up a secure WebSocket proxy server using nginx for remote connections. +categories: Infrastructure +url: https://docs.polkadot.com/node-infrastructure/run-a-node/relay-chain/secure-wss/ +--- + +# Set Up Secure WebSocket + +## Introduction + +Ensuring secure WebSocket communication is crucial for maintaining the integrity and security of a Polkadot or Kusama node when interacting with remote clients. This guide walks you through setting up a secure WebSocket (WSS) connection for your node by leveraging SSL encryption with popular web server proxies like nginx or Apache. + +By the end of this guide, you'll be able to secure your node's WebSocket port, enabling safe remote connections without exposing your node to unnecessary risks. The instructions in this guide are for UNIX-based systems. + +## Secure a WebSocket Port + +You can convert a non-secured WebSocket port to a secure WSS port by placing it behind an SSL-enabled proxy. This approach can be used to secure a bootnode or RPC server. The SSL-enabled apache2/nginx/other proxy server redirects requests to the internal WebSocket and converts it to a secure (WSS) connection. You can use a service like [LetsEncrypt](https://letsencrypt.org/){target=\_blank} to obtain an SSL certificate. + +### Obtain an SSL Certificate + +LetsEncrypt suggests using the [Certbot ACME client](https://letsencrypt.org/getting-started/#with-shell-access/){target=\_blank} for your respective web server implementation to get a free SSL certificate: + +- [nginx](https://certbot.eff.org/instructions?ws=nginx&os=ubuntufocal){target=\_blank} +- [apache2](https://certbot.eff.org/instructions?ws=apache&os=ubuntufocal){target=\_blank} + +LetsEncrypt will auto-generate an SSL certificate and include it in your configuration. + +When connecting, you can generate a self-signed certificate and rely on your node's raw IP address. However, self-signed certificates aren't optimal because you must include the certificate in an allowlist to access it from a browser. + +Use the following command to generate a self-signed certificate using OpenSSL: + +```bash +sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/ssl/private/selfsigned.key -out /etc/ssl/certs/selfsigned.crt +sudo openssl dhparam -out /etc/ssl/certs/dhparam.pem 2048 +``` +## Install a Proxy Server + +There are a lot of different implementations of a WebSocket proxy; some of the more widely used are [nginx](https://www.f5.com/go/product/welcome-to-nginx){target=\_blank} and [apache2](https://httpd.apache.org/){target=\_blank}, both of which are commonly used web server implementations. See the following section for configuration examples for both implementations. + +### Use nginx + +1. Install the `nginx` web server: + ```bash + apt install nginx + ``` + +2. In an SSL-enabled virtual host, add: + ```conf + server { + (...) + location / { + proxy_buffers 16 4k; + proxy_buffer_size 2k; + proxy_pass http://localhost:9944; + proxy_http_version 1.1; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection "Upgrade"; + proxy_set_header Host $host; + } + } + ``` +3. Optionally, you can introduce some form of rate limiting: + ```conf + http { + limit_req_zone "$http_x_forwarded_for" zone=zone:10m rate=2r/s; + (...) + } + location / { + limit_req zone=zone burst=5; + (...) + } + ``` +### Use Apache2 + +Apache2 can run in various modes, including `prefork`, `worker`, and `event`. In this example, the [`event`](https://httpd.apache.org/docs/2.4/mod/event.html){target=\_blank} mode is recommended for handling higher traffic loads, as it is optimized for performance in such environments. However, depending on the specific requirements of your setup, other modes like `prefork` or `worker` may also be appropriate. + +1. Install the `apache2` web server: + ```bash + apt install apache2 + a2dismod mpm_prefork + a2enmod mpm_event proxy proxy_html proxy_http proxy_wstunnel rewrite ssl + ``` +2. The [`mod_proxy_wstunnel`](https://httpd.apache.org/docs/2.4/mod/mod_proxy_wstunnel.html){target=\_blank} provides support for the tunneling of WebSocket connections to a backend WebSocket server. The connection is automatically upgraded to a WebSocket connection. In an SSL-enabled virtual host add: + + ```apacheconf + # (...) + SSLProxyEngine on + ProxyRequests off + ProxyPass / ws://localhost:9944 + ProxyPassReverse / ws://localhost:9944 + ``` + !!!warning + Older versions of `mod_proxy_wstunnel` don't upgrade the connection automatically and will need the following config added: + ```apacheconf + RewriteEngine on + RewriteCond %{HTTP:Upgrade} websocket [NC] + RewriteRule /(.*) ws://localhost:9944/$1 [P,L] + RewriteRule /(.*) http://localhost:9944/$1 [P,L] + ``` + +3. Optionally, some form of rate limiting can be introduced by first running the following command: + + ```bash + apt install libapache2-mod-qos + a2enmod qos + ``` + + Then edit `/etc/apache2/mods-available/qos.conf` as follows: + + ```conf + # allows max 50 connections from a single IP address: + QS_SrvMaxConnPerIP 50 + ``` + +## Connect to the Node + +1. Open [Polkadot.js Apps interface](https://polkadot.js.org/apps){target=\_blank} and click the logo in the top left to switch the node. +2. Activate the **Development** toggle and input either your node's domain or IP address. Remember to prefix with `wss://` and, if you're using the 443 port, append `:443` as follows: + + ```bash + wss://example.com:443 + ``` + +![A sync-in-progress chain connected to Polkadot.js UI](/images/nodes-and-validators/run-a-node/secure-wss/secure-wss-01.webp) diff --git a/.ai/pages/node-infrastructure-run-a-node-system-parachain-rpc.md b/.ai/pages/node-infrastructure-run-a-node-system-parachain-rpc.md new file mode 100644 index 000000000..c5d5b13cc --- /dev/null +++ b/.ai/pages/node-infrastructure-run-a-node-system-parachain-rpc.md @@ -0,0 +1,652 @@ +--- +title: Run an RPC Node for System Parachains +description: Complete guide to set up and run an RPC node for Polkadot system parachains including Bridge Hub, People Chain, and Coretime Chain. +categories: Infrastructure +url: https://docs.polkadot.com/node-infrastructure/run-a-node/system-parachain-rpc/ +--- + +# Run an RPC Node for System Parachains + +## Overview + +System parachains are core infrastructure parachains that provide essential services to the Polkadot network. Running an RPC node for these parachains enables applications, wallets, and users to interact with their specialized functionality: + +- **Bridge Hub**: Cross-chain asset transfers via trustless bridges +- **People Chain**: Identity and social credential management +- **Coretime Chain**: Blockspace allocation and core time management + +Each system parachain RPC node provides access through: +- **Polkadot SDK Node RPC** (Port 9944): Native Polkadot API (WebSocket and HTTP) + +This setup enables block explorer indexing and provides full compatibility with Polkadot SDK development tools. + +**Important Note**: The parameters and configurations in this guide are provided as illustrative examples. You may need to modify them according to your specific environment, hardware capabilities, and network conditions. + +## Choosing a System Parachain + +This guide uses **People Chain** as the example, but the same principles and setup procedures apply to all system parachains. Simply substitute the appropriate values from the table below for your chosen parachain: + +| Parachain | Para ID | Chain Spec File | Snapshot Path | Chain Name | +|-----------|---------|-----------------|---------------|------------| +| **Bridge Hub** | 1002 | `bridge-hub-polkadot.json` | `polkadot-bridge-hub-rocksdb-archive` | `bridge-hub-polkadot` | +| **People Chain** | 1004 | `people-polkadot.json` | `polkadot-people-rocksdb-archive` | `people-polkadot` | +| **Coretime Chain** | 1005 | `coretime-polkadot.json` | `polkadot-coretime-rocksdb-archive` | `coretime-polkadot` | + +**Note**: Throughout this guide, we use People Chain values. To set up a different system parachain, replace the chain spec file, snapshot path, and chain name with the corresponding values from the table above. + +## Prerequisites + +### Hardware Requirements + +RPC nodes serving production traffic require robust hardware: + +- **CPU**: 8+ cores (16+ cores for high traffic) +- **Memory**: 64 GB RAM minimum (128 GB recommended for high traffic) +- **Storage**: + - 500 GB+ NVMe SSD for parachain state (archive nodes require 2-4 TB+) + - Additional 200+ GB for relay chain pruned database + - Fast disk I/O is critical for query performance +- **Network**: + - Public IP address + - 1 Gbps connection (for high traffic scenarios) + - Stable internet connection with sufficient bandwidth + - Open ports: + - 30333 (parachain P2P) + - 30334 (relay chain P2P) + - 9944 (Polkadot SDK WebSocket RPC) + - 9933 (Polkadot SDK HTTP RPC) + - 9615 (Prometheus metrics - optional) + - Consider DDoS protection and rate limiting for production deployments + +**Note**: For development or low-traffic scenarios, you can reduce these requirements proportionally. Consider using a reverse proxy (nginx, Caddy) for production deployments. + +### Software Requirements + +Required software: + +- **Operating System**: Ubuntu 22.04 LTS (recommended) or similar Linux distribution +- **Docker**: Latest version installed and running (for Docker-based setup) +- **rclone**: (Optional but recommended) Command-line program for managing files on cloud storage (https://rclone.org/downloads/) +- **Rust Toolchain**: Version 1.86 or as specified by runtime (for manual build) + +## Setup Options + +This guide provides two deployment options: + +1. **Docker-based Setup**: Simpler to set up and maintain +2. **Manual/Systemd Setup**: For production environments requiring more control + +Choose the option that best fits your needs. + +--- + +## Option 1: Docker-Based Setup + +This option uses Docker containers for the Polkadot SDK node, making it easy to set up and manage. + +### Step 1: Download Chain Specification + +Download the official chain specification for People Chain: + +```bash +curl -L https://raw.githubusercontent.com/paritytech/polkadot-sdk/master/cumulus/parachains/chain-specs/people-polkadot.json -o people-polkadot.json +``` + +**Note**: This chain specification is the official configuration file that defines the network parameters for People Chain. + +### Step 2: Download Database Snapshots (Optional but Recommended) + +Using pre-synchronized snapshots significantly reduces initial sync time from several days to just a few hours. You need to download both parachain and relay chain data. + +**Snapshot Provider**: https://snapshots.polkadot.io/ + +#### Create Directories + +```bash +mkdir -p my-node-data/chains/people-polkadot/db +mkdir -p my-node-data/chains/polkadot/db +``` + +#### Download People Chain Snapshot + +Choose between archive (complete history) or pruned (recent state) snapshots. + +**Archive Snapshot** (recommended for RPC with historical data): + +```bash +# Check https://snapshots.polkadot.io/ for the latest snapshot URL +export SNAPSHOT_URL_PARACHAIN="https://snapshots.polkadot.io/polkadot-people-rocksdb-archive/LATEST" + +rclone copyurl $SNAPSHOT_URL_PARACHAIN/files.txt files.txt +rclone copy --progress --transfers 20 \ + --http-url $SNAPSHOT_URL_PARACHAIN \ + --no-traverse --http-no-head --disable-http2 \ + --inplace --no-gzip-encoding --size-only \ + --retries 6 --retries-sleep 10s \ + --files-from files.txt :http: my-node-data/chains/people-polkadot/db/ + +rm files.txt +``` + +**Parameter Explanation**: +- `--transfers 20`: Uses 20 parallel transfers for faster download +- `--retries 6`: Automatically retries failed transfers up to 6 times +- `--retries-sleep 10s`: Waits 10 seconds between retry attempts +- `--size-only`: Only transfers if sizes differ (prevents unnecessary re-downloads) + +**Note**: If a snapshot is not available, you can sync from scratch (which will take longer) or check https://snapshots.polkadot.io/ for alternative snapshot providers. + +#### Download Polkadot Relay Chain Snapshot + +**Pruned Snapshot** (recommended for RPC nodes): + +```bash +# Check https://snapshots.polkadot.io/ for the latest snapshot URL +export SNAPSHOT_URL_RELAY="https://snapshots.polkadot.io/polkadot-rocksdb-prune/LATEST" + +rclone copyurl $SNAPSHOT_URL_RELAY/files.txt files.txt +rclone copy --progress --transfers 20 \ + --http-url $SNAPSHOT_URL_RELAY \ + --no-traverse --http-no-head --disable-http2 \ + --inplace --no-gzip-encoding --size-only \ + --retries 6 --retries-sleep 10s \ + --files-from files.txt :http: my-node-data/chains/polkadot/db/ + +rm files.txt +``` + +**Alternative Options**: +- Pruned snapshot: `polkadot-rocksdb-prune` (smaller size, recent state) +- Archive snapshot: `polkadot-rocksdb-archive` (complete history, larger size) + +### Step 3: Start People Chain Node + +Launch the node using the official Parity Docker image. + +**Docker Image**: https://hub.docker.com/r/parity/polkadot-omni-node + +```bash +docker run -d --name people-chain-rpc --restart unless-stopped \ + -p 9944:9944 \ + -p 9933:9933 \ + -p 9615:9615 \ + -p 30334:30334 \ + -p 30333:30333 \ + -v $(pwd)/people-polkadot.json:/people-polkadot.json \ + -v $(pwd)/my-node-data:/data \ + parity/polkadot-omni-node:stable2506-4 \ + --name=PeopleChainRPC \ + --base-path=/data \ + --chain=/people-polkadot.json \ + --prometheus-external \ + --prometheus-port 9615 \ + --unsafe-rpc-external \ + --rpc-port=9944 \ + --rpc-cors=all \ + --rpc-methods=safe \ + --rpc-max-connections=1000 \ + --state-pruning=archive \ + --blocks-pruning=archive \ + -- \ + --base-path=/data \ + --chain=polkadot \ + --state-pruning=256 \ + --blocks-pruning=archive-canonical +``` + +**Critical Configuration Parameters**: + +**Port Mappings**: +- `9944`: Polkadot SDK RPC endpoint (WebSocket/HTTP) +- `9933`: Polkadot SDK HTTP RPC endpoint +- `9615`: Prometheus metrics endpoint +- `30333/30334`: P2P networking ports + +**Node Parameters**: +- `--unsafe-rpc-external`: Enables external RPC access +- `--rpc-cors=all`: Allows all origins for CORS +- `--rpc-methods=safe`: Only allows safe RPC methods +- `--state-pruning=archive`: Keeps complete state history +- `--blocks-pruning=archive`: Keeps all block data +- `--prometheus-external`: Exposes metrics externally + +**Security Warning**: The `--unsafe-rpc-external` flag should only be used in development or properly secured environments. For production, use a reverse proxy with authentication. + +### Step 4: Monitor Synchronization + +Monitor the node synchronization status: + +```bash +# Check sync status +curl -H "Content-Type: application/json" \ + -d '{"id":1, "jsonrpc":"2.0", "method": "system_syncState", "params":[]}' \ + http://localhost:9944 +``` + +**Expected Response Format**: + +```json +{ + "jsonrpc":"2.0", + "id":1, + "result":{ + "startingBlock":0, + "currentBlock":3394816, + "highestBlock":3394816 + } +} +``` + +**Synchronization Status**: +- **In Progress**: `currentBlock` < `highestBlock` +- **Complete**: `currentBlock` = `highestBlock` + +**Monitor logs**: + +```bash +# View node logs +docker logs -f people-chain-rpc + +# Filter for sync messages +docker logs people-chain-rpc 2>&1 | grep -i "syncing" +``` + +### Step 5: Verify Setup + +Let's verify the Polkadot SDK RPC endpoint is working correctly. + +#### API Endpoint + +**Polkadot SDK RPC (Port 9944)**: +- WebSocket: `ws://your-server:9944` +- HTTP: `http://your-server:9944` +- Purpose: Full Polkadot SDK API access for parachain data +- Use Cases: Polkadot SDK applications, parachain-specific operations + +#### Polkadot SDK RPC Tests + +**Get Chain Information**: + +```bash +curl -H "Content-Type: application/json" \ + -d '{"id":1, "jsonrpc":"2.0", "method": "system_chain", "params":[]}' \ + http://localhost:9944 +``` + +**Expected Response**: +```json +{ + "jsonrpc":"2.0", + "id":1, + "result":"Polkadot People" +} +``` + +**Get Latest Block**: + +```bash +curl -H "Content-Type: application/json" \ + -d '{"id":1, "jsonrpc":"2.0", "method": "chain_getHeader", "params":[]}' \ + http://localhost:9944 +``` + +**Get Node Health**: + +```bash +curl -H "Content-Type: application/json" \ + -d '{"id":1, "jsonrpc":"2.0", "method": "system_health", "params":[]}' \ + http://localhost:9944 +``` + +**Get Peer Count**: + +```bash +curl -H "Content-Type: application/json" \ + -d '{"id":1, "jsonrpc":"2.0", "method": "system_peers", "params":[]}' \ + http://localhost:9944 +``` + +### Managing Docker Containers + +**View logs**: + +```bash +docker logs -f people-chain-rpc +``` + +**Stop container**: + +```bash +docker stop people-chain-rpc +``` + +**Start container**: + +```bash +docker start people-chain-rpc +``` + +**Remove container**: + +```bash +docker rm people-chain-rpc +``` + +**Update container**: + +```bash +# Pull latest image +docker pull parity/polkadot-omni-node:stable2506-4 + +# Stop and remove old container +docker stop people-chain-rpc +docker rm people-chain-rpc + +# Start new container with same command as above +``` + +--- + +## Option 2: Manual/Systemd Setup + +This option provides more control and is recommended for production environments requiring custom configurations. + +### Step 1: Install Rust and Required Toolchain + +```bash +# Install Rust +curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh +source $HOME/.cargo/env + +# Install specific Rust version +rustup install 1.86 +rustup default 1.86 +rustup target add wasm32-unknown-unknown --toolchain 1.86 +rustup component add rust-src --toolchain 1.86 +``` + +### Step 2: Install the Polkadot Omni Node + +```bash +# Install polkadot-omni-node +cargo install --locked polkadot-omni-node@0.7.0 + +# Verify installation +polkadot-omni-node --version +``` + +### Step 3: Obtain Chain Specification + +Download the People Chain specification: + +```bash +curl -L https://raw.githubusercontent.com/paritytech/polkadot-sdk/master/cumulus/parachains/chain-specs/people-polkadot.json -o people-polkadot.json +``` + +### Step 4: Create User and Directory Structure + +```bash +# Create dedicated user (skip if already exists) +sudo useradd -r -s /bin/bash polkadot + +# Create data directory +sudo mkdir -p /var/lib/people-chain-rpc + +# Copy chain spec to the directory +sudo cp people-polkadot.json /var/lib/people-chain-rpc/ + +# Set permissions +sudo chown -R polkadot:polkadot /var/lib/people-chain-rpc +``` + +### Step 5: Create Systemd Service + +Create a service file for the People Chain RPC node: + +```bash +sudo nano /etc/systemd/system/people-chain-rpc.service +``` + +Add the following configuration: + +```ini +[Unit] +Description=People Chain RPC Node +After=network.target + +[Service] +Type=simple +User=polkadot +Group=polkadot +WorkingDirectory=/var/lib/people-chain-rpc + +ExecStart=/usr/local/bin/polkadot-omni-node \ + --name=PeopleChainRPC \ + --chain=/var/lib/people-chain-rpc/people-polkadot.json \ + --base-path=/var/lib/people-chain-rpc \ + --port=30333 \ + --rpc-port=9944 \ + --rpc-external \ + --rpc-cors=all \ + --rpc-methods=safe \ + --rpc-max-connections=1000 \ + --prometheus-port=9615 \ + --prometheus-external \ + --state-pruning=archive \ + --blocks-pruning=archive \ + -- \ + --chain=polkadot \ + --base-path=/var/lib/people-chain-rpc \ + --port=30334 \ + --state-pruning=256 \ + --blocks-pruning=archive-canonical + +Restart=always +RestartSec=10 +LimitNOFILE=65536 + +[Install] +WantedBy=multi-user.target +``` + +### Step 6: Start Service + +```bash +# Reload systemd +sudo systemctl daemon-reload + +# Enable service to start on boot +sudo systemctl enable people-chain-rpc + +# Start the node +sudo systemctl start people-chain-rpc + +# Check status and wait for sync +sudo systemctl status people-chain-rpc +sudo journalctl -u people-chain-rpc -f +``` + +### Step 7: Verify Setup + +Use the same verification tests as in the Docker setup (see Step 5 above). + +--- + +## Monitoring and Maintenance + +### Log Management + +**Docker Setup**: + +```bash +# View node logs +docker logs -f people-chain-rpc +``` + +**Systemd Setup**: + +```bash +# View node logs +sudo journalctl -u people-chain-rpc -f + +# View recent logs +sudo journalctl -u people-chain-rpc -n 100 + +# Filter for errors +sudo journalctl -u people-chain-rpc | grep -i error +``` + +### Performance Monitoring + +Monitor key metrics: +- **Sync status**: Ensure node stays fully synced +- **Peer connections**: Maintain 30+ peers for good connectivity +- **Resource usage**: Monitor CPU, RAM, and disk I/O +- **RPC request latency**: Track response times for the Polkadot SDK API +- **Connection count**: Monitor active RPC connections + +**Prometheus Metrics**: + +Metrics are available at `http://localhost:9615/metrics` + +Example Prometheus scrape configuration: + +{% raw %} +```yaml +scrape_configs: + - job_name: 'people-chain-rpc' + static_configs: + - targets: ['localhost:9615'] +``` +{% endraw %} + +**Key Metrics to Monitor**: +- `substrate_block_height`: Current block height +- `substrate_finalized_height`: Finalized block height +- `substrate_peers_count`: Number of connected peers +- `substrate_ready_transactions_number`: Transaction queue size + +### Database Maintenance + +Check database size periodically: + +```bash +# Docker setup +du -sh my-node-data + +# Systemd setup +du -sh /var/lib/people-chain-rpc +``` + +The node handles pruning automatically based on configuration unless running in archive mode. + +### Updates and Upgrades + +**Docker Setup**: + +```bash +# Pull latest image +docker pull parity/polkadot-omni-node:stable2506-4 + +# Restart container +docker stop people-chain-rpc +docker rm people-chain-rpc + +# Start new container (use same command from setup) +``` + +**Systemd Setup**: + +```bash +# Stop service +sudo systemctl stop people-chain-rpc + +# Backup data +sudo cp -r /var/lib/people-chain-rpc /var/lib/people-chain-rpc.backup + +# Update binary +cargo install --locked --force polkadot-omni-node@ + +# Restart service +sudo systemctl start people-chain-rpc +``` + +## Security Best Practices + +### Network Security + +1. **Firewall Configuration**: + - Only expose necessary ports + - Use UFW or iptables to restrict access + - Consider IP whitelisting for RPC endpoints + +2. **Reverse Proxy** (Recommended for Production): + - Use nginx or Caddy as reverse proxy + - Enable SSL/TLS (HTTPS/WSS) + - Implement authentication + - Add rate limiting + +Example nginx configuration: + +```nginx +upstream polkadot_sdk_rpc { + server 127.0.0.1:9944; +} + +server { + listen 443 ssl http2; + server_name your-domain.com; + + ssl_certificate /path/to/cert.pem; + ssl_certificate_key /path/to/key.pem; + + # Polkadot SDK RPC + location /polkadot { + proxy_pass http://polkadot_sdk_rpc; + proxy_http_version 1.1; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection "upgrade"; + + # Rate limiting + limit_req zone=rpc_limit burst=10; + } +} +``` + +### RPC Security + +- **Always use `--rpc-methods=safe`**: Prevents dangerous RPC calls +- **Restrict CORS**: Use specific domains instead of `all` in production +- **Set connection limits**: Prevent resource exhaustion +- **Monitor for abuse**: Track unusual patterns +- **Authentication**: Implement API keys or OAuth for production + +### System Security + +- Keep operating system updated +- Use dedicated user accounts (never root) +- Enable fail2ban for SSH protection +- Regular security audits +- Disable unnecessary services +- Use AppArmor or SELinux for additional isolation + +### Monitoring and Alerting + +Set up alerts for: +- Service failures +- Sync issues +- Low peer count (< 10 peers) +- High resource usage +- Unusual RPC traffic patterns +- Database errors + +## Conclusion + +Running an RPC node for system parachains provides critical infrastructure for accessing specialized Polkadot network services. By following this guide, you have set up a production-ready RPC node that: + +- Enables applications and users to interact with essential system parachain features (identity management, cross-chain bridges, or coretime allocation) +- Supports flexible deployment with both Docker and systemd options +- Implements comprehensive monitoring, security, and maintenance practices +- Can be easily adapted for any system parachain by substituting the appropriate chain specification + +Whether you're running a node for People Chain, Bridge Hub, or Coretime Chain, regular maintenance and monitoring will ensure your RPC node continues to provide reliable service. Stay updated with the latest releases and best practices to keep your infrastructure secure and performant. diff --git a/.ai/pages/node-infrastructure-run-a-validator-onboarding-and-offboarding-key-management.md b/.ai/pages/node-infrastructure-run-a-validator-onboarding-and-offboarding-key-management.md new file mode 100644 index 000000000..1f688736e --- /dev/null +++ b/.ai/pages/node-infrastructure-run-a-validator-onboarding-and-offboarding-key-management.md @@ -0,0 +1,155 @@ +--- +title: Validator Key Management +description: Learn how to generate and manage validator keys, including session keys for consensus participation and node keys for maintaining a stable network identity. +categories: Infrastructure +url: https://docs.polkadot.com/node-infrastructure/run-a-validator/onboarding-and-offboarding/key-management/ +--- + +# Key Management + +## Introduction + +After setting up your node environment as shown in the [Setup](/nodes-and-validators/run-a-validator/onboarding-and-offboarding/set-up-validator/){target=\_blank} section, you'll need to configure multiple keys for your validator to operate properly. This includes setting up session keys, which are essential for participating in the consensus process, and configuring a node key that maintains a stable network identity. This guide walks you through the key management process, showing you how to generate, store, and register these keys. + +## Set Session Keys + +Setting up your validator's session keys is essential to associate your node with your stash account on the Polkadot network. Validators use session keys to participate in the consensus process. Your validator can only perform its role in the network by properly setting session keys which consist of several key pairs for different parts of the protocol (e.g., GRANDPA, BABE). These keys must be registered on-chain and associated with your validator node to ensure it can participate in validating blocks. + +### Generate Session Keys + +There are multiple ways to create the session keys. It can be done by interacting with the [Polkadot.js Apps UI](https://polkadot.js.org/apps/#/explorer){target=\_blank}, using the curl command or by using [Subkey](https://paritytech.github.io/polkadot-sdk/master/subkey/index.html){target=\_blank}. + +=== "Polkadot.js Apps UI" + + 1. In Polkadot.js Apps, connect to your local node, navigate to the **Developer** dropdown, and select the **RPC Calls** option. + + 2. Construct an `author_rotateKeys` RPC call and execute it: + + 1. Select the **author** endpoint. + 2. Choose the **rotateKeys()** call. + 3. Click the **Submit RPC Call** button. + 4. Copy the hex-encoded public key from the response. + + ![](/images/nodes-and-validators/run-a-validator/onboarding-and-offboarding/key-management/key-management-01.webp) + +=== "Curl" + + Generate session keys by running the following command on your validator node: + + ``` bash + curl -H "Content-Type: application/json" \ + -d '{"id":1, "jsonrpc":"2.0", "method": "author_rotateKeys", "params":[]}' \ + http://localhost:9944 + ``` + + This command will return a JSON object. The `result` key is the hex-encoded public part of the newly created session key. Save this for later use. + + ```json + {"jsonrpc":"2.0","result":"0xda3861a45e0197f3ca145c2c209f9126e5053fas503e459af4255cf8011d51010","id":1} + ``` + +=== "Subkey" + + To create a keypair for your node's session keys, use the `subkey generate` command. This generates a set of cryptographic keys that must be stored in your node's keystore directory. + + When you run the command, it produces output similar to this example: + +
+ subkey generate + 
+    Secret phrase:       twist buffalo mixture excess device drastic vague mammal fitness punch match hammer
+      Network ID:        substrate
+      Secret seed:       0x5faa9e5defe42b201388d5c2b8202d6625a344abc9aa52943a71f12cb90b88a9
+      Public key (hex):  0x28cc2fdb6e28835e2bbac9a16feb65c23d448c9314ef12fe083b61bab8fc2755
+      Account ID:        0x28cc2fdb6e28835e2bbac9a16feb65c23d448c9314ef12fe083b61bab8fc2755
+      Public key (SS58): 5CzCRpXzHYhuo6G3gYFR3cgV6X3qCNwVt51m8q14ZcChsSXQ
+      SS58 Address:      5CzCRpXzHYhuo6G3gYFR3cgV6X3qCNwVt51m8q14ZcChsSXQ
+
+
+ + To properly store these keys, create a file in your keystore directory with a specific naming convention. The filename must consist of the hex string `61757261` (which represents "aura" in hex) followed by the public key without its `0x` prefix. + + Using the example above, you would create a file named: + + ``` + ./keystores/6175726128cc2fdb6e28835e2bbac9a16feb65c23d448c9314ef12fe083b61bab8fc2755 + ``` + + And store only the secret phrase in the file: + + ``` + "twist buffalo mixture excess device drastic vague mammal fitness punch match hammer" + ``` + +### Submit Transaction to Set Keys + +Now that you have generated your session keys, you must submit them to the chain. Follow these steps: + +1. Go to the **Network > Staking > Accounts** section on Polkadot.js Apps. +2. Select **Set Session Key** on the bonding account you generated earlier. +3. Paste the hex-encoded session key string you generated (from either the UI or CLI) into the input field and submit the transaction. + +![](/images/nodes-and-validators/run-a-validator/onboarding-and-offboarding/key-management/key-management-02.webp) + +Once the transaction is signed and submitted, your session keys will be registered on-chain. + +### Verify Session Key Setup + +To verify that your session keys are properly set, you can use one of two RPC calls: + +- **`hasKey`**: Checks if the node has a specific key by public key and key type. +- **`hasSessionKeys`**: Verifies if your node has the full session key string associated with the validator. + +For example, you can [check session keys on the Polkadot.js Apps](https://polkadot.js.org/apps/#/rpc){target=\_blank} interface or by running an RPC query against your node. Once this is done, your validator node is ready for its role. + +## Set the Node Key + +Validators on Polkadot need a static network key (also known as the node key) to maintain a stable node identity. This key ensures that your validator can maintain a consistent peer ID, even across restarts, which is crucial for maintaining reliable network connections. + +Starting with Polkadot version 1.11, validators without a stable network key may encounter the following error on startup: + +
+ polkadot --validator --name "INSERT_NAME_FROM_TELEMETRY" + Error: + 0: Starting an authority without network key + This is not a safe operation because other authorities in the network may depend on your node having a stable identity. + Otherwise these other authorities may not being able to reach you. + If it is the first time running your node you could use one of the following methods: + 1. [Preferred] Separately generate the key with: INSERT_NODE_BINARY key generate-node-key --base-path INSERT_YOUR_BASE_PATH + 2. [Preferred] Separately generate the key with: INSERT_NODE_BINARY key generate-node-key --file INSERT_YOUR_PATH_TO_NODE_KEY + 3. [Preferred] Separately generate the key with: INSERT_NODE_BINARY key generate-node-key --default-base-path + 4. [Unsafe] Pass --unsafe-force-node-key-generation and make sure you remove it for subsequent node restarts + +
+ +### Generate the Node Key + +Use one of the following methods to generate your node key: + +=== "Save to file" + + The recommended solution is to generate a node key and save it to a file using the following command: + + ``` bash + polkadot key generate-node-key --file INSERT_PATH_TO_NODE_KEY + ``` + +=== "Use default path" + + You can also generate the node key with the following command, which will automatically save the key to the base path of your node: + + ``` bash + polkadot key generate-node-key --default-base-path + ``` + +Save the file path for reference. You will need it in the next step to configure your node with a static identity. + +### Set Node Key + +After generating the node key, configure your node to use it by specifying the path to the key file when launching your node. Add the following flag to your validator node's startup command: + +``` bash +polkadot --node-key-file INSERT_PATH_TO_NODE_KEY +``` + +Following these steps ensures that your node retains its identity, making it discoverable by peers without the risk of conflicting identities across sessions. For further technical background, see Polkadot SDK [Pull Request #3852](https://github.com/paritytech/polkadot-sdk/pull/3852){target=\_blank} for the rationale behind requiring static keys. diff --git a/.ai/pages/node-infrastructure-run-a-validator-onboarding-and-offboarding-set-up-validator.md b/.ai/pages/node-infrastructure-run-a-validator-onboarding-and-offboarding-set-up-validator.md new file mode 100644 index 000000000..4fd33b1ef --- /dev/null +++ b/.ai/pages/node-infrastructure-run-a-validator-onboarding-and-offboarding-set-up-validator.md @@ -0,0 +1,213 @@ +--- +title: Set Up a Validator +description: Set up a Polkadot validator node to secure the network and earn staking rewards. Follow this step-by-step guide to install, configure, and manage your node. +categories: Infrastructure +url: https://docs.polkadot.com/node-infrastructure/run-a-validator/onboarding-and-offboarding/set-up-validator/ +--- + +# Set Up a Validator + +## Introduction + +Setting up a Polkadot validator node is essential for securing the network and earning staking rewards. This guide walks you through the technical steps to set up a validator, from installing the necessary software to managing keys and synchronizing your node with the chain. + +Running a validator requires a commitment to maintaining a stable, secure infrastructure. Validators are responsible for their own stakes and those of nominators who trust them with their tokens. Proper setup and ongoing management are critical to ensuring smooth operation and avoiding potential penalties such as slashing. + +## Prerequisites + +To get the most from this guide, ensure you've done the following before going forward: + +- Read [Validator Requirements](/nodes-and-validators/run-a-validator/requirements/){target=\_blank} and understand the recommended minimum skill level and hardware needs. +- Read [General Management](/nodes-and-validators/run-a-validator/operational-tasks/general-management/){target=\_blank}, [Upgrade Your Node](/nodes-and-validators/run-a-validator/operational-tasks/upgrade-your-node/){target=\_blank}, and [Pause Validating](/nodes-and-validators/run-a-validator/operational-tasks/pause-validating/){target=\_blank} and understand the tasks required to keep your validator operational. +- Read [Rewards Payout](/nodes-and-validators/run-a-validator/staking-mechanics/rewards/){target=\_blank} and understand how validator rewards are determined and paid out. +- Read [Offenses and Slashes](/nodes-and-validators/run-a-validator/staking-mechanics/offenses-and-slashes/){target=\_blank} and understand how validator performance and security can affect tokens staked by you or your nominators. + +## Initial Setup + +Before running your validator, you must configure your server environment to meet the operational and security standards required for validating. + +You must use a Linux-based operating system with Kernel 5.16 or later. Configuration includes setting up time synchronization, ensuring critical security features are active, and installing the necessary binaries. Proper setup at this stage is essential to prevent issues like block production errors or being penalized for downtime. Below are the essential steps to get your system ready. + +### Install Network Time Protocol Client + +Accurate timekeeping is critical to ensure your validator is synchronized with the network. Validators need local clocks in sync with the blockchain to avoid missing block authorship opportunities. Using [Network Time Protocol (NTP)](https://en.wikipedia.org/wiki/Network_Time_Protocol){target=\_blank} is the standard solution to keep your system's clock accurate. + +If you are using Ubuntu version 18.04 or newer, the NTP Client should be installed by default. You can check whether you have the NTP client by running: + +```sh +timedatectl +``` + +If NTP is running, you should see a message like the following: + +``` sh +System clock synchronized: yes +``` + +If NTP is not installed or running, you can install it using: + +```sh +sudo apt-get install ntp +``` + +After installation, NTP will automatically start. To check its status: + +```sh +sudo ntpq -p +``` + +This command will return a message with the status of the NTP synchronization. Skipping this step could result in your validator node missing blocks due to minor clock drift, potentially affecting its network performance. + +### Verify Landlock is Activated + +[Landlock](https://docs.kernel.org/userspace-api/landlock.html){target=\_blank} is an important security feature integrated into Linux kernels starting with version 5.13. It allows processes, even those without special privileges, to limit their access to the system to reduce the machine's attack surface. This feature is crucial for validators, as it helps ensure the security and stability of the node by preventing unauthorized access or malicious behavior. + +To use Landlock, ensure you use the reference kernel or newer versions. Most Linux distributions should already have Landlock activated. You can check if Landlock is activated on your machine by running the following command as root: + +```sh +dmesg | grep landlock || journalctl -kg landlock +``` + +If Landlock is not activated, your system logs won't show any related output. In this case, you will need to activate it manually or ensure that your Linux distribution supports it. Most modern distributions with the required kernel version should have Landlock activated by default. However, if your system lacks support, you may need to build the kernel with Landlock activated. For more information on doing so, refer to the [official kernel documentation](https://docs.kernel.org/userspace-api/landlock.html#kernel-support){target=\_blank}. + +Implementing Landlock ensures your node operates in a restricted, self-imposed sandbox, limiting potential damage from security breaches or bugs. While not a mandatory requirement, enabling this feature greatly improves the security of your validator setup. + +## Install the Polkadot Binaries + +You must install the Polkadot binaries required to run your validator node. These binaries include the main `polkadot`, `polkadot-prepare-worker`, and `polkadot-execute-worker` binaries. All three are needed to run a fully functioning validator node. + +Depending on your preference and operating system setup, there are multiple methods to install these binaries. Below are the main options: + +### Install from Official Releases + +The preferred, most straightforward method to install the required binaries is downloading the latest versions from the official releases. You can visit the [Github Releases](https://github.com/paritytech/polkadot-sdk/releases){target=\_blank} page for the most current versions of the `polkadot`, `polkadot-prepare-worker`, and `polkadot-execute-worker` binaries. + +You can also download the binaries by using the following direct links: + +=== "`polkadot`" + + ``` bash + # Download the binary + curl -LO https://github.com/paritytech/polkadot-sdk/releases/download/polkadot-stable2506-2/polkadot + + # Verify signature + curl -LO https://github.com/paritytech/polkadot-sdk/releases/download/polkadot-stable2506-2/polkadot.asc + + gpg --keyserver hkps://keyserver.ubuntu.com --receive-keys 90BD75EBBB8E95CB3DA6078F94A4029AB4B35DAE + + gpg --verify polkadot.asc + ``` + +=== "`polkadot-prepare-worker`" + + ``` bash + # Download the binary + curl -LO https://github.com/paritytech/polkadot-sdk/releases/download/polkadot-stable2506-2/polkadot-prepare-worker + + # Verify signature + curl -LO https://github.com/paritytech/polkadot-sdk/releases/download/polkadot-stable2506-2/polkadot-prepare-worker.asc + + gpg --keyserver hkps://keyserver.ubuntu.com --receive-keys 90BD75EBBB8E95CB3DA6078F94A4029AB4B35DAE + + gpg --verify polkadot-prepare-worker.asc + ``` + +=== "`polkadot-execute-worker`" + + ``` bash + # Download the binary + curl -LO https://github.com/paritytech/polkadot-sdk/releases/download/polkadot-stable2506-2/polkadot-execute-worker + + # Verify signature + curl -LO https://github.com/paritytech/polkadot-sdk/releases/download/polkadot-stable2506-2/polkadot-execute-worker.asc + + gpg --keyserver hkps://keyserver.ubuntu.com --receive-keys 90BD75EBBB8E95CB3DA6078F94A4029AB4B35DAE + + gpg --verify polkadot-execute-worker.asc + ``` + + +Signature verification cryptographically ensures the downloaded binaries are authentic and have not been tampered with by using GPG signing keys. Polkadot releases use two different signing keys: + +- ParityReleases (release-team@parity.io) with key [`90BD75EBBB8E95CB3DA6078F94A4029AB4B35DAE`](https://keyserver.ubuntu.com/pks/lookup?search=90BD75EBBB8E95CB3DA6078F94A4029AB4B35DAE&fingerprint=on&op=index){target=\_blank} for current and new releases. +- Parity Security Team (security@parity.io) with key [`9D4B2B6EB8F97156D19669A9FF0812D491B96798`](https://keyserver.ubuntu.com/pks/lookup?search=9D4B2B6EB8F97156D19669A9FF0812D491B96798&fingerprint=on&op=index){target=\_blank} for old releases. + + !!!warning + When verifying a signature, a "Good signature" message indicates successful verification, while any other output signals a potential security risk. + +### Install with Package Managers + +Users running Debian-based distributions like Ubuntu can install the binaries using the [APT](https://wiki.debian.org/Apt){target=\_blank} package manager. + +Execute the following commands as root to add the official repository and install the binaries: + +```bash +# Import the release-team@parity.io GPG key +gpg --keyserver hkps://keyserver.ubuntu.com --receive-keys 90BD75EBBB8E95CB3DA6078F94A4029AB4B35DAE +gpg --export 90BD75EBBB8E95CB3DA6078F94A4029AB4B35DAE > /usr/share/keyrings/parity.gpg + +# Add the Parity repository and update the package index +echo 'deb [signed-by=/usr/share/keyrings/parity.gpg] https://releases.parity.io/deb release main' > /etc/apt/sources.list.d/parity.list +apt update + +# Install the `parity-keyring` package - This will ensure the GPG key +# used by APT remains up-to-date +apt install parity-keyring + +# Install polkadot +apt install polkadot +``` + +Once installation completes, verify the binaries are correctly installed by following the steps in the [verify installation](#verify-installation) section. + +### Install with Ansible + +You can also manage Polkadot installations using Ansible. This approach can be beneficial for users managing multiple validator nodes or requiring automated deployment. The [Parity chain operations Ansible collection](https://github.com/paritytech/ansible-galaxy/){target=\_blank} provides a Substrate node role for this purpose. + +### Install with Docker + +If you prefer using Docker or an OCI-compatible container runtime, the official Polkadot Docker image can be pulled directly from Docker Hub. + +To pull the latest stable image, run the following command: + +```bash +docker pull parity/polkadot:stable2506-2 +``` + +### Build from Sources + +You may build the binaries from source by following the instructions on the [Polkadot SDK repository](https://github.com/paritytech/polkadot-sdk/tree/polkadot-stable2506-2/polkadot#building){target=\_blank}. + +## Verify Installation + +Once the Polkadot binaries are installed, it's essential to verify that everything is set up correctly and that all the necessary components are in place. Follow these steps to ensure the binaries are installed and functioning as expected. + +1. **Check the versions**: Run the following commands to verify the versions of the installed binaries. + + ```bash + polkadot --version + polkadot-execute-worker --version + polkadot-prepare-worker --version + ``` + + The output should show the version numbers for each of the binaries. Ensure that the versions match and are consistent, similar to the following example (the specific version may vary): + +
+ polkadot --version polkadot-execute-worker --version polkadot-prepare-worker --version + 1.16.1-36264cb36db + 1.16.1-36264cb36db + 1.16.1-36264cb36db + +
+ + If the versions do not match or if there is an error, double-check that all the binaries were correctly installed and are accessible within your `$PATH`. + +2. **Ensure all binaries are in the same directory**: All the binaries must be in the same directory for the Polkadot validator node to function properly. If the binaries are not in the same location, move them to a unified directory and ensure this directory is added to your system's `$PATH`. + + To verify the `$PATH`, run the following command: + + ```bash + echo $PATH + ``` + + If necessary, you can move the binaries to a shared location, such as `/usr/local/bin/`, and add it to your `$PATH`. diff --git a/.ai/pages/node-infrastructure-run-a-validator-onboarding-and-offboarding-start-validating.md b/.ai/pages/node-infrastructure-run-a-validator-onboarding-and-offboarding-start-validating.md new file mode 100644 index 000000000..3065d19f1 --- /dev/null +++ b/.ai/pages/node-infrastructure-run-a-validator-onboarding-and-offboarding-start-validating.md @@ -0,0 +1,255 @@ +--- +title: Start Validating +description: Learn how to start validating on Polkadot by choosing a network, syncing your node, bonding DOT tokens, and activating your validator. +categories: Infrastructure +url: https://docs.polkadot.com/node-infrastructure/run-a-validator/onboarding-and-offboarding/start-validating/ +--- + +# Start Validating + +## Introduction + +After configuring your node keys as shown in the [Key Management](/nodes-and-validators/run-a-validator/onboarding-and-offboarding/key-management/){target=\_blank} section and ensuring your system is set up, you're ready to begin the validator setup process. This guide will walk you through choosing a network, synchronizing your node with the blockchain, bonding your DOT tokens, and starting your validator. + +## Choose a Network + +Running your validator on a test network like Westend or Kusama is a smart way to familiarize yourself with the process and identify any setup issues in a lower-stakes environment before joining the Polkadot MainNet. + +- **Westend**: Polkadot's primary TestNet is open to anyone for testing purposes. Validator slots are intentionally limited to keep the network stable for the Polkadot release process, so it may not support as many validators at any given time. +- **Kusama**: Often called Polkadot's "canary network," Kusama has real economic value but operates with a faster and more experimental approach. Running a validator here provides an experience closer to MainNet with the benefit of more frequent validation opportunities with an era time of 6 hours vs 24 hours for Polkadot. +- **Polkadot**: The main network, where validators secure the Polkadot relay chain. It has a slower era time of 24 hours and requires a higher minimum bond amount to participate. + +## Synchronize Chain Data + +The next step is to sync your node with the chosen blockchain network. Synchronization is necessary to download and validate the blockchain data, ensuring your node is ready to participate as a validator. Follow these steps to sync your node: + +1. **Start syncing**: You can run a full or warp sync. + + === "Full sync" + + Polkadot defaults to using a full sync, which downloads and validates the entire blockchain history from the genesis block. Start the syncing process by running the following command: + + ```sh + polkadot + ``` + + This command starts your Polkadot node in non-validator mode, allowing you to synchronize the chain data. + + === "Warp sync" + + You can opt to use warp sync which initially downloads only GRANDPA finality proofs and the latest finalized block's state. Use the following command to start a warp sync: + + ``` bash + polkadot --sync warp + ``` + + Warp sync ensures that your node quickly updates to the latest finalized state. The historical blocks are downloaded in the background as the node continues to operate. + + If you're planning to run a validator on a TestNet, you can specify the chain using the `--chain` flag. For example, the following will run a validator on Kusama: + + ```sh + polkadot --chain=kusama + ``` + +2. **Monitor sync progress**: Once the sync starts, you will see a stream of logs providing information about the node's status and progress. Here's an example of what the output might look like: + +
+ polkadot + 2021-06-17 03:07:07 Parity Polkadot + 2021-06-17 03:07:07 ✌️ version 0.9.5-95f6aa201-x86_64-linux-gnu + 2021-06-17 03:07:07 ❤️ by Parity Technologies <admin@parity.io>, 2017-2021 + 2021-06-17 03:07:07 📋 Chain specification: Polkadot + 2021-06-17 03:07:07 🏷 Node name: boiling-pet-7554 + 2021-06-17 03:07:07 👤 Role: FULL + 2021-06-17 03:07:07 💾 Database: RocksDb at /root/.local/share/polkadot/chains/polkadot/db + 2021-06-17 03:07:07 ⛓ Native runtime: polkadot-9050 (parity-polkadot-0.tx7.au0) + 2021-06-17 03:07:10 🏷 Local node identity is: 12D3KooWLtXFWf1oGrnxMGmPKPW54xWCHAXHbFh4Eap6KXmxoi9u + 2021-06-17 03:07:10 📦 Highest known block at #17914 + 2021-06-17 03:07:10 〽️ Prometheus server started at 127.0.0.1:9615 + 2021-06-17 03:07:10 Listening for new connections on 127.0.0.1:9944 + ... +
+ + The output logs provide information such as the current block number, node name, and network connections. Monitor the sync progress and any errors that might occur during the process. Look for information about the latest processed block and compare it with the current highest block using tools like [Telemetry](https://telemetry.polkadot.io/#list/Polkadot%20CC1){target=\_blank} or [Polkadot.js Apps Explorer](https://polkadot.js.org/apps/#/explorer){target=\_blank}. + +### Database Snapshot Services + +If you'd like to speed up the process further, you can use a database snapshot. Snapshots are compressed backups of the blockchain's database directory and can significantly reduce the time required to sync a new node. Here are a few public snapshot providers: + +- [Stakeworld](https://stakeworld.io/snapshot){target=\_blank} +- [Polkachu](https://polkachu.com/substrate_snapshots){target=\_blank} +- [Polkashots](https://polkashots.io/){target=\_blank} +- [ITRocket](https://itrocket.net/services/mainnet/polkadot/#snapshot){target=\_blank} + +!!!warning + Although snapshots are convenient, syncing from scratch is recommended for security purposes. If snapshots become corrupted and most nodes rely on them, the network could inadvertently run on a non-canonical chain. + +
+ polkadot + 2021-06-17 03:07:07 Idle (0 peers), best: #0 (0x3fd7...5baf), finalized #0 (0x3fd7...5baf), ⬇ 2.9kiB/s ⬆ 3.7kiB/s + 2021-06-17 03:07:12 Idle (0 peers), best: #0 (0x3fd7...5baf), finalized #0 (0x3fd7...5baf), ⬇ 1.7kiB/s ⬆ 2.0kiB/s + 2021-06-17 03:07:17 Idle (0 peers), best: #0 (0x3fd7...5baf), finalized #0 (0x3fd7...5baf), ⬇ 0.9kiB/s ⬆ 1.2kiB/s + 2021-06-17 03:07:19 Libp2p => Random Kademlia query has yielded empty results + 2021-06-17 03:08:00 Idle (0 peers), best: #0 (0x3fd7...5baf), finalized #0 (0x3fd7...5baf), ⬇ 1.6kiB/s ⬆ 1.9kiB/s + 2021-06-17 03:08:05 Idle (0 peers), best: #0 (0x3fd7...5baf), finalized #0 (0x3fd7...5baf), ⬇ 0.6kiB/s ⬆ 0.9kiB/s + ... +
+ +If you see terminal output similar to the preceding, and you are unable to synchronize the chain due to having zero peers, make sure you have libp2p port `30333` activated. It will take some time to discover other peers over the network. + +## Bond DOT + +Once your validator node is synced, the next step is bonding DOT. A bonded account, or stash, holds your staked tokens (DOT) that back your validator node. Bonding your DOT means locking it for a period, during which it cannot be transferred or spent but is used to secure your validator's role in the network. Visit the [Minimum Bond Requirement](/nodes-and-validators/run-a-validator/requirements/#minimum-bond-requirement) section for details on how much DOT is required. + +The following sections will guide you through bonding DOT for your validator. + +### Bonding DOT on Polkadot.js Apps + +Once you're ready to bond your DOT, head over to the [Polkadot.js Apps](https://polkadot.js.org/apps/){target=\_blank} staking page by clicking the **Network** dropdown at the top of the page and selecting [**Staking**](https://polkadot.js.org/apps/#/staking/actions){target=\_blank}. + +To get started with the bond submission, click on the **Accounts** tab, then the **+ Stash** button, and then enter the following information: + +1. **Stash account**: Select your stash account (which is the account with the DOT/KSM balance). +2. **Value bonded**: Enter how much DOT from the stash account you want to bond/stake. You are not required to bond all of the DOT in that account and you may bond more DOT at a later time. Be aware, withdrawing any bonded amount requires waiting for the unbonding period. The unbonding period is seven days for Kusama and 28 days for Polkadot. +3. **Payment destination**: Add the recipient account for validator rewards. If you'd like to redirect payments to an account that is not the stash account, you can do it by entering the address here. Note that it is extremely unsafe to set an exchange address as the recipient of the staking rewards. + +Once everything is filled in properly, select **Bond** and sign the transaction with your stash account. If successful, you should see an `ExtrinsicSuccess` message. + +Your bonded account will be available under **Stashes**. After refreshing the screen, you should now see a card with all your accounts. The bonded amount on the right corresponds to the funds bonded by the stash account. + +## Validate + +Once your validator node is fully synced and ready, the next step is to ensure it's visible on the network and performing as expected. Below are steps for monitoring and managing your node on the Polkadot network. + +### Verify Sync via Telemetry + +To confirm that your validator is live and synchronized with the Polkadot network, visit the [Telemetry](https://telemetry.polkadot.io/#list/Polkadot%20CC1){target=\_blank} page. Telemetry provides real-time information on node performance and can help you check if your validator is connected properly. Search for your node by name. You can search all nodes currently active on the network, which is why you should use a unique name for easy recognition. Now, confirm that your node is fully synced by comparing the block height of your node with the network's latest block. Nodes that are fully synced will appear white in the list, while nodes that are not yet fully synced will appear gray. + +### Activate using Polkadot.js Apps + +Follow these steps to use Polkadot.js Apps to activate your validator: + +1. In Polkadot.js Apps, navigate to **Network** and select **Staking**: + + ![](/images/nodes-and-validators/run-a-validator/onboarding-and-offboarding/start-validating/start-validating-01.webp) + +2. Open the **Accounts** tab and click on **+ Validator**: + + ![](/images/nodes-and-validators/run-a-validator/onboarding-and-offboarding/start-validating/start-validating-02.webp) + +3. Set a bond amount in the **value bonded** field and then click **next**: + + ![](/images/nodes-and-validators/run-a-validator/onboarding-and-offboarding/start-validating/start-validating-03.webp) + +4. Paste the hex output from `author_rotateKeys`, set the commission, allow or block new nominations, then click **Bond & Validate** to link your validator with its session keys. + + ![](/images/nodes-and-validators/run-a-validator/onboarding-and-offboarding/start-validating/start-validating-04.webp) + + You can also set the **commission** and **blocked** nominations option via `staking.validate` extrinsic. By default, the blocked option is set to FALSE (i.e., the validator accepts nominations). + + ![](/images/nodes-and-validators/run-a-validator/onboarding-and-offboarding/start-validating/start-validating-05.webp) + + +### Monitor Validation Status and Slots + +On the [**Staking**](https://polkadot.js.org/apps/#/staking){target=\_blank} tab in Polkadot.js Apps, you can see your validator's status, the number of available validator slots, and the nodes that have signaled their intent to validate. Your node may initially appear in the waiting queue, especially if the validator slots are full. The following is an example view of the **Staking** tab: + +![staking queue](/images/nodes-and-validators/run-a-validator/onboarding-and-offboarding/start-validating/start-validating-06.webp) + +The validator set refreshes each era. If there's an available slot in the next era, your node may be selected to move from the waiting queue to the active validator set, allowing it to start validating blocks. If your validator is not selected, it remains in the waiting queue. Increasing your stake or gaining more nominators may improve your chance of being selected in future eras. + +## Run a Validator Using Systemd + +Running your Polkadot validator as a [systemd](https://en.wikipedia.org/wiki/Systemd){target=\_blank} service is an effective way to ensure its high uptime and reliability. Using systemd allows your validator to automatically restart after server reboots or unexpected crashes, significantly reducing the risk of slashing due to downtime. + +This following sections will walk you through creating and managing a systemd service for your validator, allowing you to seamlessly monitor and control it as part of your Linux system. + +Ensure the following requirements are met before proceeding with the systemd setup: + +- Confirm your system meets the [requirements](/nodes-and-validators/run-a-validator/requirements/){target=\_blank} for running a validator. +- Ensure you meet the [minimum bond requirements](https://wiki.polkadot.com/general/chain-state-values/#minimum-validator-bond){target=\_blank} for validating. +- Verify the Polkadot binary is [installed](#install-the-polkadot-binaries). + +### Create the Systemd Service File + +First create a new unit file called `polkadot-validator.service` in `/etc/systemd/system/`: + +```bash +touch /etc/systemd/system/polkadot-validator.service +``` + +In this unit file, you will write the commands that you want to run on server boot/restart: + +```systemd title="/etc/systemd/system/polkadot-validator.service" +[Unit] +Description=Polkadot Node +After=network.target +Documentation=https://github.com/paritytech/polkadot-sdk + +[Service] +EnvironmentFile=-/etc/default/polkadot +ExecStart=/usr/bin/polkadot $POLKADOT_CLI_ARGS +User=polkadot +Group=polkadot +Restart=always +RestartSec=120 +CapabilityBoundingSet= +LockPersonality=true +NoNewPrivileges=true +PrivateDevices=true +PrivateMounts=true +PrivateTmp=true +PrivateUsers=true +ProtectClock=true +ProtectControlGroups=true +ProtectHostname=true +ProtectKernelModules=true +ProtectKernelTunables=true +ProtectSystem=strict +RemoveIPC=true +RestrictAddressFamilies=AF_INET AF_INET6 AF_NETLINK AF_UNIX +RestrictNamespaces=false +RestrictSUIDSGID=true +SystemCallArchitectures=native +SystemCallFilter=@system-service +SystemCallFilter=landlock_add_rule landlock_create_ruleset landlock_restrict_self seccomp mount umount2 +SystemCallFilter=~@clock @module @reboot @swap @privileged +SystemCallFilter=pivot_root +UMask=0027 + +[Install] +WantedBy=multi-user.target +``` + +!!! warning "Restart delay and equivocation risk" + It is recommended that a node's restart be delayed with `RestartSec` in the case of a crash. It's possible that when a node crashes, consensus votes in GRANDPA aren't persisted to disk. In this case, there is potential to equivocate when immediately restarting. Delaying the restart will allow the network to progress past potentially conflicting votes. + +### Run the Service + +Activate the systemd service to start on system boot by running: + +```bash +systemctl enable polkadot-validator.service +``` + +To start the service manually, use: + +```bash +systemctl start polkadot-validator.service +``` + +Check the service's status to confirm it is running: + +```bash +systemctl status polkadot-validator.service +``` + +To view the logs in real-time, use [journalctl](https://www.freedesktop.org/software/systemd/man/latest/journalctl.html){target=\_blank} like so: + +```bash +journalctl -f -u polkadot-validator +``` + +With these steps, you can effectively manage and monitor your validator as a systemd service. + +Once your validator is active, it's officially part of Polkadot's security infrastructure. For questions or further support, you can reach out to the [Polkadot Validator chat](https://matrix.to/#/!NZrbtteFeqYKCUGQtr:matrix.parity.io?via=matrix.parity.io&via=matrix.org&via=web3.foundation){target=\_blank} for tips and troubleshooting. diff --git a/.ai/pages/node-infrastructure-run-a-validator-onboarding-and-offboarding-stop-validating.md b/.ai/pages/node-infrastructure-run-a-validator-onboarding-and-offboarding-stop-validating.md new file mode 100644 index 000000000..5cff77ad4 --- /dev/null +++ b/.ai/pages/node-infrastructure-run-a-validator-onboarding-and-offboarding-stop-validating.md @@ -0,0 +1,45 @@ +--- +title: Stop Validating +description: Learn to safely stop validating on Polkadot, including chilling, unbonding tokens, and purging validator keys. +categories: Infrastructure +url: https://docs.polkadot.com/node-infrastructure/run-a-validator/onboarding-and-offboarding/stop-validating/ +--- + +# Stop Validating + +## Introduction + +If you're ready to stop validating on Polkadot, there are essential steps to ensure a smooth transition while protecting your funds and account integrity. Whether you're taking a break for maintenance or unbonding entirely, you'll need to chill your validator, purge session keys, and unbond your tokens. This guide explains how to use Polkadot's tools and extrinsics to safely withdraw from validation activities, safeguarding your account's future usability. + +## Pause Versus Stop + +If you wish to remain a validator or nominator (for example, stopping for planned downtime or server maintenance), submitting the `chill` extrinsic in the `staking` pallet should suffice. Additional steps are only needed to unbond funds or reap an account. + +The following are steps to ensure a smooth stop to validation: + +- Chill the validator. +- Purge validator session keys. +- Unbond your tokens. + +## Chill Validator + +When stepping back from validating, the first step is to chill your validator status. This action stops your validator from being considered for the next era without fully unbonding your tokens, which can be useful for temporary pauses like maintenance or planned downtime. + +Use the `staking.chill` extrinsic to initiate this. For more guidance on chilling your node, refer to the [Pause Validating](/nodes-and-validators/run-a-validator/operational-tasks/pause-validating/){target=\_blank} guide. You may also claim any pending staking rewards at this point. + +## Purge Validator Session Keys + +Purging validator session keys is a critical step in removing the association between your validator account and its session keys, which ensures that your account is fully disassociated from validator activities. The `session.purgeKeys` extrinsic removes the reference to your session keys from the stash or staking proxy account that originally set them. + +Here are a couple of important things to know about purging keys: + +- **Account used to purge keys**: Always use the same account to purge keys you originally used to set them, usually your stash or staking proxy account. Using a different account may leave an unremovable reference to the session keys on the original account, preventing its reaping. +- **Account reaping issue**: Failing to purge keys will prevent you from reaping (fully deleting) your stash account. If you attempt to transfer tokens without purging, you'll need to rebond, purge the session keys, unbond again, and wait through the unbonding period before any transfer. + +## Unbond Your Tokens + +After chilling your node and purging session keys, the final step is to unbond your staked tokens. This action removes them from staking and begins the unbonding period (usually 28 days for Polkadot and seven days for Kusama), after which the tokens will be transferable. + +To unbond tokens, go to **Network > Staking > Account Actions** on Polkadot.js Apps. Select your stash account, click on the dropdown menu, and choose **Unbond Funds**. Alternatively, you can use the `staking.unbond` extrinsic if you handle this via a staking proxy account. + +Once the unbonding period is complete, your tokens will be available for use in transactions or transfers outside of staking. diff --git a/.ai/pages/node-infrastructure-run-a-validator-operational-tasks-general-management.md b/.ai/pages/node-infrastructure-run-a-validator-operational-tasks-general-management.md new file mode 100644 index 000000000..9c8ac6502 --- /dev/null +++ b/.ai/pages/node-infrastructure-run-a-validator-operational-tasks-general-management.md @@ -0,0 +1,722 @@ +--- +title: General Management +description: Optimize your Polkadot validator setup with advanced configuration techniques. Learn how to boost performance, enhance security, and ensure seamless operations. +categories: Infrastructure +url: https://docs.polkadot.com/node-infrastructure/run-a-validator/operational-tasks/general-management/ +--- + +# General Management + +## Introduction + +Validator performance is pivotal in maintaining the security and stability of the Polkadot network. As a validator, optimizing your setup ensures efficient transaction processing, minimizes latency, and maintains system reliability during high-demand periods. Proper configuration and proactive monitoring also help mitigate risks like slashing and service interruptions. + +This guide covers essential practices for managing a validator, including performance tuning techniques, security hardening, and tools for real-time monitoring. Whether you're fine-tuning CPU settings, configuring NUMA balancing, or setting up a robust alert system, these steps will help you build a resilient and efficient validator operation. + +## Configuration Optimization + +For those seeking to optimize their validator's performance, the following configurations can improve responsiveness, reduce latency, and ensure consistent performance during high-demand periods. + +### Deactivate Simultaneous Multithreading + +Polkadot validators operate primarily in single-threaded mode for critical tasks, so optimizing single-core CPU performance can reduce latency and improve stability. Deactivating simultaneous multithreading (SMT) can prevent virtual cores from affecting performance. SMT is called Hyper-Threading on Intel and 2-way SMT on AMD Zen. + +Take the following steps to deactivate every other (vCPU) core: + +1. Loop though all the CPU cores and deactivate the virtual cores associated with them: + + ```bash + for cpunum in $(cat /sys/devices/system/cpu/cpu*/topology/thread_siblings_list | \ + cut -s -d, -f2- | tr ',' '\n' | sort -un) + do + echo 0 > /sys/devices/system/cpu/cpu$cpunum/online + done + ``` + +2. To permanently save the changes, add `nosmt=force` to the `GRUB_CMDLINE_LINUX_DEFAULT` variable in `/etc/default/grub`: + + ```bash + sudo nano /etc/default/grub + # Add to GRUB_CMDLINE_LINUX_DEFAULT + ``` + + ```config title="/etc/default/grub" + GRUB_DEFAULT = 0; + GRUB_HIDDEN_TIMEOUT = 0; + GRUB_HIDDEN_TIMEOUT_QUIET = true; + GRUB_TIMEOUT = 10; + GRUB_DISTRIBUTOR = `lsb_release -i -s 2> /dev/null || echo Debian`; + GRUB_CMDLINE_LINUX_DEFAULT = 'nosmt=force'; + GRUB_CMDLINE_LINUX = ''; + ``` + +3. Update GRUB to apply changes: + + ```bash + sudo update-grub + ``` + +4. After the reboot, you should see that half of the cores are offline. To confirm, run: + + ```bash + lscpu --extended + ``` + +### Deactivate Automatic NUMA Balancing + +Deactivating NUMA (Non-Uniform Memory Access) balancing for multi-CPU setups helps keep processes on the same CPU node, minimizing latency. + +Follow these stpes: + +1. Deactivate NUMA balancing in runtime: + + ```bash + sysctl kernel.numa_balancing=0 + ``` + +2. Deactivate NUMA balancing permanently by adding `numa_balancing=disable` to the GRUB settings: + + ```bash + sudo nano /etc/default/grub + # Add to GRUB_CMDLINE_LINUX_DEFAULT + ``` + + ```config title="/etc/default/grub" + GRUB_DEFAULT = 0; + GRUB_HIDDEN_TIMEOUT = 0; + GRUB_HIDDEN_TIMEOUT_QUIET = true; + GRUB_TIMEOUT = 10; + GRUB_DISTRIBUTOR = `lsb_release -i -s 2> /dev/null || echo Debian`; + GRUB_CMDLINE_LINUX_DEFAULT = 'numa_balancing=disable'; + GRUB_CMDLINE_LINUX = ''; + ``` + +3. Update GRUB to apply changes: + + ```bash + sudo update-grub + ``` + +4. Confirm the deactivation: + + ```bash + sysctl -a | grep 'kernel.numa_balancing' + ``` + +If you successfully deactivated NUMA balancing, the preceding command should return `0`. + +### Spectre and Meltdown Mitigations + +[Spectre](https://en.wikipedia.org/wiki/Spectre_(security_vulnerability)){target=\_blank} and [Meltdown](https://en.wikipedia.org/wiki/Meltdown_(security_vulnerability)){target=\_blank} are well-known CPU vulnerabilities that exploit speculative execution to access sensitive data. These vulnerabilities have been patched in recent Linux kernels, but the mitigations can slightly impact performance, especially in high-throughput or containerized environments. + +If your security requirements allow it, you can deactivate specific mitigations, such as Spectre V2 and Speculative Store Bypass Disable (SSBD), to improve performance. + +To selectively deactivate the Spectre mitigations, take these steps: + +1. Update the `GRUB_CMDLINE_LINUX_DEFAULT` variable in your `/etc/default/grub` configuration: + + ```bash + sudo nano /etc/default/grub + # Add to GRUB_CMDLINE_LINUX_DEFAULT + ``` + + ```config title="/etc/default/grub" + GRUB_DEFAULT = 0; + GRUB_HIDDEN_TIMEOUT = 0; + GRUB_HIDDEN_TIMEOUT_QUIET = true; + GRUB_TIMEOUT = 10; + GRUB_DISTRIBUTOR = `lsb_release -i -s 2> /dev/null || echo Debian`; + GRUB_CMDLINE_LINUX_DEFAULT = + 'spec_store_bypass_disable=prctl spectre_v2_user=prctl'; + ``` + +2. Update GRUB to apply changes and then reboot: + + ```bash + sudo update-grub + sudo reboot + ``` + +This approach selectively deactivates the Spectre V2 and Spectre V4 mitigations, leaving other protections intact. For full security, keep mitigations activated unless there's a significant performance need, as disabling them could expose the system to potential attacks on affected CPUs. + +## Monitor Your Node + +Monitoring your node's performance is critical for network reliability and security. Tools like the following provide valuable insights: + +- **[Prometheus](https://prometheus.io/){target=\_blank}**: An open-source monitoring toolkit for collecting and querying time-series data. +- **[Grafana](https://grafana.com/){target=\_blank}**: A visualization tool for real-time metrics, providing interactive dashboards. +- **[Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/){target=\_blank}**: A tool for managing and routing alerts based on Prometheus data. + +This section covers setting up these tools and configuring alerts to notify you of potential issues. + +### Environment Setup + +Before installing Prometheus, ensure the environment is set up securely by running Prometheus with restricted user privileges. + +Follow these steps: + +1. Create a Prometheus user to ensure Prometheus runs with minimal permissions: + + ```bash + sudo useradd --no-create-home --shell /usr/sbin/nologin prometheus + ``` + +2. Create directories for configuration and data storage: + + ```bash + sudo mkdir /etc/prometheus + sudo mkdir /var/lib/prometheus + ``` + +3. Change directory ownership to ensure Prometheus has access: + + ```bash + sudo chown -R prometheus:prometheus /etc/prometheus + sudo chown -R prometheus:prometheus /var/lib/prometheus + ``` + +### Install and Configure Prometheus + +After setting up the environment, install and configure the latest version of Prometheus as follows: + +1. Download Prometheus for your system architecture from the [releases page](https://github.com/prometheus/prometheus/releases/){target=\_blank}. Replace `INSERT_RELEASE_DOWNLOAD` with the release binary URL (e.g., `https://github.com/prometheus/prometheus/releases/download/v3.0.0/prometheus-3.0.0.linux-amd64.tar.gz`): + + ```bash + sudo apt-get update && sudo apt-get upgrade + wget INSERT_RELEASE_DOWNLOAD_LINK + tar xfz prometheus-*.tar.gz + cd prometheus-3.0.0.linux-amd64 + ``` + +2. Set up Prometheus: + + 1. Copy binaries: + + ```bash + sudo cp ./prometheus /usr/local/bin/ + sudo cp ./promtool /usr/local/bin/ + sudo cp ./prometheus /usr/local/bin/ + ``` + + 2. Copy directories and assign ownership of these files to the `prometheus` user: + + ```bash + sudo cp -r ./consoles /etc/prometheus + sudo cp -r ./console_libraries /etc/prometheus + sudo chown -R prometheus:prometheus /etc/prometheus/consoles + sudo chown -R prometheus:prometheus /etc/prometheus/console_libraries + ``` + + 3. Clean up the download directory: + + ```bash + cd .. && rm -r prometheus* + ``` + +3. Create `prometheus.yml` to define global settings, rule files, and scrape targets: + + ```bash + sudo nano /etc/prometheus/prometheus.yml + ``` + + {% raw %} + ```yaml title="prometheus-config.yml" + global: + scrape_interval: 15s + evaluation_interval: 15s + + rule_files: + # - "first.rules" + # - "second.rules" + + scrape_configs: + - job_name: 'prometheus' + scrape_interval: 5s + static_configs: + - targets: ['localhost:9090'] + - job_name: 'substrate_node' + scrape_interval: 5s + static_configs: + - targets: ['localhost:9615'] + ``` + {% endraw %} + + Prometheus is scraped every 5 seconds in this example configuration file, ensuring detailed internal metrics. Node metrics with customizable intervals are scraped from port `9615` by default. + +4. Verify the configuration with `promtool`, an open source monitoring tool: + + ```bash + promtool check config /etc/prometheus/prometheus.yml + ``` + +5. Save the configuration and change the ownership of the file to `prometheus` user: + + ```bash + sudo chown prometheus:prometheus /etc/prometheus/prometheus.yml + ``` + +### Start Prometheus + +1. Launch Prometheus with the appropriate configuration file, storage location, and necessary web resources, running it with restricted privileges for security: + + ```bash + sudo -u prometheus /usr/local/bin/prometheus --config.file /etc/prometheus/prometheus.yml \ + --storage.tsdb.path /var/lib/prometheus/ \ + --web.console.templates=/etc/prometheus/consoles \ + --web.console.libraries=/etc/prometheus/console_libraries + ``` + + If you set the server up properly, you should see terminal output similar to the following: + + +2. Verify you can access the Prometheus interface by navigating to: + + ```text + http://SERVER_IP_ADDRESS:9090/graph + ``` + + If the interface appears to work as expected, exit the process using `Control + C`. + +3. Create a systemd service file to ensure Prometheus starts on boot: + + ```bash + sudo nano /etc/systemd/system/prometheus.service + ``` + + ```bash title="prometheus.service" + [Unit] + Description=Prometheus Monitoring + Wants=network-online.target + After=network-online.target + + [Service] + User=prometheus + Group=prometheus + Type=simple + ExecStart=/usr/local/bin/prometheus \ + --config.file /etc/prometheus/prometheus.yml \ + --storage.tsdb.path /var/lib/prometheus/ \ + --web.console.templates=/etc/prometheus/consoles \ + --web.console.libraries=/etc/prometheus/console_libraries + ExecReload=/bin/kill -HUP $MAINPID + + [Install] + WantedBy=multi-user.target + + ``` + +4. Reload systemd and enable the service to start on boot: + + ```bash + sudo systemctl daemon-reload && sudo systemctl enable prometheus && sudo systemctl start prometheus + ``` + +5. Verify the service is running by visiting the Prometheus interface again at: + + ```text + http://SERVER_IP_ADDRESS:9090/ + ``` + +### Install and Configure Grafana + +This guide follows [Grafana's canonical installation instructions](https://grafana.com/docs/grafana/latest/setup-grafana/installation/debian/#install-from-apt-repository){target=\_blank}. + +To install and configure Grafana, follow these steps: + +1. Install Grafana prerequisites: + + ```bash + sudo apt-get install -y apt-transport-https software-properties-common wget + ``` + +2. Import the [GPG key](https://gnupg.org/){target=\_blank}: + + ```bash + sudo mkdir -p /etc/apt/keyrings/ + wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null + ``` + +3. Configure the stable release repo and update packages: + + ```bash + echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list + sudo apt-get update + ``` + +4. Install the latest stable version of Grafana: + + ```bash + sudo apt-get install grafana + ``` + +To configure Grafana, take these steps: + +1. Configure Grafana to start automatically on boot and start the service: + + ```bash + sudo systemctl daemon-reload + sudo systemctl enable grafana-server.service + sudo systemctl start grafana-server + ``` + +2. Check if Grafana is running: + + ```bash + sudo systemctl status grafana-server + ``` + + If necessary, you can stop or restart the service with the following commands: + + ```bash + sudo systemctl stop grafana-server + sudo systemctl restart grafana-server + ``` + +3. Access Grafana by navigating to the following URL and logging in with the default username and password (`admin`): + + ```text + http://SERVER_IP_ADDRESS:3000/login + ``` + + !!! tip "Change default port" + To change Grafana's port, edit `/usr/share/grafana/conf/defaults.ini`: + + ```bash + sudo vim /usr/share/grafana/conf/defaults.ini + ``` + + Modify the `http_port` value, then restart Grafana: + + ```bash + sudo systemctl restart grafana-server + ``` + +![Grafana login screen](/images/nodes-and-validators/run-a-validator/operational-tasks/general-management/general-management-01.webp) + +To visualize node metrics, follow these steps: + +1. Select the gear icon to access **Data Sources** settings. +2. Select **Add data source** to define the data source. + + ![Select Prometheus](/images/nodes-and-validators/run-a-validator/operational-tasks/general-management/general-management-02.webp) + +3. Select **Prometheus**. + + ![Save and test](/images/nodes-and-validators/run-a-validator/operational-tasks/general-management/general-management-03.webp) + +4. Enter `http://localhost:9090` in the **URL** field and click **Save & Test**. If **"Data source is working"** appears, your connection is configured correctly. + + ![Import dashboard](/images/nodes-and-validators/run-a-validator/operational-tasks/general-management/general-management-04.webp) + +5. Select **Import** from the left menu, choose **Prometheus** from the dropdown, and click **Import**. + +6. Start your Polkadot node by running `./polkadot`. You should now be able to monitor node performance, block height, network traffic, and tasks tasks on the Grafana dashboard. + + ![Live dashboard](/images/nodes-and-validators/run-a-validator/operational-tasks/general-management/general-management-05.webp) + +The [Grafana dashboards](https://grafana.com/grafana/dashboards){target=\_blank} page features user created dashboards made available for public use. For an example, see the [Substrate Node Metrics](https://grafana.com/grafana/dashboards/21715-substrate-node-metrics/){target=\_blank} dashboard. + +### Install and Configure Alertmanager + +[Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/){target=\_blank} is an optional component that complements Prometheus by managing alerts and notifying users about potential issues. + +Follow these steps to install and configure Alertmanager: + +1. Download Alertmanager for your system architecture from the [releases page](https://github.com/prometheus/alertmanager/releases){target=\_blank}. Replace `INSERT_RELEASE_DOWNLOAD` with the release binary URL (e.g., `https://github.com/prometheus/alertmanager/releases/download/v0.28.0-rc.0/alertmanager-0.28.0-rc.0.linux-amd64.tar.gz`): + + ```bash + wget INSERT_RELEASE_DOWNLOAD_LINK + tar -xvzf alertmanager* + ``` + +2. Copy the binaries to the system directory and set permissions: + + ```bash + cd alertmanager-0.28.0-rc.0.linux-amd64 + sudo cp ./alertmanager /usr/local/bin/ + sudo cp ./amtool /usr/local/bin/ + sudo chown prometheus:prometheus /usr/local/bin/alertmanager + sudo chown prometheus:prometheus /usr/local/bin/amtool + ``` + +3. Create the `alertmanager.yml` configuration file under `/etc/alertmanager`: + + ```bash + sudo mkdir /etc/alertmanager + sudo nano /etc/alertmanager/alertmanager.yml + ``` + + Generate an [app password in your Google account](https://support.google.com/accounts/answer/185833?hl=en){target=\_blank} to enable email notifications from Alertmanager. Then, add the following code to the configuration file to define email notifications using your email and app password: + + {% raw %} + ```yml title="alertmanager.yml" + global: + resolve_timeout: 1m + + route: + receiver: 'gmail-notifications' + + receivers: + - name: 'gmail-notifications' + email_configs: + - to: INSERT_YOUR_EMAIL + from: INSERT_YOUR_EMAIL + smarthost: smtp.gmail.com:587 + auth_username: INSERT_YOUR_EMAIL + auth_identity: INSERT_YOUR_EMAIL + auth_password: INSERT_YOUR_APP_PASSWORD + send_resolved: true + + ``` + {% endraw %} + + + ```bash + sudo chown -R prometheus:prometheus /etc/alertmanager + ``` + +4. Configure Alertmanager as a service by creating a systemd service file: + + ```bash + sudo nano /etc/systemd/system/alertmanager.service + ``` + + {% raw %} + ```yml title="alertmanager.service" + [Unit] + Description=AlertManager Server Service + Wants=network-online.target + After=network-online.target + + [Service] + User=root + Group=root + Type=simple + ExecStart=/usr/local/bin/alertmanager --config.file /etc/alertmanager/alertmanager.yml --web.external-url=http://SERVER_IP:9093 --cluster.advertise-address='0.0.0.0:9093' + + [Install] + WantedBy=multi-user.target + + ``` + {% endraw %} + +5. Reload and enable the service: + + ```bash + sudo systemctl daemon-reload + sudo systemctl enable alertmanager + sudo systemctl start alertmanager + ``` + +6. Verify the service status: + + ```bash + sudo systemctl status alertmanager + ``` + + If you have configured Alertmanager properly, the **Active** field should display **active (running)** similar to below: + +
+ sudo systemctl status alertmanager + alertmanager.service - AlertManager Server Service + Loaded: loaded (/etc/systemd/system/alertmanager.service; enabled; vendor preset: enabled) + Active: active (running) since Thu 2020-08-20 22:01:21 CEST; 3 days ago + Main PID: 20592 (alertmanager) + Tasks: 70 (limit: 9830) + CGroup: /system.slice/alertmanager.service + +
+ +#### Grafana Plugin + +There is an [Alertmanager plugin in Grafana](https://grafana.com/grafana/plugins/alertmanager/){target=\_blank} that can help you monitor alert information. + +Follow these steps to use the plugin: + +1. Install the plugin: + + ```bash + sudo grafana-cli plugins install camptocamp-prometheus-alertmanager-datasource + ``` + +2. Restart Grafana: + + ```bash + sudo systemctl restart grafana-server + ``` + +3. Configure Alertmanager as a data source in your Grafana dashboard (`SERVER_IP:3000`): + + 1. Go to **Configuration** > **Data Sources** and search for **Prometheus Alertmanager**. + 2. Enter the server URL and port for the Alertmanager service, and select **Save & Test** to verify the connection. + +4. Import the [8010](https://grafana.com/grafana/dashboards/8010-prometheus-alertmanager/){target=\_blank} dashboard for Alertmanager, selecting **Prometheus Alertmanager** in the last column, then select **Import**. + +#### Integrate Alertmanager + +Complete the integration by following these steps to enable communication between Prometheus and Alertmanager and configure detection and alert rules: + +1. Update the `etc/prometheus/prometheus.yml` configuration file to include the following code: + + {% raw %} + ```yml title="prometheus.yml" + rule_files: + - 'rules.yml' + + alerting: + alertmanagers: + - static_configs: + - targets: + - localhost:9093 + ``` + {% endraw %} + + Expand the following item to view the complete `prometheus.yml` file. + + ??? code "prometheus.yml" + + {% raw %} + ```yml title="prometheus.yml" + global: + scrape_interval: 15s + evaluation_interval: 15s + + rule_files: + - 'rules.yml' + + alerting: + alertmanagers: + - static_configs: + - targets: + - localhost:9093 + + scrape_configs: + - job_name: 'prometheus' + scrape_interval: 5s + static_configs: + - targets: ['localhost:9090'] + - job_name: 'substrate_node' + scrape_interval: 5s + static_configs: + - targets: ['localhost:9615'] + + ``` + {% endraw %} + +2. Create the rules file for detection and alerts: + + ```bash + sudo nano /etc/prometheus/rules.yml + ``` + + Add a sample rule to trigger email notifications for node downtime over five minutes: + + {% raw %} + ```yml title="rules.yml" + groups: + - name: alert_rules + rules: + - alert: InstanceDown + expr: up == 0 + for: 5m + labels: + severity: critical + annotations: + summary: 'Instance [{{ $labels.instance }}] down' + description: '[{{ $labels.instance }}] of job [{{ $labels.job }}] has been down for more than 5 minutes.' + + ``` + {% endraw %} + + If any of the conditions defined in the rules file are met, an alert will be triggered. For more on alert rules, refer to [Alerting Rules](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/){target=\_blank} and [additional alerts](https://samber.github.io/awesome-prometheus-alerts/rules.html){target=\_blank}. + +3. Update the file ownership to `prometheus`: + + ```bash + sudo chown prometheus:prometheus rules.yml + ``` + +4. Validate the rules syntax: + + ```bash + sudo -u prometheus promtool check rules rules.yml + ``` + +5. Restart Prometheus and Alertmanager: + + ```bash + sudo systemctl restart prometheus && sudo systemctl restart alertmanager + ``` + +Now you will receive an email alert if one of your rule triggering conditions is met. + +## Secure Your Validator + +Validators in Polkadot's Proof of Stake (PoS) network play a critical role in maintaining network integrity and security by keeping the network in consensus and verifying state transitions. To ensure optimal performance and minimize risks, validators must adhere to strict guidelines around security and reliable operations. + +### Key Management + +Though they don't transfer funds, session keys are essential for validators as they sign messages related to consensus and parachains. Securing session keys is crucial as allowing them to be exploited or used across multiple nodes can lead to a loss of staked funds via [slashing](/nodes-and-validators/run-a-validator/staking-mechanics/offenses-and-slashes/){target=\_blank}. + +Given the current limitations in high-availability setups and the risks associated with double-signing, it’s recommended to run only a single validator instance. Keys should be securely managed, and processes automated to minimize human error. + +There are two approaches for generating session keys: + +- **Generate and store in node**: Using the `author.rotateKeys` RPC call. For most users, generating keys directly within the client is recommended. You must submit a session certificate from your staking proxy to register new keys. See the [How to Validate](/nodes-and-validators/run-a-validator/onboarding-and-offboarding/set-up-validator/){target=\_blank} guide for instructions on setting keys. + +- **Generate outside node and insert**: Using the `author.setKeys` RPC call. This flexibility accommodates advanced security setups and should only be used by experienced validator operators. + +### Signing Outside the Client + +Polkadot plans to support external signing, allowing session keys to reside in secure environments like Hardware Security Modules (HSMs). However, these modules can sign any payload they receive, potentially enabling an attacker to perform slashable actions. + +### Secure-Validator Mode + +Polkadot's Secure-Validator mode offers an extra layer of protection through strict filesystem, networking, and process sandboxing. This secure mode is activated by default if the machine meets the following requirements: + +- **Linux (x86-64 architecture)**: Usually Intel or AMD. +- **Enabled `seccomp`**: This kernel feature facilitates a more secure approach for process management on Linux. Verify by running. + + ```bash + cat /boot/config-`uname -r` | grep CONFIG_SECCOMP= + ``` + + If `seccomp` is enabled, you should see output similar to the following: + + ```bash + CONFIG_SECCOMP=y + ``` + +!!! tip + Optionally, **Linux 5.13** may also be used, as it provides access to even more strict filesystem protections. + +### Linux Best Practices + +Follow these best practices to keep your validator secure: + +- Use a non-root user for all operations. +- Regularly apply OS security patches. +- Enable and configure a firewall. +- Use key-based SSH authentication; deactivate password-based login. +- Regularly back up data and harden your SSH configuration. Visit this [SSH guide](https://blog.stribik.technology/2015/01/04/secure-secure-shell.html){target=\_blank} for more details. + +### Validator Best Practices + +Additional best practices can add an additional layer of security and operational reliability: + +- Only run the Polkadot binary, and only listen on the configured p2p port. +- Run on bare-metal machines, as opposed to virtual machines. +- Provisioning of the validator machine should be automated and defined in code which is kept in private version control, reviewed, audited, and tested. +- Generate and provide session keys in a secure way. +- Start Polkadot at boot and restart if stopped for any reason. +- Run Polkadot as a non-root user. +- Establish and maintain an on-call rotation for managing alerts. +- Establish and maintain a clear protocol with actions to perform for each level of each alert with an escalation policy. + +## Additional Resources + +- [Certus One's Knowledge Base](https://knowledgebase.certus.com/FAQ/){target=\_blank} +- [EOS Block Producer Security List](https://github.com/slowmist/eos-bp-nodes-security-checklist){target=\_blank} +- [HSM Policies and the Importance of Validator Security](https://medium.com/loom-network/hsm-policies-and-the-importance-of-validator-security-ec8a4cc1b6f){target=\_blank} + +For additional guidance, connect with other validators and the Polkadot engineering team in the [Polkadot Validator Lounge](https://matrix.to/#/#polkadotvalidatorlounge:web3.foundation){target=\_blank} on Element. diff --git a/.ai/pages/node-infrastructure-run-a-validator-operational-tasks-pause-validating.md b/.ai/pages/node-infrastructure-run-a-validator-operational-tasks-pause-validating.md new file mode 100644 index 000000000..154270d5e --- /dev/null +++ b/.ai/pages/node-infrastructure-run-a-validator-operational-tasks-pause-validating.md @@ -0,0 +1,46 @@ +--- +title: Pause Validating +description: Learn how to temporarily pause staking activity in Polkadot using the chill extrinsic, with guidance for validators and nominators. +categories: Infrastructure +url: https://docs.polkadot.com/node-infrastructure/run-a-validator/operational-tasks/pause-validating/ +--- + +# Pause Validating + +## Introduction + +If you need to temporarily stop participating in Polkadot staking activities without fully unbonding your funds, chilling your account allows you to do so efficiently. Chilling removes your node from active validation or nomination in the next era while keeping your funds bonded, making it ideal for planned downtimes or temporary pauses. + +This guide covers the steps for chilling as a validator or nominator, using the `chill` and `chillOther` extrinsics, and how these affect your staking status and nominations. + +## Chilling Your Node + +If you need to temporarily step back from staking without unbonding your funds, you can "chill" your account. Chilling pauses your active staking participation, setting your account to inactive in the next era while keeping your funds bonded. + +To chill your account, go to the **Network > Staking > Account Actions** page on [Polkadot.js Apps](https://polkadot.js.org/apps){target=\_blank}, and select **Stop**. Alternatively, you can call the [`chill`](https://paritytech.github.io/polkadot-sdk/master/pallet_staking/enum.Call.html#variant.chill){target=\_blank} extrinsic in the Staking pallet. + +## Staking Election Timing Considerations + +When a node actively participates in staking but then chills, it will continue contributing for the remainder of the current era. However, its eligibility for the next election depends on the chill status at the start of the new era: + +- **Chilled during previous era**: Will not participate in the current era election and will remain inactive until reactivated. +- **Chilled during current era**: Will not be selected for the next era's election. +- **Chilled after current era**: May be selected if it was active during the previous era and is now chilled. + +## Chilling as a Nominator + +When you choose to chill as a nominator, your active nominations are reset. Upon re-entering the nominating process, you must reselect validators to support manually. Depending on preferences, these can be the same validators as before or a new set. Remember that your previous nominations won’t be saved or automatically reactivated after chilling. + +While chilled, your nominator account remains bonded, preserving your staked funds without requiring a full unbonding process. When you’re ready to start nominating again, you can issue a new nomination call to activate your bond with a fresh set of validators. This process bypasses the need for re-bonding, allowing you to maintain your stake while adjusting your involvement in active staking. + +## Chilling as a Validator + +When you chill as a validator, your active validator status is paused. Although your nominators remain bonded to you, the validator bond will no longer appear as an active choice for new or revised nominations until reactivated. Any existing nominators who take no action will still have their stake linked to the validator, meaning they don’t need to reselect the validator upon reactivation. However, if nominators adjust their stakes while the validator is chilled, they will not be able to nominate the chilled validator until it resumes activity. + +Upon reactivating as a validator, you must also reconfigure your validator preferences, such as commission rate and other parameters. These can be set to match your previous configuration or updated as desired. This step is essential for rejoining the active validator set and regaining eligibility for nominations. + +## Chill Other + +Historical constraints in the runtime prevented unlimited nominators and validators from being supported. These constraints created a need for checks to keep the size of the staking system manageable. One of these checks is the `chillOther` extrinsic, allowing users to chill accounts that no longer met standards such as minimum staking requirements set through on-chain governance. + +This control mechanism included a `ChillThreshold`, which was structured to define how close to the maximum number of nominators or validators the staking system would be allowed to get before users could start chilling one another. With the passage of [Referendum #90](https://polkadot-old.polkassembly.io/referendum/90){target=\_blank}, the value for `maxNominatorCount` on Polkadot was set to `None`, effectively removing the limit on how many nominators and validators can participate. This means the `ChillThreshold` will never be met; thus, `chillOther` no longer has any effect. diff --git a/.ai/pages/node-infrastructure-run-a-validator-operational-tasks-upgrade-your-node.md b/.ai/pages/node-infrastructure-run-a-validator-operational-tasks-upgrade-your-node.md new file mode 100644 index 000000000..d3bf39683 --- /dev/null +++ b/.ai/pages/node-infrastructure-run-a-validator-operational-tasks-upgrade-your-node.md @@ -0,0 +1,75 @@ +--- +title: Upgrade a Validator Node +description: Guide to seamlessly upgrading your Polkadot validator node, managing session keys, and executing server maintenance while avoiding downtime and slashing risks. +categories: Infrastructure +url: https://docs.polkadot.com/node-infrastructure/run-a-validator/operational-tasks/upgrade-your-node/ +--- + +# Upgrade a Validator Node + +## Introduction + +Upgrading a Polkadot validator node is essential for staying current with network updates and maintaining optimal performance. This guide covers routine and extended maintenance scenarios, including software upgrades and major server changes. Following these steps, you can manage session keys and transition smoothly between servers without risking downtime, slashing, or network disruptions. The process requires strategic planning, especially if you need to perform long-lead maintenance, ensuring your validator remains active and compliant. + +This guide will allow validators to seamlessly substitute an active validator server to allow for maintenance operations. The process can take several hours, so ensure you understand the instructions first and plan accordingly. + +## Prerequisites + +Before beginning the upgrade process for your validator node, ensure the following: + +- You have a fully functional validator setup with all required binaries installed. See [Set Up a Validator](/nodes-and-validators/run-a-validator/onboarding-and-offboarding/set-up-validator/){target=\_blank} and [Validator Requirements](/nodes-and-validators/run-a-validator/requirements/){target=\_blank} for additional guidance. +- Your VPS infrastructure has enough capacity to run a secondary validator instance temporarily for the upgrade process. + +## Session Keys + +Session keys are used to sign validator operations and establish a connection between your validator node and your staking proxy account. These keys are stored in the client, and any change to them requires a waiting period. Specifically, if you modify your session keys, the change will take effect only after the current session is completed and two additional sessions have passed. + +Remembering this delayed effect when planning upgrades is crucial to ensure that your validator continues to function correctly and avoids interruptions. To learn more about session keys and their importance, visit the [Keys section](https://wiki.polkadot.com/learn/learn-cryptography/#keys){target=\_blank}. + +## Keystore + +Your validator server's `keystore` folder holds the private keys needed for signing network-level transactions. It is important not to duplicate or transfer this folder between validator instances. Doing so could result in multiple validators signing with the duplicate keys, leading to severe consequences such as [equivocation slashing](/nodes-and-validators/run-a-validator/staking-mechanics/offenses-and-slashes/#equivocation-slash){target=\_blank}. Instead, always generate new session keys for each validator instance. + +The default path to the `keystore` is as follows: + +```bash +/home/polkadot/.local/share/polkadot/chains//keystore +``` + +Taking care to manage your keys securely ensures that your validator operates safely and without the risk of slashing penalties. + +## Upgrade Using Backup Validator + +The following instructions outline how to temporarily switch between two validator nodes. The original active validator is referred to as Validator A and the backup node used for maintenance purposes as Validator B. + +### Session `N` + +1. **Start Validator B**: Launch a secondary node and wait until it is fully synced with the network. Once synced, start it with the `--validator` flag. This node will now act as Validator B. +2. **Generate session keys**: Create new session keys specifically for Validator B. +3. **Submit the `set_key` extrinsic**: Use your staking proxy account to submit a `set_key` extrinsic, linking the session keys for Validator B to your staking setup. +4. **Record the session**: Make a note of the session in which you executed this extrinsic. +5. **Wait for session changes**: Allow the current session to end and then wait for two additional full sessions for the new keys to take effect. + +!!! warning "Keep Validator A running" + + It is crucial to keep Validator A operational during this entire waiting period. Since `set_key` does not take effect immediately, turning off Validator A too early may result in chilling or even slashing. + +### Session `N+3` + +At this stage, Validator B becomes your active validator. You can now safely perform any maintenance tasks on Validator A. + +Complete the following steps when you are ready to bring Validator A back online: + +1. **Start Validator A**: Launch Validator A, sync the blockchain database, and ensure it is running with the `--validator` flag. +2. **Generate new session keys for Validator A**: Create fresh session keys for Validator A. +3. **Submit the `set_key` extrinsic**: Using your staking proxy account, submit a `set_key` extrinsic with the new Validator A session keys. +4. **Record the session**: Again, make a note of the session in which you executed this extrinsic. + +Keep Validator B active until the session during which you executed the `set-key` extrinsic completes plus two additional full sessions have passed. Once Validator A has successfully taken over, you can safely stop Validator B. This process helps ensure a smooth handoff between nodes and minimizes the risk of downtime or penalties. Verify the transition by checking for finalized blocks in the new session. The logs should indicate the successful change, similar to the example below: + +
+ INSERT_COMMAND + 2019-10-28 21:44:13 Applying authority set change scheduled at block #450092 + 2019-10-28 21:44:13 Applying GRANDPA set change to new set with 20 authorities + +
diff --git a/.ai/pages/node-infrastructure-run-a-validator-requirements.md b/.ai/pages/node-infrastructure-run-a-validator-requirements.md new file mode 100644 index 000000000..2997baeb9 --- /dev/null +++ b/.ai/pages/node-infrastructure-run-a-validator-requirements.md @@ -0,0 +1,85 @@ +--- +title: Validator Requirements +description: Explore the technical and system requirements for running a Polkadot validator, including setup, hardware, staking prerequisites, and security best practices. +categories: Infrastructure +url: https://docs.polkadot.com/node-infrastructure/run-a-validator/requirements/ +--- + +# Validator Requirements + +## Introduction + +Running a validator in the Polkadot ecosystem is essential for maintaining network security and decentralization. Validators are responsible for validating transactions and adding new blocks to the chain, ensuring the system operates smoothly. In return for their services, validators earn rewards. However, the role comes with inherent risks, such as slashing penalties for misbehavior or technical failures. If you’re new to validation, starting on Kusama provides a lower-stakes environment to gain valuable experience before progressing to the Polkadot network. + +This guide covers everything you need to know about becoming a validator, including system requirements, staking prerequisites, and infrastructure setup. Whether you’re deploying on a VPS or running your node on custom hardware, you’ll learn how to optimize your validator for performance and security, ensuring compliance with network standards while minimizing risks. + +## Prerequisites + +Running a validator requires solid system administration skills and a secure, well-maintained infrastructure. Below are the primary requirements you need to be aware of before getting started: + +- **System administration expertise**: Handling technical anomalies and maintaining node infrastructure is critical. Validators must be able to troubleshoot and optimize their setup. +- **Security**: Ensure your setup follows best practices for securing your node. Refer to the [Secure Your Validator](/nodes-and-validators/run-a-validator/operational-tasks/general-management/#secure-your-validator){target=\_blank} section to learn about important security measures. +- **Network choice**: Start with [Kusama](/nodes-and-validators/run-a-validator/onboarding-and-offboarding/set-up-validator/#run-a-kusama-validator){target=\_blank} to gain experience. Look for "Adjustments for Kusama" throughout these guides for tips on adapting the provided instructions for the Kusama network. +- **Staking requirements**: A minimum amount of native token (KSM or DOT) is required to be elected into the validator set. The required stake can come from your own holdings or from nominators. +- **Risk of slashing**: Any DOT you stake is at risk if your setup fails or your validator misbehaves. If you’re unsure of your ability to maintain a reliable validator, consider nominating your DOT to a trusted validator. + +## Minimum Hardware Requirements + +Polkadot validators rely on high-performance hardware to process blocks efficiently. The recommended minimum hardware requirements to ensure a fully functional and performant validator are as follows: + +- CPU: + + - x86-64 compatible. + - Eight physical cores @ 3.4 GHz. + - Processor: + - **Intel**: Ice Lake or newer (Xeon or Core series) + - **AMD**: Zen3 or newer (EPYC or Ryzen) + - Simultaneous multithreading disabled: + - **Intel**: Hyper-Threading + - **AMD**: SMT + - [Single-threaded performance](https://www.cpubenchmark.net/singleThread.html){target=\_blank} is prioritized over higher cores count. + +- Storage: + + - **NVMe SSD**: At least 2 TB for blockchain data recommended (prioritize latency rather than throughput). + - Storage requirements will increase as the chain grows. For current estimates, see the [current chain snapshot](https://stakeworld.io/docs/dbsize){target=\_blank}. + +- Memory: + + - 32 GB DDR4 ECC + +- Network: + + - Symmetric networking speed of 500 Mbit/s is required to handle large numbers of parachains and ensure congestion control during peak times. + +## VPS Provider List + +When selecting a VPS provider for your validator node, prioritize reliability, consistent performance, and adherence to the specific hardware requirements set for Polkadot validators. The following server types have been tested and showed acceptable performance in benchmark tests. However, this is not an endorsement and actual performance may vary depending on your workload and VPS provider. + +Be aware that some providers may overprovision the underlying host and use shared storage such as NVMe over TCP, which appears as local storage. These setups might result in poor or inconsistent performance. Benchmark your infrastructure before deploying. + +- **[Google Cloud Platform (GCP)](https://cloud.google.com/){target=\_blank}**: `c2` and `c2d` machine families offer high-performance configurations suitable for validators. +- **[Amazon Web Services (AWS)](https://aws.amazon.com/){target=\_blank}**: `c6id` machine family provides strong performance, particularly for I/O-intensive workloads. +- **[OVH](https://www.ovhcloud.com/en-au/){target=\_blank}**: Can be a budget-friendly solution if it meets your minimum hardware specifications. +- **[Digital Ocean](https://www.digitalocean.com/){target=\_blank}**: Popular among developers, Digital Ocean's premium droplets offer configurations suitable for medium to high-intensity workloads. +- **[Vultr](https://www.vultr.com/){target=\_blank}**: Offers flexibility with plans that may meet validator requirements, especially for high-bandwidth needs. +- **[Linode](https://www.linode.com/){target=\_blank}**: Provides detailed documentation, which can be helpful for setup. +- **[Scaleway](https://www.scaleway.com/en/){target=\_blank}**: Offers high-performance cloud instances that can be suitable for validator nodes. +- **[OnFinality](https://onfinality.io/en){target=\_blank}**: Specialized in blockchain infrastructure, OnFinality provides validator-specific support and configurations. + +!!! warning "Acceptable use policies" + Different VPS providers have varying acceptable use policies, and not all allow cryptocurrency-related activities. + + For example, Digital Ocean, requires explicit permission to use servers for cryptocurrency mining and defines unauthorized mining as [network abuse](https://www.digitalocean.com/legal/acceptable-use-policy#network-abuse){target=\_blank} in their acceptable use policy. + + Review the terms for your VPS provider to avoid account suspension or server shutdown due to policy violations. + +## Minimum Bond Requirement + +Before bonding DOT, ensure you meet the minimum bond requirement to start a validator instance. The minimum bond is the least DOT you need to stake to enter the validator set. To become eligible for rewards, your validator node must be nominated by enough staked tokens. + +For example, on November 19, 2024, the minimum stake backing a validator in Polkadot's era 1632 was 1,159,434.248 DOT. You can check the current minimum stake required using these tools: + +- [**Chain State Values**](https://wiki.polkadot.com/general/chain-state-values/){target=\_blank} +- [**Subscan**](https://polkadot.subscan.io/validator_list?status=validator){target=\_blank} +- [**Staking Dashboard**](https://staking.polkadot.cloud/#/overview){target=\_blank} diff --git a/.ai/pages/node-infrastructure-run-a-validator-staking-mechanics-offenses-and-slashes.md b/.ai/pages/node-infrastructure-run-a-validator-staking-mechanics-offenses-and-slashes.md new file mode 100644 index 000000000..6014711bd --- /dev/null +++ b/.ai/pages/node-infrastructure-run-a-validator-staking-mechanics-offenses-and-slashes.md @@ -0,0 +1,171 @@ +--- +title: Offenses and Slashes +description: Learn about how Polkadot discourages validator misconduct via an offenses and slashing system, including details on offenses and their consequences. +categories: Infrastructure +url: https://docs.polkadot.com/node-infrastructure/run-a-validator/staking-mechanics/offenses-and-slashes/ +--- + +# Offenses and Slashes + +## Introduction + +In Polkadot's Nominated Proof of Stake (NPoS) system, validator misconduct is deterred through a combination of slashing, disabling, and reputation penalties. Validators and nominators who stake tokens face consequences for validator misbehavior, which range from token slashes to restrictions on network participation. + +This page outlines the types of offenses recognized by Polkadot, including block equivocations and invalid votes, as well as the corresponding penalties. While some parachains may implement additional custom slashing mechanisms, this guide focuses on the offenses tied to staking within the Polkadot ecosystem. + +## Offenses + +Polkadot is a public permissionless network. As such, it has a mechanism to disincentivize offenses and incentivize good behavior. You can review the [parachain protocol](https://wiki.polkadot.com/learn/learn-parachains-protocol/#parachain-protocol){target=\_blank} to understand better the terminology used to describe offenses. Polkadot validator offenses fall into two categories: invalid votes and equivocations. + +### Invalid Votes + +A validator will be penalized for inappropriate voting activity during the block inclusion and approval processes. The invalid voting related offenses are as follows: + +- **Backing an invalid block**: A para-validator backs an invalid block for inclusion in a fork of the relay chain. +- **`ForInvalid` vote**: When acting as a secondary checker, the validator votes in favor of an invalid block. +- **`AgainstValid` vote**: When acting as a secondary checker, the validator votes against a valid block. This type of vote wastes network resources required to resolve the disparate votes and resulting dispute. + +### Equivocations + +Equivocation occurs when a validator produces statements that conflict with each other when producing blocks or voting. Unintentional equivocations usually occur when duplicate signing keys reside on the validator host. If keys are never duplicated, the probability of an honest equivocation slash decreases to near zero. The equivocation related offenses are as follows: + +- **Equivocation**: The validator produces two or more of the same block or vote. + - **GRANDPA and BEEFY equivocation**: The validator signs two or more votes in the same round on different chains. + - **BABE equivocation**: The validator produces two or more blocks on the relay chain in the same time slot. +- **Double seconded equivocation**: The validator attempts to second, or back, more than one block in the same round. +- **Seconded and valid equivocation**: The validator seconds, or backs, a block and then attempts to hide their role as the responsible backer by later placing a standard validation vote. + +## Penalties + +On Polkadot, offenses to the network incur different penalties depending on severity. There are three main penalties: slashing, disabling, and reputation changes. + +### Slashing + +Validators engaging in bad actor behavior in the network may be subject to slashing if they commit a qualifying offense. When a validator is slashed, they and their nominators lose a percentage of their staked DOT or KSM, from as little as 0.01% up to 100% based on the severity of the offense. Nominators are evaluated for slashing against their active validations at any given time. Validator nodes are evaluated as discrete entities, meaning an operator can't attempt to mitigate the offense on another node they operate in order to avoid a slash. + +Any slashed DOT or KSM will be added to the [Treasury](https://wiki.polkadot.com/learn/learn-polkadot-opengov-treasury/){target=\_blank} rather than burned or distributed as rewards. Moving slashed funds to the Treasury allows tokens to be quickly moved away from malicious validators while maintaining the ability to revert faulty slashes when needed. + +A nominator with a very large bond may nominate several validators in a single era. In this case, a slash is proportionate to the amount staked to the offending validator. Stake allocation and validator activation is controlled by the [Phragmén algorithm](https://wiki.polkadot.com/learn/learn-phragmen/#algorithm){target=\_blank}. + +A validator slash creates an `unapplied` state transition. You can view pending slashes on [Polkadot.js Apps](https://polkadot.js.org/apps/?rpc=wss%3A%2F%2Frpc.polkadot.io#/staking/slashes){target=\_blank}. The UI will display the slash per validator, the affected nominators, and the slash amounts. The unapplied state includes a 27-day grace period during which a governance proposal can be made to reverse the slash. Once this grace period expires, the slash is applied. + +#### Equivocation Slash + +The Web3 Foundation's [Slashing mechanisms](https://research.web3.foundation/Polkadot/security/slashing/amounts){target=\_blank} page provides guidelines for evaluating the security threat level of different offenses and determining penalties proportionate to the threat level of the offense. Offenses requiring coordination between validators or extensive computational costs to the system will typically call for harsher penalties than those more likely to be unintentional than malicious. A description of potential offenses for each threat level and the corresponding penalties is as follows: + +- **Level 1**: Honest misconduct such as isolated cases of unresponsiveness. + - **Penalty**: Validator can be kicked out or slashed up to 0.1% of stake in the validator slot. +- **Level 2**: Misconduct that can occur honestly but is a sign of bad practices. Examples include repeated cases of unresponsiveness and isolated cases of equivocation. + - **Penalty**: Slash of up to 1% of stake in the validator slot. +- **Level 3**: Misconduct that is likely intentional but of limited effect on the performance or security of the network. This level will typically include signs of coordination between validators. Examples include repeated cases of equivocation or isolated cases of unjustified voting on GRANDPA. + - **Penalty**: Reduction in networking reputation metrics, slash of up to 10% of stake in the validator slot. +- **Level 4**: Misconduct that poses severe security or monetary risk to the system or mass collusion. Examples include signs of extensive coordination, creating a serious security risk to the system, or forcing the system to use extensive resources to counter the misconduct. + - **Penalty**: Slash of up to 100% of stake in the validator slot. + +See the next section to understand how slash amounts for equivocations are calculated. If you want to know more details about slashing, please look at the research page on [Slashing mechanisms](https://research.web3.foundation/Polkadot/security/slashing/amounts){target=\_blank}. + +#### Slash Calculation for Equivocation + +The slashing penalty for GRANDPA, BABE, and BEEFY equivocations is calculated using the formula below, where `x` represents the number of offenders and `n` is the total number of validators in the active set: + +```text +min((3 * x / n )^2, 1) +``` + +The following scenarios demonstrate how this formula means slash percentages can increase exponentially based on the number of offenders involved compared to the size of the validator pool: + +- **Minor offense**: Assume 1 validator out of a 100 validator active set equivocates in a slot. A single validator committing an isolated offense is most likely a mistake rather than malicious attack on the network. This offense results in a 0.09% slash to the stake in the validator slot. + + ``` mermaid + flowchart LR + N["Total Validators = 100"] + X["Offenders = 1"] + F["min((3 * 1 / 100)^2, 1) = 0.0009"] + G["0.09% slash of stake"] + + N --> F + X --> F + F --> G + ``` + +- **Moderate offense**: Assume 5 validators out a 100 validator active set equivocate in a slot. This is a slightly more serious event as there may be some element of coordination involved. This offense results in a 2.25% slash to the stake in the validator slot. + + ``` mermaid + flowchart LR + N["Total Validators = 100"] + X["Offenders = 5"] + F["min((3 * 5 / 100)^2, 1) = 0.0225"] + G["2.25% slash of stake"] + + N --> F + X --> F + F --> G + ``` + +- **Major offense**: Assume 20 validators out a 100 validator active set equivocate in a slot. This is a major security threat as it possible represents a coordinated attack on the network. This offense results in a 36% slash and all slashed validators will also be chilled. + ``` mermaid + flowchart LR + N["Total Validators = 100"] + X["Offenders = 20"] + F["min((3 * 20 / 100)^2, 1) = 0.36"] + G["36% slash of stake"] + + N --> F + X --> F + F --> G + ``` + +The examples above show the risk of nominating or running many validators in the active set. While rewards grow linearly (two validators will get you approximately twice as many staking rewards as one), slashing grows exponentially. Going from a single validator equivocating to two validators equivocating causes a slash four time as much as the single validator. + +Validators may run their nodes on multiple machines to ensure they can still perform validation work if one of their nodes goes down. Still, validator operators should be cautious when setting these up. Equivocation is possible if they don't coordinate well in managing signing machines. + +#### Best Practices to Avoid Slashing + +The following are advised to node operators to ensure that they obtain pristine binaries or source code and to ensure the security of their node: + +- Always download either source files or binaries from the official Parity repository. +- Verify the hash of downloaded files. +- Use the W3F secure validator setup or adhere to its principles. +- Ensure essential security items are checked, use a firewall, manage user access, use SSH certificates. +- Avoid using your server as a general-purpose system. Hosting a validator on your workstation or one that hosts other services increases the risk of maleficence. +- Avoid cloning servers (copying all contents) when migrating to new hardware. If an image is needed, create it before generating keys. +- High Availability (HA) systems are generally not recommended as equivocation may occur if concurrent operations happen—such as when a failed server restarts or two servers are falsely online simultaneously. +- Copying the keystore folder when moving a database between instances can cause equivocation. Even brief use of duplicated keystores can result in slashing. + +Below are some examples of small equivocations that happened in the past: + +| Network | Era | Event Type | Details | Action Taken | +|----------|------|--------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------| +| Polkadot | 774 | Small Equivocation | [The validator](https://matrix.to/#/!NZrbtteFeqYKCUGQtr:matrix.parity.io/$165562246360408hKCfC:matrix.org?via=matrix.parity.io&via=corepaper.org&via=matrix.org){target=\_blank} migrated servers and cloned the keystore folder. The on-chain event can be viewed on [Subscan](https://polkadot.subscan.io/extrinsic/11190109-0?event=11190109-5){target=\_blank}. | The validator didn't submit a request for the slash to be canceled. | +| Kusama | 3329 | Small Equivocation | The validator operated a test machine with cloned keys. The test machine was online simultaneously as the primary, which resulted in a slash. | The validator requested a slash cancellation, but the council declined. | +| Kusama | 3995 | Small Equivocation | The validator noticed several errors, after which the client crashed, and a slash was applied. The validator recorded all events and opened GitHub issues to allow for technical opinions to be shared. | The validator requested to cancel the slash. The council approved the request as they believed the error wasn't operator-related. | + +#### Slashing Across Eras + +There are three main difficulties to account for with slashing in NPoS: + +- A nominator can nominate multiple validators and be slashed as a result of actions taken by any of them. +- Until slashed, the stake is reused from era to era. +- Slashable offenses can be found after the fact and out of order. + +To balance this, the system applies only the maximum slash a participant can receive in a given time period rather than the sum. This ensures protection from excessive slashing. + +### Disabling + +The disabling mechanism is triggered when validators commit serious infractions, such as backing invalid blocks or engaging in equivocations. Disabling stops validators from performing specific actions after they have committed an offense. Disabling is further divided into: + +- **On-chain disabling**: Lasts for a whole era and stops validators from authoring blocks, backing, and initiating a dispute. +- **Off-chain disabling**: Lasts for a session, is caused by losing a dispute, and stops validators from initiating a dispute. + +Off-chain disabling is always a lower priority than on-chain disabling. Off-chain disabling prioritizes disabling first backers and then approval checkers. + +The material in this guide reflects the changes introduced in Stage 4. For more details, see the [State of Disabling issue](https://github.com/paritytech/polkadot-sdk/issues/4359){target=\_blank} on GitHub. + + +### Reputation Changes + +Some minor offenses, such as spamming, are only punished by networking reputation changes. Validators use a reputation metric when choosing which peers to connect with. The system adds reputation if a peer provides valuable data and behaves appropriately. If they provide faulty or spam data, the system reduces their reputation. If a validator loses enough reputation, their peers will temporarily close their channels to them. This helps in fighting against Denial of Service (DoS) attacks. Performing validator tasks under reduced reputation will be harder, resulting in lower validator rewards. + +### Penalties by Offense + +Refer to the Polkadot Wiki's [offenses page](https://wiki.polkadot.com/learn/learn-offenses/){target=\_blank} for a summary of penalties for specific offenses. diff --git a/.ai/pages/node-infrastructure-run-a-validator-staking-mechanics-rewards.md b/.ai/pages/node-infrastructure-run-a-validator-staking-mechanics-rewards.md new file mode 100644 index 000000000..557244eed --- /dev/null +++ b/.ai/pages/node-infrastructure-run-a-validator-staking-mechanics-rewards.md @@ -0,0 +1,204 @@ +--- +title: Rewards Payout +description: Learn how validator rewards work on the network, including era points, payout distribution, running multiple validators, and nominator payments. +categories: Infrastructure +url: https://docs.polkadot.com/node-infrastructure/run-a-validator/staking-mechanics/rewards/ +--- + +# Rewards Payout + +## Introduction + +Understanding how rewards are distributed to validators and nominators is essential for network participants. In Polkadot and Kusama, validators earn rewards based on their era points, which are accrued through actions like block production and parachain validation. + +This guide explains the payout scheme, factors influencing rewards, and how multiple validators affect returns. Validators can also share rewards with nominators, who contribute by staking behind them. By following the payout mechanics, validators can optimize their earnings and better engage with their nominators. + +## Era Points + +The Polkadot ecosystem measures its reward cycles in a unit called an era. Kusama eras are approximately 6 hours long, and Polkadot eras are 24 hours long. At the end of each era, validators are paid proportionally to the amount of era points they have collected. Era points are reward points earned for payable actions like: + +- Issuing validity statements for [parachain blocks](/reference/parachains/blocks-transactions-fees/blocks/){target=\_blank}. +- Producing a non-uncle block in the relay chain. +- Producing a reference to a previously unreferenced uncle block. +- Producing a referenced uncle block. + +An uncle block is a relay chain block that is valid in every regard but has failed to become canonical. This can happen when two or more validators are block producers in a single slot, and the block produced by one validator reaches the next block producer before the others. The lagging blocks are called uncle blocks. + +## Reward Variance + +Rewards in Polkadot and Kusama staking systems can fluctuate due to differences in era points earned by para-validators and non-para-validators. Para-validators generally contribute more to the overall reward distribution due to their role in validating parachain blocks, thus influencing the variance in staking rewards. + +To illustrate this relationship: + +- Para-validator era points tend to have a higher impact on the expected value of staking rewards compared to non-para-validator points. +- The variance in staking rewards increases as the total number of validators grows relative to the number of para-validators. +- In simpler terms, when more validators are added to the active set without increasing the para-validator pool, the disparity in rewards between validators becomes more pronounced. + +However, despite this increased variance, rewards tend to even out over time due to the continuous rotation of para-validators across eras. The network's design ensures that over multiple eras, each validator has an equal opportunity to participate in para-validation, eventually leading to a balanced distribution of rewards. + +??? interface "Probability in Staking Rewards" + + This should only serve as a high-level overview of the probabilistic nature for staking rewards. + + Let: + + - `pe` = para-validator era points + - `ne` = non-para-validator era points + - `EV` = expected value of staking rewards + + Then, `EV(pe)` has more influence on the `EV` than `EV(ne)`. + + Since `EV(pe)` has a more weighted probability on the `EV`, the increase in variance against the `EV` becomes apparent between the different validator pools (aka. validators in the active set and the ones chosen to para-validate). + + Also, let: + + - `v` = the variance of staking rewards + - `p` = number of para-validators + - `w` = number validators in the active set + - `e` = era + + Then, `v` ↑ if `w` ↑, as this reduces `p` : `w`, with respect to `e`. + + Increased `v` is expected, and initially keeping `p` ↓ using the same para-validator set for all parachains ensures [availability](https://spec.polkadot.network/chapter-anv){target=\_blank} and [voting](https://wiki.polkadot.com/learn/learn-polkadot-opengov/){target=\_blank}. In addition, despite `v` ↑ on an `e` to `e` basis, over time, the amount of rewards each validator receives will equal out based on the continuous selection of para-validators. + + There are plans to scale the active para-validation set in the future. + +## Payout Scheme + +Validator rewards are distributed equally among all validators in the active set, regardless of the total stake behind each validator. However, individual payouts may differ based on the number of era points a validator has earned. Although factors like network connectivity can affect era points, well-performing validators should accumulate similar totals over time. + +Validators can also receive tips from users, which incentivize them to include certain transactions in their blocks. Validators retain 100% of these tips. + +Rewards are paid out in the network's native token (DOT for Polkadot and KSM for Kusama). + +The following example illustrates a four member validator set with their names, amount they have staked, and how payout of rewards is divided. This scenario assumes all validators earned the same amount of era points and no one received tips: + +``` mermaid +flowchart TD + A["Alice (18 DOT)"] + B["Bob (9 DOT)"] + C["Carol (8 DOT)"] + D["Dave (7 DOT)"] + E["Payout (8 DOT total)"] + E --"2 DOT"--> A + E --"2 DOT"--> B + E --"2 DOT"--> C + E --"2 DOT"--> D +``` + +Note that this is different than most other Proof of Stake (PoS) systems. As long as a validator is in the validator set, it will receive the same block reward as every other validator. Validator Alice, who had 18 DOT staked, received the same 2 DOT reward in this era as Dave, who had only 7 DOT staked. + +## Running Multiple Validators + +Running multiple validators can offer a more favorable risk/reward ratio compared to running a single one. If you have sufficient DOT or nominators staking on your validators, maintaining multiple validators within the active set can yield higher rewards. + +In the preceding section, with 18 DOT staked and no nominators, Alice earned 2 DOT in one era. This example uses DOT, but the same principles apply for KSM on the Kusama network. By managing stake across multiple validators, you can potentially increase overall returns. Recall the set of validators from the preceding section: + +``` mermaid +flowchart TD + A["Alice (18 DOT)"] + B["Bob (9 DOT)"] + C["Carol (8 DOT)"] + D["Dave (7 DOT)"] + E["Payout (8 DOT total)"] + E --"2 DOT"--> A + E --"2 DOT"--> B + E --"2 DOT"--> C + E --"2 DOT"--> D +``` + +Now, assume Alice decides to split their stake and run two validators, each with a nine DOT stake. This validator set only has four spots and priority is given to validators with a larger stake. In this example, Dave has the smallest stake and loses his spot in the validator set. Now, Alice will earn two shares of the total payout each era as illustrated below: + +``` mermaid +flowchart TD + A["Alice (9 DOT)"] + F["Alice (9 DOT)"] + B["Bob (9 DOT)"] + C["Carol (8 DOT)"] + E["Payout (8 DOT total)"] + E --"2 DOT"--> A + E --"2 DOT"--> B + E --"2 DOT"--> C + E --"2 DOT"--> F +``` + +With enough stake, you could run more than two validators. However, each validator must have enough stake behind it to maintain a spot in the validator set. + +## Nominators and Validator Payments + +A nominator's stake allows them to vote for validators and earn a share of the rewards without managing a validator node. Although staking rewards depend on validator activity during an era, validators themselves never control or own nominator rewards. To trigger payouts, anyone can call the `staking.payoutStakers` or `staking.payoutStakerByPage` methods, which mint and distribute rewards directly to the recipients. This trustless process ensures nominators receive their earned rewards. + +Validators set a commission rate as a percentage of the block reward, affecting how rewards are shared with nominators. A 0% commission means the validator keeps only rewards from their self-stake, while a 100% commission means they retain all rewards, leaving none for nominators. + +The following examples model splitting validator payments between nominator and validator using various commission percentages. For simplicity, these examples assume a Polkadot-SDK based relay chain that uses DOT as a native token and a single nominator per validator. Calculations of KSM reward payouts for Kusama follow the same formula. + +Start with the original validator set from the previous section: + +``` mermaid +flowchart TD + A["Alice (18 DOT)"] + B["Bob (9 DOT)"] + C["Carol (8 DOT)"] + D["Dave (7 DOT)"] + E["Payout (8 DOT total)"] + E --"2 DOT"--> A + E --"2 DOT"--> B + E --"2 DOT"--> C + E --"2 DOT"--> D +``` + +The preceding diagram shows each validator receiving a 2 DOT payout, but doesn't account for sharing rewards with nominators. The following diagram shows what nominator payout might look like for validator Alice. Alice has a 20% commission rate and holds 50% of the stake for their validator: + +``` mermaid + +flowchart TD + A["Gross Rewards = 2 DOT"] + E["Commission = 20%"] + F["Alice Validator Payment = 0.4 DOT"] + G["Total Stake Rewards = 1.6 DOT"] + B["Alice Validator Stake = 18 DOT"] + C["9 DOT Alice (50%)"] + H["Alice Stake Reward = 0.8 DOT"] + I["Total Alice Validator Reward = 1.2 DOT"] + D["9 DOT Nominator (50%)"] + J["Total Nominator Reward = 0.8 DOT"] + + A --> E + E --(2 x 0.20)--> F + F --(2 - 0.4)--> G + B --> C + B --> D + C --(1.6 x 0.50)--> H + H --(0.4 + 0.8)--> I + D --(1.60 x 0.50)--> J +``` + +Notice the validator commission rate is applied against the gross amount of rewards for the era. The validator commission is subtracted from the total rewards. After the commission is paid to the validator, the remaining amount is split among stake owners according to their percentage of the total stake. A validator's total rewards for an era include their commission plus their piece of the stake rewards. + +Now, consider a different scenario for validator Bob where the commission rate is 40%, and Bob holds 33% of the stake for their validator: + +``` mermaid + +flowchart TD + A["Gross Rewards = 2 DOT"] + E["Commission = 40%"] + F["Bob Validator Payment = 0.8 DOT"] + G["Total Stake Rewards = 1.2 DOT"] + B["Bob Validator Stake = 9 DOT"] + C["3 DOT Bob (33%)"] + H["Bob Stake Reward = 0.4 DOT"] + I["Total Bob Validator Reward = 1.2 DOT"] + D["6 DOT Nominator (67%)"] + J["Total Nominator Reward = 0.8 DOT"] + + A --> E + E --(2 x 0.4)--> F + F --(2 - 0.8)--> G + B --> C + B --> D + C --(1.2 x 0.33)--> H + H --(0.8 + 0.4)--> I + D --(1.2 x 0.67)--> J +``` + +Bob holds a smaller percentage of their node's total stake, making their stake reward smaller than Alice's. In this scenario, Bob makes up the difference by charging a 40% commission rate and ultimately ends up with the same total payment as Alice. Each validator will need to find their ideal balance between the amount of stake and commission rate to attract nominators while still making running a validator worthwhile. diff --git a/.nav.yml b/.nav.yml index 0bb75ad17..88871fcb5 100644 --- a/.nav.yml +++ b/.nav.yml @@ -4,4 +4,4 @@ nav: - 'Parachains': parachains - 'Chain Interactions': chain-interactions # 'Get Support': get-support - - 'Nodes and Validators': nodes-and-validators + - 'Node Infrastructure': node-infrastructure diff --git a/nodes-and-validators/.nav.yml b/node-infrastructure/.nav.yml similarity index 66% rename from nodes-and-validators/.nav.yml rename to node-infrastructure/.nav.yml index 17b62067c..a14f33cae 100644 --- a/nodes-and-validators/.nav.yml +++ b/node-infrastructure/.nav.yml @@ -1,3 +1,4 @@ nav: - 'Run a Node': run-a-node + - 'Run a Collator': run-a-collator - 'Run a Validator': run-a-validator diff --git a/node-infrastructure/run-a-collator/.nav.yml b/node-infrastructure/run-a-collator/.nav.yml new file mode 100644 index 000000000..a18175837 --- /dev/null +++ b/node-infrastructure/run-a-collator/.nav.yml @@ -0,0 +1,2 @@ +nav: + - 'Run a Block-Producing Collator': collator.md diff --git a/node-infrastructure/run-a-collator/collator.md b/node-infrastructure/run-a-collator/collator.md new file mode 100644 index 000000000..57ba15e1c --- /dev/null +++ b/node-infrastructure/run-a-collator/collator.md @@ -0,0 +1,557 @@ +--- +title: Run a Block-Producing Collator +description: Learn how to set up and run a block-producing collator for Polkadot system parachains, including registration and session key management. +categories: Infrastructure +--- + +# Run a Block-Producing Collator + +## Overview + +Block-producing collators are the backbone of system parachain operations. Unlike RPC nodes or archive nodes that simply maintain state, collators actively produce blocks and submit them to relay chain validators for inclusion. + +This guide covers setting up a **block-producing collator** for Polkadot system parachains. Running a collator requires: + +- Meeting hardware requirements for reliable block production +- Setting up and registering session keys +- Obtaining governance approval or meeting selection criteria +- Maintaining high uptime and performance + +**Important**: System parachain collators typically require governance approval or being added to the invulnerables list. This is different from non-system parachains where collator selection may be more permissionless. + +## Collator Responsibilities + +Block-producing collators perform critical functions: + +- **Maintain full nodes**: Both relay chain and parachain +- **Collect transactions**: Aggregate user transactions into blocks +- **Produce blocks**: Create parachain block candidates +- **Generate proofs**: Produce state transition proofs (Proof-of-Validity) +- **Submit to validators**: Send block candidates to relay chain validators +- **Facilitate XCM**: Enable cross-chain message passing + +Unlike relay chain validators, collators do not provide security guarantees—that responsibility lies with relay chain validators through the ELVES protocol. However, collators are essential for network liveness and censorship resistance. + +## Prerequisites + +### Hardware Requirements + +Block-producing collators require robust hardware for reliable operation: + +- **CPU**: 4+ cores (8+ cores recommended for optimal performance) +- **Memory**: 32 GB RAM minimum (64 GB recommended) +- **Storage**: + - 500 GB+ NVMe SSD for parachain data + - Additional 200+ GB for relay chain pruned database + - Fast disk I/O is critical for block production performance +- **Network**: + - Public IP address (required) + - 100+ Mbps connection (stable connection critical) + - Open ports: + - 30333 (parachain P2P) + - 30334 (relay chain P2P) + - 9944 (WebSocket RPC - for management) + +**Note**: Uptime is critical. Consider redundancy and monitoring to maintain block production reliability. + +### Software Requirements + +Collators use the **Polkadot Omni Node**, a universal binary that runs any parachain using a chain specification file. + +Required software: + +- **Operating System**: Ubuntu 22.04 LTS (recommended) or similar Linux distribution +- **Docker**: For running subkey utility +- **Rust Toolchain**: Version 1.86 or as specified by the runtime +- **Dependencies**: + ```bash + sudo apt update + sudo apt install -y build-essential git clang curl libssl-dev llvm libudev-dev make protobuf-compiler + ``` + +### Account Requirements + +You'll need: +- **Funded account**: For on-chain transactions and potential bonding +- **Session keys**: For collator identification (generated after node setup) +- **Node key**: For stable P2P peer ID (recommended) + +## Installation + +### Step 1: Install Rust and Required Toolchain + +```bash +# Install Rust +curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh +source $HOME/.cargo/env + +# Install specific Rust version +rustup install 1.86 +rustup default 1.86 +rustup target add wasm32-unknown-unknown --toolchain 1.86 +rustup component add rust-src --toolchain 1.86 +``` + +### Step 2: Install the Polkadot Omni Node + +```bash +# Install polkadot-omni-node +cargo install --locked polkadot-omni-node@0.5.0 + +# Verify installation +polkadot-omni-node --version +``` + +### Step 3: Generate Node Key + +Generate a stable node key for consistent peer ID: + +```bash +# Create directory for node data +sudo mkdir -p /var/lib/polkadot-collator + +# Generate node key using Docker +docker run -it parity/subkey:latest generate-node-key > /var/lib/polkadot-collator/node.key + +# The output displays your peer ID +# Example: 12D3KooWExcVYu7Mvjd4kxPVLwN2ZPnZ5NyLZ5ft477wqzfP2q6E +``` + +Save the peer ID for future reference. + +### Step 4: Generate Account Key + +Generate an account for on-chain transactions: + +```bash +# Generate account key with sr25519 scheme +docker run -it parity/subkey:latest generate --scheme sr25519 +``` + +Save the output containing: +- Secret phrase (seed) - Keep this secure! +- Public key (hex) +- Account ID +- SS58 Address + +**Security**: Store the secret phrase securely. Never share it. Consider using a hardware wallet for production collators. + +### Step 5: Obtain Chain Specification + +Download the chain specification for your target system parachain: + +**Option 1: Download from Chainspec Collection (Recommended)** + +1. Visit the [Chainspec Collection website](https://paritytech.github.io/chainspecs/) +2. Find your target system parachain +3. Download the chain specification JSON file +4. Save it as `chain-spec.json` + +**Option 2: Build from Runtime** + +```bash +# Clone the runtimes repository +git clone https://github.com/polkadot-fellows/runtimes.git +cd runtimes + +# Build the desired runtime (example for Polkadot Hub) +cargo build --release -p asset-hub-polkadot-runtime + +# Install chain-spec-builder +cargo install --locked staging-chain-spec-builder@10.0.0 + +# Generate chain spec +chain-spec-builder create \ + --relay-chain polkadot \ + --para-id 1000 \ + --runtime target/release/wbuild/asset-hub-polkadot-runtime/asset_hub_polkadot_runtime.compact.compressed.wasm \ + named-preset production > chain-spec.json +``` + +**System Parachain Para IDs:** +- Polkadot Hub: 1000 +- Bridge Hub: 1002 +- People Chain: 1004 +- Coretime Chain: 1005 + +### Step 6: Create User and Directory Structure + +```bash +# Create dedicated user +sudo useradd -r -s /bin/bash polkadot + +# Copy chain spec to directory +sudo cp chain-spec.json /var/lib/polkadot-collator/ + +# Set permissions +sudo chown -R polkadot:polkadot /var/lib/polkadot-collator +``` + +## Configuration + +### Create Systemd Service File + +Create a service file for your collator: + +```bash +sudo nano /etc/systemd/system/polkadot-collator.service +``` + +Add the following configuration: + +```ini +[Unit] +Description=Polkadot System Parachain Collator +After=network.target + +[Service] +Type=simple +User=polkadot +Group=polkadot +WorkingDirectory=/var/lib/polkadot-collator + +# Block-Producing Collator Configuration +ExecStart=/usr/local/bin/polkadot-omni-node \ + --collator \ + --chain=/var/lib/polkadot-collator/chain-spec.json \ + --base-path=/var/lib/polkadot-collator \ + --port=30333 \ + --rpc-port=9944 \ + --prometheus-port=9615 \ + --node-key-file=/var/lib/polkadot-collator/node.key \ + --name="YourCollatorName" \ + -- \ + --execution=wasm \ + --chain=polkadot \ + --port=30334 \ + --sync=warp + +Restart=always +RestartSec=10 +LimitNOFILE=65536 + +[Install] +WantedBy=multi-user.target +``` + +**Configuration Notes**: +- `--collator`: Enables block production mode +- `--node-key-file`: Uses the generated node key for stable peer ID +- `--name`: Your collator name (visible in telemetry) +- Relay chain uses `--sync=warp` for faster initial sync + +## Running the Collator + +### Step 1: Start the Service + +```bash +# Reload systemd +sudo systemctl daemon-reload + +# Enable service to start on boot +sudo systemctl enable polkadot-collator + +# Start the service +sudo systemctl start polkadot-collator + +# Check status +sudo systemctl status polkadot-collator + +# View logs +sudo journalctl -u polkadot-collator -f +``` + +### Step 2: Initial Sync + +Your collator must sync both the relay chain and parachain before producing blocks. + +Sync time depends on: +- Network bandwidth +- Disk I/O speed +- Current chain size + +The relay chain uses warp sync for faster synchronization. + +Monitor sync progress: +```bash +# Check logs for sync status +sudo journalctl -u polkadot-collator -f | grep "Syncing" + +# Wait for messages indicating full sync +# Example: "Idle" or "Imported" messages for both chains +``` + +**Important**: Do not proceed with registration until both chains are fully synced. + +### Step 3: Generate Session Keys + +Once your node is fully synced, generate session keys: + +```bash +# Generate session keys via RPC +curl -H "Content-Type: application/json" \ + -d '{"id":1, "jsonrpc":"2.0", "method": "author_rotateKeys", "params":[]}' \ + http://localhost:9944 + +# Returns session keys as a hex string +# Example: "0x1234567890abcdef..." +``` + +**Save the session keys** - you'll need them for on-chain registration. + +**Note**: Session keys are stored in the node's database. If you wipe the database, you'll need to generate new keys. + +## Registration and Governance + +### Understanding Collator Selection + +System parachains use different collator selection mechanisms: + +**Invulnerables List**: +- Fixed list of collators approved through governance +- Most common for system parachains +- Requires governance proposal and approval + +**On-chain Selection**: +- Some parachains use pallet-collator-selection +- May require bonding tokens +- Automatic selection based on criteria + +**Fellowship Decisions**: +- Technical Fellowship may manage some system parachain collators +- Requires Fellowship membership or approval + +### Registration Process + +The registration process varies by system parachain. General steps: + +#### 1. Check Current Collators + +Check the existing collators for your target parachain: + +```bash +# Using Polkadot.js Apps +# Connect to your target system parachain +# Go to Developer > Chain State +# Query: collatorSelection.invulnerables() or similar +``` + +#### 2. Prepare Governance Proposal + +For invulnerables-based selection: + +1. **Draft proposal**: Explain why you should be added as a collator +2. **Technical details**: Provide your session keys and account ID +3. **Infrastructure**: Describe your hardware and monitoring setup +4. **Experience**: Detail your relevant experience + +Submit to: +- Polkadot Forum: https://forum.polkadot.network +- Relevant governance channels + +#### 3. Set Session Keys On-Chain + +Once approved (or if using on-chain selection), set your session keys: + +**Using Polkadot.js Apps:** + +1. Navigate to Polkadot.js Apps and connect to your system parachain +2. Go to **Developer > Extrinsics** +3. Select your account +4. Choose `session.setKeys` extrinsic +5. Enter: + - `keys`: Your session keys (from `author_rotateKeys`) + - `proof`: 0x00 (typically) +6. Submit and sign the transaction + +**Using CLI (alternative):** + +```bash +# This varies by parachain - consult specific documentation +``` + +#### 4. Bond Tokens (if required) + +Some parachains require bonding tokens: + +1. Go to **Developer > Extrinsics** +2. Select `collatorSelection.registerAsCandidate` (if available) +3. Submit with required bond amount +4. Sign transaction + +#### 5. Await Governance Approval + +If using invulnerables: +- Wait for governance vote +- Monitor forum and announcements +- Once approved, you'll be added to the invulnerables list +- Your collator will begin producing blocks in the next session/era + +### Verify Collator Status + +Check if your collator is active: + +```bash +# Monitor logs for block production +sudo journalctl -u polkadot-collator -f | grep -i "imported" + +# Look for messages like: +# "Prepared block for proposing" +# "Imported #123" +``` + +## Monitoring and Maintenance + +### Essential Monitoring + +**Block Production**: +```bash +# Monitor block production +sudo journalctl -u polkadot-collator | grep -i "prepared block" +``` + +**Peer Connections**: +- Maintain 30+ peers for good connectivity +- Check peer count in logs + +**Resource Usage**: +- Monitor CPU, RAM, and disk I/O +- Set up alerts for high usage + +**Sync Status**: +- Ensure both chains stay synced +- Alert on sync issues + +### Prometheus Metrics + +Metrics available at `http://localhost:9615/metrics` + +Example Prometheus configuration: +```yaml +scrape_configs: + - job_name: 'polkadot-collator' + static_configs: + - targets: ['localhost:9615'] +``` + +Key metrics to monitor: +- `substrate_block_height`: Current block height +- `substrate_finalized_height`: Finalized block height +- `substrate_peers_count`: Peer connections +- `substrate_ready_transactions_number`: Transaction queue + +### Setting Up Alerts + +Configure alerts for: +- Service failures +- Sync issues +- Low peer count (< 10 peers) +- Block production gaps +- High resource usage +- Disk space low + +### Log Management + +```bash +# View recent logs +sudo journalctl -u polkadot-collator -n 100 + +# Follow logs in real-time +sudo journalctl -u polkadot-collator -f + +# Filter for errors +sudo journalctl -u polkadot-collator | grep -i error + +# Filter for block production +sudo journalctl -u polkadot-collator | grep -i "imported" +``` + +### Database Maintenance + +Check database size: +```bash +# Check database size +du -sh /var/lib/polkadot-collator +``` + +The node handles pruning automatically. + +### Updates and Upgrades + +**Runtime Upgrades**: +- Automatic via on-chain governance +- No manual action required +- Monitor announcements for breaking changes + +**Client Upgrades**: +- Require manual binary update +- Subscribe to announcements: + - Polkadot Forum + - Fellowship GitHub + - Matrix channels + +**Upgrade Procedure**: + +```bash +# Stop the service +sudo systemctl stop polkadot-collator + +# Backup data (recommended) +sudo cp -r /var/lib/polkadot-collator /var/lib/polkadot-collator.backup + +# Update polkadot-omni-node +cargo install --locked --force polkadot-omni-node@ + +# Verify version +polkadot-omni-node --version + +# Restart service +sudo systemctl start polkadot-collator + +# Monitor logs +sudo journalctl -u polkadot-collator -f +``` + +## Security Best Practices + +### Key Management + +- **Secure storage**: Store session keys and account keys securely +- **Never share**: Never share private keys or secret phrases +- **Hardware wallets**: Consider HSM for production +- **Backup**: Keep encrypted backups of keys +- **Rotation**: Plan for key rotation procedures + +### Network Security + +- **Firewall**: Restrict access to necessary ports only +- **SSH**: Use SSH keys, disable password auth +- **VPN**: Consider VPN for administrative access +- **DDoS protection**: Implement if running in cloud + +### System Security + +- **Updates**: Keep OS and software updated +- **Dedicated user**: Never run as root +- **Fail2ban**: Enable for SSH protection +- **Audits**: Regular security audits +- **Minimal services**: Disable unnecessary services + +### Operational Security + +- **Monitoring**: 24/7 monitoring with alerts +- **Backups**: Regular configuration backups +- **Documentation**: Document procedures +- **Incident response**: Have incident response plan +- **Redundancy**: Consider backup collator (standby) + +## Conclusion + +Running a collator node is essential for parachain operation and network security. By following this guide, you have set up a production-ready collator that: + +- Produces blocks for your parachain and maintains network consensus +- Implements comprehensive security measures to protect keys and operations +- Supports robust monitoring and alerting for reliable performance +- Follows best practices for both Docker and systemd deployments + +As a collator operator, you play a vital role in your parachain's infrastructure. Regular maintenance, security updates, and monitoring will ensure your collator continues to perform reliably. Stay engaged with your parachain community and keep up with updates to maintain optimal performance and security. diff --git a/node-infrastructure/run-a-node/.nav.yml b/node-infrastructure/run-a-node/.nav.yml new file mode 100644 index 000000000..f4e014c43 --- /dev/null +++ b/node-infrastructure/run-a-node/.nav.yml @@ -0,0 +1,4 @@ +nav: + - 'Polkadot Hub RPC Node': polkadot-hub-rpc.md + - 'System Parachain RPC Nodes': system-parachain-rpc.md + - 'Relay Chain Nodes': relay-chain diff --git a/node-infrastructure/run-a-node/polkadot-hub-rpc.md b/node-infrastructure/run-a-node/polkadot-hub-rpc.md new file mode 100644 index 000000000..92e5b2528 --- /dev/null +++ b/node-infrastructure/run-a-node/polkadot-hub-rpc.md @@ -0,0 +1,619 @@ +--- +title: Run an RPC Node for Polkadot Hub +description: Complete guide to set up and run an RPC node for Polkadot Hub with Polkadot SDK RPC endpoints. +categories: Infrastructure +--- + +# Run an RPC Node for Polkadot Hub + +## Overview + +[Polkadot Hub](/reference/polkadot-hub/){target=\_blank} is the entry point to Polkadot for all users and application developers. It provides access to essential Web3 services, including: + +- **Asset Management**: Native support for fungible and non-fungible assets +- **Governance, Staking, and Treasury**: Core protocol operations +- **Cross-chain Communication**: XCM message handling + +Running an RPC node for Polkadot Hub enables applications, wallets, and users to interact with the parachain through: +- **Polkadot SDK Node RPC** (Port 9944): Native Polkadot API (WebSocket and HTTP) + +This setup enables block explorer indexing and provides full compatibility with Polkadot SDK development tools. + +**Important Note**: The parameters and configurations in this guide are provided as illustrative examples. You may need to modify them according to your specific environment, hardware capabilities, and network conditions. + +## Prerequisites + +### Hardware Requirements + +RPC nodes serving production traffic require robust hardware: + +- **CPU**: 8+ cores (16+ cores for high traffic) +- **Memory**: 64 GB RAM minimum (128 GB recommended for high traffic) +- **Storage**: + - 500 GB+ NVMe SSD for parachain state (archive nodes require 2-4 TB+) + - Additional 200+ GB for relay chain pruned database + - Fast disk I/O is critical for query performance +- **Network**: + - Public IP address + - 1 Gbps connection (for high traffic scenarios) + - Stable internet connection with sufficient bandwidth + - Open ports: + - 30333 (parachain P2P) + - 30334 (relay chain P2P) + - 9944 (Polkadot SDK WebSocket RPC) + - 9933 (Polkadot SDK HTTP RPC) + - 9615 (Prometheus metrics - optional) + - Consider DDoS protection and rate limiting for production deployments + +**Note**: For development or low-traffic scenarios, you can reduce these requirements proportionally. Consider using a reverse proxy (nginx, Caddy) for production deployments. + +### Software Requirements + +Required software: + +- **Operating System**: Ubuntu 22.04 LTS (recommended) or similar Linux distribution +- **Docker**: Latest version installed and running (for Docker-based setup) +- **rclone**: (Optional but recommended) Command-line program for managing files on cloud storage (https://rclone.org/downloads/) +- **Rust Toolchain**: Version 1.86 or as specified by runtime (for manual build) + +## Setup Options + +This guide provides two deployment options: + +1. **Docker-based Setup**: Simpler to set up and maintain +2. **Manual/Systemd Setup**: For production environments requiring more control + +Choose the option that best fits your needs. + +--- + +## Option 1: Docker-Based Setup + +This option uses Docker containers for the Polkadot SDK node, making it easy to set up and manage. + +### Step 1: Download Chain Specification + +Download the official Polkadot Hub (formerly known as Asset Hub) chain specification: + +```bash +curl -L https://raw.githubusercontent.com/paritytech/polkadot-sdk/master/cumulus/parachains/chain-specs/asset-hub-polkadot.json -o asset-hub-polkadot.json +``` + +**Note**: This chain specification is the official configuration file that defines the network parameters for Polkadot Hub. + +### Step 2: Download Database Snapshots (Optional but Recommended) + +Using pre-synchronized snapshots significantly reduces initial sync time from several days to just a few hours. You need to download both parachain and relay chain data. + +**Snapshot Provider**: https://snapshots.polkadot.io/ + +#### Create Directories + +```bash +mkdir -p my-node-data/chains/asset-hub-polkadot/db +mkdir -p my-node-data/chains/polkadot/db +``` + +#### Download Polkadot Hub Parachain Snapshot + +Choose between archive (complete history) or pruned (recent state) snapshots: + +**Archive Snapshot** (recommended for RPC with historical data): + +```bash +# Check https://snapshots.polkadot.io/ for the latest snapshot URL +export SNAPSHOT_URL_ASSET_HUB="https://snapshots.polkadot.io/polkadot-asset-hub-rocksdb-archive/LATEST" + +rclone copyurl $SNAPSHOT_URL_ASSET_HUB/files.txt files.txt +rclone copy --progress --transfers 20 \ + --http-url $SNAPSHOT_URL_ASSET_HUB \ + --no-traverse --http-no-head --disable-http2 \ + --inplace --no-gzip-encoding --size-only \ + --retries 6 --retries-sleep 10s \ + --files-from files.txt :http: my-node-data/chains/asset-hub-polkadot/db/ + +rm files.txt +``` + +**Parameter Explanation**: +- `--transfers 20`: Uses 20 parallel transfers for faster download +- `--retries 6`: Automatically retries failed transfers up to 6 times +- `--retries-sleep 10s`: Waits 10 seconds between retry attempts +- `--size-only`: Only transfers if sizes differ (prevents unnecessary re-downloads) + +#### Download Polkadot Relay Chain Snapshot + +**Pruned Snapshot** (recommended for RPC nodes): + +```bash +# Check https://snapshots.polkadot.io/ for the latest snapshot URL +export SNAPSHOT_URL_RELAY="https://snapshots.polkadot.io/polkadot-rocksdb-prune/LATEST" + +rclone copyurl $SNAPSHOT_URL_RELAY/files.txt files.txt +rclone copy --progress --transfers 20 \ + --http-url $SNAPSHOT_URL_RELAY \ + --no-traverse --http-no-head --disable-http2 \ + --inplace --no-gzip-encoding --size-only \ + --retries 6 --retries-sleep 10s \ + --files-from files.txt :http: my-node-data/chains/polkadot/db/ + +rm files.txt +``` + +**Alternative Options**: +- Pruned snapshot: `polkadot-rocksdb-prune` (smaller size, recent state) +- Archive snapshot: `polkadot-rocksdb-archive` (complete history, larger size) + +### Step 3: Start Polkadot Hub Node + +Launch the node using the official Parity Docker image: + +**Docker Image**: https://hub.docker.com/r/parity/polkadot-omni-node + +```bash +docker run -d --name polkadot-hub-rpc --restart unless-stopped \ + -p 9944:9944 \ + -p 9933:9933 \ + -p 9615:9615 \ + -p 30334:30334 \ + -p 30333:30333 \ + -v $(pwd)/asset-hub-polkadot.json:/asset-hub-polkadot.json \ + -v $(pwd)/my-node-data:/data \ + parity/polkadot-omni-node:stable2506-4 \ + --name=PolkadotHubRPC \ + --base-path=/data \ + --chain=/asset-hub-polkadot.json \ + --prometheus-external \ + --prometheus-port 9615 \ + --unsafe-rpc-external \ + --rpc-port=9944 \ + --rpc-cors=all \ + --rpc-methods=safe \ + --rpc-max-connections=1000 \ + --state-pruning=archive \ + --blocks-pruning=archive \ + -- \ + --base-path=/data \ + --chain=polkadot \ + --state-pruning=256 \ + --blocks-pruning=archive-canonical +``` + +**Critical Configuration Parameters**: + +**Port Mappings**: +- `9944`: Polkadot SDK RPC endpoint (WebSocket/HTTP) +- `9933`: Polkadot SDK HTTP RPC endpoint +- `9615`: Prometheus metrics endpoint +- `30333/30334`: P2P networking ports + +**Node Parameters**: +- `--unsafe-rpc-external`: Enables external RPC access +- `--rpc-cors=all`: Allows all origins for CORS +- `--rpc-methods=safe`: Only allows safe RPC methods +- `--state-pruning=archive`: Keeps complete state history +- `--blocks-pruning=archive`: Keeps all block data +- `--prometheus-external`: Exposes metrics externally + +**Security Warning**: The `--unsafe-rpc-external` flag should only be used in development or properly secured environments. For production, use a reverse proxy with authentication. + +### Step 4: Monitor Synchronization + +Monitor the node synchronization status: + +```bash +# Check sync status +curl -H "Content-Type: application/json" \ + -d '{"id":1, "jsonrpc":"2.0", "method": "system_syncState", "params":[]}' \ + http://localhost:9944 +``` + +**Expected Response Format**: + +```json +{ + "jsonrpc":"2.0", + "id":1, + "result":{ + "startingBlock":0, + "currentBlock":3394816, + "highestBlock":3394816 + } +} +``` + +**Synchronization Status**: +- **In Progress**: `currentBlock` < `highestBlock` +- **Complete**: `currentBlock` = `highestBlock` + +**Monitor logs**: + +```bash +# View node logs +docker logs -f polkadot-hub-rpc + +# Filter for sync messages +docker logs polkadot-hub-rpc 2>&1 | grep -i "syncing" +``` + +### Step 5: Verify Setup + +Let's verify the Polkadot SDK RPC endpoint is working correctly. + +#### API Endpoint + +**Polkadot SDK RPC (Port 9944)**: +- WebSocket: `ws://your-server:9944` +- HTTP: `http://your-server:9944` +- Purpose: Full Polkadot SDK API access for parachain data +- Use Cases: Polkadot SDK applications, parachain-specific operations + +#### Polkadot SDK RPC Tests + +**Get Chain Information**: + +```bash +curl -H "Content-Type: application/json" \ + -d '{"id":1, "jsonrpc":"2.0", "method": "system_chain", "params":[]}' \ + http://localhost:9944 +``` + +**Get Latest Block**: + +```bash +curl -H "Content-Type: application/json" \ + -d '{"id":1, "jsonrpc":"2.0", "method": "chain_getHeader", "params":[]}' \ + http://localhost:9944 +``` + +**Get Node Health**: + +```bash +curl -H "Content-Type: application/json" \ + -d '{"id":1, "jsonrpc":"2.0", "method": "system_health", "params":[]}' \ + http://localhost:9944 +``` + +### Managing Docker Containers + +**View logs**: + +```bash +# View node logs +docker logs -f polkadot-hub-rpc +``` + +**Stop container**: + +```bash +docker stop polkadot-hub-rpc +``` + +**Start container**: + +```bash +docker start polkadot-hub-rpc +``` + +**Remove container**: + +```bash +docker rm polkadot-hub-rpc +``` + +**Update container**: + +```bash +# Pull latest image +docker pull parity/polkadot-omni-node:stable2506-4 + +# Stop and remove old container +docker stop polkadot-hub-rpc +docker rm polkadot-hub-rpc + +# Start new container with same command as above +``` + +--- + +## Option 2: Manual/Systemd Setup + +This option provides more control and is recommended for production environments requiring custom configurations. + +### Step 1: Install Rust and Required Toolchain + +```bash +# Install Rust +curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh +source $HOME/.cargo/env + +# Install specific Rust version +rustup install 1.86 +rustup default 1.86 +rustup target add wasm32-unknown-unknown --toolchain 1.86 +rustup component add rust-src --toolchain 1.86 +``` + +### Step 2: Install the Polkadot Omni Node + +```bash +# Install polkadot-omni-node +cargo install --locked polkadot-omni-node@0.7.0 + +# Verify installation +polkadot-omni-node --version +``` + +### Step 3: Obtain Chain Specification + +Download the Polkadot Hub chain specification: + +```bash +curl -L https://raw.githubusercontent.com/paritytech/polkadot-sdk/master/cumulus/parachains/chain-specs/asset-hub-polkadot.json -o asset-hub-polkadot.json +``` + +### Step 4: Create User and Directory Structure + +```bash +# Create dedicated user +sudo useradd -r -s /bin/bash polkadot + +# Create data directory +sudo mkdir -p /var/lib/polkadot-hub-rpc + +# Copy chain spec to the directory +sudo cp asset-hub-polkadot.json /var/lib/polkadot-hub-rpc/ + +# Set permissions +sudo chown -R polkadot:polkadot /var/lib/polkadot-hub-rpc +``` + +### Step 5: Create Systemd Service for Polkadot SDK Node + +Create a service file for the Polkadot SDK RPC node: + +```bash +sudo nano /etc/systemd/system/polkadot-hub-rpc.service +``` + +Add the following configuration: + +```ini +[Unit] +Description=Polkadot Hub RPC Node +After=network.target + +[Service] +Type=simple +User=polkadot +Group=polkadot +WorkingDirectory=/var/lib/polkadot-hub-rpc + +ExecStart=/usr/local/bin/polkadot-omni-node \ + --name=PolkadotHubRPC \ + --chain=/var/lib/polkadot-hub-rpc/asset-hub-polkadot.json \ + --base-path=/var/lib/polkadot-hub-rpc \ + --port=30333 \ + --rpc-port=9944 \ + --rpc-external \ + --rpc-cors=all \ + --rpc-methods=safe \ + --rpc-max-connections=1000 \ + --prometheus-port=9615 \ + --prometheus-external \ + --state-pruning=archive \ + --blocks-pruning=archive \ + -- \ + --chain=polkadot \ + --base-path=/var/lib/polkadot-hub-rpc \ + --port=30334 \ + --state-pruning=256 \ + --blocks-pruning=archive-canonical + +Restart=always +RestartSec=10 +LimitNOFILE=65536 + +[Install] +WantedBy=multi-user.target +``` + +### Step 6: Start Service + +```bash +# Reload systemd +sudo systemctl daemon-reload + +# Enable service to start on boot +sudo systemctl enable polkadot-hub-rpc + +# Start the Polkadot SDK node +sudo systemctl start polkadot-hub-rpc + +# Check status and wait for sync +sudo systemctl status polkadot-hub-rpc +sudo journalctl -u polkadot-hub-rpc -f +``` + +### Step 7: Verify Setup + +Use the same verification tests as in the Docker setup (see Step 5 above). + +--- + +## Monitoring and Maintenance + +### Log Management + +**Docker Setup**: + +```bash +# View node logs +docker logs -f polkadot-hub-rpc +``` + +**Systemd Setup**: + +```bash +# View node logs +sudo journalctl -u polkadot-hub-rpc -f + +# View recent logs +sudo journalctl -u polkadot-hub-rpc -n 100 + +# Filter for errors +sudo journalctl -u polkadot-hub-rpc | grep -i error +``` + +### Performance Monitoring + +Monitor key metrics: +- **Sync status**: Ensure node stays fully synced +- **Peer connections**: Maintain 30+ peers for good connectivity +- **Resource usage**: Monitor CPU, RAM, and disk I/O +- **RPC request latency**: Track response times for the Polkadot SDK API +- **Connection count**: Monitor active RPC connections + +**Prometheus Metrics**: + +Metrics are available at `http://localhost:9615/metrics` + +Example Prometheus scrape configuration: + +```yaml +scrape_configs: + - job_name: 'polkadot-hub-rpc' + static_configs: + - targets: ['localhost:9615'] +``` + +**Key Metrics to Monitor**: +- `substrate_block_height`: Current block height +- `substrate_finalized_height`: Finalized block height +- `substrate_peers_count`: Number of connected peers +- `substrate_ready_transactions_number`: Transaction queue size + +### Database Maintenance + +Check database size periodically: + +```bash +# Docker setup +du -sh my-node-data + +# Systemd setup +du -sh /var/lib/polkadot-hub-rpc +``` + +The node handles pruning automatically based on configuration unless running in archive mode. + +### Updates and Upgrades + +**Docker Setup**: + +```bash +# Pull latest image +docker pull parity/polkadot-omni-node:stable2506-4 + +# Restart container +docker stop polkadot-hub-rpc +docker rm polkadot-hub-rpc + +# Start new container (use same command from setup) +``` + +**Systemd Setup**: + +```bash +# Stop service +sudo systemctl stop polkadot-hub-rpc + +# Backup data +sudo cp -r /var/lib/polkadot-hub-rpc /var/lib/polkadot-hub-rpc.backup + +# Update binary +cargo install --locked --force polkadot-omni-node@ + +# Restart service +sudo systemctl start polkadot-hub-rpc +``` + +## Security Best Practices + +### Network Security + +1. **Firewall Configuration**: + - Only expose necessary ports + - Use UFW or iptables to restrict access + - Consider IP whitelisting for RPC endpoints + +2. **Reverse Proxy** (Recommended for Production): + - Use nginx or Caddy as reverse proxy + - Enable SSL/TLS (HTTPS/WSS) + - Implement authentication + - Add rate limiting + +Example nginx configuration: + +```nginx +upstream polkadot_sdk_rpc { + server 127.0.0.1:9944; +} + +server { + listen 443 ssl http2; + server_name your-domain.com; + + ssl_certificate /path/to/cert.pem; + ssl_certificate_key /path/to/key.pem; + + # Polkadot SDK RPC + location /polkadot { + proxy_pass http://polkadot_sdk_rpc; + proxy_http_version 1.1; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection "upgrade"; + + # Rate limiting + limit_req zone=rpc_limit burst=10; + } +} +``` + +### RPC Security + +- **Always use `--rpc-methods=safe`**: Prevents dangerous RPC calls +- **Restrict CORS**: Use specific domains instead of `all` in production +- **Set connection limits**: Prevent resource exhaustion +- **Monitor for abuse**: Track unusual patterns +- **Authentication**: Implement API keys or OAuth for production + +### System Security + +- Keep operating system updated +- Use dedicated user accounts (never root) +- Enable fail2ban for SSH protection +- Regular security audits +- Disable unnecessary services +- Use AppArmor or SELinux for additional isolation + +### Monitoring and Alerting + +Set up alerts for: +- Service failures +- Sync issues +- Low peer count (< 10 peers) +- High resource usage +- Unusual RPC traffic patterns +- Database errors + +## Conclusion + +Running an RPC node for Polkadot Hub provides essential infrastructure for applications and users to interact with the network. By following this guide, you have set up a production-ready RPC node that: + +- Provides reliable access to Polkadot Hub's asset management, governance, and cross-chain communication features +- Supports both Docker and systemd deployment options for flexibility +- Implements proper monitoring, security, and maintenance practices +- Serves as a foundation for building and operating Polkadot SDK applications + +Regular maintenance, security updates, and monitoring will ensure your RPC node continues to serve your users reliably. As the Polkadot network evolves, stay informed about updates and best practices through the official channels and community resources listed in this guide. diff --git a/nodes-and-validators/run-a-node/.nav.yml b/node-infrastructure/run-a-node/relay-chain/.nav.yml similarity index 100% rename from nodes-and-validators/run-a-node/.nav.yml rename to node-infrastructure/run-a-node/relay-chain/.nav.yml diff --git a/nodes-and-validators/run-a-node/bootnode.md b/node-infrastructure/run-a-node/relay-chain/bootnode.md similarity index 100% rename from nodes-and-validators/run-a-node/bootnode.md rename to node-infrastructure/run-a-node/relay-chain/bootnode.md diff --git a/nodes-and-validators/run-a-node/full-node.md b/node-infrastructure/run-a-node/relay-chain/full-node.md similarity index 100% rename from nodes-and-validators/run-a-node/full-node.md rename to node-infrastructure/run-a-node/relay-chain/full-node.md diff --git a/nodes-and-validators/run-a-node/secure-wss.md b/node-infrastructure/run-a-node/relay-chain/secure-wss.md similarity index 100% rename from nodes-and-validators/run-a-node/secure-wss.md rename to node-infrastructure/run-a-node/relay-chain/secure-wss.md diff --git a/node-infrastructure/run-a-node/system-parachain-rpc.md b/node-infrastructure/run-a-node/system-parachain-rpc.md new file mode 100644 index 000000000..82823a719 --- /dev/null +++ b/node-infrastructure/run-a-node/system-parachain-rpc.md @@ -0,0 +1,649 @@ +--- +title: Run an RPC Node for System Parachains +description: Complete guide to set up and run an RPC node for Polkadot system parachains including Bridge Hub, People Chain, and Coretime Chain. +categories: Infrastructure +--- + +# Run an RPC Node for System Parachains + +## Overview + +System parachains are core infrastructure parachains that provide essential services to the Polkadot network. Running an RPC node for these parachains enables applications, wallets, and users to interact with their specialized functionality: + +- **Bridge Hub**: Cross-chain asset transfers via trustless bridges +- **People Chain**: Identity and social credential management +- **Coretime Chain**: Blockspace allocation and core time management + +Each system parachain RPC node provides access through: +- **Polkadot SDK Node RPC** (Port 9944): Native Polkadot API (WebSocket and HTTP) + +This setup enables block explorer indexing and provides full compatibility with Polkadot SDK development tools. + +**Important Note**: The parameters and configurations in this guide are provided as illustrative examples. You may need to modify them according to your specific environment, hardware capabilities, and network conditions. + +## Choosing a System Parachain + +This guide uses **People Chain** as the example, but the same principles and setup procedures apply to all system parachains. Simply substitute the appropriate values from the table below for your chosen parachain: + +| Parachain | Para ID | Chain Spec File | Snapshot Path | Chain Name | +|-----------|---------|-----------------|---------------|------------| +| **Bridge Hub** | 1002 | `bridge-hub-polkadot.json` | `polkadot-bridge-hub-rocksdb-archive` | `bridge-hub-polkadot` | +| **People Chain** | 1004 | `people-polkadot.json` | `polkadot-people-rocksdb-archive` | `people-polkadot` | +| **Coretime Chain** | 1005 | `coretime-polkadot.json` | `polkadot-coretime-rocksdb-archive` | `coretime-polkadot` | + +**Note**: Throughout this guide, we use People Chain values. To set up a different system parachain, replace the chain spec file, snapshot path, and chain name with the corresponding values from the table above. + +## Prerequisites + +### Hardware Requirements + +RPC nodes serving production traffic require robust hardware: + +- **CPU**: 8+ cores (16+ cores for high traffic) +- **Memory**: 64 GB RAM minimum (128 GB recommended for high traffic) +- **Storage**: + - 500 GB+ NVMe SSD for parachain state (archive nodes require 2-4 TB+) + - Additional 200+ GB for relay chain pruned database + - Fast disk I/O is critical for query performance +- **Network**: + - Public IP address + - 1 Gbps connection (for high traffic scenarios) + - Stable internet connection with sufficient bandwidth + - Open ports: + - 30333 (parachain P2P) + - 30334 (relay chain P2P) + - 9944 (Polkadot SDK WebSocket RPC) + - 9933 (Polkadot SDK HTTP RPC) + - 9615 (Prometheus metrics - optional) + - Consider DDoS protection and rate limiting for production deployments + +**Note**: For development or low-traffic scenarios, you can reduce these requirements proportionally. Consider using a reverse proxy (nginx, Caddy) for production deployments. + +### Software Requirements + +Required software: + +- **Operating System**: Ubuntu 22.04 LTS (recommended) or similar Linux distribution +- **Docker**: Latest version installed and running (for Docker-based setup) +- **rclone**: (Optional but recommended) Command-line program for managing files on cloud storage (https://rclone.org/downloads/) +- **Rust Toolchain**: Version 1.86 or as specified by runtime (for manual build) + +## Setup Options + +This guide provides two deployment options: + +1. **Docker-based Setup**: Simpler to set up and maintain +2. **Manual/Systemd Setup**: For production environments requiring more control + +Choose the option that best fits your needs. + +--- + +## Option 1: Docker-Based Setup + +This option uses Docker containers for the Polkadot SDK node, making it easy to set up and manage. + +### Step 1: Download Chain Specification + +Download the official chain specification for People Chain: + +```bash +curl -L https://raw.githubusercontent.com/paritytech/polkadot-sdk/master/cumulus/parachains/chain-specs/people-polkadot.json -o people-polkadot.json +``` + +**Note**: This chain specification is the official configuration file that defines the network parameters for People Chain. + +### Step 2: Download Database Snapshots (Optional but Recommended) + +Using pre-synchronized snapshots significantly reduces initial sync time from several days to just a few hours. You need to download both parachain and relay chain data. + +**Snapshot Provider**: https://snapshots.polkadot.io/ + +#### Create Directories + +```bash +mkdir -p my-node-data/chains/people-polkadot/db +mkdir -p my-node-data/chains/polkadot/db +``` + +#### Download People Chain Snapshot + +Choose between archive (complete history) or pruned (recent state) snapshots. + +**Archive Snapshot** (recommended for RPC with historical data): + +```bash +# Check https://snapshots.polkadot.io/ for the latest snapshot URL +export SNAPSHOT_URL_PARACHAIN="https://snapshots.polkadot.io/polkadot-people-rocksdb-archive/LATEST" + +rclone copyurl $SNAPSHOT_URL_PARACHAIN/files.txt files.txt +rclone copy --progress --transfers 20 \ + --http-url $SNAPSHOT_URL_PARACHAIN \ + --no-traverse --http-no-head --disable-http2 \ + --inplace --no-gzip-encoding --size-only \ + --retries 6 --retries-sleep 10s \ + --files-from files.txt :http: my-node-data/chains/people-polkadot/db/ + +rm files.txt +``` + +**Parameter Explanation**: +- `--transfers 20`: Uses 20 parallel transfers for faster download +- `--retries 6`: Automatically retries failed transfers up to 6 times +- `--retries-sleep 10s`: Waits 10 seconds between retry attempts +- `--size-only`: Only transfers if sizes differ (prevents unnecessary re-downloads) + +**Note**: If a snapshot is not available, you can sync from scratch (which will take longer) or check https://snapshots.polkadot.io/ for alternative snapshot providers. + +#### Download Polkadot Relay Chain Snapshot + +**Pruned Snapshot** (recommended for RPC nodes): + +```bash +# Check https://snapshots.polkadot.io/ for the latest snapshot URL +export SNAPSHOT_URL_RELAY="https://snapshots.polkadot.io/polkadot-rocksdb-prune/LATEST" + +rclone copyurl $SNAPSHOT_URL_RELAY/files.txt files.txt +rclone copy --progress --transfers 20 \ + --http-url $SNAPSHOT_URL_RELAY \ + --no-traverse --http-no-head --disable-http2 \ + --inplace --no-gzip-encoding --size-only \ + --retries 6 --retries-sleep 10s \ + --files-from files.txt :http: my-node-data/chains/polkadot/db/ + +rm files.txt +``` + +**Alternative Options**: +- Pruned snapshot: `polkadot-rocksdb-prune` (smaller size, recent state) +- Archive snapshot: `polkadot-rocksdb-archive` (complete history, larger size) + +### Step 3: Start People Chain Node + +Launch the node using the official Parity Docker image. + +**Docker Image**: https://hub.docker.com/r/parity/polkadot-omni-node + +```bash +docker run -d --name people-chain-rpc --restart unless-stopped \ + -p 9944:9944 \ + -p 9933:9933 \ + -p 9615:9615 \ + -p 30334:30334 \ + -p 30333:30333 \ + -v $(pwd)/people-polkadot.json:/people-polkadot.json \ + -v $(pwd)/my-node-data:/data \ + parity/polkadot-omni-node:stable2506-4 \ + --name=PeopleChainRPC \ + --base-path=/data \ + --chain=/people-polkadot.json \ + --prometheus-external \ + --prometheus-port 9615 \ + --unsafe-rpc-external \ + --rpc-port=9944 \ + --rpc-cors=all \ + --rpc-methods=safe \ + --rpc-max-connections=1000 \ + --state-pruning=archive \ + --blocks-pruning=archive \ + -- \ + --base-path=/data \ + --chain=polkadot \ + --state-pruning=256 \ + --blocks-pruning=archive-canonical +``` + +**Critical Configuration Parameters**: + +**Port Mappings**: +- `9944`: Polkadot SDK RPC endpoint (WebSocket/HTTP) +- `9933`: Polkadot SDK HTTP RPC endpoint +- `9615`: Prometheus metrics endpoint +- `30333/30334`: P2P networking ports + +**Node Parameters**: +- `--unsafe-rpc-external`: Enables external RPC access +- `--rpc-cors=all`: Allows all origins for CORS +- `--rpc-methods=safe`: Only allows safe RPC methods +- `--state-pruning=archive`: Keeps complete state history +- `--blocks-pruning=archive`: Keeps all block data +- `--prometheus-external`: Exposes metrics externally + +**Security Warning**: The `--unsafe-rpc-external` flag should only be used in development or properly secured environments. For production, use a reverse proxy with authentication. + +### Step 4: Monitor Synchronization + +Monitor the node synchronization status: + +```bash +# Check sync status +curl -H "Content-Type: application/json" \ + -d '{"id":1, "jsonrpc":"2.0", "method": "system_syncState", "params":[]}' \ + http://localhost:9944 +``` + +**Expected Response Format**: + +```json +{ + "jsonrpc":"2.0", + "id":1, + "result":{ + "startingBlock":0, + "currentBlock":3394816, + "highestBlock":3394816 + } +} +``` + +**Synchronization Status**: +- **In Progress**: `currentBlock` < `highestBlock` +- **Complete**: `currentBlock` = `highestBlock` + +**Monitor logs**: + +```bash +# View node logs +docker logs -f people-chain-rpc + +# Filter for sync messages +docker logs people-chain-rpc 2>&1 | grep -i "syncing" +``` + +### Step 5: Verify Setup + +Let's verify the Polkadot SDK RPC endpoint is working correctly. + +#### API Endpoint + +**Polkadot SDK RPC (Port 9944)**: +- WebSocket: `ws://your-server:9944` +- HTTP: `http://your-server:9944` +- Purpose: Full Polkadot SDK API access for parachain data +- Use Cases: Polkadot SDK applications, parachain-specific operations + +#### Polkadot SDK RPC Tests + +**Get Chain Information**: + +```bash +curl -H "Content-Type: application/json" \ + -d '{"id":1, "jsonrpc":"2.0", "method": "system_chain", "params":[]}' \ + http://localhost:9944 +``` + +**Expected Response**: +```json +{ + "jsonrpc":"2.0", + "id":1, + "result":"Polkadot People" +} +``` + +**Get Latest Block**: + +```bash +curl -H "Content-Type: application/json" \ + -d '{"id":1, "jsonrpc":"2.0", "method": "chain_getHeader", "params":[]}' \ + http://localhost:9944 +``` + +**Get Node Health**: + +```bash +curl -H "Content-Type: application/json" \ + -d '{"id":1, "jsonrpc":"2.0", "method": "system_health", "params":[]}' \ + http://localhost:9944 +``` + +**Get Peer Count**: + +```bash +curl -H "Content-Type: application/json" \ + -d '{"id":1, "jsonrpc":"2.0", "method": "system_peers", "params":[]}' \ + http://localhost:9944 +``` + +### Managing Docker Containers + +**View logs**: + +```bash +docker logs -f people-chain-rpc +``` + +**Stop container**: + +```bash +docker stop people-chain-rpc +``` + +**Start container**: + +```bash +docker start people-chain-rpc +``` + +**Remove container**: + +```bash +docker rm people-chain-rpc +``` + +**Update container**: + +```bash +# Pull latest image +docker pull parity/polkadot-omni-node:stable2506-4 + +# Stop and remove old container +docker stop people-chain-rpc +docker rm people-chain-rpc + +# Start new container with same command as above +``` + +--- + +## Option 2: Manual/Systemd Setup + +This option provides more control and is recommended for production environments requiring custom configurations. + +### Step 1: Install Rust and Required Toolchain + +```bash +# Install Rust +curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh +source $HOME/.cargo/env + +# Install specific Rust version +rustup install 1.86 +rustup default 1.86 +rustup target add wasm32-unknown-unknown --toolchain 1.86 +rustup component add rust-src --toolchain 1.86 +``` + +### Step 2: Install the Polkadot Omni Node + +```bash +# Install polkadot-omni-node +cargo install --locked polkadot-omni-node@0.7.0 + +# Verify installation +polkadot-omni-node --version +``` + +### Step 3: Obtain Chain Specification + +Download the People Chain specification: + +```bash +curl -L https://raw.githubusercontent.com/paritytech/polkadot-sdk/master/cumulus/parachains/chain-specs/people-polkadot.json -o people-polkadot.json +``` + +### Step 4: Create User and Directory Structure + +```bash +# Create dedicated user (skip if already exists) +sudo useradd -r -s /bin/bash polkadot + +# Create data directory +sudo mkdir -p /var/lib/people-chain-rpc + +# Copy chain spec to the directory +sudo cp people-polkadot.json /var/lib/people-chain-rpc/ + +# Set permissions +sudo chown -R polkadot:polkadot /var/lib/people-chain-rpc +``` + +### Step 5: Create Systemd Service + +Create a service file for the People Chain RPC node: + +```bash +sudo nano /etc/systemd/system/people-chain-rpc.service +``` + +Add the following configuration: + +```ini +[Unit] +Description=People Chain RPC Node +After=network.target + +[Service] +Type=simple +User=polkadot +Group=polkadot +WorkingDirectory=/var/lib/people-chain-rpc + +ExecStart=/usr/local/bin/polkadot-omni-node \ + --name=PeopleChainRPC \ + --chain=/var/lib/people-chain-rpc/people-polkadot.json \ + --base-path=/var/lib/people-chain-rpc \ + --port=30333 \ + --rpc-port=9944 \ + --rpc-external \ + --rpc-cors=all \ + --rpc-methods=safe \ + --rpc-max-connections=1000 \ + --prometheus-port=9615 \ + --prometheus-external \ + --state-pruning=archive \ + --blocks-pruning=archive \ + -- \ + --chain=polkadot \ + --base-path=/var/lib/people-chain-rpc \ + --port=30334 \ + --state-pruning=256 \ + --blocks-pruning=archive-canonical + +Restart=always +RestartSec=10 +LimitNOFILE=65536 + +[Install] +WantedBy=multi-user.target +``` + +### Step 6: Start Service + +```bash +# Reload systemd +sudo systemctl daemon-reload + +# Enable service to start on boot +sudo systemctl enable people-chain-rpc + +# Start the node +sudo systemctl start people-chain-rpc + +# Check status and wait for sync +sudo systemctl status people-chain-rpc +sudo journalctl -u people-chain-rpc -f +``` + +### Step 7: Verify Setup + +Use the same verification tests as in the Docker setup (see Step 5 above). + +--- + +## Monitoring and Maintenance + +### Log Management + +**Docker Setup**: + +```bash +# View node logs +docker logs -f people-chain-rpc +``` + +**Systemd Setup**: + +```bash +# View node logs +sudo journalctl -u people-chain-rpc -f + +# View recent logs +sudo journalctl -u people-chain-rpc -n 100 + +# Filter for errors +sudo journalctl -u people-chain-rpc | grep -i error +``` + +### Performance Monitoring + +Monitor key metrics: +- **Sync status**: Ensure node stays fully synced +- **Peer connections**: Maintain 30+ peers for good connectivity +- **Resource usage**: Monitor CPU, RAM, and disk I/O +- **RPC request latency**: Track response times for the Polkadot SDK API +- **Connection count**: Monitor active RPC connections + +**Prometheus Metrics**: + +Metrics are available at `http://localhost:9615/metrics` + +Example Prometheus scrape configuration: + +```yaml +scrape_configs: + - job_name: 'people-chain-rpc' + static_configs: + - targets: ['localhost:9615'] +``` + +**Key Metrics to Monitor**: +- `substrate_block_height`: Current block height +- `substrate_finalized_height`: Finalized block height +- `substrate_peers_count`: Number of connected peers +- `substrate_ready_transactions_number`: Transaction queue size + +### Database Maintenance + +Check database size periodically: + +```bash +# Docker setup +du -sh my-node-data + +# Systemd setup +du -sh /var/lib/people-chain-rpc +``` + +The node handles pruning automatically based on configuration unless running in archive mode. + +### Updates and Upgrades + +**Docker Setup**: + +```bash +# Pull latest image +docker pull parity/polkadot-omni-node:stable2506-4 + +# Restart container +docker stop people-chain-rpc +docker rm people-chain-rpc + +# Start new container (use same command from setup) +``` + +**Systemd Setup**: + +```bash +# Stop service +sudo systemctl stop people-chain-rpc + +# Backup data +sudo cp -r /var/lib/people-chain-rpc /var/lib/people-chain-rpc.backup + +# Update binary +cargo install --locked --force polkadot-omni-node@ + +# Restart service +sudo systemctl start people-chain-rpc +``` + +## Security Best Practices + +### Network Security + +1. **Firewall Configuration**: + - Only expose necessary ports + - Use UFW or iptables to restrict access + - Consider IP whitelisting for RPC endpoints + +2. **Reverse Proxy** (Recommended for Production): + - Use nginx or Caddy as reverse proxy + - Enable SSL/TLS (HTTPS/WSS) + - Implement authentication + - Add rate limiting + +Example nginx configuration: + +```nginx +upstream polkadot_sdk_rpc { + server 127.0.0.1:9944; +} + +server { + listen 443 ssl http2; + server_name your-domain.com; + + ssl_certificate /path/to/cert.pem; + ssl_certificate_key /path/to/key.pem; + + # Polkadot SDK RPC + location /polkadot { + proxy_pass http://polkadot_sdk_rpc; + proxy_http_version 1.1; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection "upgrade"; + + # Rate limiting + limit_req zone=rpc_limit burst=10; + } +} +``` + +### RPC Security + +- **Always use `--rpc-methods=safe`**: Prevents dangerous RPC calls +- **Restrict CORS**: Use specific domains instead of `all` in production +- **Set connection limits**: Prevent resource exhaustion +- **Monitor for abuse**: Track unusual patterns +- **Authentication**: Implement API keys or OAuth for production + +### System Security + +- Keep operating system updated +- Use dedicated user accounts (never root) +- Enable fail2ban for SSH protection +- Regular security audits +- Disable unnecessary services +- Use AppArmor or SELinux for additional isolation + +### Monitoring and Alerting + +Set up alerts for: +- Service failures +- Sync issues +- Low peer count (< 10 peers) +- High resource usage +- Unusual RPC traffic patterns +- Database errors + +## Conclusion + +Running an RPC node for system parachains provides critical infrastructure for accessing specialized Polkadot network services. By following this guide, you have set up a production-ready RPC node that: + +- Enables applications and users to interact with essential system parachain features (identity management, cross-chain bridges, or coretime allocation) +- Supports flexible deployment with both Docker and systemd options +- Implements comprehensive monitoring, security, and maintenance practices +- Can be easily adapted for any system parachain by substituting the appropriate chain specification + +Whether you're running a node for People Chain, Bridge Hub, or Coretime Chain, regular maintenance and monitoring will ensure your RPC node continues to provide reliable service. Stay updated with the latest releases and best practices to keep your infrastructure secure and performant. diff --git a/nodes-and-validators/run-a-validator/.nav.yml b/node-infrastructure/run-a-validator/.nav.yml similarity index 100% rename from nodes-and-validators/run-a-validator/.nav.yml rename to node-infrastructure/run-a-validator/.nav.yml diff --git a/nodes-and-validators/run-a-validator/onboarding-and-offboarding/.nav.yml b/node-infrastructure/run-a-validator/onboarding-and-offboarding/.nav.yml similarity index 100% rename from nodes-and-validators/run-a-validator/onboarding-and-offboarding/.nav.yml rename to node-infrastructure/run-a-validator/onboarding-and-offboarding/.nav.yml diff --git a/nodes-and-validators/run-a-validator/onboarding-and-offboarding/key-management.md b/node-infrastructure/run-a-validator/onboarding-and-offboarding/key-management.md similarity index 100% rename from nodes-and-validators/run-a-validator/onboarding-and-offboarding/key-management.md rename to node-infrastructure/run-a-validator/onboarding-and-offboarding/key-management.md diff --git a/nodes-and-validators/run-a-validator/onboarding-and-offboarding/set-up-validator.md b/node-infrastructure/run-a-validator/onboarding-and-offboarding/set-up-validator.md similarity index 100% rename from nodes-and-validators/run-a-validator/onboarding-and-offboarding/set-up-validator.md rename to node-infrastructure/run-a-validator/onboarding-and-offboarding/set-up-validator.md diff --git a/nodes-and-validators/run-a-validator/onboarding-and-offboarding/start-validating.md b/node-infrastructure/run-a-validator/onboarding-and-offboarding/start-validating.md similarity index 100% rename from nodes-and-validators/run-a-validator/onboarding-and-offboarding/start-validating.md rename to node-infrastructure/run-a-validator/onboarding-and-offboarding/start-validating.md diff --git a/nodes-and-validators/run-a-validator/onboarding-and-offboarding/stop-validating.md b/node-infrastructure/run-a-validator/onboarding-and-offboarding/stop-validating.md similarity index 100% rename from nodes-and-validators/run-a-validator/onboarding-and-offboarding/stop-validating.md rename to node-infrastructure/run-a-validator/onboarding-and-offboarding/stop-validating.md diff --git a/nodes-and-validators/run-a-validator/operational-tasks/.nav.yml b/node-infrastructure/run-a-validator/operational-tasks/.nav.yml similarity index 100% rename from nodes-and-validators/run-a-validator/operational-tasks/.nav.yml rename to node-infrastructure/run-a-validator/operational-tasks/.nav.yml diff --git a/nodes-and-validators/run-a-validator/operational-tasks/general-management.md b/node-infrastructure/run-a-validator/operational-tasks/general-management.md similarity index 100% rename from nodes-and-validators/run-a-validator/operational-tasks/general-management.md rename to node-infrastructure/run-a-validator/operational-tasks/general-management.md diff --git a/nodes-and-validators/run-a-validator/operational-tasks/pause-validating.md b/node-infrastructure/run-a-validator/operational-tasks/pause-validating.md similarity index 100% rename from nodes-and-validators/run-a-validator/operational-tasks/pause-validating.md rename to node-infrastructure/run-a-validator/operational-tasks/pause-validating.md diff --git a/nodes-and-validators/run-a-validator/operational-tasks/upgrade-your-node.md b/node-infrastructure/run-a-validator/operational-tasks/upgrade-your-node.md similarity index 100% rename from nodes-and-validators/run-a-validator/operational-tasks/upgrade-your-node.md rename to node-infrastructure/run-a-validator/operational-tasks/upgrade-your-node.md diff --git a/nodes-and-validators/run-a-validator/requirements.md b/node-infrastructure/run-a-validator/requirements.md similarity index 100% rename from nodes-and-validators/run-a-validator/requirements.md rename to node-infrastructure/run-a-validator/requirements.md diff --git a/nodes-and-validators/run-a-validator/staking-mechanics/.nav.yml b/node-infrastructure/run-a-validator/staking-mechanics/.nav.yml similarity index 100% rename from nodes-and-validators/run-a-validator/staking-mechanics/.nav.yml rename to node-infrastructure/run-a-validator/staking-mechanics/.nav.yml diff --git a/nodes-and-validators/run-a-validator/staking-mechanics/offenses-and-slashes.md b/node-infrastructure/run-a-validator/staking-mechanics/offenses-and-slashes.md similarity index 100% rename from nodes-and-validators/run-a-validator/staking-mechanics/offenses-and-slashes.md rename to node-infrastructure/run-a-validator/staking-mechanics/offenses-and-slashes.md diff --git a/nodes-and-validators/run-a-validator/staking-mechanics/rewards.md b/node-infrastructure/run-a-validator/staking-mechanics/rewards.md similarity index 100% rename from nodes-and-validators/run-a-validator/staking-mechanics/rewards.md rename to node-infrastructure/run-a-validator/staking-mechanics/rewards.md From cb3ac550933a2317744099cbda8be0d5327b776f Mon Sep 17 00:00:00 2001 From: Bruno Galvao Date: Wed, 19 Nov 2025 14:39:18 +0700 Subject: [PATCH 02/21] Update system parachain RPC guide to latest versions Update polkadot-omni-node from stable2506-4 to v1.20.2 and Rust from 1.86 to 1.91.1 in the system parachain RPC documentation. This ensures users are working with the latest stable releases. Changes: - Docker image: parity/polkadot-omni-node:v1.20.2 - Cargo package: polkadot-omni-node@0.11.0 - Rust toolchain: 1.91.1 All versions have been tested and verified to work correctly. --- .../run-a-node/system-parachain-rpc.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/node-infrastructure/run-a-node/system-parachain-rpc.md b/node-infrastructure/run-a-node/system-parachain-rpc.md index 82823a719..4cfb4b741 100644 --- a/node-infrastructure/run-a-node/system-parachain-rpc.md +++ b/node-infrastructure/run-a-node/system-parachain-rpc.md @@ -66,7 +66,7 @@ Required software: - **Operating System**: Ubuntu 22.04 LTS (recommended) or similar Linux distribution - **Docker**: Latest version installed and running (for Docker-based setup) - **rclone**: (Optional but recommended) Command-line program for managing files on cloud storage (https://rclone.org/downloads/) -- **Rust Toolchain**: Version 1.86 or as specified by runtime (for manual build) +- **Rust Toolchain**: Version 1.91.1 or as specified by runtime (for manual build) ## Setup Options @@ -173,7 +173,7 @@ docker run -d --name people-chain-rpc --restart unless-stopped \ -p 30333:30333 \ -v $(pwd)/people-polkadot.json:/people-polkadot.json \ -v $(pwd)/my-node-data:/data \ - parity/polkadot-omni-node:stable2506-4 \ + parity/polkadot-omni-node:v1.20.2 \ --name=PeopleChainRPC \ --base-path=/data \ --chain=/people-polkadot.json \ @@ -335,7 +335,7 @@ docker rm people-chain-rpc ```bash # Pull latest image -docker pull parity/polkadot-omni-node:stable2506-4 +docker pull parity/polkadot-omni-node:v1.20.2 # Stop and remove old container docker stop people-chain-rpc @@ -358,17 +358,17 @@ curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh source $HOME/.cargo/env # Install specific Rust version -rustup install 1.86 -rustup default 1.86 -rustup target add wasm32-unknown-unknown --toolchain 1.86 -rustup component add rust-src --toolchain 1.86 +rustup install 1.91.1 +rustup default 1.91.1 +rustup target add wasm32-unknown-unknown --toolchain 1.91.1 +rustup component add rust-src --toolchain 1.91.1 ``` ### Step 2: Install the Polkadot Omni Node ```bash # Install polkadot-omni-node -cargo install --locked polkadot-omni-node@0.7.0 +cargo install --locked polkadot-omni-node@0.11.0 # Verify installation polkadot-omni-node --version @@ -543,7 +543,7 @@ The node handles pruning automatically based on configuration unless running in ```bash # Pull latest image -docker pull parity/polkadot-omni-node:stable2506-4 +docker pull parity/polkadot-omni-node:v1.20.2 # Restart container docker stop people-chain-rpc From 9105f65d9c323275dcf3687909d219d8a713af43 Mon Sep 17 00:00:00 2001 From: Bruno Galvao Date: Wed, 19 Nov 2025 17:06:56 +0700 Subject: [PATCH 03/21] Update software versions and dependencies - Update Rust toolchain to 1.91.1 (from 1.86) - Update polkadot-omni-node to v0.11.0 for collator (from v0.5.0) - Update Docker image to v1.20.2 for RPC nodes (from stable2506-4) - Update chain-spec-builder to v14.0.0 (from v10.0.0) - Update AI-generated page files to reflect latest changes - Clean up old nodes-and-validators AI page files --- ...-infrastructure-run-a-collator-collator.md | 14 +- ...rastructure-run-a-node-polkadot-hub-rpc.md | 40 +- ...ucture-run-a-node-relay-chain-full-node.md | 2 +- ...ructure-run-a-node-system-parachain-rpc.md | 18 +- ...arding-and-offboarding-start-validating.md | 1 - ...odes-and-validators-run-a-node-bootnode.md | 130 ---- ...des-and-validators-run-a-node-full-node.md | 338 -------- ...es-and-validators-run-a-node-secure-wss.md | 125 --- ...boarding-and-offboarding-key-management.md | 155 ---- ...arding-and-offboarding-set-up-validator.md | 213 ------ ...arding-and-offboarding-start-validating.md | 254 ------ ...oarding-and-offboarding-stop-validating.md | 45 -- ...or-operational-tasks-general-management.md | 722 ------------------ ...ator-operational-tasks-pause-validating.md | 46 -- ...tor-operational-tasks-upgrade-your-node.md | 75 -- ...validators-run-a-validator-requirements.md | 85 --- ...-staking-mechanics-offenses-and-slashes.md | 171 ----- ...n-a-validator-staking-mechanics-rewards.md | 204 ----- .../run-a-collator/collator.md | 14 +- .../run-a-node/polkadot-hub-rpc.md | 40 +- 20 files changed, 74 insertions(+), 2618 deletions(-) delete mode 100644 .ai/pages/nodes-and-validators-run-a-node-bootnode.md delete mode 100644 .ai/pages/nodes-and-validators-run-a-node-full-node.md delete mode 100644 .ai/pages/nodes-and-validators-run-a-node-secure-wss.md delete mode 100644 .ai/pages/nodes-and-validators-run-a-validator-onboarding-and-offboarding-key-management.md delete mode 100644 .ai/pages/nodes-and-validators-run-a-validator-onboarding-and-offboarding-set-up-validator.md delete mode 100644 .ai/pages/nodes-and-validators-run-a-validator-onboarding-and-offboarding-start-validating.md delete mode 100644 .ai/pages/nodes-and-validators-run-a-validator-onboarding-and-offboarding-stop-validating.md delete mode 100644 .ai/pages/nodes-and-validators-run-a-validator-operational-tasks-general-management.md delete mode 100644 .ai/pages/nodes-and-validators-run-a-validator-operational-tasks-pause-validating.md delete mode 100644 .ai/pages/nodes-and-validators-run-a-validator-operational-tasks-upgrade-your-node.md delete mode 100644 .ai/pages/nodes-and-validators-run-a-validator-requirements.md delete mode 100644 .ai/pages/nodes-and-validators-run-a-validator-staking-mechanics-offenses-and-slashes.md delete mode 100644 .ai/pages/nodes-and-validators-run-a-validator-staking-mechanics-rewards.md diff --git a/.ai/pages/node-infrastructure-run-a-collator-collator.md b/.ai/pages/node-infrastructure-run-a-collator-collator.md index 18babcfa8..1f6f96cc6 100644 --- a/.ai/pages/node-infrastructure-run-a-collator-collator.md +++ b/.ai/pages/node-infrastructure-run-a-collator-collator.md @@ -63,7 +63,7 @@ Required software: - **Operating System**: Ubuntu 22.04 LTS (recommended) or similar Linux distribution - **Docker**: For running subkey utility -- **Rust Toolchain**: Version 1.86 or as specified by the runtime +- **Rust Toolchain**: Version 1.91.1 or as specified by the runtime - **Dependencies**: ```bash sudo apt update @@ -87,17 +87,17 @@ curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh source $HOME/.cargo/env # Install specific Rust version -rustup install 1.86 -rustup default 1.86 -rustup target add wasm32-unknown-unknown --toolchain 1.86 -rustup component add rust-src --toolchain 1.86 +rustup install 1.91.1 +rustup default 1.91.1 +rustup target add wasm32-unknown-unknown --toolchain 1.91.1 +rustup component add rust-src --toolchain 1.91.1 ``` ### Step 2: Install the Polkadot Omni Node ```bash # Install polkadot-omni-node -cargo install --locked polkadot-omni-node@0.5.0 +cargo install --locked polkadot-omni-node@0.11.0 # Verify installation polkadot-omni-node --version @@ -159,7 +159,7 @@ cd runtimes cargo build --release -p asset-hub-polkadot-runtime # Install chain-spec-builder -cargo install --locked staging-chain-spec-builder@10.0.0 +cargo install --locked staging-chain-spec-builder@14.0.0 # Generate chain spec chain-spec-builder create \ diff --git a/.ai/pages/node-infrastructure-run-a-node-polkadot-hub-rpc.md b/.ai/pages/node-infrastructure-run-a-node-polkadot-hub-rpc.md index 7282f2ef0..ed95abd5d 100644 --- a/.ai/pages/node-infrastructure-run-a-node-polkadot-hub-rpc.md +++ b/.ai/pages/node-infrastructure-run-a-node-polkadot-hub-rpc.md @@ -55,7 +55,7 @@ Required software: - **Operating System**: Ubuntu 22.04 LTS (recommended) or similar Linux distribution - **Docker**: Latest version installed and running (for Docker-based setup) - **rclone**: (Optional but recommended) Command-line program for managing files on cloud storage (https://rclone.org/downloads/) -- **Rust Toolchain**: Version 1.86 or as specified by runtime (for manual build) +- **Rust Toolchain**: Version 1.91.1 or later (for manual build) ## Setup Options @@ -160,7 +160,7 @@ docker run -d --name polkadot-hub-rpc --restart unless-stopped \ -p 30333:30333 \ -v $(pwd)/asset-hub-polkadot.json:/asset-hub-polkadot.json \ -v $(pwd)/my-node-data:/data \ - parity/polkadot-omni-node:stable2506-4 \ + parity/polkadot-omni-node:v1.20.2 \ --name=PolkadotHubRPC \ --base-path=/data \ --chain=/asset-hub-polkadot.json \ @@ -306,7 +306,7 @@ docker rm polkadot-hub-rpc ```bash # Pull latest image -docker pull parity/polkadot-omni-node:stable2506-4 +docker pull parity/polkadot-omni-node:v1.20.2 # Stop and remove old container docker stop polkadot-hub-rpc @@ -329,23 +329,33 @@ curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh source $HOME/.cargo/env # Install specific Rust version -rustup install 1.86 -rustup default 1.86 -rustup target add wasm32-unknown-unknown --toolchain 1.86 -rustup component add rust-src --toolchain 1.86 +rustup install 1.91.1 +rustup default 1.91.1 +rustup target add wasm32-unknown-unknown --toolchain 1.91.1 +rustup component add rust-src --toolchain 1.91.1 ``` -### Step 2: Install the Polkadot Omni Node +### Step 2: Install Required Dependencies + +```bash +# Install system dependencies +sudo apt update +sudo apt install -y build-essential git clang curl libssl-dev llvm libudev-dev make protobuf-compiler +``` + +### Step 3: Install the Polkadot Omni Node ```bash # Install polkadot-omni-node -cargo install --locked polkadot-omni-node@0.7.0 +cargo install --locked polkadot-omni-node@0.11.0 # Verify installation polkadot-omni-node --version ``` -### Step 3: Obtain Chain Specification +**Note**: Compiling polkadot-omni-node from source requires significant RAM (minimum 24GB recommended). The compilation may take 10-15 minutes on systems with adequate resources. + +### Step 4: Obtain Chain Specification Download the Polkadot Hub chain specification: @@ -353,7 +363,7 @@ Download the Polkadot Hub chain specification: curl -L https://raw.githubusercontent.com/paritytech/polkadot-sdk/master/cumulus/parachains/chain-specs/asset-hub-polkadot.json -o asset-hub-polkadot.json ``` -### Step 4: Create User and Directory Structure +### Step 5: Create User and Directory Structure ```bash # Create dedicated user @@ -369,7 +379,7 @@ sudo cp asset-hub-polkadot.json /var/lib/polkadot-hub-rpc/ sudo chown -R polkadot:polkadot /var/lib/polkadot-hub-rpc ``` -### Step 5: Create Systemd Service for Polkadot SDK Node +### Step 6: Create Systemd Service for Polkadot SDK Node Create a service file for the Polkadot SDK RPC node: @@ -419,7 +429,7 @@ LimitNOFILE=65536 WantedBy=multi-user.target ``` -### Step 6: Start Service +### Step 7: Start Service ```bash # Reload systemd @@ -436,7 +446,7 @@ sudo systemctl status polkadot-hub-rpc sudo journalctl -u polkadot-hub-rpc -f ``` -### Step 7: Verify Setup +### Step 8: Verify Setup Use the same verification tests as in the Docker setup (see Step 5 above). @@ -516,7 +526,7 @@ The node handles pruning automatically based on configuration unless running in ```bash # Pull latest image -docker pull parity/polkadot-omni-node:stable2506-4 +docker pull parity/polkadot-omni-node:v1.20.2 # Restart container docker stop polkadot-hub-rpc diff --git a/.ai/pages/node-infrastructure-run-a-node-relay-chain-full-node.md b/.ai/pages/node-infrastructure-run-a-node-relay-chain-full-node.md index 17f93a8f1..05c3394fc 100644 --- a/.ai/pages/node-infrastructure-run-a-node-relay-chain-full-node.md +++ b/.ai/pages/node-infrastructure-run-a-node-relay-chain-full-node.md @@ -23,7 +23,7 @@ Now that you're familiar with the different types of nodes, this section will wa Before getting started, ensure the following prerequisites are met: -- Ensure [Rust](https://www.rust-lang.org/tools/install){target=\_blank} is installed on your operating system. +- Ensure [Rust](https://rust-lang.org/tools/install/){target=\_blank} is installed on your operating system. - [Install the necessary dependencies for the Polkadot SDK](/parachains/install-polkadot-sdk/){target=\_blank}. !!! warning diff --git a/.ai/pages/node-infrastructure-run-a-node-system-parachain-rpc.md b/.ai/pages/node-infrastructure-run-a-node-system-parachain-rpc.md index c5d5b13cc..9d3fc5735 100644 --- a/.ai/pages/node-infrastructure-run-a-node-system-parachain-rpc.md +++ b/.ai/pages/node-infrastructure-run-a-node-system-parachain-rpc.md @@ -67,7 +67,7 @@ Required software: - **Operating System**: Ubuntu 22.04 LTS (recommended) or similar Linux distribution - **Docker**: Latest version installed and running (for Docker-based setup) - **rclone**: (Optional but recommended) Command-line program for managing files on cloud storage (https://rclone.org/downloads/) -- **Rust Toolchain**: Version 1.86 or as specified by runtime (for manual build) +- **Rust Toolchain**: Version 1.91.1 or as specified by runtime (for manual build) ## Setup Options @@ -174,7 +174,7 @@ docker run -d --name people-chain-rpc --restart unless-stopped \ -p 30333:30333 \ -v $(pwd)/people-polkadot.json:/people-polkadot.json \ -v $(pwd)/my-node-data:/data \ - parity/polkadot-omni-node:stable2506-4 \ + parity/polkadot-omni-node:v1.20.2 \ --name=PeopleChainRPC \ --base-path=/data \ --chain=/people-polkadot.json \ @@ -336,7 +336,7 @@ docker rm people-chain-rpc ```bash # Pull latest image -docker pull parity/polkadot-omni-node:stable2506-4 +docker pull parity/polkadot-omni-node:v1.20.2 # Stop and remove old container docker stop people-chain-rpc @@ -359,17 +359,17 @@ curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh source $HOME/.cargo/env # Install specific Rust version -rustup install 1.86 -rustup default 1.86 -rustup target add wasm32-unknown-unknown --toolchain 1.86 -rustup component add rust-src --toolchain 1.86 +rustup install 1.91.1 +rustup default 1.91.1 +rustup target add wasm32-unknown-unknown --toolchain 1.91.1 +rustup component add rust-src --toolchain 1.91.1 ``` ### Step 2: Install the Polkadot Omni Node ```bash # Install polkadot-omni-node -cargo install --locked polkadot-omni-node@0.7.0 +cargo install --locked polkadot-omni-node@0.11.0 # Verify installation polkadot-omni-node --version @@ -546,7 +546,7 @@ The node handles pruning automatically based on configuration unless running in ```bash # Pull latest image -docker pull parity/polkadot-omni-node:stable2506-4 +docker pull parity/polkadot-omni-node:v1.20.2 # Restart container docker stop people-chain-rpc diff --git a/.ai/pages/node-infrastructure-run-a-validator-onboarding-and-offboarding-start-validating.md b/.ai/pages/node-infrastructure-run-a-validator-onboarding-and-offboarding-start-validating.md index 3065d19f1..0ca3cc0fc 100644 --- a/.ai/pages/node-infrastructure-run-a-validator-onboarding-and-offboarding-start-validating.md +++ b/.ai/pages/node-infrastructure-run-a-validator-onboarding-and-offboarding-start-validating.md @@ -149,7 +149,6 @@ Follow these steps to use Polkadot.js Apps to activate your validator: ![](/images/nodes-and-validators/run-a-validator/onboarding-and-offboarding/start-validating/start-validating-05.webp) - ### Monitor Validation Status and Slots On the [**Staking**](https://polkadot.js.org/apps/#/staking){target=\_blank} tab in Polkadot.js Apps, you can see your validator's status, the number of available validator slots, and the nodes that have signaled their intent to validate. Your node may initially appear in the waiting queue, especially if the validator slots are full. The following is an example view of the **Staking** tab: diff --git a/.ai/pages/nodes-and-validators-run-a-node-bootnode.md b/.ai/pages/nodes-and-validators-run-a-node-bootnode.md deleted file mode 100644 index 33f135611..000000000 --- a/.ai/pages/nodes-and-validators-run-a-node-bootnode.md +++ /dev/null @@ -1,130 +0,0 @@ ---- -title: Set Up a Bootnode -description: Learn how to configure and run a bootnode for Polkadot, including P2P, WS, and secure WSS connections with network key management and proxies. -categories: Infrastructure -url: https://docs.polkadot.com/nodes-and-validators/run-a-node/bootnode/ ---- - -# Set Up a Bootnode - -## Introduction - -Bootnodes are essential for helping blockchain nodes discover peers and join the network. When a node starts, it needs to find other nodes, and bootnodes provide an initial point of contact. Once connected, a node can expand its peer connections and play its role in the network, like participating as a validator. - -This guide will walk you through setting up a Polkadot bootnode, configuring P2P, WebSocket (WS), secure WSS connections, and managing network keys. You'll also learn how to test your bootnode to ensure it is running correctly and accessible to other nodes. - -## Prerequisites - -Before you start, you need to have the following prerequisites: - -- Verify a working Polkadot (`polkadot`) binary is available on your machine. -- Ensure you have nginx installed. Please refer to the [Installation Guide](https://nginx.org/en/docs/install.html){target=\_blank} for help with installation if needed. -- A VPS or other dedicated server setup. - -## Accessing the Bootnode - -Bootnodes must be accessible through three key channels to connect with other nodes in the network: - -- **P2P**: A direct peer-to-peer connection, set by. - - ```bash - - --listen-addr /ip4/0.0.0.0/tcp/INSERT_PORT - - ``` - - This is not enabled by default on non-validator nodes like archive RPC nodes. - -- **P2P/WS**: A WebSocket (WS) connection, also configured via `--listen-addr`. -- **P2P/WSS**: A secure WebSocket (WSS) connection using SSL, often required for light clients. An SSL proxy is needed, as the node itself cannot handle certificates. - -## Node Key - -A node key is the ED25519 key used by `libp2p` to assign your node an identity or peer ID. Generating a known node key for a bootnode is crucial, as it gives you a consistent key that can be placed in chain specifications as a known, reliable bootnode. - -Starting a node creates its node key in the `chains/INSERT_CHAIN/network/secret_ed25519` file. - -You can create a node key using: - - ``` bash - polkadot key generate-node-key - ``` - -This key can be used in the startup command line. - -It is imperative that you backup the node key. If it is included in the `polkadot` binary, it is hardcoded into the binary, which must be recompiled to change the key. - -## Running the Bootnode - -A bootnode can be run as follows: - - ``` bash - polkadot --chain polkadot \ - --name dot-bootnode \ - --listen-addr /ip4/0.0.0.0/tcp/30310 \ - --listen-addr /ip4/0.0.0.0/tcp/30311/ws - ``` - -This assigns the p2p to port 30310 and p2p/ws to port 30311. For the p2p/wss port, a proxy must be set up with a DNS name and a corresponding certificate. The following example is for the popular nginx server and enables p2p/wss on port 30312 by adding a proxy to the p2p/ws port 30311: - -``` conf title="/etc/nginx/sites-enabled/dot-bootnode" -server { - listen 30312 ssl http2 default_server; - server_name dot-bootnode.stakeworld.io; - root /var/www/html; - - ssl_certificate "INSERT_YOUR_CERT"; - ssl_certificate_key "INSERT_YOUR_KEY"; - - location / { - proxy_buffers 16 4k; - proxy_buffer_size 2k; - proxy_pass http://localhost:30311; - proxy_http_version 1.1; - proxy_set_header Upgrade $http_upgrade; - proxy_set_header Connection "Upgrade"; - proxy_set_header Host $host; - } - -} -``` - -## Testing Bootnode Connection - -If the preceding node is running with DNS name `dot-bootnode.stakeworld.io`, which contains a proxy with a valid certificate and node-id `12D3KooWAb5MyC1UJiEQJk4Hg4B2Vi3AJdqSUhTGYUqSnEqCFMFg` then the following commands should output `syncing 1 peers`. - -!!!tip - You can add `-lsub-libp2p=trace` on the end to get libp2p trace logging for debugging purposes. - -### P2P - -```bash -polkadot --chain polkadot \ ---base-path /tmp/node \ ---name "Bootnode testnode" \ ---reserved-only \ ---reserved-nodes "/dns/dot-bootnode.stakeworld.io/tcp/30310/p2p/12D3KooWAb5MyC1UJiEQJk4Hg4B2Vi3AJdqSUhTGYUqSnEqCFMFg" \ ---no-hardware-benchmarks -``` - -### P2P/WS - -```bash -polkadot --chain polkadot \ ---base-path /tmp/node \ ---name "Bootnode testnode" \ ---reserved-only \ ---reserved-nodes "/dns/dot-bootnode.stakeworld.io/tcp/30311/ws/p2p/12D3KooWAb5MyC1UJiEQJk4Hg4B2Vi3AJdqSUhTGYUqSnEqCFMFg" \ ---no-hardware-benchmarks -``` - -### P2P/WSS - -```bash -polkadot --chain polkadot \ ---base-path /tmp/node \ ---name "Bootnode testnode" \ ---reserved-only \ ---reserved-nodes "/dns/dot-bootnode.stakeworld.io/tcp/30312/wss/p2p/12D3KooWAb5MyC1UJiEQJk4Hg4B2Vi3AJdqSUhTGYUqSnEqCFMFg" \ ---no-hardware-benchmarks -``` diff --git a/.ai/pages/nodes-and-validators-run-a-node-full-node.md b/.ai/pages/nodes-and-validators-run-a-node-full-node.md deleted file mode 100644 index 8ba35da3c..000000000 --- a/.ai/pages/nodes-and-validators-run-a-node-full-node.md +++ /dev/null @@ -1,338 +0,0 @@ ---- -title: Set Up a Node -description: Learn how to install, configure, and run Polkadot nodes, including setting up different node types and connecting to the network. -categories: Infrastructure -url: https://docs.polkadot.com/nodes-and-validators/run-a-node/full-node/ ---- - -# Set Up a Node - -## Introduction - -Running a node on Polkadot provides direct interaction with the network, enhanced privacy, and full control over RPC requests, transactions, and data queries. As the backbone of the network, nodes ensure decentralized data propagation, transaction validation, and seamless communication across the ecosystem. - -Polkadot supports multiple node types, including pruned, archive, and light nodes, each suited to specific use cases. During setup, you can use configuration flags to choose the node type you wish to run. - -This guide walks you through configuring, securing, and maintaining a node on Polkadot or any Polkadot SDK-based chain. It covers instructions for the different node types and how to safely expose your node's RPC server for external access. Whether you're building a local development environment, powering dApps, or supporting network decentralization, this guide provides all the essentials. - -## Set Up a Node - -Now that you're familiar with the different types of nodes, this section will walk you through configuring, securing, and maintaining a node on Polkadot or any Polkadot SDK-based chain. - -### Prerequisites - -Before getting started, ensure the following prerequisites are met: - -- Ensure [Rust](https://rust-lang.org/tools/install/){target=\_blank} is installed on your operating system. -- [Install the necessary dependencies for the Polkadot SDK](/parachains/install-polkadot-sdk/){target=\_blank}. - -!!! warning - This setup is not recommended for validators. If you plan to run a validator, refer to the [Running a Validator](/nodes-and-validators/run-a-validator/onboarding-and-offboarding/set-up-validator/){target=\_blank} guide for proper instructions. - -### Install and Build the Polkadot Binary - -This section will walk you through installing and building the Polkadot binary for different operating systems and methods. - -??? interface "macOS" - - To get started, update and configure the Rust toolchain by running the following commands: - - ```bash - source ~/.cargo/env - - rustup default stable - rustup update - - rustup update nightly - rustup target add wasm32-unknown-unknown --toolchain nightly - rustup component add rust-src --toolchain stable-aarch64-apple-darwin - ``` - - You can verify your installation by running: - - ```bash - rustup show - rustup +nightly show - ``` - - You should see output similar to the following: - -
- rustup show
- rustup +nightly show - active toolchain - ---------------- - - stable-aarch64-apple-darwin (default) - rustc 1.82.0 (f6e511eec 2024-10-15) - - active toolchain - ---------------- - - nightly-aarch64-apple-darwin (overridden by +toolchain on the command line) - rustc 1.84.0-nightly (03ee48451 2024-11-18) - -
- - Then, run the following commands to clone and build the Polkadot binary: - - ```bash - git clone https://github.com/paritytech/polkadot-sdk polkadot-sdk - cd polkadot-sdk - cargo build --release - ``` - - Depending upon the specs of your machine, compiling the binary may take an hour or more. After building the Polkadot node from source, the executable binary will be located in the `./target/release/polkadot` directory. - -??? interface "Windows" - - To get started, make sure that you have [WSL and Ubuntu](https://learn.microsoft.com/en-us/windows/wsl/install){target=\_blank} installed on your Windows machine. - - Once installed, you have a couple options for installing the Polkadot binary: - - - If Rust is installed, then `cargo` can be used similar to the macOS instructions. - - Or, the instructions in the Linux section can be used. - -??? interface "Linux (pre-built binary)" - - To grab the [latest release of the Polkadot binary](https://github.com/paritytech/polkadot-sdk/releases){target=\_blank}, you can use `wget`: - - ```bash - wget https://github.com/paritytech/polkadot-sdk/releases/download/polkadot-INSERT_VERSION/polkadot - ``` - - Ensure you note the executable binary's location, as you'll need to use it when running the start-up command. If you prefer, you can specify the output location of the executable binary with the `-O` flag, for example: - - ```bash - wget https://github.com/paritytech/polkadot-sdk/releases/download/polkadot-INSERT_VERSION/polkadot \ - - O /var/lib/polkadot-data/polkadot - ``` - - !!!tip - The nature of pre-built binaries means that they may not work on your particular architecture or Linux distribution. If you see an error like `cannot execute binary file: Exec format error` it likely means the binary is incompatible with your system. You will either need to compile the binary or use [Docker](#use-docker). - - Ensure that you properly configure the permissions to make the Polkadot release binary executable: - - ```bash - sudo chmod +x polkadot - ``` - -??? interface "Linux (compile binary)" - - The most reliable (although perhaps not the fastest) way of launching a full node is to compile the binary yourself. Depending on your machine's specs, this may take an hour or more. - - To get started, run the following commands to configure the Rust toolchain: - - ```bash - rustup default stable - rustup update - rustup update nightly - rustup target add wasm32-unknown-unknown --toolchain nightly - rustup target add wasm32-unknown-unknown --toolchain stable-x86_64-unknown-linux-gnu - rustup component add rust-src --toolchain stable-x86_64-unknown-linux-gnu - ``` - - You can verify your installation by running: - - ```bash - rustup show - ``` - - You should see output similar to the following: - -
- rustup show
- rustup +nightly show - active toolchain - ---------------- - - stable-x86_64-unknown-linux-gnu (default) - rustc 1.82.0 (f6e511eec 2024-10-15) -
- - Once Rust is configured, run the following commands to clone and build Polkadot: - - ```bash - git clone https://github.com/paritytech/polkadot-sdk polkadot-sdk - cd polkadot-sdk - cargo build --release - ``` - - Compiling the binary may take an hour or more, depending on your machine's specs. After building the Polkadot node from the source, the executable binary will be located in the `./target/release/polkadot` directory. - -??? interface "Linux (snap package)" - - Polkadot can be installed as a [snap package](https://snapcraft.io/polkadot){target=\_blank}. If you don't already have Snap installed, take the following steps to install it: - - ```bash - sudo apt update - sudo apt install snapd - ``` - - Install the Polkadot snap package: - - ```bash - sudo snap install polkadot - ``` - - Before continuing on with the following instructions, check out the [Configure and Run Your Node](#configure-and-run-your-node) section to learn more about the configuration options. - - To configure your Polkadot node with your desired options, you'll run a command similar to the following: - - ```bash - sudo snap set polkadot service-args="--name=MyName --chain=polkadot" - ``` - - Then to start the node service, run: - - ```bash - sudo snap start polkadot - ``` - - You can review the logs to check on the status of the node: - - ```bash - snap logs polkadot -f - ``` - - And at any time, you can stop the node service: - - ```bash - sudo snap stop polkadot - ``` - - You can optionally prevent the service from stopping when snap is updated with the following command: - - ```bash - sudo snap set polkadot endure=true - ``` - -### Use Docker - -As an additional option, you can use Docker to run your node in a container. Doing this is more advanced, so it's best left up to those already familiar with Docker or who have completed the other set-up instructions in this guide. You can review the latest versions on [DockerHub](https://hub.docker.com/r/parity/polkadot/tags){target=\_blank}. - -Be aware that when you run Polkadot in Docker, the process only listens on `localhost` by default. If you would like to connect to your node's services (RPC and Prometheus) you need to ensure that you run the node with the `--rpc-external`, and `--prometheus-external` commands. - -```bash -docker run -p 9944:9944 -p 9615:9615 parity/polkadot:v1.16.2 --name "my-polkadot-node-calling-home" --rpc-external --prometheus-external -``` - -If you're running Docker on an Apple Silicon machine (e.g. M4), you'll need to adapt the command slightly: - -```bash -docker run --platform linux/amd64 -p 9944:9944 -p 9615:9615 parity/polkadot:v1.16.2 --name "kearsarge-calling-home" --rpc-external --prometheus-external -``` - -## Configure and Run Your Node - -Now that you've installed and built the Polkadot binary, the next step is to configure the start-up command depending on the type of node that you want to run. You'll need to modify the start-up command accordingly based on the location of the binary. In some cases, it may be located within the `./target/release/` folder, so you'll need to replace polkadot with `./target/release/polkadot` in the following commands. - -Also, note that you can use the same binary for Polkadot as you would for Kusama or any other relay chain. You'll need to use the `--chain` flag to differentiate between chains. - -The base commands for running a Polkadot node are as follows: - -=== "Default pruned node" - - This uses the default pruning value of the last 256 blocks: - - ```bash - polkadot --chain polkadot \ - --name "INSERT_NODE_NAME" - ``` - -=== "Custom pruned node" - - You can customize the pruning value, for example, to the last 1000 finalized blocks: - - ```bash - polkadot --chain polkadot \ - --name INSERT_YOUR_NODE_NAME \ - --state-pruning 1000 \ - --blocks-pruning archive \ - --rpc-cors all \ - --rpc-methods safe - ``` - -=== "Archive node" - - To support the full state, use the `archive` option: - - ```bash - polkadot --chain polkadot \ - --name INSERT_YOUR_NODE_NAME \ - --state-pruning archive \ - --blocks-pruning archive \ - ``` - -If you want to run an RPC node, please refer to the following [RPC Configurations](#rpc-configurations) section. - -To review a complete list of the available commands, flags, and options, you can use the `--help` flag: - -```bash -polkadot --help -``` - -Once you've fully configured your start-up command, you can execute it in your terminal and your node will start [syncing](#sync-your-node). - -### RPC Configurations - -The node startup settings allow you to choose what to expose, how many connections to expose, and which systems should be granted access through the RPC server. - -- You can limit the methods to use with `--rpc-methods`; an easy way to set this to a safe mode is `--rpc-methods safe`. -- You can set your maximum connections through `--rpc-max-connections`, for example, `--rpc-max-connections 200`. -- By default, localhost and Polkadot.js can access the RPC server. You can change this by setting `--rpc-cors`. To allow access from everywhere, you can use `--rpc-cors all`. - -For a list of important flags when running RPC nodes, refer to the Parity DevOps documentation: [Important Flags for Running an RPC Node](https://paritytech.github.io/devops-guide/guides/rpc_index.html?#important-flags-for-running-an-rpc-node){target=\_blank}. - -## Sync Your Node - -The syncing process will take a while, depending on your capacity, processing power, disk speed, and RAM. The process may be completed on a $10 DigitalOcean droplet in about ~36 hours. While syncing, your node name should be visible in gray on Polkadot Telemetry, and once it is fully synced, your node name will appear in white on [Polkadot Telemetry](https://telemetry.polkadot.io/#list/Polkadot){target=_blank}. - -A healthy node syncing blocks will output logs like the following: - -
- 2024-11-19 23:49:57 Parity Polkadot - 2024-11-19 23:49:57 ✌️ version 1.14.1-7c4cd60da6d - 2024-11-19 23:49:57 ❤️ by Parity Technologies <admin@parity.io>, 2017-2024 - 2024-11-19 23:49:57 📋 Chain specification: Polkadot - 2024-11-19 23:49:57 🏷 Node name: myPolkadotNode - 2024-11-19 23:49:57 👤 Role: FULL - 2024-11-19 23:49:57 💾 Database: RocksDb at /home/ubuntu/.local/share/polkadot/chains/polkadot/db/full - 2024-11-19 23:50:00 🏷 Local node identity is: 12D3KooWDmhHEgPRJUJnUpJ4TFWn28EENqvKWH4dZGCN9TS51y9h - 2024-11-19 23:50:00 Running libp2p network backend - 2024-11-19 23:50:00 💻 Operating system: linux - 2024-11-19 23:50:00 💻 CPU architecture: x86_64 - 2024-11-19 23:50:00 💻 Target environment: gnu - 2024-11-19 23:50:00 💻 CPU: Intel(R) Xeon(R) CPU E3-1245 V2 @ 3.40GHz - 2024-11-19 23:50:00 💻 CPU cores: 4 - 2024-11-19 23:50:00 💻 Memory: 32001MB - 2024-11-19 23:50:00 💻 Kernel: 5.15.0-113-generic - 2024-11-19 23:50:00 💻 Linux distribution: Ubuntu 22.04.5 LTS - 2024-11-19 23:50:00 💻 Virtual machine: no - 2024-11-19 23:50:00 📦 Highest known block at #9319 - 2024-11-19 23:50:00 〽️ Prometheus exporter started at 127.0.0.1:9615 - 2024-11-19 23:50:00 Running JSON-RPC server: addr=127.0.0.1:9944, allowed origins=["http://localhost:*", "http://127.0.0.1:*", "https://localhost:*", "https://127.0.0.1:*", "https://polkadot.js.org"] - 2024-11-19 23:50:00 🏁 CPU score: 671.67 MiBs - 2024-11-19 23:50:00 🏁 Memory score: 7.96 GiBs - 2024-11-19 23:50:00 🏁 Disk score (seq. writes): 377.87 MiBs - 2024-11-19 23:50:00 🏁 Disk score (rand. writes): 147.92 MiBs - 2024-11-19 23:50:00 🥩 BEEFY gadget waiting for BEEFY pallet to become available... - 2024-11-19 23:50:00 🔍 Discovered new external address for our node: /ip4/37.187.93.17/tcp/30333/ws/p2p/12D3KooWDmhHEgPRJUJnUpJ4TFWn28EENqvKWH4dZGCN9TS51y9h - 2024-11-19 23:50:01 🔍 Discovered new external address for our node: /ip6/2001:41d0:a:3511::1/tcp/30333/ws/p2p/12D3KooWDmhHEgPRJUJnUpJ4TFWn28EENqvKWH4dZGCN9TS51y9h - 2024-11-19 23:50:05 ⚙️ Syncing, target=#23486325 (5 peers), best: #12262 (0x8fb5…f310), finalized #11776 (0x9de1…32fb), ⬇ 430.5kiB/s ⬆ 17.8kiB/s - 2024-11-19 23:50:10 ⚙️ Syncing 628.8 bps, target=#23486326 (6 peers), best: #15406 (0x9ce1…2d76), finalized #15360 (0x0e41…a064), ⬇ 255.0kiB/s ⬆ 1.8kiB/s -
- -Congratulations, you're now syncing a Polkadot full node! Remember that the process is identical when using any other Polkadot SDK-based chain, although individual chains may have chain-specific flag requirements. - -### Connect to Your Node - -Open [Polkadot.js Apps](https://polkadot.js.org/apps/?rpc=ws%3A%2F%2F127.0.0.1%3A9944#/explorer){target=\_blank} and click the logo in the top left to switch the node. Activate the **Development** toggle and input your node's domain or IP address. The default WSS endpoint for a local node is: - -```bash -ws://127.0.0.1:9944 -``` diff --git a/.ai/pages/nodes-and-validators-run-a-node-secure-wss.md b/.ai/pages/nodes-and-validators-run-a-node-secure-wss.md deleted file mode 100644 index 27810abc3..000000000 --- a/.ai/pages/nodes-and-validators-run-a-node-secure-wss.md +++ /dev/null @@ -1,125 +0,0 @@ ---- -title: Set Up Secure WebSocket -description: Instructions on enabling SSL for your node and setting up a secure WebSocket proxy server using nginx for remote connections. -categories: Infrastructure -url: https://docs.polkadot.com/nodes-and-validators/run-a-node/secure-wss/ ---- - -# Set Up Secure WebSocket - -## Introduction - -Ensuring secure WebSocket communication is crucial for maintaining the integrity and security of a Polkadot or Kusama node when interacting with remote clients. This guide walks you through setting up a secure WebSocket (WSS) connection for your node by leveraging SSL encryption with popular web server proxies like nginx or Apache. - -By the end of this guide, you'll be able to secure your node's WebSocket port, enabling safe remote connections without exposing your node to unnecessary risks. The instructions in this guide are for UNIX-based systems. - -## Secure a WebSocket Port - -You can convert a non-secured WebSocket port to a secure WSS port by placing it behind an SSL-enabled proxy. This approach can be used to secure a bootnode or RPC server. The SSL-enabled apache2/nginx/other proxy server redirects requests to the internal WebSocket and converts it to a secure (WSS) connection. You can use a service like [LetsEncrypt](https://letsencrypt.org/){target=\_blank} to obtain an SSL certificate. - -### Obtain an SSL Certificate - -LetsEncrypt suggests using the [Certbot ACME client](https://letsencrypt.org/getting-started/#with-shell-access/){target=\_blank} for your respective web server implementation to get a free SSL certificate: - -- [nginx](https://certbot.eff.org/instructions?ws=nginx&os=ubuntufocal){target=\_blank} -- [apache2](https://certbot.eff.org/instructions?ws=apache&os=ubuntufocal){target=\_blank} - -LetsEncrypt will auto-generate an SSL certificate and include it in your configuration. - -When connecting, you can generate a self-signed certificate and rely on your node's raw IP address. However, self-signed certificates aren't optimal because you must include the certificate in an allowlist to access it from a browser. - -Use the following command to generate a self-signed certificate using OpenSSL: - -```bash -sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/ssl/private/selfsigned.key -out /etc/ssl/certs/selfsigned.crt -sudo openssl dhparam -out /etc/ssl/certs/dhparam.pem 2048 -``` -## Install a Proxy Server - -There are a lot of different implementations of a WebSocket proxy; some of the more widely used are [nginx](https://www.f5.com/go/product/welcome-to-nginx){target=\_blank} and [apache2](https://httpd.apache.org/){target=\_blank}, both of which are commonly used web server implementations. See the following section for configuration examples for both implementations. - -### Use nginx - -1. Install the `nginx` web server: - ```bash - apt install nginx - ``` - -2. In an SSL-enabled virtual host, add: - ```conf - server { - (...) - location / { - proxy_buffers 16 4k; - proxy_buffer_size 2k; - proxy_pass http://localhost:9944; - proxy_http_version 1.1; - proxy_set_header Upgrade $http_upgrade; - proxy_set_header Connection "Upgrade"; - proxy_set_header Host $host; - } - } - ``` -3. Optionally, you can introduce some form of rate limiting: - ```conf - http { - limit_req_zone "$http_x_forwarded_for" zone=zone:10m rate=2r/s; - (...) - } - location / { - limit_req zone=zone burst=5; - (...) - } - ``` -### Use Apache2 - -Apache2 can run in various modes, including `prefork`, `worker`, and `event`. In this example, the [`event`](https://httpd.apache.org/docs/2.4/mod/event.html){target=\_blank} mode is recommended for handling higher traffic loads, as it is optimized for performance in such environments. However, depending on the specific requirements of your setup, other modes like `prefork` or `worker` may also be appropriate. - -1. Install the `apache2` web server: - ```bash - apt install apache2 - a2dismod mpm_prefork - a2enmod mpm_event proxy proxy_html proxy_http proxy_wstunnel rewrite ssl - ``` -2. The [`mod_proxy_wstunnel`](https://httpd.apache.org/docs/2.4/mod/mod_proxy_wstunnel.html){target=\_blank} provides support for the tunneling of WebSocket connections to a backend WebSocket server. The connection is automatically upgraded to a WebSocket connection. In an SSL-enabled virtual host add: - - ```apacheconf - # (...) - SSLProxyEngine on - ProxyRequests off - ProxyPass / ws://localhost:9944 - ProxyPassReverse / ws://localhost:9944 - ``` - !!!warning - Older versions of `mod_proxy_wstunnel` don't upgrade the connection automatically and will need the following config added: - ```apacheconf - RewriteEngine on - RewriteCond %{HTTP:Upgrade} websocket [NC] - RewriteRule /(.*) ws://localhost:9944/$1 [P,L] - RewriteRule /(.*) http://localhost:9944/$1 [P,L] - ``` - -3. Optionally, some form of rate limiting can be introduced by first running the following command: - - ```bash - apt install libapache2-mod-qos - a2enmod qos - ``` - - Then edit `/etc/apache2/mods-available/qos.conf` as follows: - - ```conf - # allows max 50 connections from a single IP address: - QS_SrvMaxConnPerIP 50 - ``` - -## Connect to the Node - -1. Open [Polkadot.js Apps interface](https://polkadot.js.org/apps){target=\_blank} and click the logo in the top left to switch the node. -2. Activate the **Development** toggle and input either your node's domain or IP address. Remember to prefix with `wss://` and, if you're using the 443 port, append `:443` as follows: - - ```bash - wss://example.com:443 - ``` - -![A sync-in-progress chain connected to Polkadot.js UI](/images/nodes-and-validators/run-a-node/secure-wss/secure-wss-01.webp) diff --git a/.ai/pages/nodes-and-validators-run-a-validator-onboarding-and-offboarding-key-management.md b/.ai/pages/nodes-and-validators-run-a-validator-onboarding-and-offboarding-key-management.md deleted file mode 100644 index 25c74598c..000000000 --- a/.ai/pages/nodes-and-validators-run-a-validator-onboarding-and-offboarding-key-management.md +++ /dev/null @@ -1,155 +0,0 @@ ---- -title: Validator Key Management -description: Learn how to generate and manage validator keys, including session keys for consensus participation and node keys for maintaining a stable network identity. -categories: Infrastructure -url: https://docs.polkadot.com/nodes-and-validators/run-a-validator/onboarding-and-offboarding/key-management/ ---- - -# Key Management - -## Introduction - -After setting up your node environment as shown in the [Setup](/nodes-and-validators/run-a-validator/onboarding-and-offboarding/set-up-validator/){target=\_blank} section, you'll need to configure multiple keys for your validator to operate properly. This includes setting up session keys, which are essential for participating in the consensus process, and configuring a node key that maintains a stable network identity. This guide walks you through the key management process, showing you how to generate, store, and register these keys. - -## Set Session Keys - -Setting up your validator's session keys is essential to associate your node with your stash account on the Polkadot network. Validators use session keys to participate in the consensus process. Your validator can only perform its role in the network by properly setting session keys which consist of several key pairs for different parts of the protocol (e.g., GRANDPA, BABE). These keys must be registered on-chain and associated with your validator node to ensure it can participate in validating blocks. - -### Generate Session Keys - -There are multiple ways to create the session keys. It can be done by interacting with the [Polkadot.js Apps UI](https://polkadot.js.org/apps/#/explorer){target=\_blank}, using the curl command or by using [Subkey](https://paritytech.github.io/polkadot-sdk/master/subkey/index.html){target=\_blank}. - -=== "Polkadot.js Apps UI" - - 1. In Polkadot.js Apps, connect to your local node, navigate to the **Developer** dropdown, and select the **RPC Calls** option. - - 2. Construct an `author_rotateKeys` RPC call and execute it: - - 1. Select the **author** endpoint. - 2. Choose the **rotateKeys()** call. - 3. Click the **Submit RPC Call** button. - 4. Copy the hex-encoded public key from the response. - - ![](/images/nodes-and-validators/run-a-validator/onboarding-and-offboarding/key-management/key-management-01.webp) - -=== "Curl" - - Generate session keys by running the following command on your validator node: - - ``` bash - curl -H "Content-Type: application/json" \ - -d '{"id":1, "jsonrpc":"2.0", "method": "author_rotateKeys", "params":[]}' \ - http://localhost:9944 - ``` - - This command will return a JSON object. The `result` key is the hex-encoded public part of the newly created session key. Save this for later use. - - ```json - {"jsonrpc":"2.0","result":"0xda3861a45e0197f3ca145c2c209f9126e5053fas503e459af4255cf8011d51010","id":1} - ``` - -=== "Subkey" - - To create a keypair for your node's session keys, use the `subkey generate` command. This generates a set of cryptographic keys that must be stored in your node's keystore directory. - - When you run the command, it produces output similar to this example: - -
- subkey generate - 
-    Secret phrase:       twist buffalo mixture excess device drastic vague mammal fitness punch match hammer
-      Network ID:        substrate
-      Secret seed:       0x5faa9e5defe42b201388d5c2b8202d6625a344abc9aa52943a71f12cb90b88a9
-      Public key (hex):  0x28cc2fdb6e28835e2bbac9a16feb65c23d448c9314ef12fe083b61bab8fc2755
-      Account ID:        0x28cc2fdb6e28835e2bbac9a16feb65c23d448c9314ef12fe083b61bab8fc2755
-      Public key (SS58): 5CzCRpXzHYhuo6G3gYFR3cgV6X3qCNwVt51m8q14ZcChsSXQ
-      SS58 Address:      5CzCRpXzHYhuo6G3gYFR3cgV6X3qCNwVt51m8q14ZcChsSXQ
-
-
- - To properly store these keys, create a file in your keystore directory with a specific naming convention. The filename must consist of the hex string `61757261` (which represents "aura" in hex) followed by the public key without its `0x` prefix. - - Using the example above, you would create a file named: - - ``` - ./keystores/6175726128cc2fdb6e28835e2bbac9a16feb65c23d448c9314ef12fe083b61bab8fc2755 - ``` - - And store only the secret phrase in the file: - - ``` - "twist buffalo mixture excess device drastic vague mammal fitness punch match hammer" - ``` - -### Submit Transaction to Set Keys - -Now that you have generated your session keys, you must submit them to the chain. Follow these steps: - -1. Go to the **Network > Staking > Accounts** section on Polkadot.js Apps. -2. Select **Set Session Key** on the bonding account you generated earlier. -3. Paste the hex-encoded session key string you generated (from either the UI or CLI) into the input field and submit the transaction. - -![](/images/nodes-and-validators/run-a-validator/onboarding-and-offboarding/key-management/key-management-02.webp) - -Once the transaction is signed and submitted, your session keys will be registered on-chain. - -### Verify Session Key Setup - -To verify that your session keys are properly set, you can use one of two RPC calls: - -- **`hasKey`**: Checks if the node has a specific key by public key and key type. -- **`hasSessionKeys`**: Verifies if your node has the full session key string associated with the validator. - -For example, you can [check session keys on the Polkadot.js Apps](https://polkadot.js.org/apps/#/rpc){target=\_blank} interface or by running an RPC query against your node. Once this is done, your validator node is ready for its role. - -## Set the Node Key - -Validators on Polkadot need a static network key (also known as the node key) to maintain a stable node identity. This key ensures that your validator can maintain a consistent peer ID, even across restarts, which is crucial for maintaining reliable network connections. - -Starting with Polkadot version 1.11, validators without a stable network key may encounter the following error on startup: - -
- polkadot --validator --name "INSERT_NAME_FROM_TELEMETRY" - Error: - 0: Starting an authority without network key - This is not a safe operation because other authorities in the network may depend on your node having a stable identity. - Otherwise these other authorities may not being able to reach you. - If it is the first time running your node you could use one of the following methods: - 1. [Preferred] Separately generate the key with: INSERT_NODE_BINARY key generate-node-key --base-path INSERT_YOUR_BASE_PATH - 2. [Preferred] Separately generate the key with: INSERT_NODE_BINARY key generate-node-key --file INSERT_YOUR_PATH_TO_NODE_KEY - 3. [Preferred] Separately generate the key with: INSERT_NODE_BINARY key generate-node-key --default-base-path - 4. [Unsafe] Pass --unsafe-force-node-key-generation and make sure you remove it for subsequent node restarts - -
- -### Generate the Node Key - -Use one of the following methods to generate your node key: - -=== "Save to file" - - The recommended solution is to generate a node key and save it to a file using the following command: - - ``` bash - polkadot key generate-node-key --file INSERT_PATH_TO_NODE_KEY - ``` - -=== "Use default path" - - You can also generate the node key with the following command, which will automatically save the key to the base path of your node: - - ``` bash - polkadot key generate-node-key --default-base-path - ``` - -Save the file path for reference. You will need it in the next step to configure your node with a static identity. - -### Set Node Key - -After generating the node key, configure your node to use it by specifying the path to the key file when launching your node. Add the following flag to your validator node's startup command: - -``` bash -polkadot --node-key-file INSERT_PATH_TO_NODE_KEY -``` - -Following these steps ensures that your node retains its identity, making it discoverable by peers without the risk of conflicting identities across sessions. For further technical background, see Polkadot SDK [Pull Request #3852](https://github.com/paritytech/polkadot-sdk/pull/3852){target=\_blank} for the rationale behind requiring static keys. diff --git a/.ai/pages/nodes-and-validators-run-a-validator-onboarding-and-offboarding-set-up-validator.md b/.ai/pages/nodes-and-validators-run-a-validator-onboarding-and-offboarding-set-up-validator.md deleted file mode 100644 index 2f7566b64..000000000 --- a/.ai/pages/nodes-and-validators-run-a-validator-onboarding-and-offboarding-set-up-validator.md +++ /dev/null @@ -1,213 +0,0 @@ ---- -title: Set Up a Validator -description: Set up a Polkadot validator node to secure the network and earn staking rewards. Follow this step-by-step guide to install, configure, and manage your node. -categories: Infrastructure -url: https://docs.polkadot.com/nodes-and-validators/run-a-validator/onboarding-and-offboarding/set-up-validator/ ---- - -# Set Up a Validator - -## Introduction - -Setting up a Polkadot validator node is essential for securing the network and earning staking rewards. This guide walks you through the technical steps to set up a validator, from installing the necessary software to managing keys and synchronizing your node with the chain. - -Running a validator requires a commitment to maintaining a stable, secure infrastructure. Validators are responsible for their own stakes and those of nominators who trust them with their tokens. Proper setup and ongoing management are critical to ensuring smooth operation and avoiding potential penalties such as slashing. - -## Prerequisites - -To get the most from this guide, ensure you've done the following before going forward: - -- Read [Validator Requirements](/nodes-and-validators/run-a-validator/requirements/){target=\_blank} and understand the recommended minimum skill level and hardware needs. -- Read [General Management](/nodes-and-validators/run-a-validator/operational-tasks/general-management/){target=\_blank}, [Upgrade Your Node](/nodes-and-validators/run-a-validator/operational-tasks/upgrade-your-node/){target=\_blank}, and [Pause Validating](/nodes-and-validators/run-a-validator/operational-tasks/pause-validating/){target=\_blank} and understand the tasks required to keep your validator operational. -- Read [Rewards Payout](/nodes-and-validators/run-a-validator/staking-mechanics/rewards/){target=\_blank} and understand how validator rewards are determined and paid out. -- Read [Offenses and Slashes](/nodes-and-validators/run-a-validator/staking-mechanics/offenses-and-slashes/){target=\_blank} and understand how validator performance and security can affect tokens staked by you or your nominators. - -## Initial Setup - -Before running your validator, you must configure your server environment to meet the operational and security standards required for validating. - -You must use a Linux-based operating system with Kernel 5.16 or later. Configuration includes setting up time synchronization, ensuring critical security features are active, and installing the necessary binaries. Proper setup at this stage is essential to prevent issues like block production errors or being penalized for downtime. Below are the essential steps to get your system ready. - -### Install Network Time Protocol Client - -Accurate timekeeping is critical to ensure your validator is synchronized with the network. Validators need local clocks in sync with the blockchain to avoid missing block authorship opportunities. Using [Network Time Protocol (NTP)](https://en.wikipedia.org/wiki/Network_Time_Protocol){target=\_blank} is the standard solution to keep your system's clock accurate. - -If you are using Ubuntu version 18.04 or newer, the NTP Client should be installed by default. You can check whether you have the NTP client by running: - -```sh -timedatectl -``` - -If NTP is running, you should see a message like the following: - -``` sh -System clock synchronized: yes -``` - -If NTP is not installed or running, you can install it using: - -```sh -sudo apt-get install ntp -``` - -After installation, NTP will automatically start. To check its status: - -```sh -sudo ntpq -p -``` - -This command will return a message with the status of the NTP synchronization. Skipping this step could result in your validator node missing blocks due to minor clock drift, potentially affecting its network performance. - -### Verify Landlock is Activated - -[Landlock](https://docs.kernel.org/userspace-api/landlock.html){target=\_blank} is an important security feature integrated into Linux kernels starting with version 5.13. It allows processes, even those without special privileges, to limit their access to the system to reduce the machine's attack surface. This feature is crucial for validators, as it helps ensure the security and stability of the node by preventing unauthorized access or malicious behavior. - -To use Landlock, ensure you use the reference kernel or newer versions. Most Linux distributions should already have Landlock activated. You can check if Landlock is activated on your machine by running the following command as root: - -```sh -dmesg | grep landlock || journalctl -kg landlock -``` - -If Landlock is not activated, your system logs won't show any related output. In this case, you will need to activate it manually or ensure that your Linux distribution supports it. Most modern distributions with the required kernel version should have Landlock activated by default. However, if your system lacks support, you may need to build the kernel with Landlock activated. For more information on doing so, refer to the [official kernel documentation](https://docs.kernel.org/userspace-api/landlock.html#kernel-support){target=\_blank}. - -Implementing Landlock ensures your node operates in a restricted, self-imposed sandbox, limiting potential damage from security breaches or bugs. While not a mandatory requirement, enabling this feature greatly improves the security of your validator setup. - -## Install the Polkadot Binaries - -You must install the Polkadot binaries required to run your validator node. These binaries include the main `polkadot`, `polkadot-prepare-worker`, and `polkadot-execute-worker` binaries. All three are needed to run a fully functioning validator node. - -Depending on your preference and operating system setup, there are multiple methods to install these binaries. Below are the main options: - -### Install from Official Releases - -The preferred, most straightforward method to install the required binaries is downloading the latest versions from the official releases. You can visit the [Github Releases](https://github.com/paritytech/polkadot-sdk/releases){target=\_blank} page for the most current versions of the `polkadot`, `polkadot-prepare-worker`, and `polkadot-execute-worker` binaries. - -You can also download the binaries by using the following direct links: - -=== "`polkadot`" - - ``` bash - # Download the binary - curl -LO https://github.com/paritytech/polkadot-sdk/releases/download/polkadot-stable2506-2/polkadot - - # Verify signature - curl -LO https://github.com/paritytech/polkadot-sdk/releases/download/polkadot-stable2506-2/polkadot.asc - - gpg --keyserver hkps://keyserver.ubuntu.com --receive-keys 90BD75EBBB8E95CB3DA6078F94A4029AB4B35DAE - - gpg --verify polkadot.asc - ``` - -=== "`polkadot-prepare-worker`" - - ``` bash - # Download the binary - curl -LO https://github.com/paritytech/polkadot-sdk/releases/download/polkadot-stable2506-2/polkadot-prepare-worker - - # Verify signature - curl -LO https://github.com/paritytech/polkadot-sdk/releases/download/polkadot-stable2506-2/polkadot-prepare-worker.asc - - gpg --keyserver hkps://keyserver.ubuntu.com --receive-keys 90BD75EBBB8E95CB3DA6078F94A4029AB4B35DAE - - gpg --verify polkadot-prepare-worker.asc - ``` - -=== "`polkadot-execute-worker`" - - ``` bash - # Download the binary - curl -LO https://github.com/paritytech/polkadot-sdk/releases/download/polkadot-stable2506-2/polkadot-execute-worker - - # Verify signature - curl -LO https://github.com/paritytech/polkadot-sdk/releases/download/polkadot-stable2506-2/polkadot-execute-worker.asc - - gpg --keyserver hkps://keyserver.ubuntu.com --receive-keys 90BD75EBBB8E95CB3DA6078F94A4029AB4B35DAE - - gpg --verify polkadot-execute-worker.asc - ``` - - -Signature verification cryptographically ensures the downloaded binaries are authentic and have not been tampered with by using GPG signing keys. Polkadot releases use two different signing keys: - -- ParityReleases (release-team@parity.io) with key [`90BD75EBBB8E95CB3DA6078F94A4029AB4B35DAE`](https://keyserver.ubuntu.com/pks/lookup?search=90BD75EBBB8E95CB3DA6078F94A4029AB4B35DAE&fingerprint=on&op=index){target=\_blank} for current and new releases. -- Parity Security Team (security@parity.io) with key [`9D4B2B6EB8F97156D19669A9FF0812D491B96798`](https://keyserver.ubuntu.com/pks/lookup?search=9D4B2B6EB8F97156D19669A9FF0812D491B96798&fingerprint=on&op=index){target=\_blank} for old releases. - - !!!warning - When verifying a signature, a "Good signature" message indicates successful verification, while any other output signals a potential security risk. - -### Install with Package Managers - -Users running Debian-based distributions like Ubuntu can install the binaries using the [APT](https://wiki.debian.org/Apt){target=\_blank} package manager. - -Execute the following commands as root to add the official repository and install the binaries: - -```bash -# Import the release-team@parity.io GPG key -gpg --keyserver hkps://keyserver.ubuntu.com --receive-keys 90BD75EBBB8E95CB3DA6078F94A4029AB4B35DAE -gpg --export 90BD75EBBB8E95CB3DA6078F94A4029AB4B35DAE > /usr/share/keyrings/parity.gpg - -# Add the Parity repository and update the package index -echo 'deb [signed-by=/usr/share/keyrings/parity.gpg] https://releases.parity.io/deb release main' > /etc/apt/sources.list.d/parity.list -apt update - -# Install the `parity-keyring` package - This will ensure the GPG key -# used by APT remains up-to-date -apt install parity-keyring - -# Install polkadot -apt install polkadot -``` - -Once installation completes, verify the binaries are correctly installed by following the steps in the [verify installation](#verify-installation) section. - -### Install with Ansible - -You can also manage Polkadot installations using Ansible. This approach can be beneficial for users managing multiple validator nodes or requiring automated deployment. The [Parity chain operations Ansible collection](https://github.com/paritytech/ansible-galaxy/){target=\_blank} provides a Substrate node role for this purpose. - -### Install with Docker - -If you prefer using Docker or an OCI-compatible container runtime, the official Polkadot Docker image can be pulled directly from Docker Hub. - -To pull the latest stable image, run the following command: - -```bash -docker pull parity/polkadot:stable2506-2 -``` - -### Build from Sources - -You may build the binaries from source by following the instructions on the [Polkadot SDK repository](https://github.com/paritytech/polkadot-sdk/tree/polkadot-stable2506-2/polkadot#building){target=\_blank}. - -## Verify Installation - -Once the Polkadot binaries are installed, it's essential to verify that everything is set up correctly and that all the necessary components are in place. Follow these steps to ensure the binaries are installed and functioning as expected. - -1. **Check the versions**: Run the following commands to verify the versions of the installed binaries. - - ```bash - polkadot --version - polkadot-execute-worker --version - polkadot-prepare-worker --version - ``` - - The output should show the version numbers for each of the binaries. Ensure that the versions match and are consistent, similar to the following example (the specific version may vary): - -
- polkadot --version polkadot-execute-worker --version polkadot-prepare-worker --version - 1.16.1-36264cb36db - 1.16.1-36264cb36db - 1.16.1-36264cb36db - -
- - If the versions do not match or if there is an error, double-check that all the binaries were correctly installed and are accessible within your `$PATH`. - -2. **Ensure all binaries are in the same directory**: All the binaries must be in the same directory for the Polkadot validator node to function properly. If the binaries are not in the same location, move them to a unified directory and ensure this directory is added to your system's `$PATH`. - - To verify the `$PATH`, run the following command: - - ```bash - echo $PATH - ``` - - If necessary, you can move the binaries to a shared location, such as `/usr/local/bin/`, and add it to your `$PATH`. diff --git a/.ai/pages/nodes-and-validators-run-a-validator-onboarding-and-offboarding-start-validating.md b/.ai/pages/nodes-and-validators-run-a-validator-onboarding-and-offboarding-start-validating.md deleted file mode 100644 index fdec84d1d..000000000 --- a/.ai/pages/nodes-and-validators-run-a-validator-onboarding-and-offboarding-start-validating.md +++ /dev/null @@ -1,254 +0,0 @@ ---- -title: Start Validating -description: Learn how to start validating on Polkadot by choosing a network, syncing your node, bonding DOT tokens, and activating your validator. -categories: Infrastructure -url: https://docs.polkadot.com/nodes-and-validators/run-a-validator/onboarding-and-offboarding/start-validating/ ---- - -# Start Validating - -## Introduction - -After configuring your node keys as shown in the [Key Management](/nodes-and-validators/run-a-validator/onboarding-and-offboarding/key-management/){target=\_blank} section and ensuring your system is set up, you're ready to begin the validator setup process. This guide will walk you through choosing a network, synchronizing your node with the blockchain, bonding your DOT tokens, and starting your validator. - -## Choose a Network - -Running your validator on a test network like Westend or Kusama is a smart way to familiarize yourself with the process and identify any setup issues in a lower-stakes environment before joining the Polkadot MainNet. - -- **Westend**: Polkadot's primary TestNet is open to anyone for testing purposes. Validator slots are intentionally limited to keep the network stable for the Polkadot release process, so it may not support as many validators at any given time. -- **Kusama**: Often called Polkadot's "canary network," Kusama has real economic value but operates with a faster and more experimental approach. Running a validator here provides an experience closer to MainNet with the benefit of more frequent validation opportunities with an era time of 6 hours vs 24 hours for Polkadot. -- **Polkadot**: The main network, where validators secure the Polkadot relay chain. It has a slower era time of 24 hours and requires a higher minimum bond amount to participate. - -## Synchronize Chain Data - -The next step is to sync your node with the chosen blockchain network. Synchronization is necessary to download and validate the blockchain data, ensuring your node is ready to participate as a validator. Follow these steps to sync your node: - -1. **Start syncing**: You can run a full or warp sync. - - === "Full sync" - - Polkadot defaults to using a full sync, which downloads and validates the entire blockchain history from the genesis block. Start the syncing process by running the following command: - - ```sh - polkadot - ``` - - This command starts your Polkadot node in non-validator mode, allowing you to synchronize the chain data. - - === "Warp sync" - - You can opt to use warp sync which initially downloads only GRANDPA finality proofs and the latest finalized block's state. Use the following command to start a warp sync: - - ``` bash - polkadot --sync warp - ``` - - Warp sync ensures that your node quickly updates to the latest finalized state. The historical blocks are downloaded in the background as the node continues to operate. - - If you're planning to run a validator on a TestNet, you can specify the chain using the `--chain` flag. For example, the following will run a validator on Kusama: - - ```sh - polkadot --chain=kusama - ``` - -2. **Monitor sync progress**: Once the sync starts, you will see a stream of logs providing information about the node's status and progress. Here's an example of what the output might look like: - -
- polkadot - 2021-06-17 03:07:07 Parity Polkadot - 2021-06-17 03:07:07 ✌️ version 0.9.5-95f6aa201-x86_64-linux-gnu - 2021-06-17 03:07:07 ❤️ by Parity Technologies <admin@parity.io>, 2017-2021 - 2021-06-17 03:07:07 📋 Chain specification: Polkadot - 2021-06-17 03:07:07 🏷 Node name: boiling-pet-7554 - 2021-06-17 03:07:07 👤 Role: FULL - 2021-06-17 03:07:07 💾 Database: RocksDb at /root/.local/share/polkadot/chains/polkadot/db - 2021-06-17 03:07:07 ⛓ Native runtime: polkadot-9050 (parity-polkadot-0.tx7.au0) - 2021-06-17 03:07:10 🏷 Local node identity is: 12D3KooWLtXFWf1oGrnxMGmPKPW54xWCHAXHbFh4Eap6KXmxoi9u - 2021-06-17 03:07:10 📦 Highest known block at #17914 - 2021-06-17 03:07:10 〽️ Prometheus server started at 127.0.0.1:9615 - 2021-06-17 03:07:10 Listening for new connections on 127.0.0.1:9944 - ... -
- - The output logs provide information such as the current block number, node name, and network connections. Monitor the sync progress and any errors that might occur during the process. Look for information about the latest processed block and compare it with the current highest block using tools like [Telemetry](https://telemetry.polkadot.io/#list/Polkadot%20CC1){target=\_blank} or [Polkadot.js Apps Explorer](https://polkadot.js.org/apps/#/explorer){target=\_blank}. - -### Database Snapshot Services - -If you'd like to speed up the process further, you can use a database snapshot. Snapshots are compressed backups of the blockchain's database directory and can significantly reduce the time required to sync a new node. Here are a few public snapshot providers: - -- [Stakeworld](https://stakeworld.io/snapshot){target=\_blank} -- [Polkachu](https://polkachu.com/substrate_snapshots){target=\_blank} -- [Polkashots](https://polkashots.io/){target=\_blank} -- [ITRocket](https://itrocket.net/services/mainnet/polkadot/#snapshot){target=\_blank} - -!!!warning - Although snapshots are convenient, syncing from scratch is recommended for security purposes. If snapshots become corrupted and most nodes rely on them, the network could inadvertently run on a non-canonical chain. - -
- polkadot - 2021-06-17 03:07:07 Idle (0 peers), best: #0 (0x3fd7...5baf), finalized #0 (0x3fd7...5baf), ⬇ 2.9kiB/s ⬆ 3.7kiB/s - 2021-06-17 03:07:12 Idle (0 peers), best: #0 (0x3fd7...5baf), finalized #0 (0x3fd7...5baf), ⬇ 1.7kiB/s ⬆ 2.0kiB/s - 2021-06-17 03:07:17 Idle (0 peers), best: #0 (0x3fd7...5baf), finalized #0 (0x3fd7...5baf), ⬇ 0.9kiB/s ⬆ 1.2kiB/s - 2021-06-17 03:07:19 Libp2p => Random Kademlia query has yielded empty results - 2021-06-17 03:08:00 Idle (0 peers), best: #0 (0x3fd7...5baf), finalized #0 (0x3fd7...5baf), ⬇ 1.6kiB/s ⬆ 1.9kiB/s - 2021-06-17 03:08:05 Idle (0 peers), best: #0 (0x3fd7...5baf), finalized #0 (0x3fd7...5baf), ⬇ 0.6kiB/s ⬆ 0.9kiB/s - ... -
- -If you see terminal output similar to the preceding, and you are unable to synchronize the chain due to having zero peers, make sure you have libp2p port `30333` activated. It will take some time to discover other peers over the network. - -## Bond DOT - -Once your validator node is synced, the next step is bonding DOT. A bonded account, or stash, holds your staked tokens (DOT) that back your validator node. Bonding your DOT means locking it for a period, during which it cannot be transferred or spent but is used to secure your validator's role in the network. Visit the [Minimum Bond Requirement](/nodes-and-validators/run-a-validator/requirements/#minimum-bond-requirement) section for details on how much DOT is required. - -The following sections will guide you through bonding DOT for your validator. - -### Bonding DOT on Polkadot.js Apps - -Once you're ready to bond your DOT, head over to the [Polkadot.js Apps](https://polkadot.js.org/apps/){target=\_blank} staking page by clicking the **Network** dropdown at the top of the page and selecting [**Staking**](https://polkadot.js.org/apps/#/staking/actions){target=\_blank}. - -To get started with the bond submission, click on the **Accounts** tab, then the **+ Stash** button, and then enter the following information: - -1. **Stash account**: Select your stash account (which is the account with the DOT/KSM balance). -2. **Value bonded**: Enter how much DOT from the stash account you want to bond/stake. You are not required to bond all of the DOT in that account and you may bond more DOT at a later time. Be aware, withdrawing any bonded amount requires waiting for the unbonding period. The unbonding period is seven days for Kusama and 28 days for Polkadot. -3. **Payment destination**: Add the recipient account for validator rewards. If you'd like to redirect payments to an account that is not the stash account, you can do it by entering the address here. Note that it is extremely unsafe to set an exchange address as the recipient of the staking rewards. - -Once everything is filled in properly, select **Bond** and sign the transaction with your stash account. If successful, you should see an `ExtrinsicSuccess` message. - -Your bonded account will be available under **Stashes**. After refreshing the screen, you should now see a card with all your accounts. The bonded amount on the right corresponds to the funds bonded by the stash account. - -## Validate - -Once your validator node is fully synced and ready, the next step is to ensure it's visible on the network and performing as expected. Below are steps for monitoring and managing your node on the Polkadot network. - -### Verify Sync via Telemetry - -To confirm that your validator is live and synchronized with the Polkadot network, visit the [Telemetry](https://telemetry.polkadot.io/#list/Polkadot%20CC1){target=\_blank} page. Telemetry provides real-time information on node performance and can help you check if your validator is connected properly. Search for your node by name. You can search all nodes currently active on the network, which is why you should use a unique name for easy recognition. Now, confirm that your node is fully synced by comparing the block height of your node with the network's latest block. Nodes that are fully synced will appear white in the list, while nodes that are not yet fully synced will appear gray. - -### Activate using Polkadot.js Apps - -Follow these steps to use Polkadot.js Apps to activate your validator: - -1. In Polkadot.js Apps, navigate to **Network** and select **Staking**: - - ![](/images/nodes-and-validators/run-a-validator/onboarding-and-offboarding/start-validating/start-validating-01.webp) - -2. Open the **Accounts** tab and click on **+ Validator**: - - ![](/images/nodes-and-validators/run-a-validator/onboarding-and-offboarding/start-validating/start-validating-02.webp) - -3. Set a bond amount in the **value bonded** field and then click **next**: - - ![](/images/nodes-and-validators/run-a-validator/onboarding-and-offboarding/start-validating/start-validating-03.webp) - -4. Paste the hex output from `author_rotateKeys`, set the commission, allow or block new nominations, then click **Bond & Validate** to link your validator with its session keys. - - ![](/images/nodes-and-validators/run-a-validator/onboarding-and-offboarding/start-validating/start-validating-04.webp) - - You can also set the **commission** and **blocked** nominations option via `staking.validate` extrinsic. By default, the blocked option is set to FALSE (i.e., the validator accepts nominations). - - ![](/images/nodes-and-validators/run-a-validator/onboarding-and-offboarding/start-validating/start-validating-05.webp) - -### Monitor Validation Status and Slots - -On the [**Staking**](https://polkadot.js.org/apps/#/staking){target=\_blank} tab in Polkadot.js Apps, you can see your validator's status, the number of available validator slots, and the nodes that have signaled their intent to validate. Your node may initially appear in the waiting queue, especially if the validator slots are full. The following is an example view of the **Staking** tab: - -![staking queue](/images/nodes-and-validators/run-a-validator/onboarding-and-offboarding/start-validating/start-validating-06.webp) - -The validator set refreshes each era. If there's an available slot in the next era, your node may be selected to move from the waiting queue to the active validator set, allowing it to start validating blocks. If your validator is not selected, it remains in the waiting queue. Increasing your stake or gaining more nominators may improve your chance of being selected in future eras. - -## Run a Validator Using Systemd - -Running your Polkadot validator as a [systemd](https://en.wikipedia.org/wiki/Systemd){target=\_blank} service is an effective way to ensure its high uptime and reliability. Using systemd allows your validator to automatically restart after server reboots or unexpected crashes, significantly reducing the risk of slashing due to downtime. - -This following sections will walk you through creating and managing a systemd service for your validator, allowing you to seamlessly monitor and control it as part of your Linux system. - -Ensure the following requirements are met before proceeding with the systemd setup: - -- Confirm your system meets the [requirements](/nodes-and-validators/run-a-validator/requirements/){target=\_blank} for running a validator. -- Ensure you meet the [minimum bond requirements](https://wiki.polkadot.com/general/chain-state-values/#minimum-validator-bond){target=\_blank} for validating. -- Verify the Polkadot binary is [installed](#install-the-polkadot-binaries). - -### Create the Systemd Service File - -First create a new unit file called `polkadot-validator.service` in `/etc/systemd/system/`: - -```bash -touch /etc/systemd/system/polkadot-validator.service -``` - -In this unit file, you will write the commands that you want to run on server boot/restart: - -```systemd title="/etc/systemd/system/polkadot-validator.service" -[Unit] -Description=Polkadot Node -After=network.target -Documentation=https://github.com/paritytech/polkadot-sdk - -[Service] -EnvironmentFile=-/etc/default/polkadot -ExecStart=/usr/bin/polkadot $POLKADOT_CLI_ARGS -User=polkadot -Group=polkadot -Restart=always -RestartSec=120 -CapabilityBoundingSet= -LockPersonality=true -NoNewPrivileges=true -PrivateDevices=true -PrivateMounts=true -PrivateTmp=true -PrivateUsers=true -ProtectClock=true -ProtectControlGroups=true -ProtectHostname=true -ProtectKernelModules=true -ProtectKernelTunables=true -ProtectSystem=strict -RemoveIPC=true -RestrictAddressFamilies=AF_INET AF_INET6 AF_NETLINK AF_UNIX -RestrictNamespaces=false -RestrictSUIDSGID=true -SystemCallArchitectures=native -SystemCallFilter=@system-service -SystemCallFilter=landlock_add_rule landlock_create_ruleset landlock_restrict_self seccomp mount umount2 -SystemCallFilter=~@clock @module @reboot @swap @privileged -SystemCallFilter=pivot_root -UMask=0027 - -[Install] -WantedBy=multi-user.target -``` - -!!! warning "Restart delay and equivocation risk" - It is recommended that a node's restart be delayed with `RestartSec` in the case of a crash. It's possible that when a node crashes, consensus votes in GRANDPA aren't persisted to disk. In this case, there is potential to equivocate when immediately restarting. Delaying the restart will allow the network to progress past potentially conflicting votes. - -### Run the Service - -Activate the systemd service to start on system boot by running: - -```bash -systemctl enable polkadot-validator.service -``` - -To start the service manually, use: - -```bash -systemctl start polkadot-validator.service -``` - -Check the service's status to confirm it is running: - -```bash -systemctl status polkadot-validator.service -``` - -To view the logs in real-time, use [journalctl](https://www.freedesktop.org/software/systemd/man/latest/journalctl.html){target=\_blank} like so: - -```bash -journalctl -f -u polkadot-validator -``` - -With these steps, you can effectively manage and monitor your validator as a systemd service. - -Once your validator is active, it's officially part of Polkadot's security infrastructure. For questions or further support, you can reach out to the [Polkadot Validator chat](https://matrix.to/#/!NZrbtteFeqYKCUGQtr:matrix.parity.io?via=matrix.parity.io&via=matrix.org&via=web3.foundation){target=\_blank} for tips and troubleshooting. diff --git a/.ai/pages/nodes-and-validators-run-a-validator-onboarding-and-offboarding-stop-validating.md b/.ai/pages/nodes-and-validators-run-a-validator-onboarding-and-offboarding-stop-validating.md deleted file mode 100644 index 922e9784b..000000000 --- a/.ai/pages/nodes-and-validators-run-a-validator-onboarding-and-offboarding-stop-validating.md +++ /dev/null @@ -1,45 +0,0 @@ ---- -title: Stop Validating -description: Learn to safely stop validating on Polkadot, including chilling, unbonding tokens, and purging validator keys. -categories: Infrastructure -url: https://docs.polkadot.com/nodes-and-validators/run-a-validator/onboarding-and-offboarding/stop-validating/ ---- - -# Stop Validating - -## Introduction - -If you're ready to stop validating on Polkadot, there are essential steps to ensure a smooth transition while protecting your funds and account integrity. Whether you're taking a break for maintenance or unbonding entirely, you'll need to chill your validator, purge session keys, and unbond your tokens. This guide explains how to use Polkadot's tools and extrinsics to safely withdraw from validation activities, safeguarding your account's future usability. - -## Pause Versus Stop - -If you wish to remain a validator or nominator (for example, stopping for planned downtime or server maintenance), submitting the `chill` extrinsic in the `staking` pallet should suffice. Additional steps are only needed to unbond funds or reap an account. - -The following are steps to ensure a smooth stop to validation: - -- Chill the validator. -- Purge validator session keys. -- Unbond your tokens. - -## Chill Validator - -When stepping back from validating, the first step is to chill your validator status. This action stops your validator from being considered for the next era without fully unbonding your tokens, which can be useful for temporary pauses like maintenance or planned downtime. - -Use the `staking.chill` extrinsic to initiate this. For more guidance on chilling your node, refer to the [Pause Validating](/nodes-and-validators/run-a-validator/operational-tasks/pause-validating/){target=\_blank} guide. You may also claim any pending staking rewards at this point. - -## Purge Validator Session Keys - -Purging validator session keys is a critical step in removing the association between your validator account and its session keys, which ensures that your account is fully disassociated from validator activities. The `session.purgeKeys` extrinsic removes the reference to your session keys from the stash or staking proxy account that originally set them. - -Here are a couple of important things to know about purging keys: - -- **Account used to purge keys**: Always use the same account to purge keys you originally used to set them, usually your stash or staking proxy account. Using a different account may leave an unremovable reference to the session keys on the original account, preventing its reaping. -- **Account reaping issue**: Failing to purge keys will prevent you from reaping (fully deleting) your stash account. If you attempt to transfer tokens without purging, you'll need to rebond, purge the session keys, unbond again, and wait through the unbonding period before any transfer. - -## Unbond Your Tokens - -After chilling your node and purging session keys, the final step is to unbond your staked tokens. This action removes them from staking and begins the unbonding period (usually 28 days for Polkadot and seven days for Kusama), after which the tokens will be transferable. - -To unbond tokens, go to **Network > Staking > Account Actions** on Polkadot.js Apps. Select your stash account, click on the dropdown menu, and choose **Unbond Funds**. Alternatively, you can use the `staking.unbond` extrinsic if you handle this via a staking proxy account. - -Once the unbonding period is complete, your tokens will be available for use in transactions or transfers outside of staking. diff --git a/.ai/pages/nodes-and-validators-run-a-validator-operational-tasks-general-management.md b/.ai/pages/nodes-and-validators-run-a-validator-operational-tasks-general-management.md deleted file mode 100644 index 8bceb81b8..000000000 --- a/.ai/pages/nodes-and-validators-run-a-validator-operational-tasks-general-management.md +++ /dev/null @@ -1,722 +0,0 @@ ---- -title: General Management -description: Optimize your Polkadot validator setup with advanced configuration techniques. Learn how to boost performance, enhance security, and ensure seamless operations. -categories: Infrastructure -url: https://docs.polkadot.com/nodes-and-validators/run-a-validator/operational-tasks/general-management/ ---- - -# General Management - -## Introduction - -Validator performance is pivotal in maintaining the security and stability of the Polkadot network. As a validator, optimizing your setup ensures efficient transaction processing, minimizes latency, and maintains system reliability during high-demand periods. Proper configuration and proactive monitoring also help mitigate risks like slashing and service interruptions. - -This guide covers essential practices for managing a validator, including performance tuning techniques, security hardening, and tools for real-time monitoring. Whether you're fine-tuning CPU settings, configuring NUMA balancing, or setting up a robust alert system, these steps will help you build a resilient and efficient validator operation. - -## Configuration Optimization - -For those seeking to optimize their validator's performance, the following configurations can improve responsiveness, reduce latency, and ensure consistent performance during high-demand periods. - -### Deactivate Simultaneous Multithreading - -Polkadot validators operate primarily in single-threaded mode for critical tasks, so optimizing single-core CPU performance can reduce latency and improve stability. Deactivating simultaneous multithreading (SMT) can prevent virtual cores from affecting performance. SMT is called Hyper-Threading on Intel and 2-way SMT on AMD Zen. - -Take the following steps to deactivate every other (vCPU) core: - -1. Loop though all the CPU cores and deactivate the virtual cores associated with them: - - ```bash - for cpunum in $(cat /sys/devices/system/cpu/cpu*/topology/thread_siblings_list | \ - cut -s -d, -f2- | tr ',' '\n' | sort -un) - do - echo 0 > /sys/devices/system/cpu/cpu$cpunum/online - done - ``` - -2. To permanently save the changes, add `nosmt=force` to the `GRUB_CMDLINE_LINUX_DEFAULT` variable in `/etc/default/grub`: - - ```bash - sudo nano /etc/default/grub - # Add to GRUB_CMDLINE_LINUX_DEFAULT - ``` - - ```config title="/etc/default/grub" - GRUB_DEFAULT = 0; - GRUB_HIDDEN_TIMEOUT = 0; - GRUB_HIDDEN_TIMEOUT_QUIET = true; - GRUB_TIMEOUT = 10; - GRUB_DISTRIBUTOR = `lsb_release -i -s 2> /dev/null || echo Debian`; - GRUB_CMDLINE_LINUX_DEFAULT = 'nosmt=force'; - GRUB_CMDLINE_LINUX = ''; - ``` - -3. Update GRUB to apply changes: - - ```bash - sudo update-grub - ``` - -4. After the reboot, you should see that half of the cores are offline. To confirm, run: - - ```bash - lscpu --extended - ``` - -### Deactivate Automatic NUMA Balancing - -Deactivating NUMA (Non-Uniform Memory Access) balancing for multi-CPU setups helps keep processes on the same CPU node, minimizing latency. - -Follow these stpes: - -1. Deactivate NUMA balancing in runtime: - - ```bash - sysctl kernel.numa_balancing=0 - ``` - -2. Deactivate NUMA balancing permanently by adding `numa_balancing=disable` to the GRUB settings: - - ```bash - sudo nano /etc/default/grub - # Add to GRUB_CMDLINE_LINUX_DEFAULT - ``` - - ```config title="/etc/default/grub" - GRUB_DEFAULT = 0; - GRUB_HIDDEN_TIMEOUT = 0; - GRUB_HIDDEN_TIMEOUT_QUIET = true; - GRUB_TIMEOUT = 10; - GRUB_DISTRIBUTOR = `lsb_release -i -s 2> /dev/null || echo Debian`; - GRUB_CMDLINE_LINUX_DEFAULT = 'numa_balancing=disable'; - GRUB_CMDLINE_LINUX = ''; - ``` - -3. Update GRUB to apply changes: - - ```bash - sudo update-grub - ``` - -4. Confirm the deactivation: - - ```bash - sysctl -a | grep 'kernel.numa_balancing' - ``` - -If you successfully deactivated NUMA balancing, the preceding command should return `0`. - -### Spectre and Meltdown Mitigations - -[Spectre](https://en.wikipedia.org/wiki/Spectre_(security_vulnerability)){target=\_blank} and [Meltdown](https://en.wikipedia.org/wiki/Meltdown_(security_vulnerability)){target=\_blank} are well-known CPU vulnerabilities that exploit speculative execution to access sensitive data. These vulnerabilities have been patched in recent Linux kernels, but the mitigations can slightly impact performance, especially in high-throughput or containerized environments. - -If your security requirements allow it, you can deactivate specific mitigations, such as Spectre V2 and Speculative Store Bypass Disable (SSBD), to improve performance. - -To selectively deactivate the Spectre mitigations, take these steps: - -1. Update the `GRUB_CMDLINE_LINUX_DEFAULT` variable in your `/etc/default/grub` configuration: - - ```bash - sudo nano /etc/default/grub - # Add to GRUB_CMDLINE_LINUX_DEFAULT - ``` - - ```config title="/etc/default/grub" - GRUB_DEFAULT = 0; - GRUB_HIDDEN_TIMEOUT = 0; - GRUB_HIDDEN_TIMEOUT_QUIET = true; - GRUB_TIMEOUT = 10; - GRUB_DISTRIBUTOR = `lsb_release -i -s 2> /dev/null || echo Debian`; - GRUB_CMDLINE_LINUX_DEFAULT = - 'spec_store_bypass_disable=prctl spectre_v2_user=prctl'; - ``` - -2. Update GRUB to apply changes and then reboot: - - ```bash - sudo update-grub - sudo reboot - ``` - -This approach selectively deactivates the Spectre V2 and Spectre V4 mitigations, leaving other protections intact. For full security, keep mitigations activated unless there's a significant performance need, as disabling them could expose the system to potential attacks on affected CPUs. - -## Monitor Your Node - -Monitoring your node's performance is critical for network reliability and security. Tools like the following provide valuable insights: - -- **[Prometheus](https://prometheus.io/){target=\_blank}**: An open-source monitoring toolkit for collecting and querying time-series data. -- **[Grafana](https://grafana.com/){target=\_blank}**: A visualization tool for real-time metrics, providing interactive dashboards. -- **[Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/){target=\_blank}**: A tool for managing and routing alerts based on Prometheus data. - -This section covers setting up these tools and configuring alerts to notify you of potential issues. - -### Environment Setup - -Before installing Prometheus, ensure the environment is set up securely by running Prometheus with restricted user privileges. - -Follow these steps: - -1. Create a Prometheus user to ensure Prometheus runs with minimal permissions: - - ```bash - sudo useradd --no-create-home --shell /usr/sbin/nologin prometheus - ``` - -2. Create directories for configuration and data storage: - - ```bash - sudo mkdir /etc/prometheus - sudo mkdir /var/lib/prometheus - ``` - -3. Change directory ownership to ensure Prometheus has access: - - ```bash - sudo chown -R prometheus:prometheus /etc/prometheus - sudo chown -R prometheus:prometheus /var/lib/prometheus - ``` - -### Install and Configure Prometheus - -After setting up the environment, install and configure the latest version of Prometheus as follows: - -1. Download Prometheus for your system architecture from the [releases page](https://github.com/prometheus/prometheus/releases/){target=\_blank}. Replace `INSERT_RELEASE_DOWNLOAD` with the release binary URL (e.g., `https://github.com/prometheus/prometheus/releases/download/v3.0.0/prometheus-3.0.0.linux-amd64.tar.gz`): - - ```bash - sudo apt-get update && sudo apt-get upgrade - wget INSERT_RELEASE_DOWNLOAD_LINK - tar xfz prometheus-*.tar.gz - cd prometheus-3.0.0.linux-amd64 - ``` - -2. Set up Prometheus: - - 1. Copy binaries: - - ```bash - sudo cp ./prometheus /usr/local/bin/ - sudo cp ./promtool /usr/local/bin/ - sudo cp ./prometheus /usr/local/bin/ - ``` - - 2. Copy directories and assign ownership of these files to the `prometheus` user: - - ```bash - sudo cp -r ./consoles /etc/prometheus - sudo cp -r ./console_libraries /etc/prometheus - sudo chown -R prometheus:prometheus /etc/prometheus/consoles - sudo chown -R prometheus:prometheus /etc/prometheus/console_libraries - ``` - - 3. Clean up the download directory: - - ```bash - cd .. && rm -r prometheus* - ``` - -3. Create `prometheus.yml` to define global settings, rule files, and scrape targets: - - ```bash - sudo nano /etc/prometheus/prometheus.yml - ``` - - {% raw %} - ```yaml title="prometheus-config.yml" - global: - scrape_interval: 15s - evaluation_interval: 15s - - rule_files: - # - "first.rules" - # - "second.rules" - - scrape_configs: - - job_name: 'prometheus' - scrape_interval: 5s - static_configs: - - targets: ['localhost:9090'] - - job_name: 'substrate_node' - scrape_interval: 5s - static_configs: - - targets: ['localhost:9615'] - ``` - {% endraw %} - - Prometheus is scraped every 5 seconds in this example configuration file, ensuring detailed internal metrics. Node metrics with customizable intervals are scraped from port `9615` by default. - -4. Verify the configuration with `promtool`, an open source monitoring tool: - - ```bash - promtool check config /etc/prometheus/prometheus.yml - ``` - -5. Save the configuration and change the ownership of the file to `prometheus` user: - - ```bash - sudo chown prometheus:prometheus /etc/prometheus/prometheus.yml - ``` - -### Start Prometheus - -1. Launch Prometheus with the appropriate configuration file, storage location, and necessary web resources, running it with restricted privileges for security: - - ```bash - sudo -u prometheus /usr/local/bin/prometheus --config.file /etc/prometheus/prometheus.yml \ - --storage.tsdb.path /var/lib/prometheus/ \ - --web.console.templates=/etc/prometheus/consoles \ - --web.console.libraries=/etc/prometheus/console_libraries - ``` - - If you set the server up properly, you should see terminal output similar to the following: - - -2. Verify you can access the Prometheus interface by navigating to: - - ```text - http://SERVER_IP_ADDRESS:9090/graph - ``` - - If the interface appears to work as expected, exit the process using `Control + C`. - -3. Create a systemd service file to ensure Prometheus starts on boot: - - ```bash - sudo nano /etc/systemd/system/prometheus.service - ``` - - ```bash title="prometheus.service" - [Unit] - Description=Prometheus Monitoring - Wants=network-online.target - After=network-online.target - - [Service] - User=prometheus - Group=prometheus - Type=simple - ExecStart=/usr/local/bin/prometheus \ - --config.file /etc/prometheus/prometheus.yml \ - --storage.tsdb.path /var/lib/prometheus/ \ - --web.console.templates=/etc/prometheus/consoles \ - --web.console.libraries=/etc/prometheus/console_libraries - ExecReload=/bin/kill -HUP $MAINPID - - [Install] - WantedBy=multi-user.target - - ``` - -4. Reload systemd and enable the service to start on boot: - - ```bash - sudo systemctl daemon-reload && sudo systemctl enable prometheus && sudo systemctl start prometheus - ``` - -5. Verify the service is running by visiting the Prometheus interface again at: - - ```text - http://SERVER_IP_ADDRESS:9090/ - ``` - -### Install and Configure Grafana - -This guide follows [Grafana's canonical installation instructions](https://grafana.com/docs/grafana/latest/setup-grafana/installation/debian/#install-from-apt-repository){target=\_blank}. - -To install and configure Grafana, follow these steps: - -1. Install Grafana prerequisites: - - ```bash - sudo apt-get install -y apt-transport-https software-properties-common wget - ``` - -2. Import the [GPG key](https://gnupg.org/){target=\_blank}: - - ```bash - sudo mkdir -p /etc/apt/keyrings/ - wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null - ``` - -3. Configure the stable release repo and update packages: - - ```bash - echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list - sudo apt-get update - ``` - -4. Install the latest stable version of Grafana: - - ```bash - sudo apt-get install grafana - ``` - -To configure Grafana, take these steps: - -1. Configure Grafana to start automatically on boot and start the service: - - ```bash - sudo systemctl daemon-reload - sudo systemctl enable grafana-server.service - sudo systemctl start grafana-server - ``` - -2. Check if Grafana is running: - - ```bash - sudo systemctl status grafana-server - ``` - - If necessary, you can stop or restart the service with the following commands: - - ```bash - sudo systemctl stop grafana-server - sudo systemctl restart grafana-server - ``` - -3. Access Grafana by navigating to the following URL and logging in with the default username and password (`admin`): - - ```text - http://SERVER_IP_ADDRESS:3000/login - ``` - - !!! tip "Change default port" - To change Grafana's port, edit `/usr/share/grafana/conf/defaults.ini`: - - ```bash - sudo vim /usr/share/grafana/conf/defaults.ini - ``` - - Modify the `http_port` value, then restart Grafana: - - ```bash - sudo systemctl restart grafana-server - ``` - -![Grafana login screen](/images/nodes-and-validators/run-a-validator/operational-tasks/general-management/general-management-01.webp) - -To visualize node metrics, follow these steps: - -1. Select the gear icon to access **Data Sources** settings. -2. Select **Add data source** to define the data source. - - ![Select Prometheus](/images/nodes-and-validators/run-a-validator/operational-tasks/general-management/general-management-02.webp) - -3. Select **Prometheus**. - - ![Save and test](/images/nodes-and-validators/run-a-validator/operational-tasks/general-management/general-management-03.webp) - -4. Enter `http://localhost:9090` in the **URL** field and click **Save & Test**. If **"Data source is working"** appears, your connection is configured correctly. - - ![Import dashboard](/images/nodes-and-validators/run-a-validator/operational-tasks/general-management/general-management-04.webp) - -5. Select **Import** from the left menu, choose **Prometheus** from the dropdown, and click **Import**. - -6. Start your Polkadot node by running `./polkadot`. You should now be able to monitor node performance, block height, network traffic, and tasks tasks on the Grafana dashboard. - - ![Live dashboard](/images/nodes-and-validators/run-a-validator/operational-tasks/general-management/general-management-05.webp) - -The [Grafana dashboards](https://grafana.com/grafana/dashboards){target=\_blank} page features user created dashboards made available for public use. For an example, see the [Substrate Node Metrics](https://grafana.com/grafana/dashboards/21715-substrate-node-metrics/){target=\_blank} dashboard. - -### Install and Configure Alertmanager - -[Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/){target=\_blank} is an optional component that complements Prometheus by managing alerts and notifying users about potential issues. - -Follow these steps to install and configure Alertmanager: - -1. Download Alertmanager for your system architecture from the [releases page](https://github.com/prometheus/alertmanager/releases){target=\_blank}. Replace `INSERT_RELEASE_DOWNLOAD` with the release binary URL (e.g., `https://github.com/prometheus/alertmanager/releases/download/v0.28.0-rc.0/alertmanager-0.28.0-rc.0.linux-amd64.tar.gz`): - - ```bash - wget INSERT_RELEASE_DOWNLOAD_LINK - tar -xvzf alertmanager* - ``` - -2. Copy the binaries to the system directory and set permissions: - - ```bash - cd alertmanager-0.28.0-rc.0.linux-amd64 - sudo cp ./alertmanager /usr/local/bin/ - sudo cp ./amtool /usr/local/bin/ - sudo chown prometheus:prometheus /usr/local/bin/alertmanager - sudo chown prometheus:prometheus /usr/local/bin/amtool - ``` - -3. Create the `alertmanager.yml` configuration file under `/etc/alertmanager`: - - ```bash - sudo mkdir /etc/alertmanager - sudo nano /etc/alertmanager/alertmanager.yml - ``` - - Generate an [app password in your Google account](https://support.google.com/accounts/answer/185833?hl=en){target=\_blank} to enable email notifications from Alertmanager. Then, add the following code to the configuration file to define email notifications using your email and app password: - - {% raw %} - ```yml title="alertmanager.yml" - global: - resolve_timeout: 1m - - route: - receiver: 'gmail-notifications' - - receivers: - - name: 'gmail-notifications' - email_configs: - - to: INSERT_YOUR_EMAIL - from: INSERT_YOUR_EMAIL - smarthost: smtp.gmail.com:587 - auth_username: INSERT_YOUR_EMAIL - auth_identity: INSERT_YOUR_EMAIL - auth_password: INSERT_YOUR_APP_PASSWORD - send_resolved: true - - ``` - {% endraw %} - - - ```bash - sudo chown -R prometheus:prometheus /etc/alertmanager - ``` - -4. Configure Alertmanager as a service by creating a systemd service file: - - ```bash - sudo nano /etc/systemd/system/alertmanager.service - ``` - - {% raw %} - ```yml title="alertmanager.service" - [Unit] - Description=AlertManager Server Service - Wants=network-online.target - After=network-online.target - - [Service] - User=root - Group=root - Type=simple - ExecStart=/usr/local/bin/alertmanager --config.file /etc/alertmanager/alertmanager.yml --web.external-url=http://SERVER_IP:9093 --cluster.advertise-address='0.0.0.0:9093' - - [Install] - WantedBy=multi-user.target - - ``` - {% endraw %} - -5. Reload and enable the service: - - ```bash - sudo systemctl daemon-reload - sudo systemctl enable alertmanager - sudo systemctl start alertmanager - ``` - -6. Verify the service status: - - ```bash - sudo systemctl status alertmanager - ``` - - If you have configured Alertmanager properly, the **Active** field should display **active (running)** similar to below: - -
- sudo systemctl status alertmanager - alertmanager.service - AlertManager Server Service - Loaded: loaded (/etc/systemd/system/alertmanager.service; enabled; vendor preset: enabled) - Active: active (running) since Thu 2020-08-20 22:01:21 CEST; 3 days ago - Main PID: 20592 (alertmanager) - Tasks: 70 (limit: 9830) - CGroup: /system.slice/alertmanager.service - -
- -#### Grafana Plugin - -There is an [Alertmanager plugin in Grafana](https://grafana.com/grafana/plugins/alertmanager/){target=\_blank} that can help you monitor alert information. - -Follow these steps to use the plugin: - -1. Install the plugin: - - ```bash - sudo grafana-cli plugins install camptocamp-prometheus-alertmanager-datasource - ``` - -2. Restart Grafana: - - ```bash - sudo systemctl restart grafana-server - ``` - -3. Configure Alertmanager as a data source in your Grafana dashboard (`SERVER_IP:3000`): - - 1. Go to **Configuration** > **Data Sources** and search for **Prometheus Alertmanager**. - 2. Enter the server URL and port for the Alertmanager service, and select **Save & Test** to verify the connection. - -4. Import the [8010](https://grafana.com/grafana/dashboards/8010-prometheus-alertmanager/){target=\_blank} dashboard for Alertmanager, selecting **Prometheus Alertmanager** in the last column, then select **Import**. - -#### Integrate Alertmanager - -Complete the integration by following these steps to enable communication between Prometheus and Alertmanager and configure detection and alert rules: - -1. Update the `etc/prometheus/prometheus.yml` configuration file to include the following code: - - {% raw %} - ```yml title="prometheus.yml" - rule_files: - - 'rules.yml' - - alerting: - alertmanagers: - - static_configs: - - targets: - - localhost:9093 - ``` - {% endraw %} - - Expand the following item to view the complete `prometheus.yml` file. - - ??? code "prometheus.yml" - - {% raw %} - ```yml title="prometheus.yml" - global: - scrape_interval: 15s - evaluation_interval: 15s - - rule_files: - - 'rules.yml' - - alerting: - alertmanagers: - - static_configs: - - targets: - - localhost:9093 - - scrape_configs: - - job_name: 'prometheus' - scrape_interval: 5s - static_configs: - - targets: ['localhost:9090'] - - job_name: 'substrate_node' - scrape_interval: 5s - static_configs: - - targets: ['localhost:9615'] - - ``` - {% endraw %} - -2. Create the rules file for detection and alerts: - - ```bash - sudo nano /etc/prometheus/rules.yml - ``` - - Add a sample rule to trigger email notifications for node downtime over five minutes: - - {% raw %} - ```yml title="rules.yml" - groups: - - name: alert_rules - rules: - - alert: InstanceDown - expr: up == 0 - for: 5m - labels: - severity: critical - annotations: - summary: 'Instance [{{ $labels.instance }}] down' - description: '[{{ $labels.instance }}] of job [{{ $labels.job }}] has been down for more than 5 minutes.' - - ``` - {% endraw %} - - If any of the conditions defined in the rules file are met, an alert will be triggered. For more on alert rules, refer to [Alerting Rules](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/){target=\_blank} and [additional alerts](https://samber.github.io/awesome-prometheus-alerts/rules.html){target=\_blank}. - -3. Update the file ownership to `prometheus`: - - ```bash - sudo chown prometheus:prometheus rules.yml - ``` - -4. Validate the rules syntax: - - ```bash - sudo -u prometheus promtool check rules rules.yml - ``` - -5. Restart Prometheus and Alertmanager: - - ```bash - sudo systemctl restart prometheus && sudo systemctl restart alertmanager - ``` - -Now you will receive an email alert if one of your rule triggering conditions is met. - -## Secure Your Validator - -Validators in Polkadot's Proof of Stake (PoS) network play a critical role in maintaining network integrity and security by keeping the network in consensus and verifying state transitions. To ensure optimal performance and minimize risks, validators must adhere to strict guidelines around security and reliable operations. - -### Key Management - -Though they don't transfer funds, session keys are essential for validators as they sign messages related to consensus and parachains. Securing session keys is crucial as allowing them to be exploited or used across multiple nodes can lead to a loss of staked funds via [slashing](/nodes-and-validators/run-a-validator/staking-mechanics/offenses-and-slashes/){target=\_blank}. - -Given the current limitations in high-availability setups and the risks associated with double-signing, it’s recommended to run only a single validator instance. Keys should be securely managed, and processes automated to minimize human error. - -There are two approaches for generating session keys: - -- **Generate and store in node**: Using the `author.rotateKeys` RPC call. For most users, generating keys directly within the client is recommended. You must submit a session certificate from your staking proxy to register new keys. See the [How to Validate](/nodes-and-validators/run-a-validator/onboarding-and-offboarding/set-up-validator/){target=\_blank} guide for instructions on setting keys. - -- **Generate outside node and insert**: Using the `author.setKeys` RPC call. This flexibility accommodates advanced security setups and should only be used by experienced validator operators. - -### Signing Outside the Client - -Polkadot plans to support external signing, allowing session keys to reside in secure environments like Hardware Security Modules (HSMs). However, these modules can sign any payload they receive, potentially enabling an attacker to perform slashable actions. - -### Secure-Validator Mode - -Polkadot's Secure-Validator mode offers an extra layer of protection through strict filesystem, networking, and process sandboxing. This secure mode is activated by default if the machine meets the following requirements: - -- **Linux (x86-64 architecture)**: Usually Intel or AMD. -- **Enabled `seccomp`**: This kernel feature facilitates a more secure approach for process management on Linux. Verify by running. - - ```bash - cat /boot/config-`uname -r` | grep CONFIG_SECCOMP= - ``` - - If `seccomp` is enabled, you should see output similar to the following: - - ```bash - CONFIG_SECCOMP=y - ``` - -!!! tip - Optionally, **Linux 5.13** may also be used, as it provides access to even more strict filesystem protections. - -### Linux Best Practices - -Follow these best practices to keep your validator secure: - -- Use a non-root user for all operations. -- Regularly apply OS security patches. -- Enable and configure a firewall. -- Use key-based SSH authentication; deactivate password-based login. -- Regularly back up data and harden your SSH configuration. Visit this [SSH guide](https://blog.stribik.technology/2015/01/04/secure-secure-shell.html){target=\_blank} for more details. - -### Validator Best Practices - -Additional best practices can add an additional layer of security and operational reliability: - -- Only run the Polkadot binary, and only listen on the configured p2p port. -- Run on bare-metal machines, as opposed to virtual machines. -- Provisioning of the validator machine should be automated and defined in code which is kept in private version control, reviewed, audited, and tested. -- Generate and provide session keys in a secure way. -- Start Polkadot at boot and restart if stopped for any reason. -- Run Polkadot as a non-root user. -- Establish and maintain an on-call rotation for managing alerts. -- Establish and maintain a clear protocol with actions to perform for each level of each alert with an escalation policy. - -## Additional Resources - -- [Certus One's Knowledge Base](https://knowledgebase.certus.com/FAQ/){target=\_blank} -- [EOS Block Producer Security List](https://github.com/slowmist/eos-bp-nodes-security-checklist){target=\_blank} -- [HSM Policies and the Importance of Validator Security](https://medium.com/loom-network/hsm-policies-and-the-importance-of-validator-security-ec8a4cc1b6f){target=\_blank} - -For additional guidance, connect with other validators and the Polkadot engineering team in the [Polkadot Validator Lounge](https://matrix.to/#/#polkadotvalidatorlounge:web3.foundation){target=\_blank} on Element. diff --git a/.ai/pages/nodes-and-validators-run-a-validator-operational-tasks-pause-validating.md b/.ai/pages/nodes-and-validators-run-a-validator-operational-tasks-pause-validating.md deleted file mode 100644 index f29841b3e..000000000 --- a/.ai/pages/nodes-and-validators-run-a-validator-operational-tasks-pause-validating.md +++ /dev/null @@ -1,46 +0,0 @@ ---- -title: Pause Validating -description: Learn how to temporarily pause staking activity in Polkadot using the chill extrinsic, with guidance for validators and nominators. -categories: Infrastructure -url: https://docs.polkadot.com/nodes-and-validators/run-a-validator/operational-tasks/pause-validating/ ---- - -# Pause Validating - -## Introduction - -If you need to temporarily stop participating in Polkadot staking activities without fully unbonding your funds, chilling your account allows you to do so efficiently. Chilling removes your node from active validation or nomination in the next era while keeping your funds bonded, making it ideal for planned downtimes or temporary pauses. - -This guide covers the steps for chilling as a validator or nominator, using the `chill` and `chillOther` extrinsics, and how these affect your staking status and nominations. - -## Chilling Your Node - -If you need to temporarily step back from staking without unbonding your funds, you can "chill" your account. Chilling pauses your active staking participation, setting your account to inactive in the next era while keeping your funds bonded. - -To chill your account, go to the **Network > Staking > Account Actions** page on [Polkadot.js Apps](https://polkadot.js.org/apps){target=\_blank}, and select **Stop**. Alternatively, you can call the [`chill`](https://paritytech.github.io/polkadot-sdk/master/pallet_staking/enum.Call.html#variant.chill){target=\_blank} extrinsic in the Staking pallet. - -## Staking Election Timing Considerations - -When a node actively participates in staking but then chills, it will continue contributing for the remainder of the current era. However, its eligibility for the next election depends on the chill status at the start of the new era: - -- **Chilled during previous era**: Will not participate in the current era election and will remain inactive until reactivated. -- **Chilled during current era**: Will not be selected for the next era's election. -- **Chilled after current era**: May be selected if it was active during the previous era and is now chilled. - -## Chilling as a Nominator - -When you choose to chill as a nominator, your active nominations are reset. Upon re-entering the nominating process, you must reselect validators to support manually. Depending on preferences, these can be the same validators as before or a new set. Remember that your previous nominations won’t be saved or automatically reactivated after chilling. - -While chilled, your nominator account remains bonded, preserving your staked funds without requiring a full unbonding process. When you’re ready to start nominating again, you can issue a new nomination call to activate your bond with a fresh set of validators. This process bypasses the need for re-bonding, allowing you to maintain your stake while adjusting your involvement in active staking. - -## Chilling as a Validator - -When you chill as a validator, your active validator status is paused. Although your nominators remain bonded to you, the validator bond will no longer appear as an active choice for new or revised nominations until reactivated. Any existing nominators who take no action will still have their stake linked to the validator, meaning they don’t need to reselect the validator upon reactivation. However, if nominators adjust their stakes while the validator is chilled, they will not be able to nominate the chilled validator until it resumes activity. - -Upon reactivating as a validator, you must also reconfigure your validator preferences, such as commission rate and other parameters. These can be set to match your previous configuration or updated as desired. This step is essential for rejoining the active validator set and regaining eligibility for nominations. - -## Chill Other - -Historical constraints in the runtime prevented unlimited nominators and validators from being supported. These constraints created a need for checks to keep the size of the staking system manageable. One of these checks is the `chillOther` extrinsic, allowing users to chill accounts that no longer met standards such as minimum staking requirements set through on-chain governance. - -This control mechanism included a `ChillThreshold`, which was structured to define how close to the maximum number of nominators or validators the staking system would be allowed to get before users could start chilling one another. With the passage of [Referendum #90](https://polkadot-old.polkassembly.io/referendum/90){target=\_blank}, the value for `maxNominatorCount` on Polkadot was set to `None`, effectively removing the limit on how many nominators and validators can participate. This means the `ChillThreshold` will never be met; thus, `chillOther` no longer has any effect. diff --git a/.ai/pages/nodes-and-validators-run-a-validator-operational-tasks-upgrade-your-node.md b/.ai/pages/nodes-and-validators-run-a-validator-operational-tasks-upgrade-your-node.md deleted file mode 100644 index 5e26ac7a4..000000000 --- a/.ai/pages/nodes-and-validators-run-a-validator-operational-tasks-upgrade-your-node.md +++ /dev/null @@ -1,75 +0,0 @@ ---- -title: Upgrade a Validator Node -description: Guide to seamlessly upgrading your Polkadot validator node, managing session keys, and executing server maintenance while avoiding downtime and slashing risks. -categories: Infrastructure -url: https://docs.polkadot.com/nodes-and-validators/run-a-validator/operational-tasks/upgrade-your-node/ ---- - -# Upgrade a Validator Node - -## Introduction - -Upgrading a Polkadot validator node is essential for staying current with network updates and maintaining optimal performance. This guide covers routine and extended maintenance scenarios, including software upgrades and major server changes. Following these steps, you can manage session keys and transition smoothly between servers without risking downtime, slashing, or network disruptions. The process requires strategic planning, especially if you need to perform long-lead maintenance, ensuring your validator remains active and compliant. - -This guide will allow validators to seamlessly substitute an active validator server to allow for maintenance operations. The process can take several hours, so ensure you understand the instructions first and plan accordingly. - -## Prerequisites - -Before beginning the upgrade process for your validator node, ensure the following: - -- You have a fully functional validator setup with all required binaries installed. See [Set Up a Validator](/nodes-and-validators/run-a-validator/onboarding-and-offboarding/set-up-validator/){target=\_blank} and [Validator Requirements](/nodes-and-validators/run-a-validator/requirements/){target=\_blank} for additional guidance. -- Your VPS infrastructure has enough capacity to run a secondary validator instance temporarily for the upgrade process. - -## Session Keys - -Session keys are used to sign validator operations and establish a connection between your validator node and your staking proxy account. These keys are stored in the client, and any change to them requires a waiting period. Specifically, if you modify your session keys, the change will take effect only after the current session is completed and two additional sessions have passed. - -Remembering this delayed effect when planning upgrades is crucial to ensure that your validator continues to function correctly and avoids interruptions. To learn more about session keys and their importance, visit the [Keys section](https://wiki.polkadot.com/learn/learn-cryptography/#keys){target=\_blank}. - -## Keystore - -Your validator server's `keystore` folder holds the private keys needed for signing network-level transactions. It is important not to duplicate or transfer this folder between validator instances. Doing so could result in multiple validators signing with the duplicate keys, leading to severe consequences such as [equivocation slashing](/nodes-and-validators/run-a-validator/staking-mechanics/offenses-and-slashes/#equivocation-slash){target=\_blank}. Instead, always generate new session keys for each validator instance. - -The default path to the `keystore` is as follows: - -```bash -/home/polkadot/.local/share/polkadot/chains//keystore -``` - -Taking care to manage your keys securely ensures that your validator operates safely and without the risk of slashing penalties. - -## Upgrade Using Backup Validator - -The following instructions outline how to temporarily switch between two validator nodes. The original active validator is referred to as Validator A and the backup node used for maintenance purposes as Validator B. - -### Session `N` - -1. **Start Validator B**: Launch a secondary node and wait until it is fully synced with the network. Once synced, start it with the `--validator` flag. This node will now act as Validator B. -2. **Generate session keys**: Create new session keys specifically for Validator B. -3. **Submit the `set_key` extrinsic**: Use your staking proxy account to submit a `set_key` extrinsic, linking the session keys for Validator B to your staking setup. -4. **Record the session**: Make a note of the session in which you executed this extrinsic. -5. **Wait for session changes**: Allow the current session to end and then wait for two additional full sessions for the new keys to take effect. - -!!! warning "Keep Validator A running" - - It is crucial to keep Validator A operational during this entire waiting period. Since `set_key` does not take effect immediately, turning off Validator A too early may result in chilling or even slashing. - -### Session `N+3` - -At this stage, Validator B becomes your active validator. You can now safely perform any maintenance tasks on Validator A. - -Complete the following steps when you are ready to bring Validator A back online: - -1. **Start Validator A**: Launch Validator A, sync the blockchain database, and ensure it is running with the `--validator` flag. -2. **Generate new session keys for Validator A**: Create fresh session keys for Validator A. -3. **Submit the `set_key` extrinsic**: Using your staking proxy account, submit a `set_key` extrinsic with the new Validator A session keys. -4. **Record the session**: Again, make a note of the session in which you executed this extrinsic. - -Keep Validator B active until the session during which you executed the `set-key` extrinsic completes plus two additional full sessions have passed. Once Validator A has successfully taken over, you can safely stop Validator B. This process helps ensure a smooth handoff between nodes and minimizes the risk of downtime or penalties. Verify the transition by checking for finalized blocks in the new session. The logs should indicate the successful change, similar to the example below: - -
- INSERT_COMMAND - 2019-10-28 21:44:13 Applying authority set change scheduled at block #450092 - 2019-10-28 21:44:13 Applying GRANDPA set change to new set with 20 authorities - -
diff --git a/.ai/pages/nodes-and-validators-run-a-validator-requirements.md b/.ai/pages/nodes-and-validators-run-a-validator-requirements.md deleted file mode 100644 index cf6fad411..000000000 --- a/.ai/pages/nodes-and-validators-run-a-validator-requirements.md +++ /dev/null @@ -1,85 +0,0 @@ ---- -title: Validator Requirements -description: Explore the technical and system requirements for running a Polkadot validator, including setup, hardware, staking prerequisites, and security best practices. -categories: Infrastructure -url: https://docs.polkadot.com/nodes-and-validators/run-a-validator/requirements/ ---- - -# Validator Requirements - -## Introduction - -Running a validator in the Polkadot ecosystem is essential for maintaining network security and decentralization. Validators are responsible for validating transactions and adding new blocks to the chain, ensuring the system operates smoothly. In return for their services, validators earn rewards. However, the role comes with inherent risks, such as slashing penalties for misbehavior or technical failures. If you’re new to validation, starting on Kusama provides a lower-stakes environment to gain valuable experience before progressing to the Polkadot network. - -This guide covers everything you need to know about becoming a validator, including system requirements, staking prerequisites, and infrastructure setup. Whether you’re deploying on a VPS or running your node on custom hardware, you’ll learn how to optimize your validator for performance and security, ensuring compliance with network standards while minimizing risks. - -## Prerequisites - -Running a validator requires solid system administration skills and a secure, well-maintained infrastructure. Below are the primary requirements you need to be aware of before getting started: - -- **System administration expertise**: Handling technical anomalies and maintaining node infrastructure is critical. Validators must be able to troubleshoot and optimize their setup. -- **Security**: Ensure your setup follows best practices for securing your node. Refer to the [Secure Your Validator](/nodes-and-validators/run-a-validator/operational-tasks/general-management/#secure-your-validator){target=\_blank} section to learn about important security measures. -- **Network choice**: Start with [Kusama](/nodes-and-validators/run-a-validator/onboarding-and-offboarding/set-up-validator/#run-a-kusama-validator){target=\_blank} to gain experience. Look for "Adjustments for Kusama" throughout these guides for tips on adapting the provided instructions for the Kusama network. -- **Staking requirements**: A minimum amount of native token (KSM or DOT) is required to be elected into the validator set. The required stake can come from your own holdings or from nominators. -- **Risk of slashing**: Any DOT you stake is at risk if your setup fails or your validator misbehaves. If you’re unsure of your ability to maintain a reliable validator, consider nominating your DOT to a trusted validator. - -## Minimum Hardware Requirements - -Polkadot validators rely on high-performance hardware to process blocks efficiently. The recommended minimum hardware requirements to ensure a fully functional and performant validator are as follows: - -- CPU: - - - x86-64 compatible. - - Eight physical cores @ 3.4 GHz. - - Processor: - - **Intel**: Ice Lake or newer (Xeon or Core series) - - **AMD**: Zen3 or newer (EPYC or Ryzen) - - Simultaneous multithreading disabled: - - **Intel**: Hyper-Threading - - **AMD**: SMT - - [Single-threaded performance](https://www.cpubenchmark.net/singleThread.html){target=\_blank} is prioritized over higher cores count. - -- Storage: - - - **NVMe SSD**: At least 2 TB for blockchain data recommended (prioritize latency rather than throughput). - - Storage requirements will increase as the chain grows. For current estimates, see the [current chain snapshot](https://stakeworld.io/docs/dbsize){target=\_blank}. - -- Memory: - - - 32 GB DDR4 ECC - -- Network: - - - Symmetric networking speed of 500 Mbit/s is required to handle large numbers of parachains and ensure congestion control during peak times. - -## VPS Provider List - -When selecting a VPS provider for your validator node, prioritize reliability, consistent performance, and adherence to the specific hardware requirements set for Polkadot validators. The following server types have been tested and showed acceptable performance in benchmark tests. However, this is not an endorsement and actual performance may vary depending on your workload and VPS provider. - -Be aware that some providers may overprovision the underlying host and use shared storage such as NVMe over TCP, which appears as local storage. These setups might result in poor or inconsistent performance. Benchmark your infrastructure before deploying. - -- **[Google Cloud Platform (GCP)](https://cloud.google.com/){target=\_blank}**: `c2` and `c2d` machine families offer high-performance configurations suitable for validators. -- **[Amazon Web Services (AWS)](https://aws.amazon.com/){target=\_blank}**: `c6id` machine family provides strong performance, particularly for I/O-intensive workloads. -- **[OVH](https://www.ovhcloud.com/en-au/){target=\_blank}**: Can be a budget-friendly solution if it meets your minimum hardware specifications. -- **[Digital Ocean](https://www.digitalocean.com/){target=\_blank}**: Popular among developers, Digital Ocean's premium droplets offer configurations suitable for medium to high-intensity workloads. -- **[Vultr](https://www.vultr.com/){target=\_blank}**: Offers flexibility with plans that may meet validator requirements, especially for high-bandwidth needs. -- **[Linode](https://www.linode.com/){target=\_blank}**: Provides detailed documentation, which can be helpful for setup. -- **[Scaleway](https://www.scaleway.com/en/){target=\_blank}**: Offers high-performance cloud instances that can be suitable for validator nodes. -- **[OnFinality](https://onfinality.io/en){target=\_blank}**: Specialized in blockchain infrastructure, OnFinality provides validator-specific support and configurations. - -!!! warning "Acceptable use policies" - Different VPS providers have varying acceptable use policies, and not all allow cryptocurrency-related activities. - - For example, Digital Ocean, requires explicit permission to use servers for cryptocurrency mining and defines unauthorized mining as [network abuse](https://www.digitalocean.com/legal/acceptable-use-policy#network-abuse){target=\_blank} in their acceptable use policy. - - Review the terms for your VPS provider to avoid account suspension or server shutdown due to policy violations. - -## Minimum Bond Requirement - -Before bonding DOT, ensure you meet the minimum bond requirement to start a validator instance. The minimum bond is the least DOT you need to stake to enter the validator set. To become eligible for rewards, your validator node must be nominated by enough staked tokens. - -For example, on November 19, 2024, the minimum stake backing a validator in Polkadot's era 1632 was 1,159,434.248 DOT. You can check the current minimum stake required using these tools: - -- [**Chain State Values**](https://wiki.polkadot.com/general/chain-state-values/){target=\_blank} -- [**Subscan**](https://polkadot.subscan.io/validator_list?status=validator){target=\_blank} -- [**Staking Dashboard**](https://staking.polkadot.cloud/#/overview){target=\_blank} diff --git a/.ai/pages/nodes-and-validators-run-a-validator-staking-mechanics-offenses-and-slashes.md b/.ai/pages/nodes-and-validators-run-a-validator-staking-mechanics-offenses-and-slashes.md deleted file mode 100644 index 48e739f9e..000000000 --- a/.ai/pages/nodes-and-validators-run-a-validator-staking-mechanics-offenses-and-slashes.md +++ /dev/null @@ -1,171 +0,0 @@ ---- -title: Offenses and Slashes -description: Learn about how Polkadot discourages validator misconduct via an offenses and slashing system, including details on offenses and their consequences. -categories: Infrastructure -url: https://docs.polkadot.com/nodes-and-validators/run-a-validator/staking-mechanics/offenses-and-slashes/ ---- - -# Offenses and Slashes - -## Introduction - -In Polkadot's Nominated Proof of Stake (NPoS) system, validator misconduct is deterred through a combination of slashing, disabling, and reputation penalties. Validators and nominators who stake tokens face consequences for validator misbehavior, which range from token slashes to restrictions on network participation. - -This page outlines the types of offenses recognized by Polkadot, including block equivocations and invalid votes, as well as the corresponding penalties. While some parachains may implement additional custom slashing mechanisms, this guide focuses on the offenses tied to staking within the Polkadot ecosystem. - -## Offenses - -Polkadot is a public permissionless network. As such, it has a mechanism to disincentivize offenses and incentivize good behavior. You can review the [parachain protocol](https://wiki.polkadot.com/learn/learn-parachains-protocol/#parachain-protocol){target=\_blank} to understand better the terminology used to describe offenses. Polkadot validator offenses fall into two categories: invalid votes and equivocations. - -### Invalid Votes - -A validator will be penalized for inappropriate voting activity during the block inclusion and approval processes. The invalid voting related offenses are as follows: - -- **Backing an invalid block**: A para-validator backs an invalid block for inclusion in a fork of the relay chain. -- **`ForInvalid` vote**: When acting as a secondary checker, the validator votes in favor of an invalid block. -- **`AgainstValid` vote**: When acting as a secondary checker, the validator votes against a valid block. This type of vote wastes network resources required to resolve the disparate votes and resulting dispute. - -### Equivocations - -Equivocation occurs when a validator produces statements that conflict with each other when producing blocks or voting. Unintentional equivocations usually occur when duplicate signing keys reside on the validator host. If keys are never duplicated, the probability of an honest equivocation slash decreases to near zero. The equivocation related offenses are as follows: - -- **Equivocation**: The validator produces two or more of the same block or vote. - - **GRANDPA and BEEFY equivocation**: The validator signs two or more votes in the same round on different chains. - - **BABE equivocation**: The validator produces two or more blocks on the relay chain in the same time slot. -- **Double seconded equivocation**: The validator attempts to second, or back, more than one block in the same round. -- **Seconded and valid equivocation**: The validator seconds, or backs, a block and then attempts to hide their role as the responsible backer by later placing a standard validation vote. - -## Penalties - -On Polkadot, offenses to the network incur different penalties depending on severity. There are three main penalties: slashing, disabling, and reputation changes. - -### Slashing - -Validators engaging in bad actor behavior in the network may be subject to slashing if they commit a qualifying offense. When a validator is slashed, they and their nominators lose a percentage of their staked DOT or KSM, from as little as 0.01% up to 100% based on the severity of the offense. Nominators are evaluated for slashing against their active validations at any given time. Validator nodes are evaluated as discrete entities, meaning an operator can't attempt to mitigate the offense on another node they operate in order to avoid a slash. - -Any slashed DOT or KSM will be added to the [Treasury](https://wiki.polkadot.com/learn/learn-polkadot-opengov-treasury/){target=\_blank} rather than burned or distributed as rewards. Moving slashed funds to the Treasury allows tokens to be quickly moved away from malicious validators while maintaining the ability to revert faulty slashes when needed. - -A nominator with a very large bond may nominate several validators in a single era. In this case, a slash is proportionate to the amount staked to the offending validator. Stake allocation and validator activation is controlled by the [Phragmén algorithm](https://wiki.polkadot.com/learn/learn-phragmen/#algorithm){target=\_blank}. - -A validator slash creates an `unapplied` state transition. You can view pending slashes on [Polkadot.js Apps](https://polkadot.js.org/apps/?rpc=wss%3A%2F%2Frpc.polkadot.io#/staking/slashes){target=\_blank}. The UI will display the slash per validator, the affected nominators, and the slash amounts. The unapplied state includes a 27-day grace period during which a governance proposal can be made to reverse the slash. Once this grace period expires, the slash is applied. - -#### Equivocation Slash - -The Web3 Foundation's [Slashing mechanisms](https://research.web3.foundation/Polkadot/security/slashing/amounts){target=\_blank} page provides guidelines for evaluating the security threat level of different offenses and determining penalties proportionate to the threat level of the offense. Offenses requiring coordination between validators or extensive computational costs to the system will typically call for harsher penalties than those more likely to be unintentional than malicious. A description of potential offenses for each threat level and the corresponding penalties is as follows: - -- **Level 1**: Honest misconduct such as isolated cases of unresponsiveness. - - **Penalty**: Validator can be kicked out or slashed up to 0.1% of stake in the validator slot. -- **Level 2**: Misconduct that can occur honestly but is a sign of bad practices. Examples include repeated cases of unresponsiveness and isolated cases of equivocation. - - **Penalty**: Slash of up to 1% of stake in the validator slot. -- **Level 3**: Misconduct that is likely intentional but of limited effect on the performance or security of the network. This level will typically include signs of coordination between validators. Examples include repeated cases of equivocation or isolated cases of unjustified voting on GRANDPA. - - **Penalty**: Reduction in networking reputation metrics, slash of up to 10% of stake in the validator slot. -- **Level 4**: Misconduct that poses severe security or monetary risk to the system or mass collusion. Examples include signs of extensive coordination, creating a serious security risk to the system, or forcing the system to use extensive resources to counter the misconduct. - - **Penalty**: Slash of up to 100% of stake in the validator slot. - -See the next section to understand how slash amounts for equivocations are calculated. If you want to know more details about slashing, please look at the research page on [Slashing mechanisms](https://research.web3.foundation/Polkadot/security/slashing/amounts){target=\_blank}. - -#### Slash Calculation for Equivocation - -The slashing penalty for GRANDPA, BABE, and BEEFY equivocations is calculated using the formula below, where `x` represents the number of offenders and `n` is the total number of validators in the active set: - -```text -min((3 * x / n )^2, 1) -``` - -The following scenarios demonstrate how this formula means slash percentages can increase exponentially based on the number of offenders involved compared to the size of the validator pool: - -- **Minor offense**: Assume 1 validator out of a 100 validator active set equivocates in a slot. A single validator committing an isolated offense is most likely a mistake rather than malicious attack on the network. This offense results in a 0.09% slash to the stake in the validator slot. - - ``` mermaid - flowchart LR - N["Total Validators = 100"] - X["Offenders = 1"] - F["min((3 * 1 / 100)^2, 1) = 0.0009"] - G["0.09% slash of stake"] - - N --> F - X --> F - F --> G - ``` - -- **Moderate offense**: Assume 5 validators out a 100 validator active set equivocate in a slot. This is a slightly more serious event as there may be some element of coordination involved. This offense results in a 2.25% slash to the stake in the validator slot. - - ``` mermaid - flowchart LR - N["Total Validators = 100"] - X["Offenders = 5"] - F["min((3 * 5 / 100)^2, 1) = 0.0225"] - G["2.25% slash of stake"] - - N --> F - X --> F - F --> G - ``` - -- **Major offense**: Assume 20 validators out a 100 validator active set equivocate in a slot. This is a major security threat as it possible represents a coordinated attack on the network. This offense results in a 36% slash and all slashed validators will also be chilled. - ``` mermaid - flowchart LR - N["Total Validators = 100"] - X["Offenders = 20"] - F["min((3 * 20 / 100)^2, 1) = 0.36"] - G["36% slash of stake"] - - N --> F - X --> F - F --> G - ``` - -The examples above show the risk of nominating or running many validators in the active set. While rewards grow linearly (two validators will get you approximately twice as many staking rewards as one), slashing grows exponentially. Going from a single validator equivocating to two validators equivocating causes a slash four time as much as the single validator. - -Validators may run their nodes on multiple machines to ensure they can still perform validation work if one of their nodes goes down. Still, validator operators should be cautious when setting these up. Equivocation is possible if they don't coordinate well in managing signing machines. - -#### Best Practices to Avoid Slashing - -The following are advised to node operators to ensure that they obtain pristine binaries or source code and to ensure the security of their node: - -- Always download either source files or binaries from the official Parity repository. -- Verify the hash of downloaded files. -- Use the W3F secure validator setup or adhere to its principles. -- Ensure essential security items are checked, use a firewall, manage user access, use SSH certificates. -- Avoid using your server as a general-purpose system. Hosting a validator on your workstation or one that hosts other services increases the risk of maleficence. -- Avoid cloning servers (copying all contents) when migrating to new hardware. If an image is needed, create it before generating keys. -- High Availability (HA) systems are generally not recommended as equivocation may occur if concurrent operations happen—such as when a failed server restarts or two servers are falsely online simultaneously. -- Copying the keystore folder when moving a database between instances can cause equivocation. Even brief use of duplicated keystores can result in slashing. - -Below are some examples of small equivocations that happened in the past: - -| Network | Era | Event Type | Details | Action Taken | -|----------|------|--------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------| -| Polkadot | 774 | Small Equivocation | [The validator](https://matrix.to/#/!NZrbtteFeqYKCUGQtr:matrix.parity.io/$165562246360408hKCfC:matrix.org?via=matrix.parity.io&via=corepaper.org&via=matrix.org){target=\_blank} migrated servers and cloned the keystore folder. The on-chain event can be viewed on [Subscan](https://polkadot.subscan.io/extrinsic/11190109-0?event=11190109-5){target=\_blank}. | The validator didn't submit a request for the slash to be canceled. | -| Kusama | 3329 | Small Equivocation | The validator operated a test machine with cloned keys. The test machine was online simultaneously as the primary, which resulted in a slash. | The validator requested a slash cancellation, but the council declined. | -| Kusama | 3995 | Small Equivocation | The validator noticed several errors, after which the client crashed, and a slash was applied. The validator recorded all events and opened GitHub issues to allow for technical opinions to be shared. | The validator requested to cancel the slash. The council approved the request as they believed the error wasn't operator-related. | - -#### Slashing Across Eras - -There are three main difficulties to account for with slashing in NPoS: - -- A nominator can nominate multiple validators and be slashed as a result of actions taken by any of them. -- Until slashed, the stake is reused from era to era. -- Slashable offenses can be found after the fact and out of order. - -To balance this, the system applies only the maximum slash a participant can receive in a given time period rather than the sum. This ensures protection from excessive slashing. - -### Disabling - -The disabling mechanism is triggered when validators commit serious infractions, such as backing invalid blocks or engaging in equivocations. Disabling stops validators from performing specific actions after they have committed an offense. Disabling is further divided into: - -- **On-chain disabling**: Lasts for a whole era and stops validators from authoring blocks, backing, and initiating a dispute. -- **Off-chain disabling**: Lasts for a session, is caused by losing a dispute, and stops validators from initiating a dispute. - -Off-chain disabling is always a lower priority than on-chain disabling. Off-chain disabling prioritizes disabling first backers and then approval checkers. - -The material in this guide reflects the changes introduced in Stage 4. For more details, see the [State of Disabling issue](https://github.com/paritytech/polkadot-sdk/issues/4359){target=\_blank} on GitHub. - - -### Reputation Changes - -Some minor offenses, such as spamming, are only punished by networking reputation changes. Validators use a reputation metric when choosing which peers to connect with. The system adds reputation if a peer provides valuable data and behaves appropriately. If they provide faulty or spam data, the system reduces their reputation. If a validator loses enough reputation, their peers will temporarily close their channels to them. This helps in fighting against Denial of Service (DoS) attacks. Performing validator tasks under reduced reputation will be harder, resulting in lower validator rewards. - -### Penalties by Offense - -Refer to the Polkadot Wiki's [offenses page](https://wiki.polkadot.com/learn/learn-offenses/){target=\_blank} for a summary of penalties for specific offenses. diff --git a/.ai/pages/nodes-and-validators-run-a-validator-staking-mechanics-rewards.md b/.ai/pages/nodes-and-validators-run-a-validator-staking-mechanics-rewards.md deleted file mode 100644 index d301d24c4..000000000 --- a/.ai/pages/nodes-and-validators-run-a-validator-staking-mechanics-rewards.md +++ /dev/null @@ -1,204 +0,0 @@ ---- -title: Rewards Payout -description: Learn how validator rewards work on the network, including era points, payout distribution, running multiple validators, and nominator payments. -categories: Infrastructure -url: https://docs.polkadot.com/nodes-and-validators/run-a-validator/staking-mechanics/rewards/ ---- - -# Rewards Payout - -## Introduction - -Understanding how rewards are distributed to validators and nominators is essential for network participants. In Polkadot and Kusama, validators earn rewards based on their era points, which are accrued through actions like block production and parachain validation. - -This guide explains the payout scheme, factors influencing rewards, and how multiple validators affect returns. Validators can also share rewards with nominators, who contribute by staking behind them. By following the payout mechanics, validators can optimize their earnings and better engage with their nominators. - -## Era Points - -The Polkadot ecosystem measures its reward cycles in a unit called an era. Kusama eras are approximately 6 hours long, and Polkadot eras are 24 hours long. At the end of each era, validators are paid proportionally to the amount of era points they have collected. Era points are reward points earned for payable actions like: - -- Issuing validity statements for [parachain blocks](/reference/parachains/blocks-transactions-fees/blocks/){target=\_blank}. -- Producing a non-uncle block in the relay chain. -- Producing a reference to a previously unreferenced uncle block. -- Producing a referenced uncle block. - -An uncle block is a relay chain block that is valid in every regard but has failed to become canonical. This can happen when two or more validators are block producers in a single slot, and the block produced by one validator reaches the next block producer before the others. The lagging blocks are called uncle blocks. - -## Reward Variance - -Rewards in Polkadot and Kusama staking systems can fluctuate due to differences in era points earned by para-validators and non-para-validators. Para-validators generally contribute more to the overall reward distribution due to their role in validating parachain blocks, thus influencing the variance in staking rewards. - -To illustrate this relationship: - -- Para-validator era points tend to have a higher impact on the expected value of staking rewards compared to non-para-validator points. -- The variance in staking rewards increases as the total number of validators grows relative to the number of para-validators. -- In simpler terms, when more validators are added to the active set without increasing the para-validator pool, the disparity in rewards between validators becomes more pronounced. - -However, despite this increased variance, rewards tend to even out over time due to the continuous rotation of para-validators across eras. The network's design ensures that over multiple eras, each validator has an equal opportunity to participate in para-validation, eventually leading to a balanced distribution of rewards. - -??? interface "Probability in Staking Rewards" - - This should only serve as a high-level overview of the probabilistic nature for staking rewards. - - Let: - - - `pe` = para-validator era points - - `ne` = non-para-validator era points - - `EV` = expected value of staking rewards - - Then, `EV(pe)` has more influence on the `EV` than `EV(ne)`. - - Since `EV(pe)` has a more weighted probability on the `EV`, the increase in variance against the `EV` becomes apparent between the different validator pools (aka. validators in the active set and the ones chosen to para-validate). - - Also, let: - - - `v` = the variance of staking rewards - - `p` = number of para-validators - - `w` = number validators in the active set - - `e` = era - - Then, `v` ↑ if `w` ↑, as this reduces `p` : `w`, with respect to `e`. - - Increased `v` is expected, and initially keeping `p` ↓ using the same para-validator set for all parachains ensures [availability](https://spec.polkadot.network/chapter-anv){target=\_blank} and [voting](https://wiki.polkadot.com/learn/learn-polkadot-opengov/){target=\_blank}. In addition, despite `v` ↑ on an `e` to `e` basis, over time, the amount of rewards each validator receives will equal out based on the continuous selection of para-validators. - - There are plans to scale the active para-validation set in the future. - -## Payout Scheme - -Validator rewards are distributed equally among all validators in the active set, regardless of the total stake behind each validator. However, individual payouts may differ based on the number of era points a validator has earned. Although factors like network connectivity can affect era points, well-performing validators should accumulate similar totals over time. - -Validators can also receive tips from users, which incentivize them to include certain transactions in their blocks. Validators retain 100% of these tips. - -Rewards are paid out in the network's native token (DOT for Polkadot and KSM for Kusama). - -The following example illustrates a four member validator set with their names, amount they have staked, and how payout of rewards is divided. This scenario assumes all validators earned the same amount of era points and no one received tips: - -``` mermaid -flowchart TD - A["Alice (18 DOT)"] - B["Bob (9 DOT)"] - C["Carol (8 DOT)"] - D["Dave (7 DOT)"] - E["Payout (8 DOT total)"] - E --"2 DOT"--> A - E --"2 DOT"--> B - E --"2 DOT"--> C - E --"2 DOT"--> D -``` - -Note that this is different than most other Proof of Stake (PoS) systems. As long as a validator is in the validator set, it will receive the same block reward as every other validator. Validator Alice, who had 18 DOT staked, received the same 2 DOT reward in this era as Dave, who had only 7 DOT staked. - -## Running Multiple Validators - -Running multiple validators can offer a more favorable risk/reward ratio compared to running a single one. If you have sufficient DOT or nominators staking on your validators, maintaining multiple validators within the active set can yield higher rewards. - -In the preceding section, with 18 DOT staked and no nominators, Alice earned 2 DOT in one era. This example uses DOT, but the same principles apply for KSM on the Kusama network. By managing stake across multiple validators, you can potentially increase overall returns. Recall the set of validators from the preceding section: - -``` mermaid -flowchart TD - A["Alice (18 DOT)"] - B["Bob (9 DOT)"] - C["Carol (8 DOT)"] - D["Dave (7 DOT)"] - E["Payout (8 DOT total)"] - E --"2 DOT"--> A - E --"2 DOT"--> B - E --"2 DOT"--> C - E --"2 DOT"--> D -``` - -Now, assume Alice decides to split their stake and run two validators, each with a nine DOT stake. This validator set only has four spots and priority is given to validators with a larger stake. In this example, Dave has the smallest stake and loses his spot in the validator set. Now, Alice will earn two shares of the total payout each era as illustrated below: - -``` mermaid -flowchart TD - A["Alice (9 DOT)"] - F["Alice (9 DOT)"] - B["Bob (9 DOT)"] - C["Carol (8 DOT)"] - E["Payout (8 DOT total)"] - E --"2 DOT"--> A - E --"2 DOT"--> B - E --"2 DOT"--> C - E --"2 DOT"--> F -``` - -With enough stake, you could run more than two validators. However, each validator must have enough stake behind it to maintain a spot in the validator set. - -## Nominators and Validator Payments - -A nominator's stake allows them to vote for validators and earn a share of the rewards without managing a validator node. Although staking rewards depend on validator activity during an era, validators themselves never control or own nominator rewards. To trigger payouts, anyone can call the `staking.payoutStakers` or `staking.payoutStakerByPage` methods, which mint and distribute rewards directly to the recipients. This trustless process ensures nominators receive their earned rewards. - -Validators set a commission rate as a percentage of the block reward, affecting how rewards are shared with nominators. A 0% commission means the validator keeps only rewards from their self-stake, while a 100% commission means they retain all rewards, leaving none for nominators. - -The following examples model splitting validator payments between nominator and validator using various commission percentages. For simplicity, these examples assume a Polkadot-SDK based relay chain that uses DOT as a native token and a single nominator per validator. Calculations of KSM reward payouts for Kusama follow the same formula. - -Start with the original validator set from the previous section: - -``` mermaid -flowchart TD - A["Alice (18 DOT)"] - B["Bob (9 DOT)"] - C["Carol (8 DOT)"] - D["Dave (7 DOT)"] - E["Payout (8 DOT total)"] - E --"2 DOT"--> A - E --"2 DOT"--> B - E --"2 DOT"--> C - E --"2 DOT"--> D -``` - -The preceding diagram shows each validator receiving a 2 DOT payout, but doesn't account for sharing rewards with nominators. The following diagram shows what nominator payout might look like for validator Alice. Alice has a 20% commission rate and holds 50% of the stake for their validator: - -``` mermaid - -flowchart TD - A["Gross Rewards = 2 DOT"] - E["Commission = 20%"] - F["Alice Validator Payment = 0.4 DOT"] - G["Total Stake Rewards = 1.6 DOT"] - B["Alice Validator Stake = 18 DOT"] - C["9 DOT Alice (50%)"] - H["Alice Stake Reward = 0.8 DOT"] - I["Total Alice Validator Reward = 1.2 DOT"] - D["9 DOT Nominator (50%)"] - J["Total Nominator Reward = 0.8 DOT"] - - A --> E - E --(2 x 0.20)--> F - F --(2 - 0.4)--> G - B --> C - B --> D - C --(1.6 x 0.50)--> H - H --(0.4 + 0.8)--> I - D --(1.60 x 0.50)--> J -``` - -Notice the validator commission rate is applied against the gross amount of rewards for the era. The validator commission is subtracted from the total rewards. After the commission is paid to the validator, the remaining amount is split among stake owners according to their percentage of the total stake. A validator's total rewards for an era include their commission plus their piece of the stake rewards. - -Now, consider a different scenario for validator Bob where the commission rate is 40%, and Bob holds 33% of the stake for their validator: - -``` mermaid - -flowchart TD - A["Gross Rewards = 2 DOT"] - E["Commission = 40%"] - F["Bob Validator Payment = 0.8 DOT"] - G["Total Stake Rewards = 1.2 DOT"] - B["Bob Validator Stake = 9 DOT"] - C["3 DOT Bob (33%)"] - H["Bob Stake Reward = 0.4 DOT"] - I["Total Bob Validator Reward = 1.2 DOT"] - D["6 DOT Nominator (67%)"] - J["Total Nominator Reward = 0.8 DOT"] - - A --> E - E --(2 x 0.4)--> F - F --(2 - 0.8)--> G - B --> C - B --> D - C --(1.2 x 0.33)--> H - H --(0.8 + 0.4)--> I - D --(1.2 x 0.67)--> J -``` - -Bob holds a smaller percentage of their node's total stake, making their stake reward smaller than Alice's. In this scenario, Bob makes up the difference by charging a 40% commission rate and ultimately ends up with the same total payment as Alice. Each validator will need to find their ideal balance between the amount of stake and commission rate to attract nominators while still making running a validator worthwhile. diff --git a/node-infrastructure/run-a-collator/collator.md b/node-infrastructure/run-a-collator/collator.md index 57ba15e1c..805e0a4ca 100644 --- a/node-infrastructure/run-a-collator/collator.md +++ b/node-infrastructure/run-a-collator/collator.md @@ -62,7 +62,7 @@ Required software: - **Operating System**: Ubuntu 22.04 LTS (recommended) or similar Linux distribution - **Docker**: For running subkey utility -- **Rust Toolchain**: Version 1.86 or as specified by the runtime +- **Rust Toolchain**: Version 1.91.1 or as specified by the runtime - **Dependencies**: ```bash sudo apt update @@ -86,17 +86,17 @@ curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh source $HOME/.cargo/env # Install specific Rust version -rustup install 1.86 -rustup default 1.86 -rustup target add wasm32-unknown-unknown --toolchain 1.86 -rustup component add rust-src --toolchain 1.86 +rustup install 1.91.1 +rustup default 1.91.1 +rustup target add wasm32-unknown-unknown --toolchain 1.91.1 +rustup component add rust-src --toolchain 1.91.1 ``` ### Step 2: Install the Polkadot Omni Node ```bash # Install polkadot-omni-node -cargo install --locked polkadot-omni-node@0.5.0 +cargo install --locked polkadot-omni-node@0.11.0 # Verify installation polkadot-omni-node --version @@ -158,7 +158,7 @@ cd runtimes cargo build --release -p asset-hub-polkadot-runtime # Install chain-spec-builder -cargo install --locked staging-chain-spec-builder@10.0.0 +cargo install --locked staging-chain-spec-builder@14.0.0 # Generate chain spec chain-spec-builder create \ diff --git a/node-infrastructure/run-a-node/polkadot-hub-rpc.md b/node-infrastructure/run-a-node/polkadot-hub-rpc.md index 92e5b2528..ee619b4a6 100644 --- a/node-infrastructure/run-a-node/polkadot-hub-rpc.md +++ b/node-infrastructure/run-a-node/polkadot-hub-rpc.md @@ -54,7 +54,7 @@ Required software: - **Operating System**: Ubuntu 22.04 LTS (recommended) or similar Linux distribution - **Docker**: Latest version installed and running (for Docker-based setup) - **rclone**: (Optional but recommended) Command-line program for managing files on cloud storage (https://rclone.org/downloads/) -- **Rust Toolchain**: Version 1.86 or as specified by runtime (for manual build) +- **Rust Toolchain**: Version 1.91.1 or later (for manual build) ## Setup Options @@ -159,7 +159,7 @@ docker run -d --name polkadot-hub-rpc --restart unless-stopped \ -p 30333:30333 \ -v $(pwd)/asset-hub-polkadot.json:/asset-hub-polkadot.json \ -v $(pwd)/my-node-data:/data \ - parity/polkadot-omni-node:stable2506-4 \ + parity/polkadot-omni-node:v1.20.2 \ --name=PolkadotHubRPC \ --base-path=/data \ --chain=/asset-hub-polkadot.json \ @@ -305,7 +305,7 @@ docker rm polkadot-hub-rpc ```bash # Pull latest image -docker pull parity/polkadot-omni-node:stable2506-4 +docker pull parity/polkadot-omni-node:v1.20.2 # Stop and remove old container docker stop polkadot-hub-rpc @@ -328,23 +328,33 @@ curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh source $HOME/.cargo/env # Install specific Rust version -rustup install 1.86 -rustup default 1.86 -rustup target add wasm32-unknown-unknown --toolchain 1.86 -rustup component add rust-src --toolchain 1.86 +rustup install 1.91.1 +rustup default 1.91.1 +rustup target add wasm32-unknown-unknown --toolchain 1.91.1 +rustup component add rust-src --toolchain 1.91.1 ``` -### Step 2: Install the Polkadot Omni Node +### Step 2: Install Required Dependencies + +```bash +# Install system dependencies +sudo apt update +sudo apt install -y build-essential git clang curl libssl-dev llvm libudev-dev make protobuf-compiler +``` + +### Step 3: Install the Polkadot Omni Node ```bash # Install polkadot-omni-node -cargo install --locked polkadot-omni-node@0.7.0 +cargo install --locked polkadot-omni-node@0.11.0 # Verify installation polkadot-omni-node --version ``` -### Step 3: Obtain Chain Specification +**Note**: Compiling polkadot-omni-node from source requires significant RAM (minimum 24GB recommended). The compilation may take 10-15 minutes on systems with adequate resources. + +### Step 4: Obtain Chain Specification Download the Polkadot Hub chain specification: @@ -352,7 +362,7 @@ Download the Polkadot Hub chain specification: curl -L https://raw.githubusercontent.com/paritytech/polkadot-sdk/master/cumulus/parachains/chain-specs/asset-hub-polkadot.json -o asset-hub-polkadot.json ``` -### Step 4: Create User and Directory Structure +### Step 5: Create User and Directory Structure ```bash # Create dedicated user @@ -368,7 +378,7 @@ sudo cp asset-hub-polkadot.json /var/lib/polkadot-hub-rpc/ sudo chown -R polkadot:polkadot /var/lib/polkadot-hub-rpc ``` -### Step 5: Create Systemd Service for Polkadot SDK Node +### Step 6: Create Systemd Service for Polkadot SDK Node Create a service file for the Polkadot SDK RPC node: @@ -418,7 +428,7 @@ LimitNOFILE=65536 WantedBy=multi-user.target ``` -### Step 6: Start Service +### Step 7: Start Service ```bash # Reload systemd @@ -435,7 +445,7 @@ sudo systemctl status polkadot-hub-rpc sudo journalctl -u polkadot-hub-rpc -f ``` -### Step 7: Verify Setup +### Step 8: Verify Setup Use the same verification tests as in the Docker setup (see Step 5 above). @@ -513,7 +523,7 @@ The node handles pruning automatically based on configuration unless running in ```bash # Pull latest image -docker pull parity/polkadot-omni-node:stable2506-4 +docker pull parity/polkadot-omni-node:v1.20.2 # Restart container docker stop polkadot-hub-rpc From ed97e71107b0ebc96599959ea974ffe4ba8571e0 Mon Sep 17 00:00:00 2001 From: Bruno Galvao Date: Thu, 20 Nov 2025 18:01:41 +0700 Subject: [PATCH 04/21] Update content --- ...-infrastructure-run-a-collator-collator.md | 105 +++++---------- ...rastructure-run-a-node-polkadot-hub-rpc.md | 126 +++--------------- ...ructure-run-a-node-system-parachain-rpc.md | 126 +++--------------- .../run-a-collator/collator.md | 105 +++++---------- .../run-a-node/polkadot-hub-rpc.md | 126 +++--------------- .../run-a-node/system-parachain-rpc.md | 126 +++--------------- 6 files changed, 160 insertions(+), 554 deletions(-) diff --git a/.ai/pages/node-infrastructure-run-a-collator-collator.md b/.ai/pages/node-infrastructure-run-a-collator-collator.md index 1f6f96cc6..cb2adbf48 100644 --- a/.ai/pages/node-infrastructure-run-a-collator-collator.md +++ b/.ai/pages/node-infrastructure-run-a-collator-collator.md @@ -42,16 +42,16 @@ Block-producing collators require robust hardware for reliable operation: - **CPU**: 4+ cores (8+ cores recommended for optimal performance) - **Memory**: 32 GB RAM minimum (64 GB recommended) - **Storage**: - - 500 GB+ NVMe SSD for parachain data - - Additional 200+ GB for relay chain pruned database - - Fast disk I/O is critical for block production performance + - 500 GB+ NVMe SSD for parachain data + - Additional 200+ GB for relay chain pruned database + - Fast disk I/O is critical for block production performance - **Network**: - - Public IP address (required) - - 100+ Mbps connection (stable connection critical) - - Open ports: - - 30333 (parachain P2P) - - 30334 (relay chain P2P) - - 9944 (WebSocket RPC - for management) + - Public IP address (required) + - 100+ Mbps connection (stable connection critical) + - Open ports: + - 30333 (parachain P2P) + - 30334 (relay chain P2P) + - 9944 (WebSocket RPC - for management) **Note**: Uptime is critical. Consider redundancy and monitoring to maintain block production reliability. @@ -73,6 +73,7 @@ Required software: ### Account Requirements You'll need: + - **Funded account**: For on-chain transactions and potential bonding - **Session keys**: For collator identification (generated after node setup) - **Node key**: For stable P2P peer ID (recommended) @@ -130,6 +131,7 @@ docker run -it parity/subkey:latest generate --scheme sr25519 ``` Save the output containing: + - Secret phrase (seed) - Keep this secure! - Public key (hex) - Account ID @@ -170,6 +172,7 @@ chain-spec-builder create \ ``` **System Parachain Para IDs:** + - Polkadot Hub: 1000 - Bridge Hub: 1002 - People Chain: 1004 @@ -236,6 +239,7 @@ WantedBy=multi-user.target ``` **Configuration Notes**: + - `--collator`: Enables block production mode - `--node-key-file`: Uses the generated node key for stable peer ID - `--name`: Your collator name (visible in telemetry) @@ -267,22 +271,14 @@ sudo journalctl -u polkadot-collator -f Your collator must sync both the relay chain and parachain before producing blocks. Sync time depends on: + - Network bandwidth - Disk I/O speed - Current chain size The relay chain uses warp sync for faster synchronization. -Monitor sync progress: -```bash -# Check logs for sync status -sudo journalctl -u polkadot-collator -f | grep "Syncing" - -# Wait for messages indicating full sync -# Example: "Idle" or "Imported" messages for both chains -``` - -**Important**: Do not proceed with registration until both chains are fully synced. +**Important**: Do not proceed with registration until both chains are fully synced. Monitor sync progress using the log viewing commands in the [Log Management](#log-management) section. ### Step 3: Generate Session Keys @@ -309,16 +305,19 @@ curl -H "Content-Type: application/json" \ System parachains use different collator selection mechanisms: **Invulnerables List**: + - Fixed list of collators approved through governance - Most common for system parachains - Requires governance proposal and approval **On-chain Selection**: + - Some parachains use pallet-collator-selection - May require bonding tokens - Automatic selection based on criteria **Fellowship Decisions**: + - Technical Fellowship may manage some system parachain collators - Requires Fellowship membership or approval @@ -361,8 +360,8 @@ Once approved (or if using on-chain selection), set your session keys: 3. Select your account 4. Choose `session.setKeys` extrinsic 5. Enter: - - `keys`: Your session keys (from `author_rotateKeys`) - - `proof`: 0x00 (typically) + - `keys`: Your session keys (from `author_rotateKeys`) + - `proof`: 0x00 (typically) 6. Submit and sign the transaction **Using CLI (alternative):** @@ -383,6 +382,7 @@ Some parachains require bonding tokens: #### 5. Await Governance Approval If using invulnerables: + - Wait for governance vote - Monitor forum and announcements - Once approved, you'll be added to the invulnerables list @@ -390,16 +390,7 @@ If using invulnerables: ### Verify Collator Status -Check if your collator is active: - -```bash -# Monitor logs for block production -sudo journalctl -u polkadot-collator -f | grep -i "imported" - -# Look for messages like: -# "Prepared block for proposing" -# "Imported #123" -``` +Check if your collator is active by monitoring logs for block production messages like "Prepared block for proposing" and "Imported #123". See the [Log Management](#log-management) section for log viewing commands. ## Monitoring and Maintenance @@ -412,14 +403,17 @@ sudo journalctl -u polkadot-collator | grep -i "prepared block" ``` **Peer Connections**: -- Maintain 30+ peers for good connectivity + +- Maintain a sufficient amount of peers for good connectivity - Check peer count in logs **Resource Usage**: + - Monitor CPU, RAM, and disk I/O - Set up alerts for high usage **Sync Status**: + - Ensure both chains stay synced - Alert on sync issues @@ -438,6 +432,7 @@ scrape_configs: {% endraw %} Key metrics to monitor: + - `substrate_block_height`: Current block height - `substrate_finalized_height`: Finalized block height - `substrate_peers_count`: Peer connections @@ -446,6 +441,7 @@ Key metrics to monitor: ### Setting Up Alerts Configure alerts for: + - Service failures - Sync issues - Low peer count (< 10 peers) @@ -482,16 +478,18 @@ The node handles pruning automatically. ### Updates and Upgrades **Runtime Upgrades**: + - Automatic via on-chain governance - No manual action required - Monitor announcements for breaking changes **Client Upgrades**: + - Require manual binary update - Subscribe to announcements: - - Polkadot Forum - - Fellowship GitHub - - Matrix channels + - Polkadot Forum + - Fellowship GitHub + - Matrix channels **Upgrade Procedure**: @@ -511,42 +509,11 @@ polkadot-omni-node --version # Restart service sudo systemctl start polkadot-collator -# Monitor logs -sudo journalctl -u polkadot-collator -f +# Verify service is running +sudo systemctl status polkadot-collator ``` -## Security Best Practices - -### Key Management - -- **Secure storage**: Store session keys and account keys securely -- **Never share**: Never share private keys or secret phrases -- **Hardware wallets**: Consider HSM for production -- **Backup**: Keep encrypted backups of keys -- **Rotation**: Plan for key rotation procedures - -### Network Security - -- **Firewall**: Restrict access to necessary ports only -- **SSH**: Use SSH keys, disable password auth -- **VPN**: Consider VPN for administrative access -- **DDoS protection**: Implement if running in cloud - -### System Security - -- **Updates**: Keep OS and software updated -- **Dedicated user**: Never run as root -- **Fail2ban**: Enable for SSH protection -- **Audits**: Regular security audits -- **Minimal services**: Disable unnecessary services - -### Operational Security - -- **Monitoring**: 24/7 monitoring with alerts -- **Backups**: Regular configuration backups -- **Documentation**: Document procedures -- **Incident response**: Have incident response plan -- **Redundancy**: Consider backup collator (standby) +**Note**: For log monitoring, see the [Log Management](#log-management) section. ## Conclusion diff --git a/.ai/pages/node-infrastructure-run-a-node-polkadot-hub-rpc.md b/.ai/pages/node-infrastructure-run-a-node-polkadot-hub-rpc.md index ed95abd5d..80f5ce475 100644 --- a/.ai/pages/node-infrastructure-run-a-node-polkadot-hub-rpc.md +++ b/.ai/pages/node-infrastructure-run-a-node-polkadot-hub-rpc.md @@ -31,20 +31,20 @@ RPC nodes serving production traffic require robust hardware: - **CPU**: 8+ cores (16+ cores for high traffic) - **Memory**: 64 GB RAM minimum (128 GB recommended for high traffic) - **Storage**: - - 500 GB+ NVMe SSD for parachain state (archive nodes require 2-4 TB+) - - Additional 200+ GB for relay chain pruned database - - Fast disk I/O is critical for query performance + - 500 GB+ NVMe SSD for parachain state (archive nodes require 2-4 TB+) + - Additional 200+ GB for relay chain pruned database + - Fast disk I/O is critical for query performance - **Network**: - - Public IP address - - 1 Gbps connection (for high traffic scenarios) - - Stable internet connection with sufficient bandwidth - - Open ports: - - 30333 (parachain P2P) - - 30334 (relay chain P2P) - - 9944 (Polkadot SDK WebSocket RPC) - - 9933 (Polkadot SDK HTTP RPC) - - 9615 (Prometheus metrics - optional) - - Consider DDoS protection and rate limiting for production deployments + - Public IP address + - 1 Gbps connection (for high traffic scenarios) + - Stable internet connection with sufficient bandwidth + - Open ports: + - 30333 (parachain P2P) + - 30334 (relay chain P2P) + - 9944 (Polkadot SDK WebSocket RPC) + - 9933 (Polkadot SDK HTTP RPC) + - 9615 (Prometheus metrics - optional) + - Consider DDoS protection and rate limiting for production deployments **Note**: For development or low-traffic scenarios, you can reduce these requirements proportionally. Consider using a reverse proxy (nginx, Caddy) for production deployments. @@ -117,6 +117,7 @@ rm files.txt ``` **Parameter Explanation**: + - `--transfers 20`: Uses 20 parallel transfers for faster download - `--retries 6`: Automatically retries failed transfers up to 6 times - `--retries-sleep 10s`: Waits 10 seconds between retry attempts @@ -142,6 +143,7 @@ rm files.txt ``` **Alternative Options**: + - Pruned snapshot: `polkadot-rocksdb-prune` (smaller size, recent state) - Archive snapshot: `polkadot-rocksdb-archive` (complete history, larger size) @@ -183,12 +185,14 @@ docker run -d --name polkadot-hub-rpc --restart unless-stopped \ **Critical Configuration Parameters**: **Port Mappings**: + - `9944`: Polkadot SDK RPC endpoint (WebSocket/HTTP) - `9933`: Polkadot SDK HTTP RPC endpoint - `9615`: Prometheus metrics endpoint - `30333/30334`: P2P networking ports **Node Parameters**: + - `--unsafe-rpc-external`: Enables external RPC access - `--rpc-cors=all`: Allows all origins for CORS - `--rpc-methods=safe`: Only allows safe RPC methods @@ -224,19 +228,10 @@ curl -H "Content-Type: application/json" \ ``` **Synchronization Status**: + - **In Progress**: `currentBlock` < `highestBlock` - **Complete**: `currentBlock` = `highestBlock` -**Monitor logs**: - -```bash -# View node logs -docker logs -f polkadot-hub-rpc - -# Filter for sync messages -docker logs polkadot-hub-rpc 2>&1 | grep -i "syncing" -``` - ### Step 5: Verify Setup Let's verify the Polkadot SDK RPC endpoint is working correctly. @@ -244,6 +239,7 @@ Let's verify the Polkadot SDK RPC endpoint is working correctly. #### API Endpoint **Polkadot SDK RPC (Port 9944)**: + - WebSocket: `ws://your-server:9944` - HTTP: `http://your-server:9944` - Purpose: Full Polkadot SDK API access for parachain data @@ -302,18 +298,7 @@ docker start polkadot-hub-rpc docker rm polkadot-hub-rpc ``` -**Update container**: - -```bash -# Pull latest image -docker pull parity/polkadot-omni-node:v1.20.2 - -# Stop and remove old container -docker stop polkadot-hub-rpc -docker rm polkadot-hub-rpc - -# Start new container with same command as above -``` +**Note**: For update procedures, see the [Updates and Upgrades](#updates-and-upgrades) section. --- @@ -479,6 +464,7 @@ sudo journalctl -u polkadot-hub-rpc | grep -i error ### Performance Monitoring Monitor key metrics: + - **Sync status**: Ensure node stays fully synced - **Peer connections**: Maintain 30+ peers for good connectivity - **Resource usage**: Monitor CPU, RAM, and disk I/O @@ -501,6 +487,7 @@ scrape_configs: {% endraw %} **Key Metrics to Monitor**: + - `substrate_block_height`: Current block height - `substrate_finalized_height`: Finalized block height - `substrate_peers_count`: Number of connected peers @@ -551,75 +538,6 @@ cargo install --locked --force polkadot-omni-node@ sudo systemctl start polkadot-hub-rpc ``` -## Security Best Practices - -### Network Security - -1. **Firewall Configuration**: - - Only expose necessary ports - - Use UFW or iptables to restrict access - - Consider IP whitelisting for RPC endpoints - -2. **Reverse Proxy** (Recommended for Production): - - Use nginx or Caddy as reverse proxy - - Enable SSL/TLS (HTTPS/WSS) - - Implement authentication - - Add rate limiting - -Example nginx configuration: - -```nginx -upstream polkadot_sdk_rpc { - server 127.0.0.1:9944; -} - -server { - listen 443 ssl http2; - server_name your-domain.com; - - ssl_certificate /path/to/cert.pem; - ssl_certificate_key /path/to/key.pem; - - # Polkadot SDK RPC - location /polkadot { - proxy_pass http://polkadot_sdk_rpc; - proxy_http_version 1.1; - proxy_set_header Upgrade $http_upgrade; - proxy_set_header Connection "upgrade"; - - # Rate limiting - limit_req zone=rpc_limit burst=10; - } -} -``` - -### RPC Security - -- **Always use `--rpc-methods=safe`**: Prevents dangerous RPC calls -- **Restrict CORS**: Use specific domains instead of `all` in production -- **Set connection limits**: Prevent resource exhaustion -- **Monitor for abuse**: Track unusual patterns -- **Authentication**: Implement API keys or OAuth for production - -### System Security - -- Keep operating system updated -- Use dedicated user accounts (never root) -- Enable fail2ban for SSH protection -- Regular security audits -- Disable unnecessary services -- Use AppArmor or SELinux for additional isolation - -### Monitoring and Alerting - -Set up alerts for: -- Service failures -- Sync issues -- Low peer count (< 10 peers) -- High resource usage -- Unusual RPC traffic patterns -- Database errors - ## Conclusion Running an RPC node for Polkadot Hub provides essential infrastructure for applications and users to interact with the network. By following this guide, you have set up a production-ready RPC node that: diff --git a/.ai/pages/node-infrastructure-run-a-node-system-parachain-rpc.md b/.ai/pages/node-infrastructure-run-a-node-system-parachain-rpc.md index 9d3fc5735..63f5aa7ed 100644 --- a/.ai/pages/node-infrastructure-run-a-node-system-parachain-rpc.md +++ b/.ai/pages/node-infrastructure-run-a-node-system-parachain-rpc.md @@ -43,20 +43,20 @@ RPC nodes serving production traffic require robust hardware: - **CPU**: 8+ cores (16+ cores for high traffic) - **Memory**: 64 GB RAM minimum (128 GB recommended for high traffic) - **Storage**: - - 500 GB+ NVMe SSD for parachain state (archive nodes require 2-4 TB+) - - Additional 200+ GB for relay chain pruned database - - Fast disk I/O is critical for query performance + - 500 GB+ NVMe SSD for parachain state (archive nodes require 2-4 TB+) + - Additional 200+ GB for relay chain pruned database + - Fast disk I/O is critical for query performance - **Network**: - - Public IP address - - 1 Gbps connection (for high traffic scenarios) - - Stable internet connection with sufficient bandwidth - - Open ports: - - 30333 (parachain P2P) - - 30334 (relay chain P2P) - - 9944 (Polkadot SDK WebSocket RPC) - - 9933 (Polkadot SDK HTTP RPC) - - 9615 (Prometheus metrics - optional) - - Consider DDoS protection and rate limiting for production deployments + - Public IP address + - 1 Gbps connection (for high traffic scenarios) + - Stable internet connection with sufficient bandwidth + - Open ports: + - 30333 (parachain P2P) + - 30334 (relay chain P2P) + - 9944 (Polkadot SDK WebSocket RPC) + - 9933 (Polkadot SDK HTTP RPC) + - 9615 (Prometheus metrics - optional) + - Consider DDoS protection and rate limiting for production deployments **Note**: For development or low-traffic scenarios, you can reduce these requirements proportionally. Consider using a reverse proxy (nginx, Caddy) for production deployments. @@ -129,6 +129,7 @@ rm files.txt ``` **Parameter Explanation**: + - `--transfers 20`: Uses 20 parallel transfers for faster download - `--retries 6`: Automatically retries failed transfers up to 6 times - `--retries-sleep 10s`: Waits 10 seconds between retry attempts @@ -156,6 +157,7 @@ rm files.txt ``` **Alternative Options**: + - Pruned snapshot: `polkadot-rocksdb-prune` (smaller size, recent state) - Archive snapshot: `polkadot-rocksdb-archive` (complete history, larger size) @@ -197,12 +199,14 @@ docker run -d --name people-chain-rpc --restart unless-stopped \ **Critical Configuration Parameters**: **Port Mappings**: + - `9944`: Polkadot SDK RPC endpoint (WebSocket/HTTP) - `9933`: Polkadot SDK HTTP RPC endpoint - `9615`: Prometheus metrics endpoint - `30333/30334`: P2P networking ports **Node Parameters**: + - `--unsafe-rpc-external`: Enables external RPC access - `--rpc-cors=all`: Allows all origins for CORS - `--rpc-methods=safe`: Only allows safe RPC methods @@ -238,19 +242,10 @@ curl -H "Content-Type: application/json" \ ``` **Synchronization Status**: + - **In Progress**: `currentBlock` < `highestBlock` - **Complete**: `currentBlock` = `highestBlock` -**Monitor logs**: - -```bash -# View node logs -docker logs -f people-chain-rpc - -# Filter for sync messages -docker logs people-chain-rpc 2>&1 | grep -i "syncing" -``` - ### Step 5: Verify Setup Let's verify the Polkadot SDK RPC endpoint is working correctly. @@ -258,6 +253,7 @@ Let's verify the Polkadot SDK RPC endpoint is working correctly. #### API Endpoint **Polkadot SDK RPC (Port 9944)**: + - WebSocket: `ws://your-server:9944` - HTTP: `http://your-server:9944` - Purpose: Full Polkadot SDK API access for parachain data @@ -332,18 +328,7 @@ docker start people-chain-rpc docker rm people-chain-rpc ``` -**Update container**: - -```bash -# Pull latest image -docker pull parity/polkadot-omni-node:v1.20.2 - -# Stop and remove old container -docker stop people-chain-rpc -docker rm people-chain-rpc - -# Start new container with same command as above -``` +**Note**: For update procedures, see the [Updates and Upgrades](#updates-and-upgrades) section. --- @@ -499,6 +484,7 @@ sudo journalctl -u people-chain-rpc | grep -i error ### Performance Monitoring Monitor key metrics: + - **Sync status**: Ensure node stays fully synced - **Peer connections**: Maintain 30+ peers for good connectivity - **Resource usage**: Monitor CPU, RAM, and disk I/O @@ -521,6 +507,7 @@ scrape_configs: {% endraw %} **Key Metrics to Monitor**: + - `substrate_block_height`: Current block height - `substrate_finalized_height`: Finalized block height - `substrate_peers_count`: Number of connected peers @@ -571,75 +558,6 @@ cargo install --locked --force polkadot-omni-node@ sudo systemctl start people-chain-rpc ``` -## Security Best Practices - -### Network Security - -1. **Firewall Configuration**: - - Only expose necessary ports - - Use UFW or iptables to restrict access - - Consider IP whitelisting for RPC endpoints - -2. **Reverse Proxy** (Recommended for Production): - - Use nginx or Caddy as reverse proxy - - Enable SSL/TLS (HTTPS/WSS) - - Implement authentication - - Add rate limiting - -Example nginx configuration: - -```nginx -upstream polkadot_sdk_rpc { - server 127.0.0.1:9944; -} - -server { - listen 443 ssl http2; - server_name your-domain.com; - - ssl_certificate /path/to/cert.pem; - ssl_certificate_key /path/to/key.pem; - - # Polkadot SDK RPC - location /polkadot { - proxy_pass http://polkadot_sdk_rpc; - proxy_http_version 1.1; - proxy_set_header Upgrade $http_upgrade; - proxy_set_header Connection "upgrade"; - - # Rate limiting - limit_req zone=rpc_limit burst=10; - } -} -``` - -### RPC Security - -- **Always use `--rpc-methods=safe`**: Prevents dangerous RPC calls -- **Restrict CORS**: Use specific domains instead of `all` in production -- **Set connection limits**: Prevent resource exhaustion -- **Monitor for abuse**: Track unusual patterns -- **Authentication**: Implement API keys or OAuth for production - -### System Security - -- Keep operating system updated -- Use dedicated user accounts (never root) -- Enable fail2ban for SSH protection -- Regular security audits -- Disable unnecessary services -- Use AppArmor or SELinux for additional isolation - -### Monitoring and Alerting - -Set up alerts for: -- Service failures -- Sync issues -- Low peer count (< 10 peers) -- High resource usage -- Unusual RPC traffic patterns -- Database errors - ## Conclusion Running an RPC node for system parachains provides critical infrastructure for accessing specialized Polkadot network services. By following this guide, you have set up a production-ready RPC node that: diff --git a/node-infrastructure/run-a-collator/collator.md b/node-infrastructure/run-a-collator/collator.md index 805e0a4ca..8230cbb44 100644 --- a/node-infrastructure/run-a-collator/collator.md +++ b/node-infrastructure/run-a-collator/collator.md @@ -41,16 +41,16 @@ Block-producing collators require robust hardware for reliable operation: - **CPU**: 4+ cores (8+ cores recommended for optimal performance) - **Memory**: 32 GB RAM minimum (64 GB recommended) - **Storage**: - - 500 GB+ NVMe SSD for parachain data - - Additional 200+ GB for relay chain pruned database - - Fast disk I/O is critical for block production performance + - 500 GB+ NVMe SSD for parachain data + - Additional 200+ GB for relay chain pruned database + - Fast disk I/O is critical for block production performance - **Network**: - - Public IP address (required) - - 100+ Mbps connection (stable connection critical) - - Open ports: - - 30333 (parachain P2P) - - 30334 (relay chain P2P) - - 9944 (WebSocket RPC - for management) + - Public IP address (required) + - 100+ Mbps connection (stable connection critical) + - Open ports: + - 30333 (parachain P2P) + - 30334 (relay chain P2P) + - 9944 (WebSocket RPC - for management) **Note**: Uptime is critical. Consider redundancy and monitoring to maintain block production reliability. @@ -72,6 +72,7 @@ Required software: ### Account Requirements You'll need: + - **Funded account**: For on-chain transactions and potential bonding - **Session keys**: For collator identification (generated after node setup) - **Node key**: For stable P2P peer ID (recommended) @@ -129,6 +130,7 @@ docker run -it parity/subkey:latest generate --scheme sr25519 ``` Save the output containing: + - Secret phrase (seed) - Keep this secure! - Public key (hex) - Account ID @@ -169,6 +171,7 @@ chain-spec-builder create \ ``` **System Parachain Para IDs:** + - Polkadot Hub: 1000 - Bridge Hub: 1002 - People Chain: 1004 @@ -235,6 +238,7 @@ WantedBy=multi-user.target ``` **Configuration Notes**: + - `--collator`: Enables block production mode - `--node-key-file`: Uses the generated node key for stable peer ID - `--name`: Your collator name (visible in telemetry) @@ -266,22 +270,14 @@ sudo journalctl -u polkadot-collator -f Your collator must sync both the relay chain and parachain before producing blocks. Sync time depends on: + - Network bandwidth - Disk I/O speed - Current chain size The relay chain uses warp sync for faster synchronization. -Monitor sync progress: -```bash -# Check logs for sync status -sudo journalctl -u polkadot-collator -f | grep "Syncing" - -# Wait for messages indicating full sync -# Example: "Idle" or "Imported" messages for both chains -``` - -**Important**: Do not proceed with registration until both chains are fully synced. +**Important**: Do not proceed with registration until both chains are fully synced. Monitor sync progress using the log viewing commands in the [Log Management](#log-management) section. ### Step 3: Generate Session Keys @@ -308,16 +304,19 @@ curl -H "Content-Type: application/json" \ System parachains use different collator selection mechanisms: **Invulnerables List**: + - Fixed list of collators approved through governance - Most common for system parachains - Requires governance proposal and approval **On-chain Selection**: + - Some parachains use pallet-collator-selection - May require bonding tokens - Automatic selection based on criteria **Fellowship Decisions**: + - Technical Fellowship may manage some system parachain collators - Requires Fellowship membership or approval @@ -360,8 +359,8 @@ Once approved (or if using on-chain selection), set your session keys: 3. Select your account 4. Choose `session.setKeys` extrinsic 5. Enter: - - `keys`: Your session keys (from `author_rotateKeys`) - - `proof`: 0x00 (typically) + - `keys`: Your session keys (from `author_rotateKeys`) + - `proof`: 0x00 (typically) 6. Submit and sign the transaction **Using CLI (alternative):** @@ -382,6 +381,7 @@ Some parachains require bonding tokens: #### 5. Await Governance Approval If using invulnerables: + - Wait for governance vote - Monitor forum and announcements - Once approved, you'll be added to the invulnerables list @@ -389,16 +389,7 @@ If using invulnerables: ### Verify Collator Status -Check if your collator is active: - -```bash -# Monitor logs for block production -sudo journalctl -u polkadot-collator -f | grep -i "imported" - -# Look for messages like: -# "Prepared block for proposing" -# "Imported #123" -``` +Check if your collator is active by monitoring logs for block production messages like "Prepared block for proposing" and "Imported #123". See the [Log Management](#log-management) section for log viewing commands. ## Monitoring and Maintenance @@ -411,14 +402,17 @@ sudo journalctl -u polkadot-collator | grep -i "prepared block" ``` **Peer Connections**: -- Maintain 30+ peers for good connectivity + +- Maintain a sufficient amount of peers for good connectivity - Check peer count in logs **Resource Usage**: + - Monitor CPU, RAM, and disk I/O - Set up alerts for high usage **Sync Status**: + - Ensure both chains stay synced - Alert on sync issues @@ -435,6 +429,7 @@ scrape_configs: ``` Key metrics to monitor: + - `substrate_block_height`: Current block height - `substrate_finalized_height`: Finalized block height - `substrate_peers_count`: Peer connections @@ -443,6 +438,7 @@ Key metrics to monitor: ### Setting Up Alerts Configure alerts for: + - Service failures - Sync issues - Low peer count (< 10 peers) @@ -479,16 +475,18 @@ The node handles pruning automatically. ### Updates and Upgrades **Runtime Upgrades**: + - Automatic via on-chain governance - No manual action required - Monitor announcements for breaking changes **Client Upgrades**: + - Require manual binary update - Subscribe to announcements: - - Polkadot Forum - - Fellowship GitHub - - Matrix channels + - Polkadot Forum + - Fellowship GitHub + - Matrix channels **Upgrade Procedure**: @@ -508,42 +506,11 @@ polkadot-omni-node --version # Restart service sudo systemctl start polkadot-collator -# Monitor logs -sudo journalctl -u polkadot-collator -f +# Verify service is running +sudo systemctl status polkadot-collator ``` -## Security Best Practices - -### Key Management - -- **Secure storage**: Store session keys and account keys securely -- **Never share**: Never share private keys or secret phrases -- **Hardware wallets**: Consider HSM for production -- **Backup**: Keep encrypted backups of keys -- **Rotation**: Plan for key rotation procedures - -### Network Security - -- **Firewall**: Restrict access to necessary ports only -- **SSH**: Use SSH keys, disable password auth -- **VPN**: Consider VPN for administrative access -- **DDoS protection**: Implement if running in cloud - -### System Security - -- **Updates**: Keep OS and software updated -- **Dedicated user**: Never run as root -- **Fail2ban**: Enable for SSH protection -- **Audits**: Regular security audits -- **Minimal services**: Disable unnecessary services - -### Operational Security - -- **Monitoring**: 24/7 monitoring with alerts -- **Backups**: Regular configuration backups -- **Documentation**: Document procedures -- **Incident response**: Have incident response plan -- **Redundancy**: Consider backup collator (standby) +**Note**: For log monitoring, see the [Log Management](#log-management) section. ## Conclusion diff --git a/node-infrastructure/run-a-node/polkadot-hub-rpc.md b/node-infrastructure/run-a-node/polkadot-hub-rpc.md index ee619b4a6..f8e653222 100644 --- a/node-infrastructure/run-a-node/polkadot-hub-rpc.md +++ b/node-infrastructure/run-a-node/polkadot-hub-rpc.md @@ -30,20 +30,20 @@ RPC nodes serving production traffic require robust hardware: - **CPU**: 8+ cores (16+ cores for high traffic) - **Memory**: 64 GB RAM minimum (128 GB recommended for high traffic) - **Storage**: - - 500 GB+ NVMe SSD for parachain state (archive nodes require 2-4 TB+) - - Additional 200+ GB for relay chain pruned database - - Fast disk I/O is critical for query performance + - 500 GB+ NVMe SSD for parachain state (archive nodes require 2-4 TB+) + - Additional 200+ GB for relay chain pruned database + - Fast disk I/O is critical for query performance - **Network**: - - Public IP address - - 1 Gbps connection (for high traffic scenarios) - - Stable internet connection with sufficient bandwidth - - Open ports: - - 30333 (parachain P2P) - - 30334 (relay chain P2P) - - 9944 (Polkadot SDK WebSocket RPC) - - 9933 (Polkadot SDK HTTP RPC) - - 9615 (Prometheus metrics - optional) - - Consider DDoS protection and rate limiting for production deployments + - Public IP address + - 1 Gbps connection (for high traffic scenarios) + - Stable internet connection with sufficient bandwidth + - Open ports: + - 30333 (parachain P2P) + - 30334 (relay chain P2P) + - 9944 (Polkadot SDK WebSocket RPC) + - 9933 (Polkadot SDK HTTP RPC) + - 9615 (Prometheus metrics - optional) + - Consider DDoS protection and rate limiting for production deployments **Note**: For development or low-traffic scenarios, you can reduce these requirements proportionally. Consider using a reverse proxy (nginx, Caddy) for production deployments. @@ -116,6 +116,7 @@ rm files.txt ``` **Parameter Explanation**: + - `--transfers 20`: Uses 20 parallel transfers for faster download - `--retries 6`: Automatically retries failed transfers up to 6 times - `--retries-sleep 10s`: Waits 10 seconds between retry attempts @@ -141,6 +142,7 @@ rm files.txt ``` **Alternative Options**: + - Pruned snapshot: `polkadot-rocksdb-prune` (smaller size, recent state) - Archive snapshot: `polkadot-rocksdb-archive` (complete history, larger size) @@ -182,12 +184,14 @@ docker run -d --name polkadot-hub-rpc --restart unless-stopped \ **Critical Configuration Parameters**: **Port Mappings**: + - `9944`: Polkadot SDK RPC endpoint (WebSocket/HTTP) - `9933`: Polkadot SDK HTTP RPC endpoint - `9615`: Prometheus metrics endpoint - `30333/30334`: P2P networking ports **Node Parameters**: + - `--unsafe-rpc-external`: Enables external RPC access - `--rpc-cors=all`: Allows all origins for CORS - `--rpc-methods=safe`: Only allows safe RPC methods @@ -223,19 +227,10 @@ curl -H "Content-Type: application/json" \ ``` **Synchronization Status**: + - **In Progress**: `currentBlock` < `highestBlock` - **Complete**: `currentBlock` = `highestBlock` -**Monitor logs**: - -```bash -# View node logs -docker logs -f polkadot-hub-rpc - -# Filter for sync messages -docker logs polkadot-hub-rpc 2>&1 | grep -i "syncing" -``` - ### Step 5: Verify Setup Let's verify the Polkadot SDK RPC endpoint is working correctly. @@ -243,6 +238,7 @@ Let's verify the Polkadot SDK RPC endpoint is working correctly. #### API Endpoint **Polkadot SDK RPC (Port 9944)**: + - WebSocket: `ws://your-server:9944` - HTTP: `http://your-server:9944` - Purpose: Full Polkadot SDK API access for parachain data @@ -301,18 +297,7 @@ docker start polkadot-hub-rpc docker rm polkadot-hub-rpc ``` -**Update container**: - -```bash -# Pull latest image -docker pull parity/polkadot-omni-node:v1.20.2 - -# Stop and remove old container -docker stop polkadot-hub-rpc -docker rm polkadot-hub-rpc - -# Start new container with same command as above -``` +**Note**: For update procedures, see the [Updates and Upgrades](#updates-and-upgrades) section. --- @@ -478,6 +463,7 @@ sudo journalctl -u polkadot-hub-rpc | grep -i error ### Performance Monitoring Monitor key metrics: + - **Sync status**: Ensure node stays fully synced - **Peer connections**: Maintain 30+ peers for good connectivity - **Resource usage**: Monitor CPU, RAM, and disk I/O @@ -498,6 +484,7 @@ scrape_configs: ``` **Key Metrics to Monitor**: + - `substrate_block_height`: Current block height - `substrate_finalized_height`: Finalized block height - `substrate_peers_count`: Number of connected peers @@ -548,75 +535,6 @@ cargo install --locked --force polkadot-omni-node@ sudo systemctl start polkadot-hub-rpc ``` -## Security Best Practices - -### Network Security - -1. **Firewall Configuration**: - - Only expose necessary ports - - Use UFW or iptables to restrict access - - Consider IP whitelisting for RPC endpoints - -2. **Reverse Proxy** (Recommended for Production): - - Use nginx or Caddy as reverse proxy - - Enable SSL/TLS (HTTPS/WSS) - - Implement authentication - - Add rate limiting - -Example nginx configuration: - -```nginx -upstream polkadot_sdk_rpc { - server 127.0.0.1:9944; -} - -server { - listen 443 ssl http2; - server_name your-domain.com; - - ssl_certificate /path/to/cert.pem; - ssl_certificate_key /path/to/key.pem; - - # Polkadot SDK RPC - location /polkadot { - proxy_pass http://polkadot_sdk_rpc; - proxy_http_version 1.1; - proxy_set_header Upgrade $http_upgrade; - proxy_set_header Connection "upgrade"; - - # Rate limiting - limit_req zone=rpc_limit burst=10; - } -} -``` - -### RPC Security - -- **Always use `--rpc-methods=safe`**: Prevents dangerous RPC calls -- **Restrict CORS**: Use specific domains instead of `all` in production -- **Set connection limits**: Prevent resource exhaustion -- **Monitor for abuse**: Track unusual patterns -- **Authentication**: Implement API keys or OAuth for production - -### System Security - -- Keep operating system updated -- Use dedicated user accounts (never root) -- Enable fail2ban for SSH protection -- Regular security audits -- Disable unnecessary services -- Use AppArmor or SELinux for additional isolation - -### Monitoring and Alerting - -Set up alerts for: -- Service failures -- Sync issues -- Low peer count (< 10 peers) -- High resource usage -- Unusual RPC traffic patterns -- Database errors - ## Conclusion Running an RPC node for Polkadot Hub provides essential infrastructure for applications and users to interact with the network. By following this guide, you have set up a production-ready RPC node that: diff --git a/node-infrastructure/run-a-node/system-parachain-rpc.md b/node-infrastructure/run-a-node/system-parachain-rpc.md index 4cfb4b741..47440b93c 100644 --- a/node-infrastructure/run-a-node/system-parachain-rpc.md +++ b/node-infrastructure/run-a-node/system-parachain-rpc.md @@ -42,20 +42,20 @@ RPC nodes serving production traffic require robust hardware: - **CPU**: 8+ cores (16+ cores for high traffic) - **Memory**: 64 GB RAM minimum (128 GB recommended for high traffic) - **Storage**: - - 500 GB+ NVMe SSD for parachain state (archive nodes require 2-4 TB+) - - Additional 200+ GB for relay chain pruned database - - Fast disk I/O is critical for query performance + - 500 GB+ NVMe SSD for parachain state (archive nodes require 2-4 TB+) + - Additional 200+ GB for relay chain pruned database + - Fast disk I/O is critical for query performance - **Network**: - - Public IP address - - 1 Gbps connection (for high traffic scenarios) - - Stable internet connection with sufficient bandwidth - - Open ports: - - 30333 (parachain P2P) - - 30334 (relay chain P2P) - - 9944 (Polkadot SDK WebSocket RPC) - - 9933 (Polkadot SDK HTTP RPC) - - 9615 (Prometheus metrics - optional) - - Consider DDoS protection and rate limiting for production deployments + - Public IP address + - 1 Gbps connection (for high traffic scenarios) + - Stable internet connection with sufficient bandwidth + - Open ports: + - 30333 (parachain P2P) + - 30334 (relay chain P2P) + - 9944 (Polkadot SDK WebSocket RPC) + - 9933 (Polkadot SDK HTTP RPC) + - 9615 (Prometheus metrics - optional) + - Consider DDoS protection and rate limiting for production deployments **Note**: For development or low-traffic scenarios, you can reduce these requirements proportionally. Consider using a reverse proxy (nginx, Caddy) for production deployments. @@ -128,6 +128,7 @@ rm files.txt ``` **Parameter Explanation**: + - `--transfers 20`: Uses 20 parallel transfers for faster download - `--retries 6`: Automatically retries failed transfers up to 6 times - `--retries-sleep 10s`: Waits 10 seconds between retry attempts @@ -155,6 +156,7 @@ rm files.txt ``` **Alternative Options**: + - Pruned snapshot: `polkadot-rocksdb-prune` (smaller size, recent state) - Archive snapshot: `polkadot-rocksdb-archive` (complete history, larger size) @@ -196,12 +198,14 @@ docker run -d --name people-chain-rpc --restart unless-stopped \ **Critical Configuration Parameters**: **Port Mappings**: + - `9944`: Polkadot SDK RPC endpoint (WebSocket/HTTP) - `9933`: Polkadot SDK HTTP RPC endpoint - `9615`: Prometheus metrics endpoint - `30333/30334`: P2P networking ports **Node Parameters**: + - `--unsafe-rpc-external`: Enables external RPC access - `--rpc-cors=all`: Allows all origins for CORS - `--rpc-methods=safe`: Only allows safe RPC methods @@ -237,19 +241,10 @@ curl -H "Content-Type: application/json" \ ``` **Synchronization Status**: + - **In Progress**: `currentBlock` < `highestBlock` - **Complete**: `currentBlock` = `highestBlock` -**Monitor logs**: - -```bash -# View node logs -docker logs -f people-chain-rpc - -# Filter for sync messages -docker logs people-chain-rpc 2>&1 | grep -i "syncing" -``` - ### Step 5: Verify Setup Let's verify the Polkadot SDK RPC endpoint is working correctly. @@ -257,6 +252,7 @@ Let's verify the Polkadot SDK RPC endpoint is working correctly. #### API Endpoint **Polkadot SDK RPC (Port 9944)**: + - WebSocket: `ws://your-server:9944` - HTTP: `http://your-server:9944` - Purpose: Full Polkadot SDK API access for parachain data @@ -331,18 +327,7 @@ docker start people-chain-rpc docker rm people-chain-rpc ``` -**Update container**: - -```bash -# Pull latest image -docker pull parity/polkadot-omni-node:v1.20.2 - -# Stop and remove old container -docker stop people-chain-rpc -docker rm people-chain-rpc - -# Start new container with same command as above -``` +**Note**: For update procedures, see the [Updates and Upgrades](#updates-and-upgrades) section. --- @@ -498,6 +483,7 @@ sudo journalctl -u people-chain-rpc | grep -i error ### Performance Monitoring Monitor key metrics: + - **Sync status**: Ensure node stays fully synced - **Peer connections**: Maintain 30+ peers for good connectivity - **Resource usage**: Monitor CPU, RAM, and disk I/O @@ -518,6 +504,7 @@ scrape_configs: ``` **Key Metrics to Monitor**: + - `substrate_block_height`: Current block height - `substrate_finalized_height`: Finalized block height - `substrate_peers_count`: Number of connected peers @@ -568,75 +555,6 @@ cargo install --locked --force polkadot-omni-node@ sudo systemctl start people-chain-rpc ``` -## Security Best Practices - -### Network Security - -1. **Firewall Configuration**: - - Only expose necessary ports - - Use UFW or iptables to restrict access - - Consider IP whitelisting for RPC endpoints - -2. **Reverse Proxy** (Recommended for Production): - - Use nginx or Caddy as reverse proxy - - Enable SSL/TLS (HTTPS/WSS) - - Implement authentication - - Add rate limiting - -Example nginx configuration: - -```nginx -upstream polkadot_sdk_rpc { - server 127.0.0.1:9944; -} - -server { - listen 443 ssl http2; - server_name your-domain.com; - - ssl_certificate /path/to/cert.pem; - ssl_certificate_key /path/to/key.pem; - - # Polkadot SDK RPC - location /polkadot { - proxy_pass http://polkadot_sdk_rpc; - proxy_http_version 1.1; - proxy_set_header Upgrade $http_upgrade; - proxy_set_header Connection "upgrade"; - - # Rate limiting - limit_req zone=rpc_limit burst=10; - } -} -``` - -### RPC Security - -- **Always use `--rpc-methods=safe`**: Prevents dangerous RPC calls -- **Restrict CORS**: Use specific domains instead of `all` in production -- **Set connection limits**: Prevent resource exhaustion -- **Monitor for abuse**: Track unusual patterns -- **Authentication**: Implement API keys or OAuth for production - -### System Security - -- Keep operating system updated -- Use dedicated user accounts (never root) -- Enable fail2ban for SSH protection -- Regular security audits -- Disable unnecessary services -- Use AppArmor or SELinux for additional isolation - -### Monitoring and Alerting - -Set up alerts for: -- Service failures -- Sync issues -- Low peer count (< 10 peers) -- High resource usage -- Unusual RPC traffic patterns -- Database errors - ## Conclusion Running an RPC node for system parachains provides critical infrastructure for accessing specialized Polkadot network services. By following this guide, you have set up a production-ready RPC node that: From 656920e92734429a3d1cc4e5a4b94905569ceffb Mon Sep 17 00:00:00 2001 From: DAWN KELLY Date: Thu, 20 Nov 2025 16:26:49 -0500 Subject: [PATCH 05/21] initial look at collator.md --- .../run-a-collator/collator.md | 303 +++++++++--------- 1 file changed, 158 insertions(+), 145 deletions(-) diff --git a/node-infrastructure/run-a-collator/collator.md b/node-infrastructure/run-a-collator/collator.md index 8230cbb44..f4b7b4937 100644 --- a/node-infrastructure/run-a-collator/collator.md +++ b/node-infrastructure/run-a-collator/collator.md @@ -8,35 +8,35 @@ categories: Infrastructure ## Overview -Block-producing collators are the backbone of system parachain operations. Unlike RPC nodes or archive nodes that simply maintain state, collators actively produce blocks and submit them to relay chain validators for inclusion. +Block-producing collators are the backbone of system parachain operations. Unlike RPC or archive nodes, which maintain state, collators actively produce blocks and submit them to relay chain validators for inclusion. -This guide covers setting up a **block-producing collator** for Polkadot system parachains. Running a collator requires: +This guide covers setting up a block-producing collator for Polkadot system parachains. Running a collator requires: - Meeting hardware requirements for reliable block production - Setting up and registering session keys - Obtaining governance approval or meeting selection criteria - Maintaining high uptime and performance -**Important**: System parachain collators typically require governance approval or being added to the invulnerables list. This is different from non-system parachains where collator selection may be more permissionless. +System parachain collators typically require governance approval or being added to the invulnerables list. This is different from non-system parachains where collator selection may be more permissionless. ## Collator Responsibilities Block-producing collators perform critical functions: -- **Maintain full nodes**: Both relay chain and parachain -- **Collect transactions**: Aggregate user transactions into blocks -- **Produce blocks**: Create parachain block candidates -- **Generate proofs**: Produce state transition proofs (Proof-of-Validity) -- **Submit to validators**: Send block candidates to relay chain validators -- **Facilitate XCM**: Enable cross-chain message passing +- Maintain full nodes for relay chain and parachain. +- Aggregate user transactions into blocks. +- Create parachain block candidates. +- Produce state transition proofs (Proof-of-Validity). +- Send block candidates to relay chain validators. +- Enable cross-chain message passing using XCM -Unlike relay chain validators, collators do not provide security guarantees—that responsibility lies with relay chain validators through the ELVES protocol. However, collators are essential for network liveness and censorship resistance. +Unlike relay chain validators, collators do not provide security guarantees—that responsibility lies with relay chain validators through the ELVES protocol. Rather, collators are essential for network liveness and censorship resistance. ## Prerequisites ### Hardware Requirements -Block-producing collators require robust hardware for reliable operation: +Block-producing collators require robust hardware for reliable operation including the following: - **CPU**: 4+ cores (8+ cores recommended for optimal performance) - **Memory**: 32 GB RAM minimum (64 GB recommended) @@ -52,13 +52,13 @@ Block-producing collators require robust hardware for reliable operation: - 30334 (relay chain P2P) - 9944 (WebSocket RPC - for management) -**Note**: Uptime is critical. Consider redundancy and monitoring to maintain block production reliability. +Uptime is critical. Consider redundancy and monitoring to maintain block production reliability. ### Software Requirements Collators use the **Polkadot Omni Node**, a universal binary that runs any parachain using a chain specification file. -Required software: +Required software includes the following: - **Operating System**: Ubuntu 22.04 LTS (recommended) or similar Linux distribution - **Docker**: For running subkey utility @@ -71,182 +71,195 @@ Required software: ### Account Requirements -You'll need: +Your account must meet the following requirements: - **Funded account**: For on-chain transactions and potential bonding - **Session keys**: For collator identification (generated after node setup) - **Node key**: For stable P2P peer ID (recommended) -## Installation +## Install Dependencies -### Step 1: Install Rust and Required Toolchain - -```bash -# Install Rust -curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -source $HOME/.cargo/env - -# Install specific Rust version -rustup install 1.91.1 -rustup default 1.91.1 -rustup target add wasm32-unknown-unknown --toolchain 1.91.1 -rustup component add rust-src --toolchain 1.91.1 -``` - -### Step 2: Install the Polkadot Omni Node +1. Install Rust using the following commands: + ```bash + # Install Rust + curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh + source $HOME/.cargo/env + + # Install specific Rust version + rustup install 1.91.1 + rustup default 1.91.1 + rustup target add wasm32-unknown-unknown --toolchain 1.91.1 + rustup component add rust-src --toolchain 1.91.1 + ``` -```bash -# Install polkadot-omni-node -cargo install --locked polkadot-omni-node@0.11.0 +2. Install the Polkadot Omni Node using the following command: + ```bash + cargo install --locked polkadot-omni-node@0.11.0 + ``` -# Verify installation -polkadot-omni-node --version -``` +3. Verify a successful installation using the `--version` flag: + ```bash + polkadot-omni-node --version + ``` -### Step 3: Generate Node Key +## Generate Node Key -Generate a stable node key for consistent peer ID: +Generating a stable node key enables a consistent peer ID across the network. Follow these steps to generate a node key: -```bash -# Create directory for node data -sudo mkdir -p /var/lib/polkadot-collator +1. Create a directory for node data using the following command: + ```bash + sudo mkdir -p /var/lib/polkadot-collator + ``` -# Generate node key using Docker -docker run -it parity/subkey:latest generate-node-key > /var/lib/polkadot-collator/node.key +2. Generate your node key using Docker with the following command: + ```bash + docker run -it parity/subkey:latest generate-node-key > /var/lib/polkadot-collator/node.key + ``` -# The output displays your peer ID -# Example: 12D3KooWExcVYu7Mvjd4kxPVLwN2ZPnZ5NyLZ5ft477wqzfP2q6E -``` +3. Locate your peer ID in the displayed output. It will be similar to the following example: + ```bash + 12D3KooWExcVYu7Mvjd4kxPVLwN2ZPnZ5NyLZ5ft477wqzfP2q6E + ``` -Save the peer ID for future reference. +Be sure to save the peer ID for future reference. -### Step 4: Generate Account Key +## Generate Account Key -Generate an account for on-chain transactions: +Generate an account for on-chain transactions as follows: -```bash -# Generate account key with sr25519 scheme -docker run -it parity/subkey:latest generate --scheme sr25519 -``` +1. Generate an account key with `sr25519` scheme using the following command: + ```bash + docker run -it parity/subkey:latest generate --scheme sr25519 + ``` -Save the output containing: +2. Save the following items displayed in the output: + - Secret phrase (seed) - Keep this secure! + - Public key (hex) + - Account ID + - SS58 Address -- Secret phrase (seed) - Keep this secure! -- Public key (hex) -- Account ID -- SS58 Address + !!! warning + Store the secret phrase securely. Never share it. Consider using a hardware wallet for production collators. -**Security**: Store the secret phrase securely. Never share it. Consider using a hardware wallet for production collators. +## Obtain Chain Specification -### Step 5: Obtain Chain Specification +Download the chain specification for your target system parachain using one of the following options: -Download the chain specification for your target system parachain: +### Download from Chainspec Collection (Recommended) -**Option 1: Download from Chainspec Collection (Recommended)** +Follow these steps to download your specification from the Chainspec Collection: 1. Visit the [Chainspec Collection website](https://paritytech.github.io/chainspecs/) 2. Find your target system parachain 3. Download the chain specification JSON file 4. Save it as `chain-spec.json` -**Option 2: Build from Runtime** +### Build Chainspec from Runtime -```bash -# Clone the runtimes repository -git clone https://github.com/polkadot-fellows/runtimes.git -cd runtimes - -# Build the desired runtime (example for Polkadot Hub) -cargo build --release -p asset-hub-polkadot-runtime - -# Install chain-spec-builder -cargo install --locked staging-chain-spec-builder@14.0.0 - -# Generate chain spec -chain-spec-builder create \ - --relay-chain polkadot \ - --para-id 1000 \ - --runtime target/release/wbuild/asset-hub-polkadot-runtime/asset_hub_polkadot_runtime.compact.compressed.wasm \ - named-preset production > chain-spec.json -``` +Follow these steps to build a chainspec from the runtime: -**System Parachain Para IDs:** +1. Clone the runtimes repository and navigate into it using the following commands: + ```bash + git clone https://github.com/polkadot-fellows/runtimes.git + cd runtimes + ``` -- Polkadot Hub: 1000 -- Bridge Hub: 1002 -- People Chain: 1004 -- Coretime Chain: 1005 +2. Build the desired runtime. Use the following command for Polkadot Hub: + ```bash + cargo build --release -p asset-hub-polkadot-runtime + ``` -### Step 6: Create User and Directory Structure +3. Install the `chain-spec-builder` dependency using the following command: + ```bash + cargo install --locked staging-chain-spec-builder@14.0.0 + ``` -```bash -# Create dedicated user -sudo useradd -r -s /bin/bash polkadot +4. Finally, generate the chain spec using the following commands: + ```bash + chain-spec-builder create \ + --relay-chain polkadot \ + --para-id 1000 \ + --runtime target/release/wbuild/asset-hub-polkadot-runtime/asset_hub_polkadot_runtime.compact.compressed.wasm \ + named-preset production > chain-spec.json + ``` -# Copy chain spec to directory -sudo cp chain-spec.json /var/lib/polkadot-collator/ +??? tip "System Parachain Para IDs" -# Set permissions -sudo chown -R polkadot:polkadot /var/lib/polkadot-collator -``` + - Polkadot Hub: 1000 + - Bridge Hub: 1002 + - People Chain: 1004 + - Coretime Chain: 1005 -## Configuration +## Create User and Directory Structure -### Create Systemd Service File +1. Create a dedicated user with the following command: + ```bash + sudo useradd -r -s /bin/bash polkadot + ``` -Create a service file for your collator: +2. Use the following command to copy your chain spec to the directory: + ```bash + sudo cp chain-spec.json /var/lib/polkadot-collator/ + ``` -```bash -sudo nano /etc/systemd/system/polkadot-collator.service -``` +3. Set permissions using the following command: + ```bash + sudo chown -R polkadot:polkadot /var/lib/polkadot-collator + ``` -Add the following configuration: - -```ini -[Unit] -Description=Polkadot System Parachain Collator -After=network.target - -[Service] -Type=simple -User=polkadot -Group=polkadot -WorkingDirectory=/var/lib/polkadot-collator - -# Block-Producing Collator Configuration -ExecStart=/usr/local/bin/polkadot-omni-node \ - --collator \ - --chain=/var/lib/polkadot-collator/chain-spec.json \ - --base-path=/var/lib/polkadot-collator \ - --port=30333 \ - --rpc-port=9944 \ - --prometheus-port=9615 \ - --node-key-file=/var/lib/polkadot-collator/node.key \ - --name="YourCollatorName" \ - -- \ - --execution=wasm \ - --chain=polkadot \ - --port=30334 \ - --sync=warp - -Restart=always -RestartSec=10 -LimitNOFILE=65536 - -[Install] -WantedBy=multi-user.target -``` +## Create Systemd Service File + +1. Create a service file to hold the configuration for your collator: + ```bash + sudo nano /etc/systemd/system/polkadot-collator.service + ``` + +2. Open the new file and add the following configuration code: + ```ini title="systemd/system/polkadot-collator.service" + [Unit] + Description=Polkadot System Parachain Collator + After=network.target + + [Service] + Type=simple + User=polkadot + Group=polkadot + WorkingDirectory=/var/lib/polkadot-collator + + # Block-Producing Collator Configuration + ExecStart=/usr/local/bin/polkadot-omni-node \ + --collator \ + --chain=/var/lib/polkadot-collator/chain-spec.json \ + --base-path=/var/lib/polkadot-collator \ + --port=30333 \ + --rpc-port=9944 \ + --prometheus-port=9615 \ + --node-key-file=/var/lib/polkadot-collator/node.key \ + --name="YourCollatorName" \ + -- \ + --execution=wasm \ + --chain=polkadot \ + --port=30334 \ + --sync=warp + + Restart=always + RestartSec=10 + LimitNOFILE=65536 + + [Install] + WantedBy=multi-user.target + ``` -**Configuration Notes**: +??? tip -- `--collator`: Enables block production mode -- `--node-key-file`: Uses the generated node key for stable peer ID -- `--name`: Your collator name (visible in telemetry) -- Relay chain uses `--sync=warp` for faster initial sync + - `--collator`: Enables block production mode + - `--node-key-file`: Uses the generated node key for stable peer ID + - `--name`: Your collator name (visible in telemetry) + - Relay chain uses `--sync=warp` for faster initial sync ## Running the Collator -### Step 1: Start the Service +### Start the Service ```bash # Reload systemd @@ -265,7 +278,7 @@ sudo systemctl status polkadot-collator sudo journalctl -u polkadot-collator -f ``` -### Step 2: Initial Sync +### Initial Sync Your collator must sync both the relay chain and parachain before producing blocks. @@ -279,7 +292,7 @@ The relay chain uses warp sync for faster synchronization. **Important**: Do not proceed with registration until both chains are fully synced. Monitor sync progress using the log viewing commands in the [Log Management](#log-management) section. -### Step 3: Generate Session Keys +### Generate Session Keys Once your node is fully synced, generate session keys: From b04156ae9cb6ea43b5a19dcee297d057450c12b4 Mon Sep 17 00:00:00 2001 From: DAWN KELLY Date: Thu, 20 Nov 2025 16:28:53 -0500 Subject: [PATCH 06/21] tweak admo --- node-infrastructure/run-a-collator/collator.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/node-infrastructure/run-a-collator/collator.md b/node-infrastructure/run-a-collator/collator.md index f4b7b4937..b0b9699f0 100644 --- a/node-infrastructure/run-a-collator/collator.md +++ b/node-infrastructure/run-a-collator/collator.md @@ -250,7 +250,7 @@ Follow these steps to build a chainspec from the runtime: WantedBy=multi-user.target ``` -??? tip +??? note "Configuration notes" - `--collator`: Enables block production mode - `--node-key-file`: Uses the generated node key for stable peer ID From dbd2d9805e7274080a60fdc822ea11f3d0f5cca7 Mon Sep 17 00:00:00 2001 From: DAWN KELLY Date: Fri, 21 Nov 2025 10:39:10 -0500 Subject: [PATCH 07/21] wip - collator page --- .../run-a-collator/collator.md | 35 +++++++++++-------- 1 file changed, 21 insertions(+), 14 deletions(-) diff --git a/node-infrastructure/run-a-collator/collator.md b/node-infrastructure/run-a-collator/collator.md index b0b9699f0..645024cd4 100644 --- a/node-infrastructure/run-a-collator/collator.md +++ b/node-infrastructure/run-a-collator/collator.md @@ -257,28 +257,35 @@ Follow these steps to build a chainspec from the runtime: - `--name`: Your collator name (visible in telemetry) - Relay chain uses `--sync=warp` for faster initial sync -## Running the Collator +## Run the Collator -### Start the Service +Follow these steps to run your collator node: -```bash -# Reload systemd -sudo systemctl daemon-reload - -# Enable service to start on boot -sudo systemctl enable polkadot-collator +1. Reload systemd using the following command: + ```bash + sudo systemctl daemon-reload + ``` -# Start the service -sudo systemctl start polkadot-collator +2. Next, enable the service to start on boot using the command: + ```bash + sudo systemctl enable polkadot-collator + ``` +3. Now, start the service with the following command: + ```bash + sudo systemctl start polkadot-collator + ``` -# Check status -sudo systemctl status polkadot-collator +4. Finally, you can check the status of the service using the command: + ```bash + sudo systemctl status polkadot-collator + ``` -# View logs +To view collator service logs, use the command: +```bash sudo journalctl -u polkadot-collator -f ``` -### Initial Sync +## Complete Initial Sync Your collator must sync both the relay chain and parachain before producing blocks. From fc25788bd1fc39d7bdfee79d3795dff7cc2653f9 Mon Sep 17 00:00:00 2001 From: DAWN KELLY Date: Fri, 21 Nov 2025 12:39:10 -0500 Subject: [PATCH 08/21] finish collator pre-tech review pass --- .../run-a-collator/collator.md | 336 +++++++----------- 1 file changed, 136 insertions(+), 200 deletions(-) diff --git a/node-infrastructure/run-a-collator/collator.md b/node-infrastructure/run-a-collator/collator.md index 645024cd4..a72c1e5a6 100644 --- a/node-infrastructure/run-a-collator/collator.md +++ b/node-infrastructure/run-a-collator/collator.md @@ -133,13 +133,14 @@ Generate an account for on-chain transactions as follows: ``` 2. Save the following items displayed in the output: - - Secret phrase (seed) - Keep this secure! - - Public key (hex) - - Account ID - - SS58 Address + - Secret phrase (seed) - Keep this secure! + - Public key (hex) + - Account ID + - SS58 Address - !!! warning - Store the secret phrase securely. Never share it. Consider using a hardware wallet for production collators. + !!! warning + + Store the secret phrase securely. Never share it. Consider using a hardware wallet for production collators. ## Obtain Chain Specification @@ -149,10 +150,10 @@ Download the chain specification for your target system parachain using one of t Follow these steps to download your specification from the Chainspec Collection: -1. Visit the [Chainspec Collection website](https://paritytech.github.io/chainspecs/) -2. Find your target system parachain -3. Download the chain specification JSON file -4. Save it as `chain-spec.json` +1. Visit the [Chainspec Collection](https://paritytech.github.io/chainspecs/) website. +2. Find your target system parachain. +3. Download the chain specification JSON file. +4. Save it as `chain-spec.json`. ### Build Chainspec from Runtime @@ -299,246 +300,181 @@ The relay chain uses warp sync for faster synchronization. **Important**: Do not proceed with registration until both chains are fully synced. Monitor sync progress using the log viewing commands in the [Log Management](#log-management) section. -### Generate Session Keys +## Generate Session Keys -Once your node is fully synced, generate session keys: +Once your node is fully synced, use the following command to generate session keys via RPC: ```bash -# Generate session keys via RPC curl -H "Content-Type: application/json" \ -d '{"id":1, "jsonrpc":"2.0", "method": "author_rotateKeys", "params":[]}' \ http://localhost:9944 - -# Returns session keys as a hex string -# Example: "0x1234567890abcdef..." ``` -**Save the session keys** - you'll need them for on-chain registration. - -**Note**: Session keys are stored in the node's database. If you wipe the database, you'll need to generate new keys. +This command returns session keys as a hexstring in the terminal. You must save these session keys as you'll need them for on-chain registration. As session keys are stored in the node's database, if you wipe the database, you'll also need to generate new keys. -## Registration and Governance +## Register Collator for Selection -### Understanding Collator Selection +System parachains use different collator selection mechanisms. Explore the following tabs to see how mechanisms compare and determine the most suitable mechanism for registering your collator. -System parachains use different collator selection mechanisms: +=== "Invulnerables List" -**Invulnerables List**: + - Fixed list of collators approved through governance. + - Most common for system parachains. + - Requires governance proposal and approval. -- Fixed list of collators approved through governance -- Most common for system parachains -- Requires governance proposal and approval -**On-chain Selection**: +=== "On-chain Selection" -- Some parachains use pallet-collator-selection -- May require bonding tokens -- Automatic selection based on criteria + - Some parachains use pallet-collator-selection. + - May require bonding tokens. + - Automatic selection based on criteria. -**Fellowship Decisions**: +=== "Fellowship Decisions" -- Technical Fellowship may manage some system parachain collators -- Requires Fellowship membership or approval + - Technical Fellowship may manage some system parachain collators. + - Requires Fellowship membership or approval. ### Registration Process -The registration process varies by system parachain. General steps: - -#### 1. Check Current Collators - -Check the existing collators for your target parachain: - -```bash -# Using Polkadot.js Apps -# Connect to your target system parachain -# Go to Developer > Chain State -# Query: collatorSelection.invulnerables() or similar -``` - -#### 2. Prepare Governance Proposal - -For invulnerables-based selection: - -1. **Draft proposal**: Explain why you should be added as a collator -2. **Technical details**: Provide your session keys and account ID -3. **Infrastructure**: Describe your hardware and monitoring setup -4. **Experience**: Detail your relevant experience - -Submit to: -- Polkadot Forum: https://forum.polkadot.network -- Relevant governance channels - -#### 3. Set Session Keys On-Chain - -Once approved (or if using on-chain selection), set your session keys: - -**Using Polkadot.js Apps:** - -1. Navigate to Polkadot.js Apps and connect to your system parachain -2. Go to **Developer > Extrinsics** -3. Select your account -4. Choose `session.setKeys` extrinsic -5. Enter: - - `keys`: Your session keys (from `author_rotateKeys`) - - `proof`: 0x00 (typically) -6. Submit and sign the transaction - -**Using CLI (alternative):** - -```bash -# This varies by parachain - consult specific documentation -``` - -#### 4. Bond Tokens (if required) - -Some parachains require bonding tokens: - -1. Go to **Developer > Extrinsics** -2. Select `collatorSelection.registerAsCandidate` (if available) -3. Submit with required bond amount -4. Sign transaction - -#### 5. Await Governance Approval - -If using invulnerables: - -- Wait for governance vote -- Monitor forum and announcements -- Once approved, you'll be added to the invulnerables list -- Your collator will begin producing blocks in the next session/era - -### Verify Collator Status - -Check if your collator is active by monitoring logs for block production messages like "Prepared block for proposing" and "Imported #123". See the [Log Management](#log-management) section for log viewing commands. - -## Monitoring and Maintenance - -### Essential Monitoring - -**Block Production**: -```bash -# Monitor block production -sudo journalctl -u polkadot-collator | grep -i "prepared block" -``` - -**Peer Connections**: - -- Maintain a sufficient amount of peers for good connectivity -- Check peer count in logs - -**Resource Usage**: - -- Monitor CPU, RAM, and disk I/O -- Set up alerts for high usage - -**Sync Status**: - -- Ensure both chains stay synced -- Alert on sync issues +The registration process varies by system parachain. General steps include the following: + +1. Check the existing collators for your target parachain: + + 1. Navigate to Polkadot.js Apps and connect to your system parachain. + 2. Locate **Developer > Chain State**. + 3. Query `collatorSelection.invulnerables()` + +2. Prepare a governance proposal for invulnerables-based selection including the following information: + - **Draft proposal**: Explain why you should be added as a collator + - **Technical details**: Provide your session keys and account ID + - **Infrastructure**: Describe your hardware and monitoring setup + - **Experience**: Detail your relevant experience + + Submit the proposal to the relevant governance channels on the [Polkadot Forum](https://forum.polkadot.network){target=\_blank}. + +3. Once approved (or if using on-chain selection), follow these steps to register session keys using Polkadot.js Apps: + 1. Navigate to Polkadot.js Apps and connect to your system parachain. + 2. Locate **Developer > Extrinsics**. + 3. Select your account. + 4. Choose the `session.setKeys` extrinsic. + 5. Enter the following information: + - `keys`: Your session keys (from `author_rotateKeys`) + - `proof`: 0x00 (typically) + 6. Submit and sign the transaction. + +4. If the parachain requires bonding tokens, use the follow these steps to submit them using Polkadot.js Apps: + 1. Locate **Developer > Extrinsics**. + 2. Select `collatorSelection.registerAsCandidate`. + 3. Submit the transaction with the required bond amount. + 4. Sign the transaction. + +5. If applying to join the invulnerables list, you must now await governance approval for your proposal. Monitor the [Polkadot Forum](https://forum.polkadot.network){target=\_blank} governance channels and announcements. Once approved, your collator is added to the invulnerables list and will begin producing blocks in the next session or era. + +6. Verify your collator is active by monitoring logs for block production messages like "Prepared block for proposing" and "Imported #123". See the [Log Management](#log-management) section for log viewing commands. + +## Monitor and Maintain Your Collator + +Monitoring the following items will help ensure your collator runs efficiently to avoid service interruptions or down time: + +- **Block Production**: monitor for block production using the following command: + ```bash + sudo journalctl -u polkadot-collator | grep -i "prepared block" + ``` +- **Peer Connections**: Maintain a sufficient amount of peers for good connectivity and set up alerts to notify if peer connections falls below ten. +- **Resource Usage**: Monitor CPU, RAM, and disk I/O and set up alerts for unusual or high usage. +- **Sync Status**: Ensure both chains stay synced and set up alerts for sync issues. ### Prometheus Metrics -Metrics available at `http://localhost:9615/metrics` +You can use the following information to configure [Prometheus](https://prometheus.io/docs/introduction/first_steps/){target=\_blank} to monitor, collect, and store your collator node metrics: -Example Prometheus configuration: -```yaml -scrape_configs: - - job_name: 'polkadot-collator' - static_configs: - - targets: ['localhost:9615'] -``` +- **URL**: Metrics are available to view at `http://localhost:9615/metrics` +- **Example Prometheus configuration**: Update your `prometheus.yml` to add the following code: + ```yaml + scrape_configs: + - job_name: 'polkadot-collator' + static_configs: + - targets: ['localhost:9615'] + ``` -Key metrics to monitor: +Key metrics to monitor via Prometheus include the following: - `substrate_block_height`: Current block height - `substrate_finalized_height`: Finalized block height - `substrate_peers_count`: Peer connections - `substrate_ready_transactions_number`: Transaction queue -### Setting Up Alerts -Configure alerts for: +## Log Management -- Service failures -- Sync issues -- Low peer count (< 10 peers) -- Block production gaps -- High resource usage -- Disk space low +Effecient log management is essential to ensure collator performance and uptime. Use the following commands to help you manage logs to monitor and maintain your collator: -### Log Management +- View recent logs: + ```bash + sudo journalctl -u polkadot-collator -n 100 + ``` +- Follow logs in real-time: + ```bash + sudo journalctl -u polkadot-collator -f + ``` +- Filter for errors: + ```bash + sudo journalctl -u polkadot-collator | grep -i error + ``` +- Filter for block production: + ```bash + sudo journalctl -u polkadot-collator | grep -i "imported" + ``` -```bash -# View recent logs -sudo journalctl -u polkadot-collator -n 100 +## Database Maintenance -# Follow logs in real-time -sudo journalctl -u polkadot-collator -f - -# Filter for errors -sudo journalctl -u polkadot-collator | grep -i error - -# Filter for block production -sudo journalctl -u polkadot-collator | grep -i "imported" -``` - -### Database Maintenance - -Check database size: +You can check database size using the following command: ```bash -# Check database size du -sh /var/lib/polkadot-collator ``` +The collator node handles pruning automatically. -The node handles pruning automatically. - -### Updates and Upgrades +## Updates and Upgrades -**Runtime Upgrades**: +Updates or upgrades can happen on either the runtime or client. Runtime upgrades happen automatically via on-chain governance and do not require manual action on your part. Client upgrades do require a manual binary update process performed via terminal commands as follows: -- Automatic via on-chain governance -- No manual action required -- Monitor announcements for breaking changes +1. Stop the service: + ```bash + sudo systemctl stop polkadot-collator + ``` -**Client Upgrades**: +2. Backup data (recommended): + ```bash + sudo cp -r /var/lib/polkadot-collator /var/lib/polkadot-collator.backup + ``` -- Require manual binary update -- Subscribe to announcements: - - Polkadot Forum - - Fellowship GitHub - - Matrix channels +3. Update `polkadot-omni-node`: + ```bash + cargo install --locked --force polkadot-omni-node@ + ``` -**Upgrade Procedure**: +4. Verify `polkadot-omni-node` version to confirm successful update: + ```bash + polkadot-omni-node --version + ``` -```bash -# Stop the service -sudo systemctl stop polkadot-collator - -# Backup data (recommended) -sudo cp -r /var/lib/polkadot-collator /var/lib/polkadot-collator.backup - -# Update polkadot-omni-node -cargo install --locked --force polkadot-omni-node@ - -# Verify version -polkadot-omni-node --version - -# Restart service -sudo systemctl start polkadot-collator - -# Verify service is running -sudo systemctl status polkadot-collator -``` +5. Restart the service: + ```bash + sudo systemctl start polkadot-collator + ``` -**Note**: For log monitoring, see the [Log Management](#log-management) section. +6. Verify the service is running: + ```bash + sudo systemctl status polkadot-collator + ``` ## Conclusion Running a collator node is essential for parachain operation and network security. By following this guide, you have set up a production-ready collator that: -- Produces blocks for your parachain and maintains network consensus -- Implements comprehensive security measures to protect keys and operations -- Supports robust monitoring and alerting for reliable performance -- Follows best practices for both Docker and systemd deployments +- Produces blocks for your parachain and maintains network consensus. +- Implements comprehensive security measures to protect keys and operations. +- Supports robust monitoring and alerting for reliable performance. +- Follows best practices for both Docker and systemd deployments. -As a collator operator, you play a vital role in your parachain's infrastructure. Regular maintenance, security updates, and monitoring will ensure your collator continues to perform reliably. Stay engaged with your parachain community and keep up with updates to maintain optimal performance and security. +As a collator operator, you play a vital role in your parachain's infrastructure. Regular maintenance, security updates, and monitoring will ensure your collator continues to perform reliably. Stay engaged with your parachain community and keep up with updates to maintain optimal performance and security. \ No newline at end of file From 2de9f460f4229471a42d11dbccb0f297c13a97d7 Mon Sep 17 00:00:00 2001 From: DAWN KELLY Date: Fri, 21 Nov 2025 15:56:33 -0500 Subject: [PATCH 09/21] clean up formatting for RPC node page --- .../run-a-node/polkadot-hub-rpc.md | 903 +++++++++--------- 1 file changed, 431 insertions(+), 472 deletions(-) diff --git a/node-infrastructure/run-a-node/polkadot-hub-rpc.md b/node-infrastructure/run-a-node/polkadot-hub-rpc.md index f8e653222..247651263 100644 --- a/node-infrastructure/run-a-node/polkadot-hub-rpc.md +++ b/node-infrastructure/run-a-node/polkadot-hub-rpc.md @@ -1,6 +1,6 @@ --- title: Run an RPC Node for Polkadot Hub -description: Complete guide to set up and run an RPC node for Polkadot Hub with Polkadot SDK RPC endpoints. +description: Follow this guide to understand hardware and software requirements and how to set up and run an RPC node for Polkadot Hub with Polkadot SDK RPC endpoints. categories: Infrastructure --- @@ -19,13 +19,15 @@ Running an RPC node for Polkadot Hub enables applications, wallets, and users to This setup enables block explorer indexing and provides full compatibility with Polkadot SDK development tools. -**Important Note**: The parameters and configurations in this guide are provided as illustrative examples. You may need to modify them according to your specific environment, hardware capabilities, and network conditions. +!!! note + + The parameters and configurations in this guide are provided as illustrative examples. You may need to modify them according to your specific environment, hardware capabilities, and network conditions. ## Prerequisites ### Hardware Requirements -RPC nodes serving production traffic require robust hardware: +RPC nodes serving production traffic require robust hardware. The following should be considered the minimum standard to effectively operate an RPC node: - **CPU**: 8+ cores (16+ cores for high traffic) - **Memory**: 64 GB RAM minimum (128 GB recommended for high traffic) @@ -58,490 +60,447 @@ Required software: ## Setup Options -This guide provides two deployment options: +This guide provides two options for deployment: + +- **Docker-based Setup**: Best for simpler set up and maintainance +- **Manual/systemd Setup**: Best for production environments requiring more control + +Select the best option for your project, then use the steps in the following tabs to complete set up. + + +=== "Docker-Based Setup" + + This option uses Docker containers for the Polkadot SDK node, making it easy to set up and manage. Follow these steps to set your RPC node using Docker: + + 1. Download the official Polkadot Hub (formerly known as Asset Hub) chain specification: + ```bash + curl -L https://raw.githubusercontent.com/paritytech/polkadot-sdk/master/cumulus/parachains/chain-specs/asset-hub-polkadot.json -o asset-hub-polkadot.json + ``` + + !!! note + + This chain specification is the official configuration file that defines the network parameters for Polkadot Hub. + + 2. (Optional but recommended) Download database snapshots + - Using pre-synchronized snapshots significantly reduces initial sync time from several days to just a few hours. You need to download both parachain and relay chain data. + - You can obtain the lastest snapshot from the [Snapshot Provider](https://snapshots.polkadot.io/){target=\_blank}. Follow these steps to download and use snapshots: + 1. Create new directories with the following commands: + ```bash + mkdir -p my-node-data/chains/asset-hub-polkadot/db + mkdir -p my-node-data/chains/polkadot/db + ``` + 2. Download the appropriate snapshot using the following commands: + + === "Archive snapshot" + + Contains complete history, recommended for RPC with historical data. + ```bash + # Check https://snapshots.polkadot.io/ for the latest snapshot URL + export SNAPSHOT_URL_ASSET_HUB="https://snapshots.polkadot.io/polkadot-asset-hub-rocksdb-archive/LATEST" + + rclone copyurl $SNAPSHOT_URL_ASSET_HUB/files.txt files.txt + rclone copy --progress --transfers 20 \ + --http-url $SNAPSHOT_URL_ASSET_HUB \ + --no-traverse --http-no-head --disable-http2 \ + --inplace --no-gzip-encoding --size-only \ + --retries 6 --retries-sleep 10s \ + --files-from files.txt :http: my-node-data/chains/asset-hub-polkadot/db/ + + rm files.txt + ``` + + - `--transfers 20`: Uses 20 parallel transfers for faster download. + - `--retries 6`: Automatically retries failed transfers up to 6 times. + - `--retries-sleep 10s`: Waits 10 seconds between retry attempts. + - `--size-only`: Only transfers if sizes differ (prevents unnecessary re-downloads). + + === "Pruned snapshot" + + Contains recent state for a smaller package size, recommended for RPC nodes. + ```bash + # Check https://snapshots.polkadot.io/ for the latest snapshot URL + export SNAPSHOT_URL_RELAY="https://snapshots.polkadot.io/polkadot-rocksdb-prune/LATEST" + + rclone copyurl $SNAPSHOT_URL_RELAY/files.txt files.txt + rclone copy --progress --transfers 20 \ + --http-url $SNAPSHOT_URL_RELAY \ + --no-traverse --http-no-head --disable-http2 \ + --inplace --no-gzip-encoding --size-only \ + --retries 6 --retries-sleep 10s \ + --files-from files.txt :http: my-node-data/chains/polkadot/db/ + + rm files.txt + ``` + 3. Launch Polkadot Hub Node using the official [Parity Docker image](https://hub.docker.com/r/parity/polkadot-omni-node){target=\_blank} with the following command: + ```bash + docker run -d --name polkadot-hub-rpc --restart unless-stopped \ + -p 9944:9944 \ + -p 9933:9933 \ + -p 9615:9615 \ + -p 30334:30334 \ + -p 30333:30333 \ + -v $(pwd)/asset-hub-polkadot.json:/asset-hub-polkadot.json \ + -v $(pwd)/my-node-data:/data \ + parity/polkadot-omni-node:v1.20.2 \ + --name=PolkadotHubRPC \ + --base-path=/data \ + --chain=/asset-hub-polkadot.json \ + --prometheus-external \ + --prometheus-port 9615 \ + --unsafe-rpc-external \ + --rpc-port=9944 \ + --rpc-cors=all \ + --rpc-methods=safe \ + --rpc-max-connections=1000 \ + --state-pruning=archive \ + --blocks-pruning=archive \ + -- \ + --base-path=/data \ + --chain=polkadot \ + --state-pruning=256 \ + --blocks-pruning=archive-canonical + ``` + + Critical configuration parameters include port mappings and node parameters: + + === "Port mappings" + + - `9944`: Polkadot SDK RPC endpoint (WebSocket/HTTP) + - `9933`: Polkadot SDK HTTP RPC endpoint + - `9615`: Prometheus metrics endpoint + - `30333/30334`: P2P networking ports + + === "Node parameters" + + - `--unsafe-rpc-external`: Enables external RPC access + - `--rpc-cors=all`: Allows all origins for CORS + - `--rpc-methods=safe`: Only allows safe RPC methods + - `--state-pruning=archive`: Keeps complete state history + - `--blocks-pruning=archive`: Keeps all block data + - `--prometheus-external`: Exposes metrics externally + + !!! warning + + The `--unsafe-rpc-external` flag should only be used in development or properly secured environments. For production, use a reverse proxy with authentication. + + 4. Monitor the node synchronization status using the following command: + ```bash + curl -H "Content-Type: application/json" \ + -d '{"id":1, "jsonrpc":"2.0", "method": "system_syncState", "params":[]}' \ + http://localhost:9944 + ``` + + You should see a response similar to the following: + + ```json + { + "jsonrpc":"2.0", + "id":1, + "result":{ + "startingBlock":0, + "currentBlock":3394816, + "highestBlock":3394816 + } + } + ``` + + When synchronization is complete, `currentBlock` will be equal to `highestBlock`. + + 5. You can use a few different commands to verify your node is running properly: + + - Get chain information: + ```bash + curl -H "Content-Type: application/json" \ + -d '{"id":1, "jsonrpc":"2.0", "method": "system_chain", "params":[]}' \ + http://localhost:9944 + ``` + - Get the latest block: + ```bash + curl -H "Content-Type: application/json" \ + -d '{"id":1, "jsonrpc":"2.0", "method": "chain_getHeader", "params":[]}' \ + http://localhost:9944 + ``` + - Query node health: + ```bash + curl -H "Content-Type: application/json" \ + -d '{"id":1, "jsonrpc":"2.0", "method": "system_health", "params":[]}' \ + http://localhost:9944 + ``` + + 6. Use the following commands to manage your Docker containers: + + - View node logs: + ```bash + docker logs -f polkadot-hub-rpc + ``` + - Stop container: + ```bash + docker stop polkadot-hub-rpc + ``` + - Start container: + ```bash + docker start polkadot-hub-rpc + ``` + - Remove container: + ```bash + docker rm polkadot-hub-rpc + ``` +=== "Manual systemd Setup" + + This option provides more control and is recommended for production environments requiring custom configurations. + + 1. Install Rust and required toolchain using the following command: + ```bash + # Install Rust + curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh + source $HOME/.cargo/env + + # Install specific Rust version + rustup install 1.91.1 + rustup default 1.91.1 + rustup target add wasm32-unknown-unknown --toolchain 1.91.1 + rustup component add rust-src --toolchain 1.91.1 + ``` + + 2. Install required dependencies using the following commands: + - System dependencies: + ```bash + sudo apt update + sudo apt install -y build-essential git clang curl libssl-dev llvm libudev-dev make protobuf-compiler + ``` + - Polkadot Omni node: + ```bash + cargo install --locked polkadot-omni-node@0.11.0 + ``` + You can verify successful installation of Polkadot Omni node using the version command: + ```bash + polkadot-omni-node --version + ``` + + !!! tip + + Compiling polkadot-omni-node from source requires significant RAM (minimum 24GB recommended). The compilation may take 10-15 minutes on systems with adequate resources. + + 3. Download the Polkadot Hub chain specification: + ```bash + curl -L https://raw.githubusercontent.com/paritytech/polkadot-sdk/master/cumulus/parachains/chain-specs/asset-hub-polkadot.json -o asset-hub-polkadot.json + ``` + + 4. Create user and directory structures using the following commands: + - Create a dedicated user: + ```bash + sudo useradd -r -s /bin/bash polkadot + ``` + - Create data directory: + ```bash + sudo mkdir -p /var/lib/polkadot-hub-rpc + ``` + - Copy the chain spec to the directory: + ```bash + sudo cp asset-hub-polkadot.json /var/lib/polkadot-hub-rpc/ + ``` + - Set permissions: + ```bash + sudo chown -R polkadot:polkadot /var/lib/polkadot-hub-rpc + ``` + + 5. Create a systemd service file for the Polkadot SDK RPC node: + ```bash + sudo nano /etc/systemd/system/polkadot-hub-rpc.service + ``` + + 6. Open the new service file and add the following configuration: + ```ini + [Unit] + Description=Polkadot Hub RPC Node + After=network.target + + [Service] + Type=simple + User=polkadot + Group=polkadot + WorkingDirectory=/var/lib/polkadot-hub-rpc + + ExecStart=/usr/local/bin/polkadot-omni-node \ + --name=PolkadotHubRPC \ + --chain=/var/lib/polkadot-hub-rpc/asset-hub-polkadot.json \ + --base-path=/var/lib/polkadot-hub-rpc \ + --port=30333 \ + --rpc-port=9944 \ + --rpc-external \ + --rpc-cors=all \ + --rpc-methods=safe \ + --rpc-max-connections=1000 \ + --prometheus-port=9615 \ + --prometheus-external \ + --state-pruning=archive \ + --blocks-pruning=archive \ + -- \ + --chain=polkadot \ + --base-path=/var/lib/polkadot-hub-rpc \ + --port=30334 \ + --state-pruning=256 \ + --blocks-pruning=archive-canonical + + Restart=always + RestartSec=10 + LimitNOFILE=65536 + + [Install] + WantedBy=multi-user.target + ``` + + 7. Start the service using the following commands: + - Reload systemd: + ```bash + sudo systemctl daemon-reload + ``` + - Enable service to start on boot: + ```bash + sudo systemctl enable polkadot-hub-rpc + ``` + - Start the Polkadot SDK node: + ```bash + sudo systemctl start polkadot-hub-rpc + ``` + - Check status and wait for sync: + ```bash + sudo systemctl status polkadot-hub-rpc + sudo journalctl -u polkadot-hub-rpc -f + ``` + + 8. You can use a few different commands to verify your node is running properly: + - Get chain information: + ```bash + curl -H "Content-Type: application/json" \ + -d '{"id":1, "jsonrpc":"2.0", "method": "system_chain", "params":[]}' \ + http://localhost:9944 + ``` + - Get the latest block: + ```bash + curl -H "Content-Type: application/json" \ + -d '{"id":1, "jsonrpc":"2.0", "method": "chain_getHeader", "params":[]}' \ + http://localhost:9944 + ``` + - Query node health: + ```bash + curl -H "Content-Type: application/json" \ + -d '{"id":1, "jsonrpc":"2.0", "method": "system_health", "params":[]}' \ + http://localhost:9944 + ``` + +## Monitor and Maintain RPC Node + +There are a few key monitoring and maintenance commands that can help you ensure RPC node uptime and performance including log monitoring and metrics. + +To view node logs, use the appropriate command for your setup: + +=== "Docker Setup" + + ```bash + docker logs -f polkadot-hub-rpc + ``` + +=== "systemd Setup" + + - View node logs: + ```bash + sudo journalctl -u polkadot-hub-rpc -f + ``` + - View recent logs: + ```bash + sudo journalctl -u polkadot-hub-rpc -n 100 + ``` + - Filter for errors: + ```bash + sudo journalctl -u polkadot-hub-rpc | grep -i error + ``` + +Monitoring key metrics like the following items can help you stay up to date on the performance and health of your node and allow you to intervene at the first sign of issues: + +- **Sync status**: Ensure node stays fully synced. +- **Peer connections**: Maintain 30+ peers for good connectivity. +- **Resource usage**: Monitor CPU, RAM, and disk I/O. +- **RPC request latency**: Track response times for the Polkadot SDK API. +- **Connection count**: Monitor active RPC connections. + +You can use the following information to configure [Prometheus](https://prometheus.io/docs/introduction/first_steps/){target=\_blank} to monitor, collect, and store your collator node metrics: + +- **URL**: Metrics are available to view at `http://localhost:9615/metrics` +- **Example Prometheus configuration**: Update your `prometheus.yml` to add the following code: + ```yaml + scrape_configs: + - job_name: 'polkadot-hub-rpc' + static_configs: + - targets: ['localhost:9615'] + ``` + +Key metric to monitor via Prometheus include: + - `substrate_block_height`: Current block height + - `substrate_finalized_height`: Finalized block height + - `substrate_peers_count`: Number of connected peers + - `substrate_ready_transactions_number`: Transaction queue size -1. **Docker-based Setup**: Simpler to set up and maintain -2. **Manual/Systemd Setup**: For production environments requiring more control - -Choose the option that best fits your needs. - ---- - -## Option 1: Docker-Based Setup - -This option uses Docker containers for the Polkadot SDK node, making it easy to set up and manage. - -### Step 1: Download Chain Specification - -Download the official Polkadot Hub (formerly known as Asset Hub) chain specification: - -```bash -curl -L https://raw.githubusercontent.com/paritytech/polkadot-sdk/master/cumulus/parachains/chain-specs/asset-hub-polkadot.json -o asset-hub-polkadot.json -``` - -**Note**: This chain specification is the official configuration file that defines the network parameters for Polkadot Hub. - -### Step 2: Download Database Snapshots (Optional but Recommended) - -Using pre-synchronized snapshots significantly reduces initial sync time from several days to just a few hours. You need to download both parachain and relay chain data. - -**Snapshot Provider**: https://snapshots.polkadot.io/ - -#### Create Directories - -```bash -mkdir -p my-node-data/chains/asset-hub-polkadot/db -mkdir -p my-node-data/chains/polkadot/db -``` - -#### Download Polkadot Hub Parachain Snapshot - -Choose between archive (complete history) or pruned (recent state) snapshots: - -**Archive Snapshot** (recommended for RPC with historical data): - -```bash -# Check https://snapshots.polkadot.io/ for the latest snapshot URL -export SNAPSHOT_URL_ASSET_HUB="https://snapshots.polkadot.io/polkadot-asset-hub-rocksdb-archive/LATEST" - -rclone copyurl $SNAPSHOT_URL_ASSET_HUB/files.txt files.txt -rclone copy --progress --transfers 20 \ - --http-url $SNAPSHOT_URL_ASSET_HUB \ - --no-traverse --http-no-head --disable-http2 \ - --inplace --no-gzip-encoding --size-only \ - --retries 6 --retries-sleep 10s \ - --files-from files.txt :http: my-node-data/chains/asset-hub-polkadot/db/ - -rm files.txt -``` - -**Parameter Explanation**: - -- `--transfers 20`: Uses 20 parallel transfers for faster download -- `--retries 6`: Automatically retries failed transfers up to 6 times -- `--retries-sleep 10s`: Waits 10 seconds between retry attempts -- `--size-only`: Only transfers if sizes differ (prevents unnecessary re-downloads) - -#### Download Polkadot Relay Chain Snapshot - -**Pruned Snapshot** (recommended for RPC nodes): - -```bash -# Check https://snapshots.polkadot.io/ for the latest snapshot URL -export SNAPSHOT_URL_RELAY="https://snapshots.polkadot.io/polkadot-rocksdb-prune/LATEST" - -rclone copyurl $SNAPSHOT_URL_RELAY/files.txt files.txt -rclone copy --progress --transfers 20 \ - --http-url $SNAPSHOT_URL_RELAY \ - --no-traverse --http-no-head --disable-http2 \ - --inplace --no-gzip-encoding --size-only \ - --retries 6 --retries-sleep 10s \ - --files-from files.txt :http: my-node-data/chains/polkadot/db/ - -rm files.txt -``` - -**Alternative Options**: - -- Pruned snapshot: `polkadot-rocksdb-prune` (smaller size, recent state) -- Archive snapshot: `polkadot-rocksdb-archive` (complete history, larger size) - -### Step 3: Start Polkadot Hub Node - -Launch the node using the official Parity Docker image: - -**Docker Image**: https://hub.docker.com/r/parity/polkadot-omni-node - -```bash -docker run -d --name polkadot-hub-rpc --restart unless-stopped \ - -p 9944:9944 \ - -p 9933:9933 \ - -p 9615:9615 \ - -p 30334:30334 \ - -p 30333:30333 \ - -v $(pwd)/asset-hub-polkadot.json:/asset-hub-polkadot.json \ - -v $(pwd)/my-node-data:/data \ - parity/polkadot-omni-node:v1.20.2 \ - --name=PolkadotHubRPC \ - --base-path=/data \ - --chain=/asset-hub-polkadot.json \ - --prometheus-external \ - --prometheus-port 9615 \ - --unsafe-rpc-external \ - --rpc-port=9944 \ - --rpc-cors=all \ - --rpc-methods=safe \ - --rpc-max-connections=1000 \ - --state-pruning=archive \ - --blocks-pruning=archive \ - -- \ - --base-path=/data \ - --chain=polkadot \ - --state-pruning=256 \ - --blocks-pruning=archive-canonical -``` - -**Critical Configuration Parameters**: - -**Port Mappings**: - -- `9944`: Polkadot SDK RPC endpoint (WebSocket/HTTP) -- `9933`: Polkadot SDK HTTP RPC endpoint -- `9615`: Prometheus metrics endpoint -- `30333/30334`: P2P networking ports - -**Node Parameters**: - -- `--unsafe-rpc-external`: Enables external RPC access -- `--rpc-cors=all`: Allows all origins for CORS -- `--rpc-methods=safe`: Only allows safe RPC methods -- `--state-pruning=archive`: Keeps complete state history -- `--blocks-pruning=archive`: Keeps all block data -- `--prometheus-external`: Exposes metrics externally - -**Security Warning**: The `--unsafe-rpc-external` flag should only be used in development or properly secured environments. For production, use a reverse proxy with authentication. - -### Step 4: Monitor Synchronization - -Monitor the node synchronization status: - -```bash -# Check sync status -curl -H "Content-Type: application/json" \ - -d '{"id":1, "jsonrpc":"2.0", "method": "system_syncState", "params":[]}' \ - http://localhost:9944 -``` - -**Expected Response Format**: - -```json -{ - "jsonrpc":"2.0", - "id":1, - "result":{ - "startingBlock":0, - "currentBlock":3394816, - "highestBlock":3394816 - } -} -``` - -**Synchronization Status**: - -- **In Progress**: `currentBlock` < `highestBlock` -- **Complete**: `currentBlock` = `highestBlock` - -### Step 5: Verify Setup - -Let's verify the Polkadot SDK RPC endpoint is working correctly. - -#### API Endpoint - -**Polkadot SDK RPC (Port 9944)**: - -- WebSocket: `ws://your-server:9944` -- HTTP: `http://your-server:9944` -- Purpose: Full Polkadot SDK API access for parachain data -- Use Cases: Polkadot SDK applications, parachain-specific operations - -#### Polkadot SDK RPC Tests - -**Get Chain Information**: - -```bash -curl -H "Content-Type: application/json" \ - -d '{"id":1, "jsonrpc":"2.0", "method": "system_chain", "params":[]}' \ - http://localhost:9944 -``` - -**Get Latest Block**: - -```bash -curl -H "Content-Type: application/json" \ - -d '{"id":1, "jsonrpc":"2.0", "method": "chain_getHeader", "params":[]}' \ - http://localhost:9944 -``` - -**Get Node Health**: - -```bash -curl -H "Content-Type: application/json" \ - -d '{"id":1, "jsonrpc":"2.0", "method": "system_health", "params":[]}' \ - http://localhost:9944 -``` - -### Managing Docker Containers - -**View logs**: - -```bash -# View node logs -docker logs -f polkadot-hub-rpc -``` - -**Stop container**: - -```bash -docker stop polkadot-hub-rpc -``` - -**Start container**: - -```bash -docker start polkadot-hub-rpc -``` - -**Remove container**: - -```bash -docker rm polkadot-hub-rpc -``` - -**Note**: For update procedures, see the [Updates and Upgrades](#updates-and-upgrades) section. - ---- - -## Option 2: Manual/Systemd Setup - -This option provides more control and is recommended for production environments requiring custom configurations. - -### Step 1: Install Rust and Required Toolchain - -```bash -# Install Rust -curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -source $HOME/.cargo/env - -# Install specific Rust version -rustup install 1.91.1 -rustup default 1.91.1 -rustup target add wasm32-unknown-unknown --toolchain 1.91.1 -rustup component add rust-src --toolchain 1.91.1 -``` - -### Step 2: Install Required Dependencies - -```bash -# Install system dependencies -sudo apt update -sudo apt install -y build-essential git clang curl libssl-dev llvm libudev-dev make protobuf-compiler -``` - -### Step 3: Install the Polkadot Omni Node - -```bash -# Install polkadot-omni-node -cargo install --locked polkadot-omni-node@0.11.0 - -# Verify installation -polkadot-omni-node --version -``` - -**Note**: Compiling polkadot-omni-node from source requires significant RAM (minimum 24GB recommended). The compilation may take 10-15 minutes on systems with adequate resources. - -### Step 4: Obtain Chain Specification - -Download the Polkadot Hub chain specification: - -```bash -curl -L https://raw.githubusercontent.com/paritytech/polkadot-sdk/master/cumulus/parachains/chain-specs/asset-hub-polkadot.json -o asset-hub-polkadot.json -``` - -### Step 5: Create User and Directory Structure - -```bash -# Create dedicated user -sudo useradd -r -s /bin/bash polkadot - -# Create data directory -sudo mkdir -p /var/lib/polkadot-hub-rpc - -# Copy chain spec to the directory -sudo cp asset-hub-polkadot.json /var/lib/polkadot-hub-rpc/ - -# Set permissions -sudo chown -R polkadot:polkadot /var/lib/polkadot-hub-rpc -``` - -### Step 6: Create Systemd Service for Polkadot SDK Node - -Create a service file for the Polkadot SDK RPC node: - -```bash -sudo nano /etc/systemd/system/polkadot-hub-rpc.service -``` - -Add the following configuration: - -```ini -[Unit] -Description=Polkadot Hub RPC Node -After=network.target - -[Service] -Type=simple -User=polkadot -Group=polkadot -WorkingDirectory=/var/lib/polkadot-hub-rpc - -ExecStart=/usr/local/bin/polkadot-omni-node \ - --name=PolkadotHubRPC \ - --chain=/var/lib/polkadot-hub-rpc/asset-hub-polkadot.json \ - --base-path=/var/lib/polkadot-hub-rpc \ - --port=30333 \ - --rpc-port=9944 \ - --rpc-external \ - --rpc-cors=all \ - --rpc-methods=safe \ - --rpc-max-connections=1000 \ - --prometheus-port=9615 \ - --prometheus-external \ - --state-pruning=archive \ - --blocks-pruning=archive \ - -- \ - --chain=polkadot \ - --base-path=/var/lib/polkadot-hub-rpc \ - --port=30334 \ - --state-pruning=256 \ - --blocks-pruning=archive-canonical - -Restart=always -RestartSec=10 -LimitNOFILE=65536 - -[Install] -WantedBy=multi-user.target -``` - -### Step 7: Start Service - -```bash -# Reload systemd -sudo systemctl daemon-reload - -# Enable service to start on boot -sudo systemctl enable polkadot-hub-rpc - -# Start the Polkadot SDK node -sudo systemctl start polkadot-hub-rpc - -# Check status and wait for sync -sudo systemctl status polkadot-hub-rpc -sudo journalctl -u polkadot-hub-rpc -f -``` - -### Step 8: Verify Setup - -Use the same verification tests as in the Docker setup (see Step 5 above). - ---- - -## Monitoring and Maintenance - -### Log Management - -**Docker Setup**: - -```bash -# View node logs -docker logs -f polkadot-hub-rpc -``` - -**Systemd Setup**: - -```bash -# View node logs -sudo journalctl -u polkadot-hub-rpc -f - -# View recent logs -sudo journalctl -u polkadot-hub-rpc -n 100 - -# Filter for errors -sudo journalctl -u polkadot-hub-rpc | grep -i error -``` - -### Performance Monitoring - -Monitor key metrics: - -- **Sync status**: Ensure node stays fully synced -- **Peer connections**: Maintain 30+ peers for good connectivity -- **Resource usage**: Monitor CPU, RAM, and disk I/O -- **RPC request latency**: Track response times for the Polkadot SDK API -- **Connection count**: Monitor active RPC connections - -**Prometheus Metrics**: - -Metrics are available at `http://localhost:9615/metrics` - -Example Prometheus scrape configuration: - -```yaml -scrape_configs: - - job_name: 'polkadot-hub-rpc' - static_configs: - - targets: ['localhost:9615'] -``` - -**Key Metrics to Monitor**: +### Database Maintenance -- `substrate_block_height`: Current block height -- `substrate_finalized_height`: Finalized block height -- `substrate_peers_count`: Number of connected peers -- `substrate_ready_transactions_number`: Transaction queue size +Check database size periodically using the commands for your selected setup: -### Database Maintenance +=== "Docker Setup" -Check database size periodically: + ```bash + du -sh my-node-data + ``` -```bash -# Docker setup -du -sh my-node-data +=== "systemd Setup" -# Systemd setup -du -sh /var/lib/polkadot-hub-rpc -``` + ```bash + du -sh /var/lib/polkadot-hub-rpc + ``` The node handles pruning automatically based on configuration unless running in archive mode. ### Updates and Upgrades -**Docker Setup**: - -```bash -# Pull latest image -docker pull parity/polkadot-omni-node:v1.20.2 - -# Restart container -docker stop polkadot-hub-rpc -docker rm polkadot-hub-rpc - -# Start new container (use same command from setup) -``` - -**Systemd Setup**: - -```bash -# Stop service -sudo systemctl stop polkadot-hub-rpc - -# Backup data -sudo cp -r /var/lib/polkadot-hub-rpc /var/lib/polkadot-hub-rpc.backup - -# Update binary -cargo install --locked --force polkadot-omni-node@ - -# Restart service -sudo systemctl start polkadot-hub-rpc -``` +Use the following commands for updating or upgrading your RPC node according to your setup: + +=== "Docker Setup" + + 1. Pull the latest image: + ```bash + docker pull parity/polkadot-omni-node:v1.20.2 + ``` + 2. Restart the container: + ```bash + docker stop polkadot-hub-rpc + docker rm polkadot-hub-rpc + ``` + +=== "systemd Setup" + + 1. Stop the service: + ```bash + sudo systemctl stop polkadot-hub-rpc + ``` + 2. Backup data: + ```bash + sudo cp -r /var/lib/polkadot-hub-rpc /var/lib/polkadot-hub-rpc.backup + ``` + 3. Update the binary: + ```bash + cargo install --locked --force polkadot-omni-node@ + ``` + 4. Restart the service: + ```bash + sudo systemctl start polkadot-hub-rpc + ``` ## Conclusion Running an RPC node for Polkadot Hub provides essential infrastructure for applications and users to interact with the network. By following this guide, you have set up a production-ready RPC node that: -- Provides reliable access to Polkadot Hub's asset management, governance, and cross-chain communication features -- Supports both Docker and systemd deployment options for flexibility -- Implements proper monitoring, security, and maintenance practices -- Serves as a foundation for building and operating Polkadot SDK applications +- Provides reliable access to Polkadot Hub's asset management, governance, and cross-chain communication features. +- Supports both Docker and systemd deployment options for flexibility. +- Implements proper monitoring, security, and maintenance practices. +- Serves as a foundation for building and operating Polkadot SDK applications. Regular maintenance, security updates, and monitoring will ensure your RPC node continues to serve your users reliably. As the Polkadot network evolves, stay informed about updates and best practices through the official channels and community resources listed in this guide. From 3b791ace30a33bb0c469f1263aeb182b41f271a3 Mon Sep 17 00:00:00 2001 From: DAWN KELLY Date: Fri, 21 Nov 2025 16:00:44 -0500 Subject: [PATCH 10/21] formatting edits --- node-infrastructure/run-a-node/polkadot-hub-rpc.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/node-infrastructure/run-a-node/polkadot-hub-rpc.md b/node-infrastructure/run-a-node/polkadot-hub-rpc.md index 247651263..7af6e5ddc 100644 --- a/node-infrastructure/run-a-node/polkadot-hub-rpc.md +++ b/node-infrastructure/run-a-node/polkadot-hub-rpc.md @@ -247,7 +247,7 @@ Select the best option for your project, then use the steps in the following tab ``` === "Manual systemd Setup" - This option provides more control and is recommended for production environments requiring custom configurations. + This option uses systemd to provide more control and is recommended for production environments requiring custom configurations. 1. Install Rust and required toolchain using the following command: ```bash @@ -435,11 +435,12 @@ You can use the following information to configure [Prometheus](https://promethe - targets: ['localhost:9615'] ``` -Key metric to monitor via Prometheus include: - - `substrate_block_height`: Current block height - - `substrate_finalized_height`: Finalized block height - - `substrate_peers_count`: Number of connected peers - - `substrate_ready_transactions_number`: Transaction queue size +Key metrics to monitor via Prometheus include: + +- `substrate_block_height`: Current block height +- `substrate_finalized_height`: Finalized block height +- `substrate_peers_count`: Number of connected peers +- `substrate_ready_transactions_number`: Transaction queue size ### Database Maintenance From b84d71de716ce49299d8fafdf3c9155f08f4855c Mon Sep 17 00:00:00 2001 From: Bruno Galvao Date: Mon, 24 Nov 2025 16:24:55 +0700 Subject: [PATCH 11/21] Update guides --- .ai/pages/node-infrastructure-overview.md | 81 +++++++++ ...-infrastructure-run-a-collator-collator.md | 169 ++++++++++++++---- ...infrastructure-run-a-node-parachain-rpc.md | 158 ++++++++-------- ...rastructure-run-a-node-polkadot-hub-rpc.md | 65 +++---- node-infrastructure/.nav.yml | 1 + node-infrastructure/overview.md | 80 +++++++++ .../run-a-collator/collator.md | 169 ++++++++++++++---- node-infrastructure/run-a-node/.nav.yml | 2 +- .../run-a-node/parachain-rpc.md | 158 ++++++++-------- .../run-a-node/polkadot-hub-rpc.md | 65 +++---- 10 files changed, 640 insertions(+), 308 deletions(-) create mode 100644 .ai/pages/node-infrastructure-overview.md rename node-infrastructure/run-a-node/system-parachain-rpc.md => .ai/pages/node-infrastructure-run-a-node-parachain-rpc.md (70%) create mode 100644 node-infrastructure/overview.md rename .ai/pages/node-infrastructure-run-a-node-system-parachain-rpc.md => node-infrastructure/run-a-node/parachain-rpc.md (71%) diff --git a/.ai/pages/node-infrastructure-overview.md b/.ai/pages/node-infrastructure-overview.md new file mode 100644 index 000000000..fe7e4f507 --- /dev/null +++ b/.ai/pages/node-infrastructure-overview.md @@ -0,0 +1,81 @@ +--- +title: Node Infrastructure +description: Overview of running nodes in the Polkadot ecosystem, including RPC nodes, collators, and validators. +categories: Infrastructure +url: https://docs.polkadot.com/node-infrastructure/overview/ +--- + +# Node Infrastructure Overview + +## Introduction + +The Polkadot network relies on various types of nodes to maintain security, provide data access, and produce blocks. This section covers everything you need to know about running infrastructure for the Polkadot ecosystem. + +Whether you want to provide RPC endpoints for applications, produce blocks for a parachain, or secure the relay chain as a validator, this guide will help you understand your options and get started. + +## Node Types + +### RPC Nodes + +RPC nodes provide API access to blockchain data without participating in consensus. They are essential infrastructure for: + +- **Applications and dApps**: Query blockchain state and submit transactions +- **Block explorers**: Index and display blockchain data +- **Wallets**: Check balances and broadcast transactions +- **Development**: Test and debug applications + +RPC nodes can be run for both the relay chain and parachains, with varying levels of data retention (pruned vs archive). + +### Collators + +Collators are block producers for parachains. They perform critical functions: + +- **Collect transactions**: Aggregate user transactions into blocks +- **Produce blocks**: Create parachain block candidates +- **Generate proofs**: Produce state transition proofs (Proof-of-Validity) +- **Submit to validators**: Send block candidates to relay chain validators + +Unlike validators, collators do not provide security guarantees—that responsibility lies with relay chain validators. However, collators are essential for parachain liveness and censorship resistance. + +### Validators + +Validators secure the Polkadot relay chain through Nominated Proof of Stake (NPoS). They: + +- **Validate blocks**: Verify parachain blocks and relay chain transactions +- **Participate in consensus**: Run BABE and GRANDPA protocols +- **Earn rewards**: Receive staking rewards for honest behavior +- **Risk slashing**: Face penalties for misbehavior or downtime + +Running a validator requires significant technical expertise, reliable infrastructure, and a stake of DOT tokens. + +## Next Steps + +Choose your path based on your goals: + +
+ +- :material-api:{ .lg .middle } **Run RPC Nodes** + + --- + + Provide API access for applications, explorers, and wallets + + [:octicons-arrow-right-24: Run a Node](/node-infrastructure/run-a-node/parachain-rpc/) + +- :material-cube-outline:{ .lg .middle } **Run a Collator** + + --- + + Produce blocks for system parachains or your own parachain + + [:octicons-arrow-right-24: Run a Collator](/node-infrastructure/run-a-collator/collator/) + +- :material-shield-check:{ .lg .middle } **Run a Validator** + + --- + + Secure the relay chain and earn staking rewards + + [:octicons-arrow-right-24: Run a Validator](/node-infrastructure/run-a-validator/requirements/) + +
diff --git a/.ai/pages/node-infrastructure-run-a-collator-collator.md b/.ai/pages/node-infrastructure-run-a-collator-collator.md index cb2adbf48..e1bed1380 100644 --- a/.ai/pages/node-infrastructure-run-a-collator-collator.md +++ b/.ai/pages/node-infrastructure-run-a-collator-collator.md @@ -57,18 +57,12 @@ Block-producing collators require robust hardware for reliable operation: ### Software Requirements -Collators use the **Polkadot Omni Node**, a universal binary that runs any parachain using a chain specification file. +Collators use the **Polkadot Parachain** binary, the standard client for running Polkadot system parachains. Required software: - **Operating System**: Ubuntu 22.04 LTS (recommended) or similar Linux distribution -- **Docker**: For running subkey utility -- **Rust Toolchain**: Version 1.91.1 or as specified by the runtime -- **Dependencies**: - ```bash - sudo apt update - sudo apt install -y build-essential git clang curl libssl-dev llvm libudev-dev make protobuf-compiler - ``` +- **Docker**: Required for obtaining binaries and running containers ### Account Requirements @@ -80,54 +74,81 @@ You'll need: ## Installation -### Step 1: Install Rust and Required Toolchain +### Step 1: Install the Polkadot Parachain Binary + +**For Docker deployment:** ```bash -# Install Rust -curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -source $HOME/.cargo/env - -# Install specific Rust version -rustup install 1.91.1 -rustup default 1.91.1 -rustup target add wasm32-unknown-unknown --toolchain 1.91.1 -rustup component add rust-src --toolchain 1.91.1 +# Pull the polkadot-parachain image +docker pull parity/polkadot-parachain:stable2509-2 + +# Verify installation +docker run --rm parity/polkadot-parachain:stable2509-2 --version ``` -### Step 2: Install the Polkadot Omni Node +**For bare-metal deployment:** + +Extract the binary from the Docker image: ```bash -# Install polkadot-omni-node -cargo install --locked polkadot-omni-node@0.11.0 +# Create a temporary container and copy the binary +docker create --name temp-parachain parity/polkadot-parachain:stable2509-2 +sudo docker cp temp-parachain:/usr/local/bin/polkadot-parachain /usr/local/bin/ +docker rm temp-parachain # Verify installation -polkadot-omni-node --version +polkadot-parachain --version ``` -### Step 3: Generate Node Key +Check [Docker Hub](https://hub.docker.com/r/parity/polkadot-parachain/tags) for the latest stable tags. + +### Step 2: Generate Node Key Generate a stable node key for consistent peer ID: ```bash # Create directory for node data sudo mkdir -p /var/lib/polkadot-collator +``` + +**Using Docker:** -# Generate node key using Docker -docker run -it parity/subkey:latest generate-node-key > /var/lib/polkadot-collator/node.key +```bash +# Generate node key (outputs peer ID to console, saves key to file) +docker run --rm -v /var/lib/polkadot-collator:/data \ + parity/polkadot-parachain:stable2509-2 \ + key generate-node-key --file /data/node.key -# The output displays your peer ID -# Example: 12D3KooWExcVYu7Mvjd4kxPVLwN2ZPnZ5NyLZ5ft477wqzfP2q6E +# Example output: 12D3KooWExcVYu7Mvjd4kxPVLwN2ZPnZ5NyLZ5ft477wqzfP2q6E +``` + +**Using native binary:** + +```bash +# Generate node key +polkadot-parachain key generate-node-key --file /var/lib/polkadot-collator/node.key + +# Example output: 12D3KooWExcVYu7Mvjd4kxPVLwN2ZPnZ5NyLZ5ft477wqzfP2q6E ``` Save the peer ID for future reference. -### Step 4: Generate Account Key +### Step 3: Generate Account Key Generate an account for on-chain transactions: +**Using Docker:** + +```bash +# Generate account key with sr25519 scheme +docker run --rm parity/polkadot-parachain:stable2509-2 key generate --scheme sr25519 +``` + +**Using native binary:** + ```bash # Generate account key with sr25519 scheme -docker run -it parity/subkey:latest generate --scheme sr25519 +polkadot-parachain key generate --scheme sr25519 ``` Save the output containing: @@ -139,7 +160,7 @@ Save the output containing: **Security**: Store the secret phrase securely. Never share it. Consider using a hardware wallet for production collators. -### Step 5: Obtain Chain Specification +### Step 4: Obtain Chain Specification Download the chain specification for your target system parachain: @@ -178,7 +199,7 @@ chain-spec-builder create \ - People Chain: 1004 - Coretime Chain: 1005 -### Step 6: Create User and Directory Structure +### Step 5: Create User and Directory Structure ```bash # Create dedicated user @@ -193,7 +214,7 @@ sudo chown -R polkadot:polkadot /var/lib/polkadot-collator ## Configuration -### Create Systemd Service File +### Native Binary Systemd Service Create a service file for your collator: @@ -215,7 +236,7 @@ Group=polkadot WorkingDirectory=/var/lib/polkadot-collator # Block-Producing Collator Configuration -ExecStart=/usr/local/bin/polkadot-omni-node \ +ExecStart=/usr/local/bin/polkadot-parachain \ --collator \ --chain=/var/lib/polkadot-collator/chain-spec.json \ --base-path=/var/lib/polkadot-collator \ @@ -245,6 +266,55 @@ WantedBy=multi-user.target - `--name`: Your collator name (visible in telemetry) - Relay chain uses `--sync=warp` for faster initial sync +### Docker Systemd Service + +If using Docker, create a service file with Docker configuration: + +```bash +sudo nano /etc/systemd/system/polkadot-collator.service +``` + +Add the following configuration: + +```ini +[Unit] +Description=Polkadot System Parachain Collator (Docker) +After=network.target docker.service +Requires=docker.service + +[Service] +Type=simple +Restart=always +RestartSec=10 +TimeoutStartSec=0 + +ExecStartPre=-/usr/bin/docker stop polkadot-collator +ExecStartPre=-/usr/bin/docker rm polkadot-collator + +ExecStart=/usr/bin/docker run --rm \ + --name polkadot-collator \ + --network host \ + -v /var/lib/polkadot-collator:/data \ + parity/polkadot-parachain:stable2509-2 \ + --collator \ + --chain=/data/chain-spec.json \ + --base-path=/data \ + --port=30333 \ + --rpc-port=9944 \ + --prometheus-port=9615 \ + --node-key-file=/data/node.key \ + --name="YourCollatorName" \ + -- \ + --chain=polkadot \ + --port=30334 \ + --sync=warp + +ExecStop=/usr/bin/docker stop polkadot-collator + +[Install] +WantedBy=multi-user.target +``` + ## Running the Collator ### Step 1: Start the Service @@ -491,7 +561,7 @@ The node handles pruning automatically. - Fellowship GitHub - Matrix channels -**Upgrade Procedure**: +**Upgrade Procedure (Bare-Metal)**: ```bash # Stop the service @@ -500,11 +570,14 @@ sudo systemctl stop polkadot-collator # Backup data (recommended) sudo cp -r /var/lib/polkadot-collator /var/lib/polkadot-collator.backup -# Update polkadot-omni-node -cargo install --locked --force polkadot-omni-node@ +# Pull new image and extract binary +docker pull parity/polkadot-parachain: +docker create --name temp-parachain parity/polkadot-parachain: +sudo docker cp temp-parachain:/usr/local/bin/polkadot-parachain /usr/local/bin/ +docker rm temp-parachain # Verify version -polkadot-omni-node --version +polkadot-parachain --version # Restart service sudo systemctl start polkadot-collator @@ -513,6 +586,28 @@ sudo systemctl start polkadot-collator sudo systemctl status polkadot-collator ``` +**Upgrade Procedure (Docker)**: + +```bash +# Stop the service +sudo systemctl stop polkadot-collator + +# Backup data (recommended) +sudo cp -r /var/lib/polkadot-collator /var/lib/polkadot-collator.backup + +# Pull new image +docker pull parity/polkadot-parachain: + +# Update the image tag in /etc/systemd/system/polkadot-collator.service + +# Reload systemd and restart +sudo systemctl daemon-reload +sudo systemctl start polkadot-collator + +# Verify service is running +sudo systemctl status polkadot-collator +``` + **Note**: For log monitoring, see the [Log Management](#log-management) section. ## Conclusion diff --git a/node-infrastructure/run-a-node/system-parachain-rpc.md b/.ai/pages/node-infrastructure-run-a-node-parachain-rpc.md similarity index 70% rename from node-infrastructure/run-a-node/system-parachain-rpc.md rename to .ai/pages/node-infrastructure-run-a-node-parachain-rpc.md index 47440b93c..e38c1a1fd 100644 --- a/node-infrastructure/run-a-node/system-parachain-rpc.md +++ b/.ai/pages/node-infrastructure-run-a-node-parachain-rpc.md @@ -1,37 +1,60 @@ --- -title: Run an RPC Node for System Parachains -description: Complete guide to set up and run an RPC node for Polkadot system parachains including Bridge Hub, People Chain, and Coretime Chain. +title: Run a Parachain RPC Node +description: Complete guide to set up and run an RPC node for any Polkadot parachain, with system parachains as examples. categories: Infrastructure +url: https://docs.polkadot.com/node-infrastructure/run-a-node/parachain-rpc/ --- -# Run an RPC Node for System Parachains +# Run a Parachain RPC Node ## Overview -System parachains are core infrastructure parachains that provide essential services to the Polkadot network. Running an RPC node for these parachains enables applications, wallets, and users to interact with their specialized functionality: +Running an RPC node for a parachain enables applications, wallets, and users to interact with the parachain's functionality. This guide applies to **any parachain** in the Polkadot ecosystem, including: -- **Bridge Hub**: Cross-chain asset transfers via trustless bridges -- **People Chain**: Identity and social credential management -- **Coretime Chain**: Blockspace allocation and core time management +- **System parachains**: Bridge Hub, People Chain, Coretime Chain +- **Common good parachains**: Collectives, Encointer +- **Commercial parachains**: Any parachain with a publicly available chain specification -Each system parachain RPC node provides access through: -- **Polkadot SDK Node RPC** (Port 9944): Native Polkadot API (WebSocket and HTTP) - -This setup enables block explorer indexing and provides full compatibility with Polkadot SDK development tools. +Each parachain RPC node provides access through the Polkadot SDK Node RPC (Port 9944), offering native Polkadot API access via WebSocket and HTTP. This setup enables block explorer indexing and provides full compatibility with Polkadot SDK development tools. **Important Note**: The parameters and configurations in this guide are provided as illustrative examples. You may need to modify them according to your specific environment, hardware capabilities, and network conditions. -## Choosing a System Parachain +## Obtaining a Chain Specification + +To run an RPC node for any parachain, you need its **chain specification file**. This JSON file defines the network parameters, genesis state, and bootnodes. + +### System Parachains + +System parachain chain specs are available from multiple sources: -This guide uses **People Chain** as the example, but the same principles and setup procedures apply to all system parachains. Simply substitute the appropriate values from the table below for your chosen parachain: +**Option 1: Chainspec Collection (Recommended)** -| Parachain | Para ID | Chain Spec File | Snapshot Path | Chain Name | -|-----------|---------|-----------------|---------------|------------| -| **Bridge Hub** | 1002 | `bridge-hub-polkadot.json` | `polkadot-bridge-hub-rocksdb-archive` | `bridge-hub-polkadot` | -| **People Chain** | 1004 | `people-polkadot.json` | `polkadot-people-rocksdb-archive` | `people-polkadot` | -| **Coretime Chain** | 1005 | `coretime-polkadot.json` | `polkadot-coretime-rocksdb-archive` | `coretime-polkadot` | +Visit the [Chainspec Collection](https://paritytech.github.io/chainspecs/) to download official chain specifications. -**Note**: Throughout this guide, we use People Chain values. To set up a different system parachain, replace the chain spec file, snapshot path, and chain name with the corresponding values from the table above. +**Option 2: Polkadot SDK Repository** + +Download directly from the Polkadot SDK repository: + +```bash +# Example for People Chain +curl -L https://raw.githubusercontent.com/paritytech/polkadot-sdk/master/cumulus/parachains/chain-specs/people-polkadot.json -o chain-spec.json +``` + +| System Parachain | Para ID | Chain Spec File | Snapshot Path | +|------------------|---------|-----------------|---------------| +| **Bridge Hub** | 1002 | `bridge-hub-polkadot.json` | `polkadot-bridge-hub-rocksdb-archive` | +| **People Chain** | 1004 | `people-polkadot.json` | `polkadot-people-rocksdb-archive` | +| **Coretime Chain** | 1005 | `coretime-polkadot.json` | `polkadot-coretime-rocksdb-archive` | + +### Other Parachains + +For non-system parachains: + +- **Check the parachain's documentation** for official chain specification files +- **Contact the parachain team** if no public chain spec is available +- **Export from a running node** using `system_chainSpec` RPC method if you have access to an existing node + +**Note**: Throughout this guide, we use **People Chain** as the example. To set up a different parachain, substitute the chain spec file, snapshot path, and chain name with values for your target parachain. ## Prerequisites @@ -64,9 +87,8 @@ RPC nodes serving production traffic require robust hardware: Required software: - **Operating System**: Ubuntu 22.04 LTS (recommended) or similar Linux distribution -- **Docker**: Latest version installed and running (for Docker-based setup) +- **Docker**: Required for obtaining binaries and running containers - **rclone**: (Optional but recommended) Command-line program for managing files on cloud storage (https://rclone.org/downloads/) -- **Rust Toolchain**: Version 1.91.1 or as specified by runtime (for manual build) ## Setup Options @@ -83,22 +105,14 @@ Choose the option that best fits your needs. This option uses Docker containers for the Polkadot SDK node, making it easy to set up and manage. -### Step 1: Download Chain Specification - -Download the official chain specification for People Chain: - -```bash -curl -L https://raw.githubusercontent.com/paritytech/polkadot-sdk/master/cumulus/parachains/chain-specs/people-polkadot.json -o people-polkadot.json -``` - -**Note**: This chain specification is the official configuration file that defines the network parameters for People Chain. - -### Step 2: Download Database Snapshots (Optional but Recommended) +### Step 1: Download Database Snapshots (Optional but Recommended) Using pre-synchronized snapshots significantly reduces initial sync time from several days to just a few hours. You need to download both parachain and relay chain data. **Snapshot Provider**: https://snapshots.polkadot.io/ +**Note**: Snapshots are available for system parachains and the Polkadot relay chain. For other parachains, check with the parachain team for snapshot availability or sync from genesis. + #### Create Directories ```bash @@ -160,11 +174,13 @@ rm files.txt - Pruned snapshot: `polkadot-rocksdb-prune` (smaller size, recent state) - Archive snapshot: `polkadot-rocksdb-archive` (complete history, larger size) -### Step 3: Start People Chain Node +### Step 2: Start the Parachain Node Launch the node using the official Parity Docker image. -**Docker Image**: https://hub.docker.com/r/parity/polkadot-omni-node +**Docker Image**: https://hub.docker.com/r/parity/polkadot-parachain + +**Note**: The `parity/polkadot-parachain` image works for system parachains and parachains built with standard Cumulus templates. For parachains with custom runtimes, check the parachain's documentation for their specific Docker image or binary. ```bash docker run -d --name people-chain-rpc --restart unless-stopped \ @@ -175,7 +191,7 @@ docker run -d --name people-chain-rpc --restart unless-stopped \ -p 30333:30333 \ -v $(pwd)/people-polkadot.json:/people-polkadot.json \ -v $(pwd)/my-node-data:/data \ - parity/polkadot-omni-node:v1.20.2 \ + parity/polkadot-parachain:stable2509-2 \ --name=PeopleChainRPC \ --base-path=/data \ --chain=/people-polkadot.json \ @@ -215,7 +231,7 @@ docker run -d --name people-chain-rpc --restart unless-stopped \ **Security Warning**: The `--unsafe-rpc-external` flag should only be used in development or properly secured environments. For production, use a reverse proxy with authentication. -### Step 4: Monitor Synchronization +### Step 3: Monitor Synchronization Monitor the node synchronization status: @@ -245,7 +261,7 @@ curl -H "Content-Type: application/json" \ - **In Progress**: `currentBlock` < `highestBlock` - **Complete**: `currentBlock` = `highestBlock` -### Step 5: Verify Setup +### Step 4: Verify Setup Let's verify the Polkadot SDK RPC endpoint is working correctly. @@ -335,39 +351,26 @@ docker rm people-chain-rpc This option provides more control and is recommended for production environments requiring custom configurations. -### Step 1: Install Rust and Required Toolchain +### Step 1: Install the Polkadot Parachain Binary -```bash -# Install Rust -curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -source $HOME/.cargo/env - -# Install specific Rust version -rustup install 1.91.1 -rustup default 1.91.1 -rustup target add wasm32-unknown-unknown --toolchain 1.91.1 -rustup component add rust-src --toolchain 1.91.1 -``` - -### Step 2: Install the Polkadot Omni Node +Extract the binary from the official Docker image: ```bash -# Install polkadot-omni-node -cargo install --locked polkadot-omni-node@0.11.0 +# Pull the image and extract binary +docker pull parity/polkadot-parachain:stable2509-2 +docker create --name temp-parachain parity/polkadot-parachain:stable2509-2 +sudo docker cp temp-parachain:/usr/local/bin/polkadot-parachain /usr/local/bin/ +docker rm temp-parachain # Verify installation -polkadot-omni-node --version +polkadot-parachain --version ``` -### Step 3: Obtain Chain Specification +Check [Docker Hub](https://hub.docker.com/r/parity/polkadot-parachain/tags) for the latest stable tags. -Download the People Chain specification: +### Step 2: Create User and Directory Structure -```bash -curl -L https://raw.githubusercontent.com/paritytech/polkadot-sdk/master/cumulus/parachains/chain-specs/people-polkadot.json -o people-polkadot.json -``` - -### Step 4: Create User and Directory Structure +Ensure you have downloaded your parachain's chain specification as described in [Obtaining a Chain Specification](#obtaining-a-chain-specification). ```bash # Create dedicated user (skip if already exists) @@ -383,9 +386,9 @@ sudo cp people-polkadot.json /var/lib/people-chain-rpc/ sudo chown -R polkadot:polkadot /var/lib/people-chain-rpc ``` -### Step 5: Create Systemd Service +### Step 3: Create Systemd Service -Create a service file for the People Chain RPC node: +Create a service file for your parachain RPC node: ```bash sudo nano /etc/systemd/system/people-chain-rpc.service @@ -404,7 +407,7 @@ User=polkadot Group=polkadot WorkingDirectory=/var/lib/people-chain-rpc -ExecStart=/usr/local/bin/polkadot-omni-node \ +ExecStart=/usr/local/bin/polkadot-parachain \ --name=PeopleChainRPC \ --chain=/var/lib/people-chain-rpc/people-polkadot.json \ --base-path=/var/lib/people-chain-rpc \ @@ -433,7 +436,7 @@ LimitNOFILE=65536 WantedBy=multi-user.target ``` -### Step 6: Start Service +### Step 4: Start Service ```bash # Reload systemd @@ -450,9 +453,9 @@ sudo systemctl status people-chain-rpc sudo journalctl -u people-chain-rpc -f ``` -### Step 7: Verify Setup +### Step 5: Verify Setup -Use the same verification tests as in the Docker setup (see Step 5 above). +Use the same verification tests as in the Docker setup (see Step 4 above). --- @@ -496,12 +499,14 @@ Metrics are available at `http://localhost:9615/metrics` Example Prometheus scrape configuration: +{% raw %} ```yaml scrape_configs: - job_name: 'people-chain-rpc' static_configs: - targets: ['localhost:9615'] ``` +{% endraw %} **Key Metrics to Monitor**: @@ -530,13 +535,13 @@ The node handles pruning automatically based on configuration unless running in ```bash # Pull latest image -docker pull parity/polkadot-omni-node:v1.20.2 +docker pull parity/polkadot-parachain: # Restart container docker stop people-chain-rpc docker rm people-chain-rpc -# Start new container (use same command from setup) +# Start new container (use same command from setup with updated image tag) ``` **Systemd Setup**: @@ -548,8 +553,11 @@ sudo systemctl stop people-chain-rpc # Backup data sudo cp -r /var/lib/people-chain-rpc /var/lib/people-chain-rpc.backup -# Update binary -cargo install --locked --force polkadot-omni-node@ +# Pull new image and extract binary +docker pull parity/polkadot-parachain: +docker create --name temp-parachain parity/polkadot-parachain: +sudo docker cp temp-parachain:/usr/local/bin/polkadot-parachain /usr/local/bin/ +docker rm temp-parachain # Restart service sudo systemctl start people-chain-rpc @@ -557,11 +565,11 @@ sudo systemctl start people-chain-rpc ## Conclusion -Running an RPC node for system parachains provides critical infrastructure for accessing specialized Polkadot network services. By following this guide, you have set up a production-ready RPC node that: +Running a parachain RPC node provides critical infrastructure for accessing Polkadot network services. By following this guide, you have set up a production-ready RPC node that: -- Enables applications and users to interact with essential system parachain features (identity management, cross-chain bridges, or coretime allocation) +- Provides reliable access to parachain functionality for applications and users - Supports flexible deployment with both Docker and systemd options - Implements comprehensive monitoring, security, and maintenance practices -- Can be easily adapted for any system parachain by substituting the appropriate chain specification +- Can be adapted for any parachain by substituting the appropriate chain specification -Whether you're running a node for People Chain, Bridge Hub, or Coretime Chain, regular maintenance and monitoring will ensure your RPC node continues to provide reliable service. Stay updated with the latest releases and best practices to keep your infrastructure secure and performant. +Whether you're running a node for system parachains (People Chain, Bridge Hub, Coretime Chain) or other parachains in the ecosystem, regular maintenance and monitoring will ensure your RPC node continues to provide reliable service. Stay updated with the latest releases and best practices to keep your infrastructure secure and performant. diff --git a/.ai/pages/node-infrastructure-run-a-node-polkadot-hub-rpc.md b/.ai/pages/node-infrastructure-run-a-node-polkadot-hub-rpc.md index 80f5ce475..7a8368545 100644 --- a/.ai/pages/node-infrastructure-run-a-node-polkadot-hub-rpc.md +++ b/.ai/pages/node-infrastructure-run-a-node-polkadot-hub-rpc.md @@ -53,9 +53,8 @@ RPC nodes serving production traffic require robust hardware: Required software: - **Operating System**: Ubuntu 22.04 LTS (recommended) or similar Linux distribution -- **Docker**: Latest version installed and running (for Docker-based setup) +- **Docker**: Required for obtaining binaries and running containers - **rclone**: (Optional but recommended) Command-line program for managing files on cloud storage (https://rclone.org/downloads/) -- **Rust Toolchain**: Version 1.91.1 or later (for manual build) ## Setup Options @@ -151,7 +150,7 @@ rm files.txt Launch the node using the official Parity Docker image: -**Docker Image**: https://hub.docker.com/r/parity/polkadot-omni-node +**Docker Image**: https://hub.docker.com/r/parity/polkadot-parachain ```bash docker run -d --name polkadot-hub-rpc --restart unless-stopped \ @@ -162,7 +161,7 @@ docker run -d --name polkadot-hub-rpc --restart unless-stopped \ -p 30333:30333 \ -v $(pwd)/asset-hub-polkadot.json:/asset-hub-polkadot.json \ -v $(pwd)/my-node-data:/data \ - parity/polkadot-omni-node:v1.20.2 \ + parity/polkadot-parachain:stable2509-2 \ --name=PolkadotHubRPC \ --base-path=/data \ --chain=/asset-hub-polkadot.json \ @@ -306,41 +305,24 @@ docker rm polkadot-hub-rpc This option provides more control and is recommended for production environments requiring custom configurations. -### Step 1: Install Rust and Required Toolchain +### Step 1: Install the Polkadot Parachain Binary -```bash -# Install Rust -curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -source $HOME/.cargo/env - -# Install specific Rust version -rustup install 1.91.1 -rustup default 1.91.1 -rustup target add wasm32-unknown-unknown --toolchain 1.91.1 -rustup component add rust-src --toolchain 1.91.1 -``` - -### Step 2: Install Required Dependencies - -```bash -# Install system dependencies -sudo apt update -sudo apt install -y build-essential git clang curl libssl-dev llvm libudev-dev make protobuf-compiler -``` - -### Step 3: Install the Polkadot Omni Node +Extract the binary from the official Docker image: ```bash -# Install polkadot-omni-node -cargo install --locked polkadot-omni-node@0.11.0 +# Pull the image and extract binary +docker pull parity/polkadot-parachain:stable2509-2 +docker create --name temp-parachain parity/polkadot-parachain:stable2509-2 +sudo docker cp temp-parachain:/usr/local/bin/polkadot-parachain /usr/local/bin/ +docker rm temp-parachain # Verify installation -polkadot-omni-node --version +polkadot-parachain --version ``` -**Note**: Compiling polkadot-omni-node from source requires significant RAM (minimum 24GB recommended). The compilation may take 10-15 minutes on systems with adequate resources. +Check [Docker Hub](https://hub.docker.com/r/parity/polkadot-parachain/tags) for the latest stable tags. -### Step 4: Obtain Chain Specification +### Step 2: Obtain Chain Specification Download the Polkadot Hub chain specification: @@ -348,7 +330,7 @@ Download the Polkadot Hub chain specification: curl -L https://raw.githubusercontent.com/paritytech/polkadot-sdk/master/cumulus/parachains/chain-specs/asset-hub-polkadot.json -o asset-hub-polkadot.json ``` -### Step 5: Create User and Directory Structure +### Step 3: Create User and Directory Structure ```bash # Create dedicated user @@ -364,7 +346,7 @@ sudo cp asset-hub-polkadot.json /var/lib/polkadot-hub-rpc/ sudo chown -R polkadot:polkadot /var/lib/polkadot-hub-rpc ``` -### Step 6: Create Systemd Service for Polkadot SDK Node +### Step 4: Create Systemd Service Create a service file for the Polkadot SDK RPC node: @@ -385,7 +367,7 @@ User=polkadot Group=polkadot WorkingDirectory=/var/lib/polkadot-hub-rpc -ExecStart=/usr/local/bin/polkadot-omni-node \ +ExecStart=/usr/local/bin/polkadot-parachain \ --name=PolkadotHubRPC \ --chain=/var/lib/polkadot-hub-rpc/asset-hub-polkadot.json \ --base-path=/var/lib/polkadot-hub-rpc \ @@ -414,7 +396,7 @@ LimitNOFILE=65536 WantedBy=multi-user.target ``` -### Step 7: Start Service +### Step 5: Start Service ```bash # Reload systemd @@ -431,7 +413,7 @@ sudo systemctl status polkadot-hub-rpc sudo journalctl -u polkadot-hub-rpc -f ``` -### Step 8: Verify Setup +### Step 6: Verify Setup Use the same verification tests as in the Docker setup (see Step 5 above). @@ -513,13 +495,13 @@ The node handles pruning automatically based on configuration unless running in ```bash # Pull latest image -docker pull parity/polkadot-omni-node:v1.20.2 +docker pull parity/polkadot-parachain: # Restart container docker stop polkadot-hub-rpc docker rm polkadot-hub-rpc -# Start new container (use same command from setup) +# Start new container (use same command from setup with updated image tag) ``` **Systemd Setup**: @@ -531,8 +513,11 @@ sudo systemctl stop polkadot-hub-rpc # Backup data sudo cp -r /var/lib/polkadot-hub-rpc /var/lib/polkadot-hub-rpc.backup -# Update binary -cargo install --locked --force polkadot-omni-node@ +# Pull new image and extract binary +docker pull parity/polkadot-parachain: +docker create --name temp-parachain parity/polkadot-parachain: +sudo docker cp temp-parachain:/usr/local/bin/polkadot-parachain /usr/local/bin/ +docker rm temp-parachain # Restart service sudo systemctl start polkadot-hub-rpc diff --git a/node-infrastructure/.nav.yml b/node-infrastructure/.nav.yml index a14f33cae..d907a96bb 100644 --- a/node-infrastructure/.nav.yml +++ b/node-infrastructure/.nav.yml @@ -1,4 +1,5 @@ nav: + - 'Overview': overview.md - 'Run a Node': run-a-node - 'Run a Collator': run-a-collator - 'Run a Validator': run-a-validator diff --git a/node-infrastructure/overview.md b/node-infrastructure/overview.md new file mode 100644 index 000000000..13098d5ad --- /dev/null +++ b/node-infrastructure/overview.md @@ -0,0 +1,80 @@ +--- +title: Node Infrastructure +description: Overview of running nodes in the Polkadot ecosystem, including RPC nodes, collators, and validators. +categories: Infrastructure +--- + +# Node Infrastructure Overview + +## Introduction + +The Polkadot network relies on various types of nodes to maintain security, provide data access, and produce blocks. This section covers everything you need to know about running infrastructure for the Polkadot ecosystem. + +Whether you want to provide RPC endpoints for applications, produce blocks for a parachain, or secure the relay chain as a validator, this guide will help you understand your options and get started. + +## Node Types + +### RPC Nodes + +RPC nodes provide API access to blockchain data without participating in consensus. They are essential infrastructure for: + +- **Applications and dApps**: Query blockchain state and submit transactions +- **Block explorers**: Index and display blockchain data +- **Wallets**: Check balances and broadcast transactions +- **Development**: Test and debug applications + +RPC nodes can be run for both the relay chain and parachains, with varying levels of data retention (pruned vs archive). + +### Collators + +Collators are block producers for parachains. They perform critical functions: + +- **Collect transactions**: Aggregate user transactions into blocks +- **Produce blocks**: Create parachain block candidates +- **Generate proofs**: Produce state transition proofs (Proof-of-Validity) +- **Submit to validators**: Send block candidates to relay chain validators + +Unlike validators, collators do not provide security guarantees—that responsibility lies with relay chain validators. However, collators are essential for parachain liveness and censorship resistance. + +### Validators + +Validators secure the Polkadot relay chain through Nominated Proof of Stake (NPoS). They: + +- **Validate blocks**: Verify parachain blocks and relay chain transactions +- **Participate in consensus**: Run BABE and GRANDPA protocols +- **Earn rewards**: Receive staking rewards for honest behavior +- **Risk slashing**: Face penalties for misbehavior or downtime + +Running a validator requires significant technical expertise, reliable infrastructure, and a stake of DOT tokens. + +## Next Steps + +Choose your path based on your goals: + +
+ +- :material-api:{ .lg .middle } **Run RPC Nodes** + + --- + + Provide API access for applications, explorers, and wallets + + [:octicons-arrow-right-24: Run a Node](/node-infrastructure/run-a-node/parachain-rpc/) + +- :material-cube-outline:{ .lg .middle } **Run a Collator** + + --- + + Produce blocks for system parachains or your own parachain + + [:octicons-arrow-right-24: Run a Collator](/node-infrastructure/run-a-collator/collator/) + +- :material-shield-check:{ .lg .middle } **Run a Validator** + + --- + + Secure the relay chain and earn staking rewards + + [:octicons-arrow-right-24: Run a Validator](/node-infrastructure/run-a-validator/requirements/) + +
diff --git a/node-infrastructure/run-a-collator/collator.md b/node-infrastructure/run-a-collator/collator.md index 8230cbb44..18ec9445b 100644 --- a/node-infrastructure/run-a-collator/collator.md +++ b/node-infrastructure/run-a-collator/collator.md @@ -56,18 +56,12 @@ Block-producing collators require robust hardware for reliable operation: ### Software Requirements -Collators use the **Polkadot Omni Node**, a universal binary that runs any parachain using a chain specification file. +Collators use the **Polkadot Parachain** binary, the standard client for running Polkadot system parachains. Required software: - **Operating System**: Ubuntu 22.04 LTS (recommended) or similar Linux distribution -- **Docker**: For running subkey utility -- **Rust Toolchain**: Version 1.91.1 or as specified by the runtime -- **Dependencies**: - ```bash - sudo apt update - sudo apt install -y build-essential git clang curl libssl-dev llvm libudev-dev make protobuf-compiler - ``` +- **Docker**: Required for obtaining binaries and running containers ### Account Requirements @@ -79,54 +73,81 @@ You'll need: ## Installation -### Step 1: Install Rust and Required Toolchain +### Step 1: Install the Polkadot Parachain Binary + +**For Docker deployment:** ```bash -# Install Rust -curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -source $HOME/.cargo/env - -# Install specific Rust version -rustup install 1.91.1 -rustup default 1.91.1 -rustup target add wasm32-unknown-unknown --toolchain 1.91.1 -rustup component add rust-src --toolchain 1.91.1 +# Pull the polkadot-parachain image +docker pull parity/polkadot-parachain:stable2509-2 + +# Verify installation +docker run --rm parity/polkadot-parachain:stable2509-2 --version ``` -### Step 2: Install the Polkadot Omni Node +**For bare-metal deployment:** + +Extract the binary from the Docker image: ```bash -# Install polkadot-omni-node -cargo install --locked polkadot-omni-node@0.11.0 +# Create a temporary container and copy the binary +docker create --name temp-parachain parity/polkadot-parachain:stable2509-2 +sudo docker cp temp-parachain:/usr/local/bin/polkadot-parachain /usr/local/bin/ +docker rm temp-parachain # Verify installation -polkadot-omni-node --version +polkadot-parachain --version ``` -### Step 3: Generate Node Key +Check [Docker Hub](https://hub.docker.com/r/parity/polkadot-parachain/tags) for the latest stable tags. + +### Step 2: Generate Node Key Generate a stable node key for consistent peer ID: ```bash # Create directory for node data sudo mkdir -p /var/lib/polkadot-collator +``` + +**Using Docker:** -# Generate node key using Docker -docker run -it parity/subkey:latest generate-node-key > /var/lib/polkadot-collator/node.key +```bash +# Generate node key (outputs peer ID to console, saves key to file) +docker run --rm -v /var/lib/polkadot-collator:/data \ + parity/polkadot-parachain:stable2509-2 \ + key generate-node-key --file /data/node.key -# The output displays your peer ID -# Example: 12D3KooWExcVYu7Mvjd4kxPVLwN2ZPnZ5NyLZ5ft477wqzfP2q6E +# Example output: 12D3KooWExcVYu7Mvjd4kxPVLwN2ZPnZ5NyLZ5ft477wqzfP2q6E +``` + +**Using native binary:** + +```bash +# Generate node key +polkadot-parachain key generate-node-key --file /var/lib/polkadot-collator/node.key + +# Example output: 12D3KooWExcVYu7Mvjd4kxPVLwN2ZPnZ5NyLZ5ft477wqzfP2q6E ``` Save the peer ID for future reference. -### Step 4: Generate Account Key +### Step 3: Generate Account Key Generate an account for on-chain transactions: +**Using Docker:** + +```bash +# Generate account key with sr25519 scheme +docker run --rm parity/polkadot-parachain:stable2509-2 key generate --scheme sr25519 +``` + +**Using native binary:** + ```bash # Generate account key with sr25519 scheme -docker run -it parity/subkey:latest generate --scheme sr25519 +polkadot-parachain key generate --scheme sr25519 ``` Save the output containing: @@ -138,7 +159,7 @@ Save the output containing: **Security**: Store the secret phrase securely. Never share it. Consider using a hardware wallet for production collators. -### Step 5: Obtain Chain Specification +### Step 4: Obtain Chain Specification Download the chain specification for your target system parachain: @@ -177,7 +198,7 @@ chain-spec-builder create \ - People Chain: 1004 - Coretime Chain: 1005 -### Step 6: Create User and Directory Structure +### Step 5: Create User and Directory Structure ```bash # Create dedicated user @@ -192,7 +213,7 @@ sudo chown -R polkadot:polkadot /var/lib/polkadot-collator ## Configuration -### Create Systemd Service File +### Native Binary Systemd Service Create a service file for your collator: @@ -214,7 +235,7 @@ Group=polkadot WorkingDirectory=/var/lib/polkadot-collator # Block-Producing Collator Configuration -ExecStart=/usr/local/bin/polkadot-omni-node \ +ExecStart=/usr/local/bin/polkadot-parachain \ --collator \ --chain=/var/lib/polkadot-collator/chain-spec.json \ --base-path=/var/lib/polkadot-collator \ @@ -244,6 +265,55 @@ WantedBy=multi-user.target - `--name`: Your collator name (visible in telemetry) - Relay chain uses `--sync=warp` for faster initial sync +### Docker Systemd Service + +If using Docker, create a service file with Docker configuration: + +```bash +sudo nano /etc/systemd/system/polkadot-collator.service +``` + +Add the following configuration: + +```ini +[Unit] +Description=Polkadot System Parachain Collator (Docker) +After=network.target docker.service +Requires=docker.service + +[Service] +Type=simple +Restart=always +RestartSec=10 +TimeoutStartSec=0 + +ExecStartPre=-/usr/bin/docker stop polkadot-collator +ExecStartPre=-/usr/bin/docker rm polkadot-collator + +ExecStart=/usr/bin/docker run --rm \ + --name polkadot-collator \ + --network host \ + -v /var/lib/polkadot-collator:/data \ + parity/polkadot-parachain:stable2509-2 \ + --collator \ + --chain=/data/chain-spec.json \ + --base-path=/data \ + --port=30333 \ + --rpc-port=9944 \ + --prometheus-port=9615 \ + --node-key-file=/data/node.key \ + --name="YourCollatorName" \ + -- \ + --chain=polkadot \ + --port=30334 \ + --sync=warp + +ExecStop=/usr/bin/docker stop polkadot-collator + +[Install] +WantedBy=multi-user.target +``` + ## Running the Collator ### Step 1: Start the Service @@ -488,7 +558,7 @@ The node handles pruning automatically. - Fellowship GitHub - Matrix channels -**Upgrade Procedure**: +**Upgrade Procedure (Bare-Metal)**: ```bash # Stop the service @@ -497,11 +567,14 @@ sudo systemctl stop polkadot-collator # Backup data (recommended) sudo cp -r /var/lib/polkadot-collator /var/lib/polkadot-collator.backup -# Update polkadot-omni-node -cargo install --locked --force polkadot-omni-node@ +# Pull new image and extract binary +docker pull parity/polkadot-parachain: +docker create --name temp-parachain parity/polkadot-parachain: +sudo docker cp temp-parachain:/usr/local/bin/polkadot-parachain /usr/local/bin/ +docker rm temp-parachain # Verify version -polkadot-omni-node --version +polkadot-parachain --version # Restart service sudo systemctl start polkadot-collator @@ -510,6 +583,28 @@ sudo systemctl start polkadot-collator sudo systemctl status polkadot-collator ``` +**Upgrade Procedure (Docker)**: + +```bash +# Stop the service +sudo systemctl stop polkadot-collator + +# Backup data (recommended) +sudo cp -r /var/lib/polkadot-collator /var/lib/polkadot-collator.backup + +# Pull new image +docker pull parity/polkadot-parachain: + +# Update the image tag in /etc/systemd/system/polkadot-collator.service + +# Reload systemd and restart +sudo systemctl daemon-reload +sudo systemctl start polkadot-collator + +# Verify service is running +sudo systemctl status polkadot-collator +``` + **Note**: For log monitoring, see the [Log Management](#log-management) section. ## Conclusion diff --git a/node-infrastructure/run-a-node/.nav.yml b/node-infrastructure/run-a-node/.nav.yml index f4e014c43..747174907 100644 --- a/node-infrastructure/run-a-node/.nav.yml +++ b/node-infrastructure/run-a-node/.nav.yml @@ -1,4 +1,4 @@ nav: - 'Polkadot Hub RPC Node': polkadot-hub-rpc.md - - 'System Parachain RPC Nodes': system-parachain-rpc.md + - 'Parachain RPC Nodes': parachain-rpc.md - 'Relay Chain Nodes': relay-chain diff --git a/.ai/pages/node-infrastructure-run-a-node-system-parachain-rpc.md b/node-infrastructure/run-a-node/parachain-rpc.md similarity index 71% rename from .ai/pages/node-infrastructure-run-a-node-system-parachain-rpc.md rename to node-infrastructure/run-a-node/parachain-rpc.md index 63f5aa7ed..b5a0fec82 100644 --- a/.ai/pages/node-infrastructure-run-a-node-system-parachain-rpc.md +++ b/node-infrastructure/run-a-node/parachain-rpc.md @@ -1,38 +1,59 @@ --- -title: Run an RPC Node for System Parachains -description: Complete guide to set up and run an RPC node for Polkadot system parachains including Bridge Hub, People Chain, and Coretime Chain. +title: Run a Parachain RPC Node +description: Complete guide to set up and run an RPC node for any Polkadot parachain, with system parachains as examples. categories: Infrastructure -url: https://docs.polkadot.com/node-infrastructure/run-a-node/system-parachain-rpc/ --- -# Run an RPC Node for System Parachains +# Run a Parachain RPC Node ## Overview -System parachains are core infrastructure parachains that provide essential services to the Polkadot network. Running an RPC node for these parachains enables applications, wallets, and users to interact with their specialized functionality: +Running an RPC node for a parachain enables applications, wallets, and users to interact with the parachain's functionality. This guide applies to **any parachain** in the Polkadot ecosystem, including: -- **Bridge Hub**: Cross-chain asset transfers via trustless bridges -- **People Chain**: Identity and social credential management -- **Coretime Chain**: Blockspace allocation and core time management +- **System parachains**: Bridge Hub, People Chain, Coretime Chain +- **Common good parachains**: Collectives, Encointer +- **Commercial parachains**: Any parachain with a publicly available chain specification -Each system parachain RPC node provides access through: -- **Polkadot SDK Node RPC** (Port 9944): Native Polkadot API (WebSocket and HTTP) - -This setup enables block explorer indexing and provides full compatibility with Polkadot SDK development tools. +Each parachain RPC node provides access through the Polkadot SDK Node RPC (Port 9944), offering native Polkadot API access via WebSocket and HTTP. This setup enables block explorer indexing and provides full compatibility with Polkadot SDK development tools. **Important Note**: The parameters and configurations in this guide are provided as illustrative examples. You may need to modify them according to your specific environment, hardware capabilities, and network conditions. -## Choosing a System Parachain +## Obtaining a Chain Specification + +To run an RPC node for any parachain, you need its **chain specification file**. This JSON file defines the network parameters, genesis state, and bootnodes. + +### System Parachains + +System parachain chain specs are available from multiple sources: -This guide uses **People Chain** as the example, but the same principles and setup procedures apply to all system parachains. Simply substitute the appropriate values from the table below for your chosen parachain: +**Option 1: Chainspec Collection (Recommended)** -| Parachain | Para ID | Chain Spec File | Snapshot Path | Chain Name | -|-----------|---------|-----------------|---------------|------------| -| **Bridge Hub** | 1002 | `bridge-hub-polkadot.json` | `polkadot-bridge-hub-rocksdb-archive` | `bridge-hub-polkadot` | -| **People Chain** | 1004 | `people-polkadot.json` | `polkadot-people-rocksdb-archive` | `people-polkadot` | -| **Coretime Chain** | 1005 | `coretime-polkadot.json` | `polkadot-coretime-rocksdb-archive` | `coretime-polkadot` | +Visit the [Chainspec Collection](https://paritytech.github.io/chainspecs/) to download official chain specifications. -**Note**: Throughout this guide, we use People Chain values. To set up a different system parachain, replace the chain spec file, snapshot path, and chain name with the corresponding values from the table above. +**Option 2: Polkadot SDK Repository** + +Download directly from the Polkadot SDK repository: + +```bash +# Example for People Chain +curl -L https://raw.githubusercontent.com/paritytech/polkadot-sdk/master/cumulus/parachains/chain-specs/people-polkadot.json -o chain-spec.json +``` + +| System Parachain | Para ID | Chain Spec File | Snapshot Path | +|------------------|---------|-----------------|---------------| +| **Bridge Hub** | 1002 | `bridge-hub-polkadot.json` | `polkadot-bridge-hub-rocksdb-archive` | +| **People Chain** | 1004 | `people-polkadot.json` | `polkadot-people-rocksdb-archive` | +| **Coretime Chain** | 1005 | `coretime-polkadot.json` | `polkadot-coretime-rocksdb-archive` | + +### Other Parachains + +For non-system parachains: + +- **Check the parachain's documentation** for official chain specification files +- **Contact the parachain team** if no public chain spec is available +- **Export from a running node** using `system_chainSpec` RPC method if you have access to an existing node + +**Note**: Throughout this guide, we use **People Chain** as the example. To set up a different parachain, substitute the chain spec file, snapshot path, and chain name with values for your target parachain. ## Prerequisites @@ -65,9 +86,8 @@ RPC nodes serving production traffic require robust hardware: Required software: - **Operating System**: Ubuntu 22.04 LTS (recommended) or similar Linux distribution -- **Docker**: Latest version installed and running (for Docker-based setup) +- **Docker**: Required for obtaining binaries and running containers - **rclone**: (Optional but recommended) Command-line program for managing files on cloud storage (https://rclone.org/downloads/) -- **Rust Toolchain**: Version 1.91.1 or as specified by runtime (for manual build) ## Setup Options @@ -84,22 +104,14 @@ Choose the option that best fits your needs. This option uses Docker containers for the Polkadot SDK node, making it easy to set up and manage. -### Step 1: Download Chain Specification - -Download the official chain specification for People Chain: - -```bash -curl -L https://raw.githubusercontent.com/paritytech/polkadot-sdk/master/cumulus/parachains/chain-specs/people-polkadot.json -o people-polkadot.json -``` - -**Note**: This chain specification is the official configuration file that defines the network parameters for People Chain. - -### Step 2: Download Database Snapshots (Optional but Recommended) +### Step 1: Download Database Snapshots (Optional but Recommended) Using pre-synchronized snapshots significantly reduces initial sync time from several days to just a few hours. You need to download both parachain and relay chain data. **Snapshot Provider**: https://snapshots.polkadot.io/ +**Note**: Snapshots are available for system parachains and the Polkadot relay chain. For other parachains, check with the parachain team for snapshot availability or sync from genesis. + #### Create Directories ```bash @@ -161,11 +173,13 @@ rm files.txt - Pruned snapshot: `polkadot-rocksdb-prune` (smaller size, recent state) - Archive snapshot: `polkadot-rocksdb-archive` (complete history, larger size) -### Step 3: Start People Chain Node +### Step 2: Start the Parachain Node Launch the node using the official Parity Docker image. -**Docker Image**: https://hub.docker.com/r/parity/polkadot-omni-node +**Docker Image**: https://hub.docker.com/r/parity/polkadot-parachain + +**Note**: The `parity/polkadot-parachain` image works for system parachains and parachains built with standard Cumulus templates. For parachains with custom runtimes, check the parachain's documentation for their specific Docker image or binary. ```bash docker run -d --name people-chain-rpc --restart unless-stopped \ @@ -176,7 +190,7 @@ docker run -d --name people-chain-rpc --restart unless-stopped \ -p 30333:30333 \ -v $(pwd)/people-polkadot.json:/people-polkadot.json \ -v $(pwd)/my-node-data:/data \ - parity/polkadot-omni-node:v1.20.2 \ + parity/polkadot-parachain:stable2509-2 \ --name=PeopleChainRPC \ --base-path=/data \ --chain=/people-polkadot.json \ @@ -216,7 +230,7 @@ docker run -d --name people-chain-rpc --restart unless-stopped \ **Security Warning**: The `--unsafe-rpc-external` flag should only be used in development or properly secured environments. For production, use a reverse proxy with authentication. -### Step 4: Monitor Synchronization +### Step 3: Monitor Synchronization Monitor the node synchronization status: @@ -246,7 +260,7 @@ curl -H "Content-Type: application/json" \ - **In Progress**: `currentBlock` < `highestBlock` - **Complete**: `currentBlock` = `highestBlock` -### Step 5: Verify Setup +### Step 4: Verify Setup Let's verify the Polkadot SDK RPC endpoint is working correctly. @@ -336,39 +350,26 @@ docker rm people-chain-rpc This option provides more control and is recommended for production environments requiring custom configurations. -### Step 1: Install Rust and Required Toolchain +### Step 1: Install the Polkadot Parachain Binary -```bash -# Install Rust -curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -source $HOME/.cargo/env - -# Install specific Rust version -rustup install 1.91.1 -rustup default 1.91.1 -rustup target add wasm32-unknown-unknown --toolchain 1.91.1 -rustup component add rust-src --toolchain 1.91.1 -``` - -### Step 2: Install the Polkadot Omni Node +Extract the binary from the official Docker image: ```bash -# Install polkadot-omni-node -cargo install --locked polkadot-omni-node@0.11.0 +# Pull the image and extract binary +docker pull parity/polkadot-parachain:stable2509-2 +docker create --name temp-parachain parity/polkadot-parachain:stable2509-2 +sudo docker cp temp-parachain:/usr/local/bin/polkadot-parachain /usr/local/bin/ +docker rm temp-parachain # Verify installation -polkadot-omni-node --version +polkadot-parachain --version ``` -### Step 3: Obtain Chain Specification +Check [Docker Hub](https://hub.docker.com/r/parity/polkadot-parachain/tags) for the latest stable tags. -Download the People Chain specification: +### Step 2: Create User and Directory Structure -```bash -curl -L https://raw.githubusercontent.com/paritytech/polkadot-sdk/master/cumulus/parachains/chain-specs/people-polkadot.json -o people-polkadot.json -``` - -### Step 4: Create User and Directory Structure +Ensure you have downloaded your parachain's chain specification as described in [Obtaining a Chain Specification](#obtaining-a-chain-specification). ```bash # Create dedicated user (skip if already exists) @@ -384,9 +385,9 @@ sudo cp people-polkadot.json /var/lib/people-chain-rpc/ sudo chown -R polkadot:polkadot /var/lib/people-chain-rpc ``` -### Step 5: Create Systemd Service +### Step 3: Create Systemd Service -Create a service file for the People Chain RPC node: +Create a service file for your parachain RPC node: ```bash sudo nano /etc/systemd/system/people-chain-rpc.service @@ -405,7 +406,7 @@ User=polkadot Group=polkadot WorkingDirectory=/var/lib/people-chain-rpc -ExecStart=/usr/local/bin/polkadot-omni-node \ +ExecStart=/usr/local/bin/polkadot-parachain \ --name=PeopleChainRPC \ --chain=/var/lib/people-chain-rpc/people-polkadot.json \ --base-path=/var/lib/people-chain-rpc \ @@ -434,7 +435,7 @@ LimitNOFILE=65536 WantedBy=multi-user.target ``` -### Step 6: Start Service +### Step 4: Start Service ```bash # Reload systemd @@ -451,9 +452,9 @@ sudo systemctl status people-chain-rpc sudo journalctl -u people-chain-rpc -f ``` -### Step 7: Verify Setup +### Step 5: Verify Setup -Use the same verification tests as in the Docker setup (see Step 5 above). +Use the same verification tests as in the Docker setup (see Step 4 above). --- @@ -497,14 +498,12 @@ Metrics are available at `http://localhost:9615/metrics` Example Prometheus scrape configuration: -{% raw %} ```yaml scrape_configs: - job_name: 'people-chain-rpc' static_configs: - targets: ['localhost:9615'] ``` -{% endraw %} **Key Metrics to Monitor**: @@ -533,13 +532,13 @@ The node handles pruning automatically based on configuration unless running in ```bash # Pull latest image -docker pull parity/polkadot-omni-node:v1.20.2 +docker pull parity/polkadot-parachain: # Restart container docker stop people-chain-rpc docker rm people-chain-rpc -# Start new container (use same command from setup) +# Start new container (use same command from setup with updated image tag) ``` **Systemd Setup**: @@ -551,8 +550,11 @@ sudo systemctl stop people-chain-rpc # Backup data sudo cp -r /var/lib/people-chain-rpc /var/lib/people-chain-rpc.backup -# Update binary -cargo install --locked --force polkadot-omni-node@ +# Pull new image and extract binary +docker pull parity/polkadot-parachain: +docker create --name temp-parachain parity/polkadot-parachain: +sudo docker cp temp-parachain:/usr/local/bin/polkadot-parachain /usr/local/bin/ +docker rm temp-parachain # Restart service sudo systemctl start people-chain-rpc @@ -560,11 +562,11 @@ sudo systemctl start people-chain-rpc ## Conclusion -Running an RPC node for system parachains provides critical infrastructure for accessing specialized Polkadot network services. By following this guide, you have set up a production-ready RPC node that: +Running a parachain RPC node provides critical infrastructure for accessing Polkadot network services. By following this guide, you have set up a production-ready RPC node that: -- Enables applications and users to interact with essential system parachain features (identity management, cross-chain bridges, or coretime allocation) +- Provides reliable access to parachain functionality for applications and users - Supports flexible deployment with both Docker and systemd options - Implements comprehensive monitoring, security, and maintenance practices -- Can be easily adapted for any system parachain by substituting the appropriate chain specification +- Can be adapted for any parachain by substituting the appropriate chain specification -Whether you're running a node for People Chain, Bridge Hub, or Coretime Chain, regular maintenance and monitoring will ensure your RPC node continues to provide reliable service. Stay updated with the latest releases and best practices to keep your infrastructure secure and performant. +Whether you're running a node for system parachains (People Chain, Bridge Hub, Coretime Chain) or other parachains in the ecosystem, regular maintenance and monitoring will ensure your RPC node continues to provide reliable service. Stay updated with the latest releases and best practices to keep your infrastructure secure and performant. diff --git a/node-infrastructure/run-a-node/polkadot-hub-rpc.md b/node-infrastructure/run-a-node/polkadot-hub-rpc.md index f8e653222..c08af26c3 100644 --- a/node-infrastructure/run-a-node/polkadot-hub-rpc.md +++ b/node-infrastructure/run-a-node/polkadot-hub-rpc.md @@ -52,9 +52,8 @@ RPC nodes serving production traffic require robust hardware: Required software: - **Operating System**: Ubuntu 22.04 LTS (recommended) or similar Linux distribution -- **Docker**: Latest version installed and running (for Docker-based setup) +- **Docker**: Required for obtaining binaries and running containers - **rclone**: (Optional but recommended) Command-line program for managing files on cloud storage (https://rclone.org/downloads/) -- **Rust Toolchain**: Version 1.91.1 or later (for manual build) ## Setup Options @@ -150,7 +149,7 @@ rm files.txt Launch the node using the official Parity Docker image: -**Docker Image**: https://hub.docker.com/r/parity/polkadot-omni-node +**Docker Image**: https://hub.docker.com/r/parity/polkadot-parachain ```bash docker run -d --name polkadot-hub-rpc --restart unless-stopped \ @@ -161,7 +160,7 @@ docker run -d --name polkadot-hub-rpc --restart unless-stopped \ -p 30333:30333 \ -v $(pwd)/asset-hub-polkadot.json:/asset-hub-polkadot.json \ -v $(pwd)/my-node-data:/data \ - parity/polkadot-omni-node:v1.20.2 \ + parity/polkadot-parachain:stable2509-2 \ --name=PolkadotHubRPC \ --base-path=/data \ --chain=/asset-hub-polkadot.json \ @@ -305,41 +304,24 @@ docker rm polkadot-hub-rpc This option provides more control and is recommended for production environments requiring custom configurations. -### Step 1: Install Rust and Required Toolchain +### Step 1: Install the Polkadot Parachain Binary -```bash -# Install Rust -curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -source $HOME/.cargo/env - -# Install specific Rust version -rustup install 1.91.1 -rustup default 1.91.1 -rustup target add wasm32-unknown-unknown --toolchain 1.91.1 -rustup component add rust-src --toolchain 1.91.1 -``` - -### Step 2: Install Required Dependencies - -```bash -# Install system dependencies -sudo apt update -sudo apt install -y build-essential git clang curl libssl-dev llvm libudev-dev make protobuf-compiler -``` - -### Step 3: Install the Polkadot Omni Node +Extract the binary from the official Docker image: ```bash -# Install polkadot-omni-node -cargo install --locked polkadot-omni-node@0.11.0 +# Pull the image and extract binary +docker pull parity/polkadot-parachain:stable2509-2 +docker create --name temp-parachain parity/polkadot-parachain:stable2509-2 +sudo docker cp temp-parachain:/usr/local/bin/polkadot-parachain /usr/local/bin/ +docker rm temp-parachain # Verify installation -polkadot-omni-node --version +polkadot-parachain --version ``` -**Note**: Compiling polkadot-omni-node from source requires significant RAM (minimum 24GB recommended). The compilation may take 10-15 minutes on systems with adequate resources. +Check [Docker Hub](https://hub.docker.com/r/parity/polkadot-parachain/tags) for the latest stable tags. -### Step 4: Obtain Chain Specification +### Step 2: Obtain Chain Specification Download the Polkadot Hub chain specification: @@ -347,7 +329,7 @@ Download the Polkadot Hub chain specification: curl -L https://raw.githubusercontent.com/paritytech/polkadot-sdk/master/cumulus/parachains/chain-specs/asset-hub-polkadot.json -o asset-hub-polkadot.json ``` -### Step 5: Create User and Directory Structure +### Step 3: Create User and Directory Structure ```bash # Create dedicated user @@ -363,7 +345,7 @@ sudo cp asset-hub-polkadot.json /var/lib/polkadot-hub-rpc/ sudo chown -R polkadot:polkadot /var/lib/polkadot-hub-rpc ``` -### Step 6: Create Systemd Service for Polkadot SDK Node +### Step 4: Create Systemd Service Create a service file for the Polkadot SDK RPC node: @@ -384,7 +366,7 @@ User=polkadot Group=polkadot WorkingDirectory=/var/lib/polkadot-hub-rpc -ExecStart=/usr/local/bin/polkadot-omni-node \ +ExecStart=/usr/local/bin/polkadot-parachain \ --name=PolkadotHubRPC \ --chain=/var/lib/polkadot-hub-rpc/asset-hub-polkadot.json \ --base-path=/var/lib/polkadot-hub-rpc \ @@ -413,7 +395,7 @@ LimitNOFILE=65536 WantedBy=multi-user.target ``` -### Step 7: Start Service +### Step 5: Start Service ```bash # Reload systemd @@ -430,7 +412,7 @@ sudo systemctl status polkadot-hub-rpc sudo journalctl -u polkadot-hub-rpc -f ``` -### Step 8: Verify Setup +### Step 6: Verify Setup Use the same verification tests as in the Docker setup (see Step 5 above). @@ -510,13 +492,13 @@ The node handles pruning automatically based on configuration unless running in ```bash # Pull latest image -docker pull parity/polkadot-omni-node:v1.20.2 +docker pull parity/polkadot-parachain: # Restart container docker stop polkadot-hub-rpc docker rm polkadot-hub-rpc -# Start new container (use same command from setup) +# Start new container (use same command from setup with updated image tag) ``` **Systemd Setup**: @@ -528,8 +510,11 @@ sudo systemctl stop polkadot-hub-rpc # Backup data sudo cp -r /var/lib/polkadot-hub-rpc /var/lib/polkadot-hub-rpc.backup -# Update binary -cargo install --locked --force polkadot-omni-node@ +# Pull new image and extract binary +docker pull parity/polkadot-parachain: +docker create --name temp-parachain parity/polkadot-parachain: +sudo docker cp temp-parachain:/usr/local/bin/polkadot-parachain /usr/local/bin/ +docker rm temp-parachain # Restart service sudo systemctl start polkadot-hub-rpc From a9f83404d7476d5dcb5f25a02bf0705c4332cda5 Mon Sep 17 00:00:00 2001 From: Bruno Galvao Date: Mon, 24 Nov 2025 16:34:47 +0700 Subject: [PATCH 12/21] Use polkadot-parachain binary instead of polkadot-omni-node --- .../run-a-collator/collator.md | 42 +++++------- .../run-a-node/polkadot-hub-rpc.md | 67 +++++++------------ 2 files changed, 43 insertions(+), 66 deletions(-) diff --git a/node-infrastructure/run-a-collator/collator.md b/node-infrastructure/run-a-collator/collator.md index 843a5f863..745d37dac 100644 --- a/node-infrastructure/run-a-collator/collator.md +++ b/node-infrastructure/run-a-collator/collator.md @@ -96,28 +96,17 @@ This guide provides two deployment options. Select the option that best fits you === "Manual Setup" - 1. Install Rust using the following commands: - ```bash - # Install Rust - curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh - source $HOME/.cargo/env - - # Install specific Rust version - rustup install 1.91.1 - rustup default 1.91.1 - rustup target add wasm32-unknown-unknown --toolchain 1.91.1 - rustup component add rust-src --toolchain 1.91.1 - ``` + Extract the binary from the official Docker image: - 2. Install the Polkadot Omni Node using the following command: - ```bash - cargo install --locked polkadot-omni-node@0.11.0 - ``` + ```bash + # Create a temporary container and copy the binary + docker create --name temp-parachain parity/polkadot-parachain:stable2509-2 + sudo docker cp temp-parachain:/usr/local/bin/polkadot-parachain /usr/local/bin/ + docker rm temp-parachain - 3. Verify a successful installation using the `--version` flag: - ```bash - polkadot-omni-node --version - ``` + # Verify installation + polkadot-parachain --version + ``` ## Generate Node Key @@ -245,7 +234,7 @@ Follow these steps to build a chainspec from the runtime: WorkingDirectory=/var/lib/polkadot-collator # Block-Producing Collator Configuration - ExecStart=/usr/local/bin/polkadot-omni-node \ + ExecStart=/usr/local/bin/polkadot-parachain \ --collator \ --chain=/var/lib/polkadot-collator/chain-spec.json \ --base-path=/var/lib/polkadot-collator \ @@ -500,14 +489,17 @@ Updates or upgrades can happen on either the runtime or client. Runtime upgrades sudo cp -r /var/lib/polkadot-collator /var/lib/polkadot-collator.backup ``` - 3. Update `polkadot-omni-node`: + 3. Pull the new image and extract the binary: ```bash - cargo install --locked --force polkadot-omni-node@ + docker pull parity/polkadot-parachain: + docker create --name temp-parachain parity/polkadot-parachain: + sudo docker cp temp-parachain:/usr/local/bin/polkadot-parachain /usr/local/bin/ + docker rm temp-parachain ``` - 4. Verify `polkadot-omni-node` version to confirm successful update: + 4. Verify `polkadot-parachain` version to confirm successful update: ```bash - polkadot-omni-node --version + polkadot-parachain --version ``` 5. Restart the service: diff --git a/node-infrastructure/run-a-node/polkadot-hub-rpc.md b/node-infrastructure/run-a-node/polkadot-hub-rpc.md index 7ef0bad0c..e33f65eca 100644 --- a/node-infrastructure/run-a-node/polkadot-hub-rpc.md +++ b/node-infrastructure/run-a-node/polkadot-hub-rpc.md @@ -129,7 +129,7 @@ Select the best option for your project, then use the steps in the following tab rm files.txt ``` - 3. Launch Polkadot Hub Node using the official [Parity Docker image](https://hub.docker.com/r/parity/polkadot-omni-node){target=\_blank} with the following command: + 3. Launch Polkadot Hub Node using the official [Parity Docker image](https://hub.docker.com/r/parity/polkadot-parachain){target=\_blank} with the following command: ```bash docker run -d --name polkadot-hub-rpc --restart unless-stopped \ -p 9944:9944 \ @@ -139,7 +139,7 @@ Select the best option for your project, then use the steps in the following tab -p 30333:30333 \ -v $(pwd)/asset-hub-polkadot.json:/asset-hub-polkadot.json \ -v $(pwd)/my-node-data:/data \ - parity/polkadot-omni-node:v1.20.2 \ + parity/polkadot-parachain:stable2509-2 \ --name=PolkadotHubRPC \ --base-path=/data \ --chain=/asset-hub-polkadot.json \ @@ -246,46 +246,28 @@ Select the best option for your project, then use the steps in the following tab === "Manual systemd Setup" - This option uses systemd to provide more control and is recommended for production environments requiring custom configurations. + This option provides more control and is recommended for production environments requiring custom configurations. - 1. Install Rust and required toolchain using the following command: + 1. Install the Polkadot Parachain binary by extracting it from the official Docker image: ```bash - # Install Rust - curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh - source $HOME/.cargo/env - - # Install specific Rust version - rustup install 1.91.1 - rustup default 1.91.1 - rustup target add wasm32-unknown-unknown --toolchain 1.91.1 - rustup component add rust-src --toolchain 1.91.1 + # Pull the image and extract binary + docker pull parity/polkadot-parachain:stable2509-2 + docker create --name temp-parachain parity/polkadot-parachain:stable2509-2 + sudo docker cp temp-parachain:/usr/local/bin/polkadot-parachain /usr/local/bin/ + docker rm temp-parachain + + # Verify installation + polkadot-parachain --version ``` - 2. Install required dependencies using the following commands: - - System dependencies: - ```bash - sudo apt update - sudo apt install -y build-essential git clang curl libssl-dev llvm libudev-dev make protobuf-compiler - ``` - - Polkadot Omni node: - ```bash - cargo install --locked polkadot-omni-node@0.11.0 - ``` - You can verify successful installation of Polkadot Omni node using the version command: - ```bash - polkadot-omni-node --version - ``` - - !!! tip - - Compiling polkadot-omni-node from source requires significant RAM (minimum 24GB recommended). The compilation may take 10-15 minutes on systems with adequate resources. + Check [Docker Hub](https://hub.docker.com/r/parity/polkadot-parachain/tags){target=\_blank} for the latest stable tags. - 3. Download the Polkadot Hub chain specification: + 2. Download the Polkadot Hub chain specification: ```bash curl -L https://raw.githubusercontent.com/paritytech/polkadot-sdk/master/cumulus/parachains/chain-specs/asset-hub-polkadot.json -o asset-hub-polkadot.json ``` - 4. Create user and directory structures using the following commands: + 3. Create user and directory structures using the following commands: - Create a dedicated user: ```bash sudo useradd -r -s /bin/bash polkadot @@ -303,12 +285,12 @@ Select the best option for your project, then use the steps in the following tab sudo chown -R polkadot:polkadot /var/lib/polkadot-hub-rpc ``` - 5. Create a systemd service file for the Polkadot SDK RPC node: + 4. Create a systemd service file for the Polkadot SDK RPC node: ```bash sudo nano /etc/systemd/system/polkadot-hub-rpc.service ``` - 6. Open the new service file and add the following configuration: + 5. Open the new service file and add the following configuration: ```ini [Unit] Description=Polkadot Hub RPC Node @@ -320,7 +302,7 @@ Select the best option for your project, then use the steps in the following tab Group=polkadot WorkingDirectory=/var/lib/polkadot-hub-rpc - ExecStart=/usr/local/bin/polkadot-omni-node \ + ExecStart=/usr/local/bin/polkadot-parachain \ --name=PolkadotHubRPC \ --chain=/var/lib/polkadot-hub-rpc/asset-hub-polkadot.json \ --base-path=/var/lib/polkadot-hub-rpc \ @@ -349,7 +331,7 @@ Select the best option for your project, then use the steps in the following tab WantedBy=multi-user.target ``` - 7. Start the service using the following commands: + 6. Start the service using the following commands: - Reload systemd: ```bash sudo systemctl daemon-reload @@ -368,7 +350,7 @@ Select the best option for your project, then use the steps in the following tab sudo journalctl -u polkadot-hub-rpc -f ``` - 8. You can use a few different commands to verify your node is running properly: + 7. You can use a few different commands to verify your node is running properly: - Get chain information: ```bash curl -H "Content-Type: application/json" \ @@ -472,7 +454,7 @@ Use the following commands for updating or upgrading your RPC node according to ``` 2. Pull the latest image: ```bash - docker pull parity/polkadot-omni-node: + docker pull parity/polkadot-parachain: ``` 3. Start the new container using the same command from the setup section with the updated image tag. @@ -486,9 +468,12 @@ Use the following commands for updating or upgrading your RPC node according to ```bash sudo cp -r /var/lib/polkadot-hub-rpc /var/lib/polkadot-hub-rpc.backup ``` - 3. Update the binary: + 3. Pull the new image and extract the binary: ```bash - cargo install --locked --force polkadot-omni-node@ + docker pull parity/polkadot-parachain: + docker create --name temp-parachain parity/polkadot-parachain: + sudo docker cp temp-parachain:/usr/local/bin/polkadot-parachain /usr/local/bin/ + docker rm temp-parachain ``` 4. Restart the service: ```bash From 35d51a0b3951c6b9b11bc0d3195817bb5b682ed2 Mon Sep 17 00:00:00 2001 From: Bruno Galvao Date: Mon, 24 Nov 2025 16:35:34 +0700 Subject: [PATCH 13/21] Update llms --- ...-infrastructure-run-a-collator-collator.md | 837 ++++++++--------- ...rastructure-run-a-node-polkadot-hub-rpc.md | 872 +++++++++--------- 2 files changed, 787 insertions(+), 922 deletions(-) diff --git a/.ai/pages/node-infrastructure-run-a-collator-collator.md b/.ai/pages/node-infrastructure-run-a-collator-collator.md index e1bed1380..bb8fd588e 100644 --- a/.ai/pages/node-infrastructure-run-a-collator-collator.md +++ b/.ai/pages/node-infrastructure-run-a-collator-collator.md @@ -9,35 +9,35 @@ url: https://docs.polkadot.com/node-infrastructure/run-a-collator/collator/ ## Overview -Block-producing collators are the backbone of system parachain operations. Unlike RPC nodes or archive nodes that simply maintain state, collators actively produce blocks and submit them to relay chain validators for inclusion. +Block-producing collators are the backbone of system parachain operations. Unlike RPC or archive nodes, which maintain state, collators actively produce blocks and submit them to relay chain validators for inclusion. -This guide covers setting up a **block-producing collator** for Polkadot system parachains. Running a collator requires: +This guide covers setting up a block-producing collator for Polkadot system parachains. Running a collator requires: - Meeting hardware requirements for reliable block production - Setting up and registering session keys - Obtaining governance approval or meeting selection criteria - Maintaining high uptime and performance -**Important**: System parachain collators typically require governance approval or being added to the invulnerables list. This is different from non-system parachains where collator selection may be more permissionless. +System parachain collators typically require governance approval or being added to the invulnerables list. This is different from non-system parachains where collator selection may be more permissionless. ## Collator Responsibilities Block-producing collators perform critical functions: -- **Maintain full nodes**: Both relay chain and parachain -- **Collect transactions**: Aggregate user transactions into blocks -- **Produce blocks**: Create parachain block candidates -- **Generate proofs**: Produce state transition proofs (Proof-of-Validity) -- **Submit to validators**: Send block candidates to relay chain validators -- **Facilitate XCM**: Enable cross-chain message passing +- Maintain full nodes for relay chain and parachain. +- Aggregate user transactions into blocks. +- Create parachain block candidates. +- Produce state transition proofs (Proof-of-Validity). +- Send block candidates to relay chain validators. +- Enable cross-chain message passing using XCM -Unlike relay chain validators, collators do not provide security guarantees—that responsibility lies with relay chain validators through the ELVES protocol. However, collators are essential for network liveness and censorship resistance. +Unlike relay chain validators, collators do not provide security guarantees—that responsibility lies with relay chain validators through the ELVES protocol. Rather, collators are essential for network liveness and censorship resistance. ## Prerequisites ### Hardware Requirements -Block-producing collators require robust hardware for reliable operation: +Block-producing collators require robust hardware for reliable operation including the following: - **CPU**: 4+ cores (8+ cores recommended for optimal performance) - **Memory**: 32 GB RAM minimum (64 GB recommended) @@ -53,290 +53,247 @@ Block-producing collators require robust hardware for reliable operation: - 30334 (relay chain P2P) - 9944 (WebSocket RPC - for management) -**Note**: Uptime is critical. Consider redundancy and monitoring to maintain block production reliability. +Uptime is critical. Consider redundancy and monitoring to maintain block production reliability. ### Software Requirements Collators use the **Polkadot Parachain** binary, the standard client for running Polkadot system parachains. -Required software: +Required software includes the following: - **Operating System**: Ubuntu 22.04 LTS (recommended) or similar Linux distribution - **Docker**: Required for obtaining binaries and running containers ### Account Requirements -You'll need: +Your account must meet the following requirements: - **Funded account**: For on-chain transactions and potential bonding - **Session keys**: For collator identification (generated after node setup) - **Node key**: For stable P2P peer ID (recommended) -## Installation +## Install Dependencies -### Step 1: Install the Polkadot Parachain Binary +This guide provides two deployment options. Select the option that best fits your needs: -**For Docker deployment:** +- **Docker-based Setup**: Best for simpler setup and maintenance +- **Manual/systemd Setup**: Best for production environments requiring more control -```bash -# Pull the polkadot-parachain image -docker pull parity/polkadot-parachain:stable2509-2 - -# Verify installation -docker run --rm parity/polkadot-parachain:stable2509-2 --version -``` - -**For bare-metal deployment:** - -Extract the binary from the Docker image: - -```bash -# Create a temporary container and copy the binary -docker create --name temp-parachain parity/polkadot-parachain:stable2509-2 -sudo docker cp temp-parachain:/usr/local/bin/polkadot-parachain /usr/local/bin/ -docker rm temp-parachain - -# Verify installation -polkadot-parachain --version -``` - -Check [Docker Hub](https://hub.docker.com/r/parity/polkadot-parachain/tags) for the latest stable tags. - -### Step 2: Generate Node Key - -Generate a stable node key for consistent peer ID: - -```bash -# Create directory for node data -sudo mkdir -p /var/lib/polkadot-collator -``` - -**Using Docker:** - -```bash -# Generate node key (outputs peer ID to console, saves key to file) -docker run --rm -v /var/lib/polkadot-collator:/data \ - parity/polkadot-parachain:stable2509-2 \ - key generate-node-key --file /data/node.key - -# Example output: 12D3KooWExcVYu7Mvjd4kxPVLwN2ZPnZ5NyLZ5ft477wqzfP2q6E -``` - -**Using native binary:** - -```bash -# Generate node key -polkadot-parachain key generate-node-key --file /var/lib/polkadot-collator/node.key - -# Example output: 12D3KooWExcVYu7Mvjd4kxPVLwN2ZPnZ5NyLZ5ft477wqzfP2q6E -``` - -Save the peer ID for future reference. - -### Step 3: Generate Account Key - -Generate an account for on-chain transactions: - -**Using Docker:** - -```bash -# Generate account key with sr25519 scheme -docker run --rm parity/polkadot-parachain:stable2509-2 key generate --scheme sr25519 -``` +=== "Docker Setup" -**Using native binary:** + Pull the Polkadot Parachain Docker image: -```bash -# Generate account key with sr25519 scheme -polkadot-parachain key generate --scheme sr25519 -``` + ```bash + docker pull parity/polkadot-parachain:stable2509-2 + ``` -Save the output containing: + Verify the installation: -- Secret phrase (seed) - Keep this secure! -- Public key (hex) -- Account ID -- SS58 Address + ```bash + docker run --rm parity/polkadot-parachain:stable2509-2 --version + ``` -**Security**: Store the secret phrase securely. Never share it. Consider using a hardware wallet for production collators. + Check [Docker Hub](https://hub.docker.com/r/parity/polkadot-parachain/tags){target=\_blank} for the latest stable tags. -### Step 4: Obtain Chain Specification +=== "Manual Setup" -Download the chain specification for your target system parachain: + Extract the binary from the official Docker image: -**Option 1: Download from Chainspec Collection (Recommended)** + ```bash + # Create a temporary container and copy the binary + docker create --name temp-parachain parity/polkadot-parachain:stable2509-2 + sudo docker cp temp-parachain:/usr/local/bin/polkadot-parachain /usr/local/bin/ + docker rm temp-parachain -1. Visit the [Chainspec Collection website](https://paritytech.github.io/chainspecs/) -2. Find your target system parachain -3. Download the chain specification JSON file -4. Save it as `chain-spec.json` + # Verify installation + polkadot-parachain --version + ``` -**Option 2: Build from Runtime** +## Generate Node Key -```bash -# Clone the runtimes repository -git clone https://github.com/polkadot-fellows/runtimes.git -cd runtimes - -# Build the desired runtime (example for Polkadot Hub) -cargo build --release -p asset-hub-polkadot-runtime - -# Install chain-spec-builder -cargo install --locked staging-chain-spec-builder@14.0.0 - -# Generate chain spec -chain-spec-builder create \ - --relay-chain polkadot \ - --para-id 1000 \ - --runtime target/release/wbuild/asset-hub-polkadot-runtime/asset_hub_polkadot_runtime.compact.compressed.wasm \ - named-preset production > chain-spec.json -``` +Generating a stable node key enables a consistent peer ID across the network. Follow these steps to generate a node key: -**System Parachain Para IDs:** +1. Create a directory for node data using the following command: + ```bash + sudo mkdir -p /var/lib/polkadot-collator + ``` -- Polkadot Hub: 1000 -- Bridge Hub: 1002 -- People Chain: 1004 -- Coretime Chain: 1005 +2. Generate your node key using Docker with the following command: + ```bash + docker run -it parity/subkey:latest generate-node-key > /var/lib/polkadot-collator/node.key + ``` -### Step 5: Create User and Directory Structure +3. Locate your peer ID in the displayed output. It will be similar to the following example: + ```bash + 12D3KooWExcVYu7Mvjd4kxPVLwN2ZPnZ5NyLZ5ft477wqzfP2q6E + ``` -```bash -# Create dedicated user -sudo useradd -r -s /bin/bash polkadot +Be sure to save the peer ID for future reference. -# Copy chain spec to directory -sudo cp chain-spec.json /var/lib/polkadot-collator/ +## Generate Account Key -# Set permissions -sudo chown -R polkadot:polkadot /var/lib/polkadot-collator -``` +Generate an account for on-chain transactions as follows: -## Configuration +1. Generate an account key with `sr25519` scheme using the following command: + ```bash + docker run -it parity/subkey:latest generate --scheme sr25519 + ``` -### Native Binary Systemd Service +2. Save the following items displayed in the output: + - Secret phrase (seed) - Keep this secure! + - Public key (hex) + - Account ID + - SS58 Address -Create a service file for your collator: + !!! warning + + Store the secret phrase securely. Never share it. Consider using a hardware wallet for production collators. -```bash -sudo nano /etc/systemd/system/polkadot-collator.service -``` +## Obtain Chain Specification -Add the following configuration: - -```ini -[Unit] -Description=Polkadot System Parachain Collator -After=network.target - -[Service] -Type=simple -User=polkadot -Group=polkadot -WorkingDirectory=/var/lib/polkadot-collator - -# Block-Producing Collator Configuration -ExecStart=/usr/local/bin/polkadot-parachain \ - --collator \ - --chain=/var/lib/polkadot-collator/chain-spec.json \ - --base-path=/var/lib/polkadot-collator \ - --port=30333 \ - --rpc-port=9944 \ - --prometheus-port=9615 \ - --node-key-file=/var/lib/polkadot-collator/node.key \ - --name="YourCollatorName" \ - -- \ - --execution=wasm \ - --chain=polkadot \ - --port=30334 \ - --sync=warp - -Restart=always -RestartSec=10 -LimitNOFILE=65536 - -[Install] -WantedBy=multi-user.target -``` +Download the chain specification for your target system parachain using one of the following options: -**Configuration Notes**: +### Download from Chainspec Collection (Recommended) -- `--collator`: Enables block production mode -- `--node-key-file`: Uses the generated node key for stable peer ID -- `--name`: Your collator name (visible in telemetry) -- Relay chain uses `--sync=warp` for faster initial sync +Follow these steps to download your specification from the Chainspec Collection: -### Docker Systemd Service +1. Visit the [Chainspec Collection](https://paritytech.github.io/chainspecs/) website. +2. Find your target system parachain. +3. Download the chain specification JSON file. +4. Save it as `chain-spec.json`. -If using Docker, create a service file with Docker configuration: +### Build Chainspec from Runtime -```bash -sudo nano /etc/systemd/system/polkadot-collator.service -``` - -Add the following configuration: - -```ini -[Unit] -Description=Polkadot System Parachain Collator (Docker) -After=network.target docker.service -Requires=docker.service - -[Service] -Type=simple -Restart=always -RestartSec=10 -TimeoutStartSec=0 - -ExecStartPre=-/usr/bin/docker stop polkadot-collator -ExecStartPre=-/usr/bin/docker rm polkadot-collator - -ExecStart=/usr/bin/docker run --rm \ - --name polkadot-collator \ - --network host \ - -v /var/lib/polkadot-collator:/data \ - parity/polkadot-parachain:stable2509-2 \ - --collator \ - --chain=/data/chain-spec.json \ - --base-path=/data \ - --port=30333 \ - --rpc-port=9944 \ - --prometheus-port=9615 \ - --node-key-file=/data/node.key \ - --name="YourCollatorName" \ - -- \ - --chain=polkadot \ - --port=30334 \ - --sync=warp - -ExecStop=/usr/bin/docker stop polkadot-collator - -[Install] -WantedBy=multi-user.target -``` +Follow these steps to build a chainspec from the runtime: -## Running the Collator +1. Clone the runtimes repository and navigate into it using the following commands: + ```bash + git clone https://github.com/polkadot-fellows/runtimes.git + cd runtimes + ``` -### Step 1: Start the Service +2. Build the desired runtime. Use the following command for Polkadot Hub: + ```bash + cargo build --release -p asset-hub-polkadot-runtime + ``` +3. Install the `chain-spec-builder` dependency using the following command: + ```bash + cargo install --locked staging-chain-spec-builder@14.0.0 + ``` + +4. Finally, generate the chain spec using the following commands: + ```bash + chain-spec-builder create \ + --relay-chain polkadot \ + --para-id 1000 \ + --runtime target/release/wbuild/asset-hub-polkadot-runtime/asset_hub_polkadot_runtime.compact.compressed.wasm \ + named-preset production > chain-spec.json + ``` + +??? tip "System Parachain Para IDs" + + - Polkadot Hub: 1000 + - Bridge Hub: 1002 + - People Chain: 1004 + - Coretime Chain: 1005 + +## Create User and Directory Structure + +1. Create a dedicated user with the following command: + ```bash + sudo useradd -r -s /bin/bash polkadot + ``` + +2. Use the following command to copy your chain spec to the directory: + ```bash + sudo cp chain-spec.json /var/lib/polkadot-collator/ + ``` + +3. Set permissions using the following command: + ```bash + sudo chown -R polkadot:polkadot /var/lib/polkadot-collator + ``` + +## Create Systemd Service File + +1. Create a service file to hold the configuration for your collator: + ```bash + sudo nano /etc/systemd/system/polkadot-collator.service + ``` + +2. Open the new file and add the following configuration code: + ```ini title="systemd/system/polkadot-collator.service" + [Unit] + Description=Polkadot System Parachain Collator + After=network.target + + [Service] + Type=simple + User=polkadot + Group=polkadot + WorkingDirectory=/var/lib/polkadot-collator + + # Block-Producing Collator Configuration + ExecStart=/usr/local/bin/polkadot-parachain \ + --collator \ + --chain=/var/lib/polkadot-collator/chain-spec.json \ + --base-path=/var/lib/polkadot-collator \ + --port=30333 \ + --rpc-port=9944 \ + --prometheus-port=9615 \ + --node-key-file=/var/lib/polkadot-collator/node.key \ + --name="YourCollatorName" \ + -- \ + --execution=wasm \ + --chain=polkadot \ + --port=30334 \ + --sync=warp + + Restart=always + RestartSec=10 + LimitNOFILE=65536 + + [Install] + WantedBy=multi-user.target + ``` + +??? note "Configuration notes" + + - `--collator`: Enables block production mode + - `--node-key-file`: Uses the generated node key for stable peer ID + - `--name`: Your collator name (visible in telemetry) + - Relay chain uses `--sync=warp` for faster initial sync + +## Run the Collator + +Follow these steps to run your collator node: + +1. Reload systemd using the following command: + ```bash + sudo systemctl daemon-reload + ``` + +2. Next, enable the service to start on boot using the command: + ```bash + sudo systemctl enable polkadot-collator + ``` +3. Now, start the service with the following command: + ```bash + sudo systemctl start polkadot-collator + ``` + +4. Finally, you can check the status of the service using the command: + ```bash + sudo systemctl status polkadot-collator + ``` + +To view collator service logs, use the command: ```bash -# Reload systemd -sudo systemctl daemon-reload - -# Enable service to start on boot -sudo systemctl enable polkadot-collator - -# Start the service -sudo systemctl start polkadot-collator - -# Check status -sudo systemctl status polkadot-collator - -# View logs sudo journalctl -u polkadot-collator -f ``` -### Step 2: Initial Sync +## Complete Initial Sync Your collator must sync both the relay chain and parachain before producing blocks. @@ -350,273 +307,221 @@ The relay chain uses warp sync for faster synchronization. **Important**: Do not proceed with registration until both chains are fully synced. Monitor sync progress using the log viewing commands in the [Log Management](#log-management) section. -### Step 3: Generate Session Keys +## Generate Session Keys -Once your node is fully synced, generate session keys: +Once your node is fully synced, use the following command to generate session keys via RPC: ```bash -# Generate session keys via RPC curl -H "Content-Type: application/json" \ -d '{"id":1, "jsonrpc":"2.0", "method": "author_rotateKeys", "params":[]}' \ http://localhost:9944 - -# Returns session keys as a hex string -# Example: "0x1234567890abcdef..." ``` -**Save the session keys** - you'll need them for on-chain registration. - -**Note**: Session keys are stored in the node's database. If you wipe the database, you'll need to generate new keys. +This command returns session keys as a hexstring in the terminal. You must save these session keys as you'll need them for on-chain registration. As session keys are stored in the node's database, if you wipe the database, you'll also need to generate new keys. -## Registration and Governance +## Register Collator for Selection -### Understanding Collator Selection +System parachains use different collator selection mechanisms. Explore the following tabs to see how mechanisms compare and determine the most suitable mechanism for registering your collator. -System parachains use different collator selection mechanisms: +=== "Invulnerables List" -**Invulnerables List**: + - Fixed list of collators approved through governance. + - Most common for system parachains. + - Requires governance proposal and approval. -- Fixed list of collators approved through governance -- Most common for system parachains -- Requires governance proposal and approval -**On-chain Selection**: +=== "On-chain Selection" -- Some parachains use pallet-collator-selection -- May require bonding tokens -- Automatic selection based on criteria + - Some parachains use pallet-collator-selection. + - May require bonding tokens. + - Automatic selection based on criteria. -**Fellowship Decisions**: +=== "Fellowship Decisions" -- Technical Fellowship may manage some system parachain collators -- Requires Fellowship membership or approval + - Technical Fellowship may manage some system parachain collators. + - Requires Fellowship membership or approval. ### Registration Process -The registration process varies by system parachain. General steps: - -#### 1. Check Current Collators - -Check the existing collators for your target parachain: - -```bash -# Using Polkadot.js Apps -# Connect to your target system parachain -# Go to Developer > Chain State -# Query: collatorSelection.invulnerables() or similar -``` - -#### 2. Prepare Governance Proposal - -For invulnerables-based selection: - -1. **Draft proposal**: Explain why you should be added as a collator -2. **Technical details**: Provide your session keys and account ID -3. **Infrastructure**: Describe your hardware and monitoring setup -4. **Experience**: Detail your relevant experience - -Submit to: -- Polkadot Forum: https://forum.polkadot.network -- Relevant governance channels - -#### 3. Set Session Keys On-Chain - -Once approved (or if using on-chain selection), set your session keys: - -**Using Polkadot.js Apps:** - -1. Navigate to Polkadot.js Apps and connect to your system parachain -2. Go to **Developer > Extrinsics** -3. Select your account -4. Choose `session.setKeys` extrinsic -5. Enter: - - `keys`: Your session keys (from `author_rotateKeys`) - - `proof`: 0x00 (typically) -6. Submit and sign the transaction - -**Using CLI (alternative):** - -```bash -# This varies by parachain - consult specific documentation -``` - -#### 4. Bond Tokens (if required) - -Some parachains require bonding tokens: - -1. Go to **Developer > Extrinsics** -2. Select `collatorSelection.registerAsCandidate` (if available) -3. Submit with required bond amount -4. Sign transaction - -#### 5. Await Governance Approval - -If using invulnerables: - -- Wait for governance vote -- Monitor forum and announcements -- Once approved, you'll be added to the invulnerables list -- Your collator will begin producing blocks in the next session/era - -### Verify Collator Status - -Check if your collator is active by monitoring logs for block production messages like "Prepared block for proposing" and "Imported #123". See the [Log Management](#log-management) section for log viewing commands. - -## Monitoring and Maintenance - -### Essential Monitoring - -**Block Production**: -```bash -# Monitor block production -sudo journalctl -u polkadot-collator | grep -i "prepared block" -``` - -**Peer Connections**: - -- Maintain a sufficient amount of peers for good connectivity -- Check peer count in logs - -**Resource Usage**: - -- Monitor CPU, RAM, and disk I/O -- Set up alerts for high usage - -**Sync Status**: - -- Ensure both chains stay synced -- Alert on sync issues +The registration process varies by system parachain. General steps include the following: + +1. Check the existing collators for your target parachain: + + 1. Navigate to Polkadot.js Apps and connect to your system parachain. + 2. Locate **Developer > Chain State**. + 3. Query `collatorSelection.invulnerables()` + +2. Prepare a governance proposal for invulnerables-based selection including the following information: + - **Draft proposal**: Explain why you should be added as a collator + - **Technical details**: Provide your session keys and account ID + - **Infrastructure**: Describe your hardware and monitoring setup + - **Experience**: Detail your relevant experience + + Submit the proposal to the relevant governance channels on the [Polkadot Forum](https://forum.polkadot.network){target=\_blank}. + +3. Once approved (or if using on-chain selection), follow these steps to register session keys using Polkadot.js Apps: + 1. Navigate to Polkadot.js Apps and connect to your system parachain. + 2. Locate **Developer > Extrinsics**. + 3. Select your account. + 4. Choose the `session.setKeys` extrinsic. + 5. Enter the following information: + - `keys`: Your session keys (from `author_rotateKeys`) + - `proof`: 0x00 (typically) + 6. Submit and sign the transaction. + +4. If the parachain requires bonding tokens, use the follow these steps to submit them using Polkadot.js Apps: + 1. Locate **Developer > Extrinsics**. + 2. Select `collatorSelection.registerAsCandidate`. + 3. Submit the transaction with the required bond amount. + 4. Sign the transaction. + +5. If applying to join the invulnerables list, you must now await governance approval for your proposal. Monitor the [Polkadot Forum](https://forum.polkadot.network){target=\_blank} governance channels and announcements. Once approved, your collator is added to the invulnerables list and will begin producing blocks in the next session or era. + +6. Verify your collator is active by monitoring logs for block production messages like "Prepared block for proposing" and "Imported #123". See the [Log Management](#log-management) section for log viewing commands. + +## Monitor and Maintain Your Collator + +Monitoring the following items will help ensure your collator runs efficiently to avoid service interruptions or down time: + +- **Block Production**: monitor for block production using the following command: + ```bash + sudo journalctl -u polkadot-collator | grep -i "prepared block" + ``` +- **Peer Connections**: Maintain a sufficient amount of peers for good connectivity and set up alerts to notify if peer connections falls below ten. +- **Resource Usage**: Monitor CPU, RAM, and disk I/O and set up alerts for unusual or high usage. +- **Sync Status**: Ensure both chains stay synced and set up alerts for sync issues. ### Prometheus Metrics -Metrics available at `http://localhost:9615/metrics` +You can use the following information to configure [Prometheus](https://prometheus.io/docs/introduction/first_steps/){target=\_blank} to monitor, collect, and store your collator node metrics: -Example Prometheus configuration: -{% raw %} -```yaml -scrape_configs: - - job_name: 'polkadot-collator' - static_configs: - - targets: ['localhost:9615'] -``` -{% endraw %} +- **URL**: Metrics are available to view at `http://localhost:9615/metrics` +- **Example Prometheus configuration**: Update your `prometheus.yml` to add the following code: + {% raw %} + ```yaml + scrape_configs: + - job_name: 'polkadot-collator' + static_configs: + - targets: ['localhost:9615'] + ``` + {% endraw %} -Key metrics to monitor: +Key metrics to monitor via Prometheus include the following: - `substrate_block_height`: Current block height - `substrate_finalized_height`: Finalized block height - `substrate_peers_count`: Peer connections - `substrate_ready_transactions_number`: Transaction queue -### Setting Up Alerts - -Configure alerts for: - -- Service failures -- Sync issues -- Low peer count (< 10 peers) -- Block production gaps -- High resource usage -- Disk space low -### Log Management +## Log Management -```bash -# View recent logs -sudo journalctl -u polkadot-collator -n 100 - -# Follow logs in real-time -sudo journalctl -u polkadot-collator -f - -# Filter for errors -sudo journalctl -u polkadot-collator | grep -i error +Effecient log management is essential to ensure collator performance and uptime. Use the following commands to help you manage logs to monitor and maintain your collator: -# Filter for block production -sudo journalctl -u polkadot-collator | grep -i "imported" -``` +- View recent logs: + ```bash + sudo journalctl -u polkadot-collator -n 100 + ``` +- Follow logs in real-time: + ```bash + sudo journalctl -u polkadot-collator -f + ``` +- Filter for errors: + ```bash + sudo journalctl -u polkadot-collator | grep -i error + ``` +- Filter for block production: + ```bash + sudo journalctl -u polkadot-collator | grep -i "imported" + ``` -### Database Maintenance +## Database Maintenance -Check database size: +You can check database size using the following command: ```bash -# Check database size du -sh /var/lib/polkadot-collator ``` - -The node handles pruning automatically. - -### Updates and Upgrades - -**Runtime Upgrades**: - -- Automatic via on-chain governance -- No manual action required -- Monitor announcements for breaking changes - -**Client Upgrades**: - -- Require manual binary update -- Subscribe to announcements: - - Polkadot Forum - - Fellowship GitHub - - Matrix channels - -**Upgrade Procedure (Bare-Metal)**: - -```bash -# Stop the service -sudo systemctl stop polkadot-collator - -# Backup data (recommended) -sudo cp -r /var/lib/polkadot-collator /var/lib/polkadot-collator.backup - -# Pull new image and extract binary -docker pull parity/polkadot-parachain: -docker create --name temp-parachain parity/polkadot-parachain: -sudo docker cp temp-parachain:/usr/local/bin/polkadot-parachain /usr/local/bin/ -docker rm temp-parachain - -# Verify version -polkadot-parachain --version - -# Restart service -sudo systemctl start polkadot-collator - -# Verify service is running -sudo systemctl status polkadot-collator -``` - -**Upgrade Procedure (Docker)**: - -```bash -# Stop the service -sudo systemctl stop polkadot-collator - -# Backup data (recommended) -sudo cp -r /var/lib/polkadot-collator /var/lib/polkadot-collator.backup - -# Pull new image -docker pull parity/polkadot-parachain: - -# Update the image tag in /etc/systemd/system/polkadot-collator.service - -# Reload systemd and restart -sudo systemctl daemon-reload -sudo systemctl start polkadot-collator - -# Verify service is running -sudo systemctl status polkadot-collator -``` - -**Note**: For log monitoring, see the [Log Management](#log-management) section. +The collator node handles pruning automatically. + +## Updates and Upgrades + +Updates or upgrades can happen on either the runtime or client. Runtime upgrades happen automatically via on-chain governance and do not require manual action on your part. Client upgrades do require a manual binary update process performed via terminal commands as follows: + +=== "Docker Setup" + + 1. Stop the service: + ```bash + sudo systemctl stop polkadot-collator + ``` + + 2. Backup data (recommended): + ```bash + sudo cp -r /var/lib/polkadot-collator /var/lib/polkadot-collator.backup + ``` + + 3. Pull the new Docker image: + ```bash + docker pull parity/polkadot-parachain: + ``` + + 4. Update the image tag in your systemd service file: + ```bash + sudo nano /etc/systemd/system/polkadot-collator.service + ``` + + 5. Reload systemd and restart the service: + ```bash + sudo systemctl daemon-reload + sudo systemctl start polkadot-collator + ``` + + 6. Verify the service is running: + ```bash + sudo systemctl status polkadot-collator + ``` + +=== "Manual Setup" + + 1. Stop the service: + ```bash + sudo systemctl stop polkadot-collator + ``` + + 2. Backup data (recommended): + ```bash + sudo cp -r /var/lib/polkadot-collator /var/lib/polkadot-collator.backup + ``` + + 3. Pull the new image and extract the binary: + ```bash + docker pull parity/polkadot-parachain: + docker create --name temp-parachain parity/polkadot-parachain: + sudo docker cp temp-parachain:/usr/local/bin/polkadot-parachain /usr/local/bin/ + docker rm temp-parachain + ``` + + 4. Verify `polkadot-parachain` version to confirm successful update: + ```bash + polkadot-parachain --version + ``` + + 5. Restart the service: + ```bash + sudo systemctl start polkadot-collator + ``` + + 6. Verify the service is running: + ```bash + sudo systemctl status polkadot-collator + ``` ## Conclusion Running a collator node is essential for parachain operation and network security. By following this guide, you have set up a production-ready collator that: -- Produces blocks for your parachain and maintains network consensus -- Implements comprehensive security measures to protect keys and operations -- Supports robust monitoring and alerting for reliable performance -- Follows best practices for both Docker and systemd deployments +- Produces blocks for your parachain and maintains network consensus. +- Implements comprehensive security measures to protect keys and operations. +- Supports robust monitoring and alerting for reliable performance. +- Follows best practices for both Docker and systemd deployments. As a collator operator, you play a vital role in your parachain's infrastructure. Regular maintenance, security updates, and monitoring will ensure your collator continues to perform reliably. Stay engaged with your parachain community and keep up with updates to maintain optimal performance and security. diff --git a/.ai/pages/node-infrastructure-run-a-node-polkadot-hub-rpc.md b/.ai/pages/node-infrastructure-run-a-node-polkadot-hub-rpc.md index 7a8368545..e1dab7997 100644 --- a/.ai/pages/node-infrastructure-run-a-node-polkadot-hub-rpc.md +++ b/.ai/pages/node-infrastructure-run-a-node-polkadot-hub-rpc.md @@ -1,6 +1,6 @@ --- title: Run an RPC Node for Polkadot Hub -description: Complete guide to set up and run an RPC node for Polkadot Hub with Polkadot SDK RPC endpoints. +description: Follow this guide to understand hardware and software requirements and how to set up and run an RPC node for Polkadot Hub with Polkadot SDK RPC endpoints. categories: Infrastructure url: https://docs.polkadot.com/node-infrastructure/run-a-node/polkadot-hub-rpc/ --- @@ -20,13 +20,15 @@ Running an RPC node for Polkadot Hub enables applications, wallets, and users to This setup enables block explorer indexing and provides full compatibility with Polkadot SDK development tools. -**Important Note**: The parameters and configurations in this guide are provided as illustrative examples. You may need to modify them according to your specific environment, hardware capabilities, and network conditions. +!!! note + + The parameters and configurations in this guide are provided as illustrative examples. You may need to modify them according to your specific environment, hardware capabilities, and network conditions. ## Prerequisites ### Hardware Requirements -RPC nodes serving production traffic require robust hardware: +RPC nodes serving production traffic require robust hardware. The following should be considered the minimum standard to effectively operate an RPC node: - **CPU**: 8+ cores (16+ cores for high traffic) - **Memory**: 64 GB RAM minimum (128 GB recommended for high traffic) @@ -58,417 +60,366 @@ Required software: ## Setup Options -This guide provides two deployment options: - -1. **Docker-based Setup**: Simpler to set up and maintain -2. **Manual/Systemd Setup**: For production environments requiring more control - -Choose the option that best fits your needs. - ---- - -## Option 1: Docker-Based Setup - -This option uses Docker containers for the Polkadot SDK node, making it easy to set up and manage. - -### Step 1: Download Chain Specification - -Download the official Polkadot Hub (formerly known as Asset Hub) chain specification: - -```bash -curl -L https://raw.githubusercontent.com/paritytech/polkadot-sdk/master/cumulus/parachains/chain-specs/asset-hub-polkadot.json -o asset-hub-polkadot.json -``` - -**Note**: This chain specification is the official configuration file that defines the network parameters for Polkadot Hub. - -### Step 2: Download Database Snapshots (Optional but Recommended) - -Using pre-synchronized snapshots significantly reduces initial sync time from several days to just a few hours. You need to download both parachain and relay chain data. - -**Snapshot Provider**: https://snapshots.polkadot.io/ - -#### Create Directories - -```bash -mkdir -p my-node-data/chains/asset-hub-polkadot/db -mkdir -p my-node-data/chains/polkadot/db -``` - -#### Download Polkadot Hub Parachain Snapshot - -Choose between archive (complete history) or pruned (recent state) snapshots: - -**Archive Snapshot** (recommended for RPC with historical data): - -```bash -# Check https://snapshots.polkadot.io/ for the latest snapshot URL -export SNAPSHOT_URL_ASSET_HUB="https://snapshots.polkadot.io/polkadot-asset-hub-rocksdb-archive/LATEST" - -rclone copyurl $SNAPSHOT_URL_ASSET_HUB/files.txt files.txt -rclone copy --progress --transfers 20 \ - --http-url $SNAPSHOT_URL_ASSET_HUB \ - --no-traverse --http-no-head --disable-http2 \ - --inplace --no-gzip-encoding --size-only \ - --retries 6 --retries-sleep 10s \ - --files-from files.txt :http: my-node-data/chains/asset-hub-polkadot/db/ - -rm files.txt -``` - -**Parameter Explanation**: - -- `--transfers 20`: Uses 20 parallel transfers for faster download -- `--retries 6`: Automatically retries failed transfers up to 6 times -- `--retries-sleep 10s`: Waits 10 seconds between retry attempts -- `--size-only`: Only transfers if sizes differ (prevents unnecessary re-downloads) - -#### Download Polkadot Relay Chain Snapshot - -**Pruned Snapshot** (recommended for RPC nodes): - -```bash -# Check https://snapshots.polkadot.io/ for the latest snapshot URL -export SNAPSHOT_URL_RELAY="https://snapshots.polkadot.io/polkadot-rocksdb-prune/LATEST" - -rclone copyurl $SNAPSHOT_URL_RELAY/files.txt files.txt -rclone copy --progress --transfers 20 \ - --http-url $SNAPSHOT_URL_RELAY \ - --no-traverse --http-no-head --disable-http2 \ - --inplace --no-gzip-encoding --size-only \ - --retries 6 --retries-sleep 10s \ - --files-from files.txt :http: my-node-data/chains/polkadot/db/ - -rm files.txt -``` - -**Alternative Options**: - -- Pruned snapshot: `polkadot-rocksdb-prune` (smaller size, recent state) -- Archive snapshot: `polkadot-rocksdb-archive` (complete history, larger size) - -### Step 3: Start Polkadot Hub Node - -Launch the node using the official Parity Docker image: - -**Docker Image**: https://hub.docker.com/r/parity/polkadot-parachain - -```bash -docker run -d --name polkadot-hub-rpc --restart unless-stopped \ - -p 9944:9944 \ - -p 9933:9933 \ - -p 9615:9615 \ - -p 30334:30334 \ - -p 30333:30333 \ - -v $(pwd)/asset-hub-polkadot.json:/asset-hub-polkadot.json \ - -v $(pwd)/my-node-data:/data \ - parity/polkadot-parachain:stable2509-2 \ - --name=PolkadotHubRPC \ - --base-path=/data \ - --chain=/asset-hub-polkadot.json \ - --prometheus-external \ - --prometheus-port 9615 \ - --unsafe-rpc-external \ - --rpc-port=9944 \ - --rpc-cors=all \ - --rpc-methods=safe \ - --rpc-max-connections=1000 \ - --state-pruning=archive \ - --blocks-pruning=archive \ - -- \ - --base-path=/data \ - --chain=polkadot \ - --state-pruning=256 \ - --blocks-pruning=archive-canonical -``` - -**Critical Configuration Parameters**: - -**Port Mappings**: - -- `9944`: Polkadot SDK RPC endpoint (WebSocket/HTTP) -- `9933`: Polkadot SDK HTTP RPC endpoint -- `9615`: Prometheus metrics endpoint -- `30333/30334`: P2P networking ports - -**Node Parameters**: - -- `--unsafe-rpc-external`: Enables external RPC access -- `--rpc-cors=all`: Allows all origins for CORS -- `--rpc-methods=safe`: Only allows safe RPC methods -- `--state-pruning=archive`: Keeps complete state history -- `--blocks-pruning=archive`: Keeps all block data -- `--prometheus-external`: Exposes metrics externally - -**Security Warning**: The `--unsafe-rpc-external` flag should only be used in development or properly secured environments. For production, use a reverse proxy with authentication. - -### Step 4: Monitor Synchronization - -Monitor the node synchronization status: - -```bash -# Check sync status -curl -H "Content-Type: application/json" \ - -d '{"id":1, "jsonrpc":"2.0", "method": "system_syncState", "params":[]}' \ - http://localhost:9944 -``` - -**Expected Response Format**: - -```json -{ - "jsonrpc":"2.0", - "id":1, - "result":{ - "startingBlock":0, - "currentBlock":3394816, - "highestBlock":3394816 - } -} -``` - -**Synchronization Status**: - -- **In Progress**: `currentBlock` < `highestBlock` -- **Complete**: `currentBlock` = `highestBlock` - -### Step 5: Verify Setup - -Let's verify the Polkadot SDK RPC endpoint is working correctly. - -#### API Endpoint - -**Polkadot SDK RPC (Port 9944)**: - -- WebSocket: `ws://your-server:9944` -- HTTP: `http://your-server:9944` -- Purpose: Full Polkadot SDK API access for parachain data -- Use Cases: Polkadot SDK applications, parachain-specific operations - -#### Polkadot SDK RPC Tests - -**Get Chain Information**: - -```bash -curl -H "Content-Type: application/json" \ - -d '{"id":1, "jsonrpc":"2.0", "method": "system_chain", "params":[]}' \ - http://localhost:9944 -``` - -**Get Latest Block**: - -```bash -curl -H "Content-Type: application/json" \ - -d '{"id":1, "jsonrpc":"2.0", "method": "chain_getHeader", "params":[]}' \ - http://localhost:9944 -``` - -**Get Node Health**: - -```bash -curl -H "Content-Type: application/json" \ - -d '{"id":1, "jsonrpc":"2.0", "method": "system_health", "params":[]}' \ - http://localhost:9944 -``` - -### Managing Docker Containers - -**View logs**: - -```bash -# View node logs -docker logs -f polkadot-hub-rpc -``` - -**Stop container**: - -```bash -docker stop polkadot-hub-rpc -``` - -**Start container**: - -```bash -docker start polkadot-hub-rpc -``` - -**Remove container**: - -```bash -docker rm polkadot-hub-rpc -``` - -**Note**: For update procedures, see the [Updates and Upgrades](#updates-and-upgrades) section. - ---- - -## Option 2: Manual/Systemd Setup - -This option provides more control and is recommended for production environments requiring custom configurations. - -### Step 1: Install the Polkadot Parachain Binary - -Extract the binary from the official Docker image: - -```bash -# Pull the image and extract binary -docker pull parity/polkadot-parachain:stable2509-2 -docker create --name temp-parachain parity/polkadot-parachain:stable2509-2 -sudo docker cp temp-parachain:/usr/local/bin/polkadot-parachain /usr/local/bin/ -docker rm temp-parachain - -# Verify installation -polkadot-parachain --version -``` - -Check [Docker Hub](https://hub.docker.com/r/parity/polkadot-parachain/tags) for the latest stable tags. - -### Step 2: Obtain Chain Specification - -Download the Polkadot Hub chain specification: - -```bash -curl -L https://raw.githubusercontent.com/paritytech/polkadot-sdk/master/cumulus/parachains/chain-specs/asset-hub-polkadot.json -o asset-hub-polkadot.json -``` - -### Step 3: Create User and Directory Structure - -```bash -# Create dedicated user -sudo useradd -r -s /bin/bash polkadot - -# Create data directory -sudo mkdir -p /var/lib/polkadot-hub-rpc - -# Copy chain spec to the directory -sudo cp asset-hub-polkadot.json /var/lib/polkadot-hub-rpc/ - -# Set permissions -sudo chown -R polkadot:polkadot /var/lib/polkadot-hub-rpc -``` - -### Step 4: Create Systemd Service - -Create a service file for the Polkadot SDK RPC node: - -```bash -sudo nano /etc/systemd/system/polkadot-hub-rpc.service -``` - -Add the following configuration: - -```ini -[Unit] -Description=Polkadot Hub RPC Node -After=network.target - -[Service] -Type=simple -User=polkadot -Group=polkadot -WorkingDirectory=/var/lib/polkadot-hub-rpc - -ExecStart=/usr/local/bin/polkadot-parachain \ - --name=PolkadotHubRPC \ - --chain=/var/lib/polkadot-hub-rpc/asset-hub-polkadot.json \ - --base-path=/var/lib/polkadot-hub-rpc \ - --port=30333 \ - --rpc-port=9944 \ - --rpc-external \ - --rpc-cors=all \ - --rpc-methods=safe \ - --rpc-max-connections=1000 \ - --prometheus-port=9615 \ - --prometheus-external \ - --state-pruning=archive \ - --blocks-pruning=archive \ - -- \ - --chain=polkadot \ - --base-path=/var/lib/polkadot-hub-rpc \ - --port=30334 \ - --state-pruning=256 \ - --blocks-pruning=archive-canonical - -Restart=always -RestartSec=10 -LimitNOFILE=65536 - -[Install] -WantedBy=multi-user.target -``` - -### Step 5: Start Service - -```bash -# Reload systemd -sudo systemctl daemon-reload - -# Enable service to start on boot -sudo systemctl enable polkadot-hub-rpc - -# Start the Polkadot SDK node -sudo systemctl start polkadot-hub-rpc - -# Check status and wait for sync -sudo systemctl status polkadot-hub-rpc -sudo journalctl -u polkadot-hub-rpc -f -``` - -### Step 6: Verify Setup - -Use the same verification tests as in the Docker setup (see Step 5 above). - ---- - -## Monitoring and Maintenance - -### Log Management - -**Docker Setup**: - -```bash -# View node logs -docker logs -f polkadot-hub-rpc -``` - -**Systemd Setup**: - -```bash -# View node logs -sudo journalctl -u polkadot-hub-rpc -f - -# View recent logs -sudo journalctl -u polkadot-hub-rpc -n 100 - -# Filter for errors -sudo journalctl -u polkadot-hub-rpc | grep -i error -``` - -### Performance Monitoring - -Monitor key metrics: - -- **Sync status**: Ensure node stays fully synced -- **Peer connections**: Maintain 30+ peers for good connectivity -- **Resource usage**: Monitor CPU, RAM, and disk I/O -- **RPC request latency**: Track response times for the Polkadot SDK API -- **Connection count**: Monitor active RPC connections - -**Prometheus Metrics**: - -Metrics are available at `http://localhost:9615/metrics` - -Example Prometheus scrape configuration: - -{% raw %} -```yaml -scrape_configs: - - job_name: 'polkadot-hub-rpc' - static_configs: - - targets: ['localhost:9615'] -``` -{% endraw %} - -**Key Metrics to Monitor**: +This guide provides two options for deployment: + +- **Docker-based Setup**: Best for simpler set up and maintenance +- **Manual/systemd Setup**: Best for production environments requiring more control + +Select the best option for your project, then use the steps in the following tabs to complete set up. + +=== "Docker-Based Setup" + + This option uses Docker containers for the Polkadot SDK node, making it easy to set up and manage. Follow these steps to set your RPC node using Docker: + + 1. Download the official Polkadot Hub (formerly known as Asset Hub) chain specification: + ```bash + curl -L https://raw.githubusercontent.com/paritytech/polkadot-sdk/master/cumulus/parachains/chain-specs/asset-hub-polkadot.json -o asset-hub-polkadot.json + ``` + + !!! note + + This chain specification is the official configuration file that defines the network parameters for Polkadot Hub. + + 2. (Optional but recommended) Download database snapshots + - Using pre-synchronized snapshots significantly reduces initial sync time from several days to just a few hours. You need to download both parachain and relay chain data. + - You can obtain the latest snapshot from the [Snapshot Provider](https://snapshots.polkadot.io/){target=\_blank}. Follow these steps to download and use snapshots: + 1. Create new directories with the following commands: + ```bash + mkdir -p my-node-data/chains/asset-hub-polkadot/db + mkdir -p my-node-data/chains/polkadot/db + ``` + 2. Download the appropriate snapshot using the following commands: + + === "Archive snapshot" + + Contains complete history, recommended for RPC with historical data. + ```bash + # Check https://snapshots.polkadot.io/ for the latest snapshot URL + export SNAPSHOT_URL_ASSET_HUB="https://snapshots.polkadot.io/polkadot-asset-hub-rocksdb-archive/LATEST" + + rclone copyurl $SNAPSHOT_URL_ASSET_HUB/files.txt files.txt + rclone copy --progress --transfers 20 \ + --http-url $SNAPSHOT_URL_ASSET_HUB \ + --no-traverse --http-no-head --disable-http2 \ + --inplace --no-gzip-encoding --size-only \ + --retries 6 --retries-sleep 10s \ + --files-from files.txt :http: my-node-data/chains/asset-hub-polkadot/db/ + + rm files.txt + ``` + + - `--transfers 20`: Uses 20 parallel transfers for faster download. + - `--retries 6`: Automatically retries failed transfers up to 6 times. + - `--retries-sleep 10s`: Waits 10 seconds between retry attempts. + - `--size-only`: Only transfers if sizes differ (prevents unnecessary re-downloads). + + === "Pruned snapshot" + + Contains recent state for a smaller package size, recommended for RPC nodes. + ```bash + # Check https://snapshots.polkadot.io/ for the latest snapshot URL + export SNAPSHOT_URL_RELAY="https://snapshots.polkadot.io/polkadot-rocksdb-prune/LATEST" + + rclone copyurl $SNAPSHOT_URL_RELAY/files.txt files.txt + rclone copy --progress --transfers 20 \ + --http-url $SNAPSHOT_URL_RELAY \ + --no-traverse --http-no-head --disable-http2 \ + --inplace --no-gzip-encoding --size-only \ + --retries 6 --retries-sleep 10s \ + --files-from files.txt :http: my-node-data/chains/polkadot/db/ + + rm files.txt + ``` + 3. Launch Polkadot Hub Node using the official [Parity Docker image](https://hub.docker.com/r/parity/polkadot-parachain){target=\_blank} with the following command: + ```bash + docker run -d --name polkadot-hub-rpc --restart unless-stopped \ + -p 9944:9944 \ + -p 9933:9933 \ + -p 9615:9615 \ + -p 30334:30334 \ + -p 30333:30333 \ + -v $(pwd)/asset-hub-polkadot.json:/asset-hub-polkadot.json \ + -v $(pwd)/my-node-data:/data \ + parity/polkadot-parachain:stable2509-2 \ + --name=PolkadotHubRPC \ + --base-path=/data \ + --chain=/asset-hub-polkadot.json \ + --prometheus-external \ + --prometheus-port 9615 \ + --unsafe-rpc-external \ + --rpc-port=9944 \ + --rpc-cors=all \ + --rpc-methods=safe \ + --rpc-max-connections=1000 \ + --state-pruning=archive \ + --blocks-pruning=archive \ + -- \ + --base-path=/data \ + --chain=polkadot \ + --state-pruning=256 \ + --blocks-pruning=archive-canonical + ``` + + Critical configuration parameters include port mappings and node parameters: + + === "Port mappings" + + - `9944`: Polkadot SDK RPC endpoint (WebSocket/HTTP) + - `9933`: Polkadot SDK HTTP RPC endpoint + - `9615`: Prometheus metrics endpoint + - `30333/30334`: P2P networking ports + + === "Node parameters" + + - `--unsafe-rpc-external`: Enables external RPC access + - `--rpc-cors=all`: Allows all origins for CORS + - `--rpc-methods=safe`: Only allows safe RPC methods + - `--state-pruning=archive`: Keeps complete state history + - `--blocks-pruning=archive`: Keeps all block data + - `--prometheus-external`: Exposes metrics externally + + !!! warning + + The `--unsafe-rpc-external` flag should only be used in development or properly secured environments. For production, use a reverse proxy with authentication. + + 4. Monitor the node synchronization status using the following command: + ```bash + curl -H "Content-Type: application/json" \ + -d '{"id":1, "jsonrpc":"2.0", "method": "system_syncState", "params":[]}' \ + http://localhost:9944 + ``` + + You should see a response similar to the following: + + ```json + { + "jsonrpc":"2.0", + "id":1, + "result":{ + "startingBlock":0, + "currentBlock":3394816, + "highestBlock":3394816 + } + } + ``` + + When synchronization is complete, `currentBlock` will be equal to `highestBlock`. + + 5. You can use a few different commands to verify your node is running properly: + + - Get chain information: + ```bash + curl -H "Content-Type: application/json" \ + -d '{"id":1, "jsonrpc":"2.0", "method": "system_chain", "params":[]}' \ + http://localhost:9944 + ``` + - Get the latest block: + ```bash + curl -H "Content-Type: application/json" \ + -d '{"id":1, "jsonrpc":"2.0", "method": "chain_getHeader", "params":[]}' \ + http://localhost:9944 + ``` + - Query node health: + ```bash + curl -H "Content-Type: application/json" \ + -d '{"id":1, "jsonrpc":"2.0", "method": "system_health", "params":[]}' \ + http://localhost:9944 + ``` + + 6. Use the following commands to manage your Docker containers: + + - View node logs: + ```bash + docker logs -f polkadot-hub-rpc + ``` + - Stop container: + ```bash + docker stop polkadot-hub-rpc + ``` + - Start container: + ```bash + docker start polkadot-hub-rpc + ``` + - Remove container: + ```bash + docker rm polkadot-hub-rpc + ``` + +=== "Manual systemd Setup" + + This option provides more control and is recommended for production environments requiring custom configurations. + + 1. Install the Polkadot Parachain binary by extracting it from the official Docker image: + ```bash + # Pull the image and extract binary + docker pull parity/polkadot-parachain:stable2509-2 + docker create --name temp-parachain parity/polkadot-parachain:stable2509-2 + sudo docker cp temp-parachain:/usr/local/bin/polkadot-parachain /usr/local/bin/ + docker rm temp-parachain + + # Verify installation + polkadot-parachain --version + ``` + + Check [Docker Hub](https://hub.docker.com/r/parity/polkadot-parachain/tags){target=\_blank} for the latest stable tags. + + 2. Download the Polkadot Hub chain specification: + ```bash + curl -L https://raw.githubusercontent.com/paritytech/polkadot-sdk/master/cumulus/parachains/chain-specs/asset-hub-polkadot.json -o asset-hub-polkadot.json + ``` + + 3. Create user and directory structures using the following commands: + - Create a dedicated user: + ```bash + sudo useradd -r -s /bin/bash polkadot + ``` + - Create data directory: + ```bash + sudo mkdir -p /var/lib/polkadot-hub-rpc + ``` + - Copy the chain spec to the directory: + ```bash + sudo cp asset-hub-polkadot.json /var/lib/polkadot-hub-rpc/ + ``` + - Set permissions: + ```bash + sudo chown -R polkadot:polkadot /var/lib/polkadot-hub-rpc + ``` + + 4. Create a systemd service file for the Polkadot SDK RPC node: + ```bash + sudo nano /etc/systemd/system/polkadot-hub-rpc.service + ``` + + 5. Open the new service file and add the following configuration: + ```ini + [Unit] + Description=Polkadot Hub RPC Node + After=network.target + + [Service] + Type=simple + User=polkadot + Group=polkadot + WorkingDirectory=/var/lib/polkadot-hub-rpc + + ExecStart=/usr/local/bin/polkadot-parachain \ + --name=PolkadotHubRPC \ + --chain=/var/lib/polkadot-hub-rpc/asset-hub-polkadot.json \ + --base-path=/var/lib/polkadot-hub-rpc \ + --port=30333 \ + --rpc-port=9944 \ + --rpc-external \ + --rpc-cors=all \ + --rpc-methods=safe \ + --rpc-max-connections=1000 \ + --prometheus-port=9615 \ + --prometheus-external \ + --state-pruning=archive \ + --blocks-pruning=archive \ + -- \ + --chain=polkadot \ + --base-path=/var/lib/polkadot-hub-rpc \ + --port=30334 \ + --state-pruning=256 \ + --blocks-pruning=archive-canonical + + Restart=always + RestartSec=10 + LimitNOFILE=65536 + + [Install] + WantedBy=multi-user.target + ``` + + 6. Start the service using the following commands: + - Reload systemd: + ```bash + sudo systemctl daemon-reload + ``` + - Enable service to start on boot: + ```bash + sudo systemctl enable polkadot-hub-rpc + ``` + - Start the Polkadot SDK node: + ```bash + sudo systemctl start polkadot-hub-rpc + ``` + - Check status and wait for sync: + ```bash + sudo systemctl status polkadot-hub-rpc + sudo journalctl -u polkadot-hub-rpc -f + ``` + + 7. You can use a few different commands to verify your node is running properly: + - Get chain information: + ```bash + curl -H "Content-Type: application/json" \ + -d '{"id":1, "jsonrpc":"2.0", "method": "system_chain", "params":[]}' \ + http://localhost:9944 + ``` + - Get the latest block: + ```bash + curl -H "Content-Type: application/json" \ + -d '{"id":1, "jsonrpc":"2.0", "method": "chain_getHeader", "params":[]}' \ + http://localhost:9944 + ``` + - Query node health: + ```bash + curl -H "Content-Type: application/json" \ + -d '{"id":1, "jsonrpc":"2.0", "method": "system_health", "params":[]}' \ + http://localhost:9944 + ``` + +## Monitor and Maintain RPC Node + +There are a few key monitoring and maintenance commands that can help you ensure RPC node uptime and performance including log monitoring and metrics. + +To view node logs, use the appropriate command for your setup: + +=== "Docker Setup" + + ```bash + docker logs -f polkadot-hub-rpc + ``` + +=== "systemd Setup" + + - View node logs: + ```bash + sudo journalctl -u polkadot-hub-rpc -f + ``` + - View recent logs: + ```bash + sudo journalctl -u polkadot-hub-rpc -n 100 + ``` + - Filter for errors: + ```bash + sudo journalctl -u polkadot-hub-rpc | grep -i error + ``` + +Monitoring key metrics like the following items can help you stay up to date on the performance and health of your node and allow you to intervene at the first sign of issues: + +- **Sync status**: Ensure node stays fully synced. +- **Peer connections**: Maintain 30+ peers for good connectivity. +- **Resource usage**: Monitor CPU, RAM, and disk I/O. +- **RPC request latency**: Track response times for the Polkadot SDK API. +- **Connection count**: Monitor active RPC connections. + +You can use the following information to configure [Prometheus](https://prometheus.io/docs/introduction/first_steps/){target=\_blank} to monitor, collect, and store your RPC node metrics: + +- **URL**: Metrics are available to view at `http://localhost:9615/metrics` +- **Example Prometheus configuration**: Update your `prometheus.yml` to add the following code: + {% raw %} + ```yaml + scrape_configs: + - job_name: 'polkadot-hub-rpc' + static_configs: + - targets: ['localhost:9615'] + ``` + {% endraw %} + +Key metrics to monitor via Prometheus include: - `substrate_block_height`: Current block height - `substrate_finalized_height`: Finalized block height @@ -477,59 +428,68 @@ scrape_configs: ### Database Maintenance -Check database size periodically: - -```bash -# Docker setup -du -sh my-node-data - -# Systemd setup -du -sh /var/lib/polkadot-hub-rpc -``` +Check database size periodically using the commands for your selected setup: -The node handles pruning automatically based on configuration unless running in archive mode. - -### Updates and Upgrades - -**Docker Setup**: +=== "Docker Setup" -```bash -# Pull latest image -docker pull parity/polkadot-parachain: + ```bash + du -sh my-node-data + ``` -# Restart container -docker stop polkadot-hub-rpc -docker rm polkadot-hub-rpc +=== "systemd Setup" -# Start new container (use same command from setup with updated image tag) -``` + ```bash + du -sh /var/lib/polkadot-hub-rpc + ``` -**Systemd Setup**: - -```bash -# Stop service -sudo systemctl stop polkadot-hub-rpc - -# Backup data -sudo cp -r /var/lib/polkadot-hub-rpc /var/lib/polkadot-hub-rpc.backup +The node handles pruning automatically based on configuration unless running in archive mode. -# Pull new image and extract binary -docker pull parity/polkadot-parachain: -docker create --name temp-parachain parity/polkadot-parachain: -sudo docker cp temp-parachain:/usr/local/bin/polkadot-parachain /usr/local/bin/ -docker rm temp-parachain +### Updates and Upgrades -# Restart service -sudo systemctl start polkadot-hub-rpc -``` +Use the following commands for updating or upgrading your RPC node according to your setup: + +=== "Docker Setup" + + 1. Stop and remove the existing container: + ```bash + docker stop polkadot-hub-rpc + docker rm polkadot-hub-rpc + ``` + 2. Pull the latest image: + ```bash + docker pull parity/polkadot-parachain: + ``` + 3. Start the new container using the same command from the setup section with the updated image tag. + +=== "systemd Setup" + + 1. Stop the service: + ```bash + sudo systemctl stop polkadot-hub-rpc + ``` + 2. Backup data: + ```bash + sudo cp -r /var/lib/polkadot-hub-rpc /var/lib/polkadot-hub-rpc.backup + ``` + 3. Pull the new image and extract the binary: + ```bash + docker pull parity/polkadot-parachain: + docker create --name temp-parachain parity/polkadot-parachain: + sudo docker cp temp-parachain:/usr/local/bin/polkadot-parachain /usr/local/bin/ + docker rm temp-parachain + ``` + 4. Restart the service: + ```bash + sudo systemctl start polkadot-hub-rpc + ``` ## Conclusion Running an RPC node for Polkadot Hub provides essential infrastructure for applications and users to interact with the network. By following this guide, you have set up a production-ready RPC node that: -- Provides reliable access to Polkadot Hub's asset management, governance, and cross-chain communication features -- Supports both Docker and systemd deployment options for flexibility -- Implements proper monitoring, security, and maintenance practices -- Serves as a foundation for building and operating Polkadot SDK applications +- Provides reliable access to Polkadot Hub's asset management, governance, and cross-chain communication features. +- Supports both Docker and systemd deployment options for flexibility. +- Implements proper monitoring, security, and maintenance practices. +- Serves as a foundation for building and operating Polkadot SDK applications. Regular maintenance, security updates, and monitoring will ensure your RPC node continues to serve your users reliably. As the Polkadot network evolves, stay informed about updates and best practices through the official channels and community resources listed in this guide. From fda3e1ce0b607fbc9a448d08f922449c684bf7ba Mon Sep 17 00:00:00 2001 From: Bruno Galvao Date: Mon, 24 Nov 2025 16:51:55 +0700 Subject: [PATCH 14/21] Download polkadot-parachain binary from GitHub releases instead of Docker --- .../run-a-collator/collator.md | 23 ++++++------ .../run-a-node/parachain-rpc.md | 36 ++++++++----------- .../run-a-node/polkadot-hub-rpc.md | 24 ++++++------- 3 files changed, 39 insertions(+), 44 deletions(-) diff --git a/node-infrastructure/run-a-collator/collator.md b/node-infrastructure/run-a-collator/collator.md index 745d37dac..31e6c60bc 100644 --- a/node-infrastructure/run-a-collator/collator.md +++ b/node-infrastructure/run-a-collator/collator.md @@ -96,18 +96,22 @@ This guide provides two deployment options. Select the option that best fits you === "Manual Setup" - Extract the binary from the official Docker image: + Download the `polkadot-parachain` binary from the latest stable [Polkadot SDK release](https://github.com/paritytech/polkadot-sdk/releases){target=\_blank}: ```bash - # Create a temporary container and copy the binary - docker create --name temp-parachain parity/polkadot-parachain:stable2509-2 - sudo docker cp temp-parachain:/usr/local/bin/polkadot-parachain /usr/local/bin/ - docker rm temp-parachain + # Download the latest stable release (check releases page for current version) + wget https://github.com/paritytech/polkadot-sdk/releases/download/stable2409-2/polkadot-parachain + + # Make it executable and move to system path + chmod +x polkadot-parachain + sudo mv polkadot-parachain /usr/local/bin/ # Verify installation polkadot-parachain --version ``` + Check the [Polkadot SDK releases](https://github.com/paritytech/polkadot-sdk/releases){target=\_blank} page for the latest stable version. + ## Generate Node Key Generating a stable node key enables a consistent peer ID across the network. Follow these steps to generate a node key: @@ -489,12 +493,11 @@ Updates or upgrades can happen on either the runtime or client. Runtime upgrades sudo cp -r /var/lib/polkadot-collator /var/lib/polkadot-collator.backup ``` - 3. Pull the new image and extract the binary: + 3. Download the new binary from [GitHub releases](https://github.com/paritytech/polkadot-sdk/releases){target=\_blank}: ```bash - docker pull parity/polkadot-parachain: - docker create --name temp-parachain parity/polkadot-parachain: - sudo docker cp temp-parachain:/usr/local/bin/polkadot-parachain /usr/local/bin/ - docker rm temp-parachain + wget https://github.com/paritytech/polkadot-sdk/releases/download//polkadot-parachain + chmod +x polkadot-parachain + sudo mv polkadot-parachain /usr/local/bin/ ``` 4. Verify `polkadot-parachain` version to confirm successful update: diff --git a/node-infrastructure/run-a-node/parachain-rpc.md b/node-infrastructure/run-a-node/parachain-rpc.md index b5a0fec82..3feb1c941 100644 --- a/node-infrastructure/run-a-node/parachain-rpc.md +++ b/node-infrastructure/run-a-node/parachain-rpc.md @@ -8,11 +8,7 @@ categories: Infrastructure ## Overview -Running an RPC node for a parachain enables applications, wallets, and users to interact with the parachain's functionality. This guide applies to **any parachain** in the Polkadot ecosystem, including: - -- **System parachains**: Bridge Hub, People Chain, Coretime Chain -- **Common good parachains**: Collectives, Encointer -- **Commercial parachains**: Any parachain with a publicly available chain specification +Running an RPC node for a parachain enables applications, wallets, and users to interact with the parachain's functionality. Each parachain RPC node provides access through the Polkadot SDK Node RPC (Port 9944), offering native Polkadot API access via WebSocket and HTTP. This setup enables block explorer indexing and provides full compatibility with Polkadot SDK development tools. @@ -47,11 +43,7 @@ curl -L https://raw.githubusercontent.com/paritytech/polkadot-sdk/master/cumulus ### Other Parachains -For non-system parachains: - -- **Check the parachain's documentation** for official chain specification files -- **Contact the parachain team** if no public chain spec is available -- **Export from a running node** using `system_chainSpec` RPC method if you have access to an existing node +For non-system parachains, check the parachain's documentation for official chain specification files. **Note**: Throughout this guide, we use **People Chain** as the example. To set up a different parachain, substitute the chain spec file, snapshot path, and chain name with values for your target parachain. @@ -352,20 +344,21 @@ This option provides more control and is recommended for production environments ### Step 1: Install the Polkadot Parachain Binary -Extract the binary from the official Docker image: +Download the `polkadot-parachain` binary from the latest stable [Polkadot SDK release](https://github.com/paritytech/polkadot-sdk/releases){target=\_blank}: ```bash -# Pull the image and extract binary -docker pull parity/polkadot-parachain:stable2509-2 -docker create --name temp-parachain parity/polkadot-parachain:stable2509-2 -sudo docker cp temp-parachain:/usr/local/bin/polkadot-parachain /usr/local/bin/ -docker rm temp-parachain +# Download the latest stable release (check releases page for current version) +wget https://github.com/paritytech/polkadot-sdk/releases/download/stable2409-2/polkadot-parachain + +# Make it executable and move to system path +chmod +x polkadot-parachain +sudo mv polkadot-parachain /usr/local/bin/ # Verify installation polkadot-parachain --version ``` -Check [Docker Hub](https://hub.docker.com/r/parity/polkadot-parachain/tags) for the latest stable tags. +Check the [Polkadot SDK releases](https://github.com/paritytech/polkadot-sdk/releases){target=\_blank} page for the latest stable version. ### Step 2: Create User and Directory Structure @@ -550,11 +543,10 @@ sudo systemctl stop people-chain-rpc # Backup data sudo cp -r /var/lib/people-chain-rpc /var/lib/people-chain-rpc.backup -# Pull new image and extract binary -docker pull parity/polkadot-parachain: -docker create --name temp-parachain parity/polkadot-parachain: -sudo docker cp temp-parachain:/usr/local/bin/polkadot-parachain /usr/local/bin/ -docker rm temp-parachain +# Download new binary from GitHub releases +wget https://github.com/paritytech/polkadot-sdk/releases/download//polkadot-parachain +chmod +x polkadot-parachain +sudo mv polkadot-parachain /usr/local/bin/ # Restart service sudo systemctl start people-chain-rpc diff --git a/node-infrastructure/run-a-node/polkadot-hub-rpc.md b/node-infrastructure/run-a-node/polkadot-hub-rpc.md index e33f65eca..ad4a6656f 100644 --- a/node-infrastructure/run-a-node/polkadot-hub-rpc.md +++ b/node-infrastructure/run-a-node/polkadot-hub-rpc.md @@ -248,19 +248,20 @@ Select the best option for your project, then use the steps in the following tab This option provides more control and is recommended for production environments requiring custom configurations. - 1. Install the Polkadot Parachain binary by extracting it from the official Docker image: + 1. Download the `polkadot-parachain` binary from the latest stable [Polkadot SDK release](https://github.com/paritytech/polkadot-sdk/releases){target=\_blank}: ```bash - # Pull the image and extract binary - docker pull parity/polkadot-parachain:stable2509-2 - docker create --name temp-parachain parity/polkadot-parachain:stable2509-2 - sudo docker cp temp-parachain:/usr/local/bin/polkadot-parachain /usr/local/bin/ - docker rm temp-parachain + # Download the latest stable release (check releases page for current version) + wget https://github.com/paritytech/polkadot-sdk/releases/download/stable2409-2/polkadot-parachain + + # Make it executable and move to system path + chmod +x polkadot-parachain + sudo mv polkadot-parachain /usr/local/bin/ # Verify installation polkadot-parachain --version ``` - Check [Docker Hub](https://hub.docker.com/r/parity/polkadot-parachain/tags){target=\_blank} for the latest stable tags. + Check the [Polkadot SDK releases](https://github.com/paritytech/polkadot-sdk/releases){target=\_blank} page for the latest stable version. 2. Download the Polkadot Hub chain specification: ```bash @@ -468,12 +469,11 @@ Use the following commands for updating or upgrading your RPC node according to ```bash sudo cp -r /var/lib/polkadot-hub-rpc /var/lib/polkadot-hub-rpc.backup ``` - 3. Pull the new image and extract the binary: + 3. Download the new binary from [GitHub releases](https://github.com/paritytech/polkadot-sdk/releases){target=\_blank}: ```bash - docker pull parity/polkadot-parachain: - docker create --name temp-parachain parity/polkadot-parachain: - sudo docker cp temp-parachain:/usr/local/bin/polkadot-parachain /usr/local/bin/ - docker rm temp-parachain + wget https://github.com/paritytech/polkadot-sdk/releases/download//polkadot-parachain + chmod +x polkadot-parachain + sudo mv polkadot-parachain /usr/local/bin/ ``` 4. Restart the service: ```bash From e5175947d98a207e0f495a19daa0cf8627061d5d Mon Sep 17 00:00:00 2001 From: Bruno Galvao Date: Mon, 24 Nov 2025 16:57:09 +0700 Subject: [PATCH 15/21] Use MkDocs admonition syntax for notes and warnings --- .../run-a-collator/collator.md | 4 +++- .../run-a-node/parachain-rpc.md | 24 ++++++++++++++----- 2 files changed, 21 insertions(+), 7 deletions(-) diff --git a/node-infrastructure/run-a-collator/collator.md b/node-infrastructure/run-a-collator/collator.md index 31e6c60bc..6414709ac 100644 --- a/node-infrastructure/run-a-collator/collator.md +++ b/node-infrastructure/run-a-collator/collator.md @@ -308,7 +308,9 @@ Sync time depends on: The relay chain uses warp sync for faster synchronization. -**Important**: Do not proceed with registration until both chains are fully synced. Monitor sync progress using the log viewing commands in the [Log Management](#log-management) section. +!!! warning + + Do not proceed with registration until both chains are fully synced. Monitor sync progress using the log viewing commands in the [Log Management](#log-management) section. ## Generate Session Keys diff --git a/node-infrastructure/run-a-node/parachain-rpc.md b/node-infrastructure/run-a-node/parachain-rpc.md index 3feb1c941..960a22535 100644 --- a/node-infrastructure/run-a-node/parachain-rpc.md +++ b/node-infrastructure/run-a-node/parachain-rpc.md @@ -45,7 +45,9 @@ curl -L https://raw.githubusercontent.com/paritytech/polkadot-sdk/master/cumulus For non-system parachains, check the parachain's documentation for official chain specification files. -**Note**: Throughout this guide, we use **People Chain** as the example. To set up a different parachain, substitute the chain spec file, snapshot path, and chain name with values for your target parachain. +!!! note + + Throughout this guide, we use **People Chain** as the example. To set up a different parachain, substitute the chain spec file, snapshot path, and chain name with values for your target parachain. ## Prerequisites @@ -71,7 +73,9 @@ RPC nodes serving production traffic require robust hardware: - 9615 (Prometheus metrics - optional) - Consider DDoS protection and rate limiting for production deployments -**Note**: For development or low-traffic scenarios, you can reduce these requirements proportionally. Consider using a reverse proxy (nginx, Caddy) for production deployments. +!!! note + + For development or low-traffic scenarios, you can reduce these requirements proportionally. Consider using a reverse proxy (nginx, Caddy) for production deployments. ### Software Requirements @@ -102,7 +106,9 @@ Using pre-synchronized snapshots significantly reduces initial sync time from se **Snapshot Provider**: https://snapshots.polkadot.io/ -**Note**: Snapshots are available for system parachains and the Polkadot relay chain. For other parachains, check with the parachain team for snapshot availability or sync from genesis. +!!! note + + Snapshots are available for system parachains and the Polkadot relay chain. For other parachains, check with the parachain team for snapshot availability or sync from genesis. #### Create Directories @@ -139,7 +145,9 @@ rm files.txt - `--retries-sleep 10s`: Waits 10 seconds between retry attempts - `--size-only`: Only transfers if sizes differ (prevents unnecessary re-downloads) -**Note**: If a snapshot is not available, you can sync from scratch (which will take longer) or check https://snapshots.polkadot.io/ for alternative snapshot providers. +!!! note + + If a snapshot is not available, you can sync from scratch (which will take longer) or check https://snapshots.polkadot.io/ for alternative snapshot providers. #### Download Polkadot Relay Chain Snapshot @@ -171,7 +179,9 @@ Launch the node using the official Parity Docker image. **Docker Image**: https://hub.docker.com/r/parity/polkadot-parachain -**Note**: The `parity/polkadot-parachain` image works for system parachains and parachains built with standard Cumulus templates. For parachains with custom runtimes, check the parachain's documentation for their specific Docker image or binary. +!!! note + + The `parity/polkadot-parachain` image works for system parachains and parachains built with standard Cumulus templates. For parachains with custom runtimes, check the parachain's documentation for their specific Docker image or binary. ```bash docker run -d --name people-chain-rpc --restart unless-stopped \ @@ -334,7 +344,9 @@ docker start people-chain-rpc docker rm people-chain-rpc ``` -**Note**: For update procedures, see the [Updates and Upgrades](#updates-and-upgrades) section. +!!! note + + For update procedures, see the [Updates and Upgrades](#updates-and-upgrades) section. --- From 1c3288fdb24bb606ea143a91bb1f1c01c4d3bb4f Mon Sep 17 00:00:00 2001 From: Bruno Galvao Date: Mon, 24 Nov 2025 16:59:35 +0700 Subject: [PATCH 16/21] Fix GitHub release tag to polkadot-stable2509-2 --- node-infrastructure/run-a-collator/collator.md | 2 +- node-infrastructure/run-a-node/parachain-rpc.md | 2 +- node-infrastructure/run-a-node/polkadot-hub-rpc.md | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/node-infrastructure/run-a-collator/collator.md b/node-infrastructure/run-a-collator/collator.md index 6414709ac..e4f645733 100644 --- a/node-infrastructure/run-a-collator/collator.md +++ b/node-infrastructure/run-a-collator/collator.md @@ -100,7 +100,7 @@ This guide provides two deployment options. Select the option that best fits you ```bash # Download the latest stable release (check releases page for current version) - wget https://github.com/paritytech/polkadot-sdk/releases/download/stable2409-2/polkadot-parachain + wget https://github.com/paritytech/polkadot-sdk/releases/download/polkadot-stable2509-2/polkadot-parachain # Make it executable and move to system path chmod +x polkadot-parachain diff --git a/node-infrastructure/run-a-node/parachain-rpc.md b/node-infrastructure/run-a-node/parachain-rpc.md index 960a22535..ba58c5a7c 100644 --- a/node-infrastructure/run-a-node/parachain-rpc.md +++ b/node-infrastructure/run-a-node/parachain-rpc.md @@ -360,7 +360,7 @@ Download the `polkadot-parachain` binary from the latest stable [Polkadot SDK re ```bash # Download the latest stable release (check releases page for current version) -wget https://github.com/paritytech/polkadot-sdk/releases/download/stable2409-2/polkadot-parachain +wget https://github.com/paritytech/polkadot-sdk/releases/download/polkadot-stable2509-2/polkadot-parachain # Make it executable and move to system path chmod +x polkadot-parachain diff --git a/node-infrastructure/run-a-node/polkadot-hub-rpc.md b/node-infrastructure/run-a-node/polkadot-hub-rpc.md index ad4a6656f..8104bff42 100644 --- a/node-infrastructure/run-a-node/polkadot-hub-rpc.md +++ b/node-infrastructure/run-a-node/polkadot-hub-rpc.md @@ -251,7 +251,7 @@ Select the best option for your project, then use the steps in the following tab 1. Download the `polkadot-parachain` binary from the latest stable [Polkadot SDK release](https://github.com/paritytech/polkadot-sdk/releases){target=\_blank}: ```bash # Download the latest stable release (check releases page for current version) - wget https://github.com/paritytech/polkadot-sdk/releases/download/stable2409-2/polkadot-parachain + wget https://github.com/paritytech/polkadot-sdk/releases/download/polkadot-stable2509-2/polkadot-parachain # Make it executable and move to system path chmod +x polkadot-parachain From 18599158d02a1f0442cbe9f94037e85180a8bae2 Mon Sep 17 00:00:00 2001 From: Bruno Galvao Date: Mon, 24 Nov 2025 17:05:36 +0700 Subject: [PATCH 17/21] Restructure parachain-rpc.md to use tabbed content layout --- .../run-a-node/parachain-rpc.md | 878 ++++++++---------- 1 file changed, 412 insertions(+), 466 deletions(-) diff --git a/node-infrastructure/run-a-node/parachain-rpc.md b/node-infrastructure/run-a-node/parachain-rpc.md index ba58c5a7c..e857f6ec6 100644 --- a/node-infrastructure/run-a-node/parachain-rpc.md +++ b/node-infrastructure/run-a-node/parachain-rpc.md @@ -12,7 +12,9 @@ Running an RPC node for a parachain enables applications, wallets, and users to Each parachain RPC node provides access through the Polkadot SDK Node RPC (Port 9944), offering native Polkadot API access via WebSocket and HTTP. This setup enables block explorer indexing and provides full compatibility with Polkadot SDK development tools. -**Important Note**: The parameters and configurations in this guide are provided as illustrative examples. You may need to modify them according to your specific environment, hardware capabilities, and network conditions. +!!! note + + The parameters and configurations in this guide are provided as illustrative examples. You may need to modify them according to your specific environment, hardware capabilities, and network conditions. ## Obtaining a Chain Specification @@ -87,430 +89,365 @@ Required software: ## Setup Options -This guide provides two deployment options: - -1. **Docker-based Setup**: Simpler to set up and maintain -2. **Manual/Systemd Setup**: For production environments requiring more control - -Choose the option that best fits your needs. - ---- - -## Option 1: Docker-Based Setup - -This option uses Docker containers for the Polkadot SDK node, making it easy to set up and manage. - -### Step 1: Download Database Snapshots (Optional but Recommended) - -Using pre-synchronized snapshots significantly reduces initial sync time from several days to just a few hours. You need to download both parachain and relay chain data. - -**Snapshot Provider**: https://snapshots.polkadot.io/ - -!!! note - - Snapshots are available for system parachains and the Polkadot relay chain. For other parachains, check with the parachain team for snapshot availability or sync from genesis. - -#### Create Directories - -```bash -mkdir -p my-node-data/chains/people-polkadot/db -mkdir -p my-node-data/chains/polkadot/db -``` - -#### Download People Chain Snapshot - -Choose between archive (complete history) or pruned (recent state) snapshots. - -**Archive Snapshot** (recommended for RPC with historical data): - -```bash -# Check https://snapshots.polkadot.io/ for the latest snapshot URL -export SNAPSHOT_URL_PARACHAIN="https://snapshots.polkadot.io/polkadot-people-rocksdb-archive/LATEST" - -rclone copyurl $SNAPSHOT_URL_PARACHAIN/files.txt files.txt -rclone copy --progress --transfers 20 \ - --http-url $SNAPSHOT_URL_PARACHAIN \ - --no-traverse --http-no-head --disable-http2 \ - --inplace --no-gzip-encoding --size-only \ - --retries 6 --retries-sleep 10s \ - --files-from files.txt :http: my-node-data/chains/people-polkadot/db/ - -rm files.txt -``` - -**Parameter Explanation**: - -- `--transfers 20`: Uses 20 parallel transfers for faster download -- `--retries 6`: Automatically retries failed transfers up to 6 times -- `--retries-sleep 10s`: Waits 10 seconds between retry attempts -- `--size-only`: Only transfers if sizes differ (prevents unnecessary re-downloads) - -!!! note - - If a snapshot is not available, you can sync from scratch (which will take longer) or check https://snapshots.polkadot.io/ for alternative snapshot providers. - -#### Download Polkadot Relay Chain Snapshot - -**Pruned Snapshot** (recommended for RPC nodes): - -```bash -# Check https://snapshots.polkadot.io/ for the latest snapshot URL -export SNAPSHOT_URL_RELAY="https://snapshots.polkadot.io/polkadot-rocksdb-prune/LATEST" - -rclone copyurl $SNAPSHOT_URL_RELAY/files.txt files.txt -rclone copy --progress --transfers 20 \ - --http-url $SNAPSHOT_URL_RELAY \ - --no-traverse --http-no-head --disable-http2 \ - --inplace --no-gzip-encoding --size-only \ - --retries 6 --retries-sleep 10s \ - --files-from files.txt :http: my-node-data/chains/polkadot/db/ - -rm files.txt -``` - -**Alternative Options**: - -- Pruned snapshot: `polkadot-rocksdb-prune` (smaller size, recent state) -- Archive snapshot: `polkadot-rocksdb-archive` (complete history, larger size) - -### Step 2: Start the Parachain Node - -Launch the node using the official Parity Docker image. - -**Docker Image**: https://hub.docker.com/r/parity/polkadot-parachain - -!!! note - - The `parity/polkadot-parachain` image works for system parachains and parachains built with standard Cumulus templates. For parachains with custom runtimes, check the parachain's documentation for their specific Docker image or binary. - -```bash -docker run -d --name people-chain-rpc --restart unless-stopped \ - -p 9944:9944 \ - -p 9933:9933 \ - -p 9615:9615 \ - -p 30334:30334 \ - -p 30333:30333 \ - -v $(pwd)/people-polkadot.json:/people-polkadot.json \ - -v $(pwd)/my-node-data:/data \ - parity/polkadot-parachain:stable2509-2 \ - --name=PeopleChainRPC \ - --base-path=/data \ - --chain=/people-polkadot.json \ - --prometheus-external \ - --prometheus-port 9615 \ - --unsafe-rpc-external \ - --rpc-port=9944 \ - --rpc-cors=all \ - --rpc-methods=safe \ - --rpc-max-connections=1000 \ - --state-pruning=archive \ - --blocks-pruning=archive \ - -- \ - --base-path=/data \ - --chain=polkadot \ - --state-pruning=256 \ - --blocks-pruning=archive-canonical -``` - -**Critical Configuration Parameters**: - -**Port Mappings**: - -- `9944`: Polkadot SDK RPC endpoint (WebSocket/HTTP) -- `9933`: Polkadot SDK HTTP RPC endpoint -- `9615`: Prometheus metrics endpoint -- `30333/30334`: P2P networking ports - -**Node Parameters**: - -- `--unsafe-rpc-external`: Enables external RPC access -- `--rpc-cors=all`: Allows all origins for CORS -- `--rpc-methods=safe`: Only allows safe RPC methods -- `--state-pruning=archive`: Keeps complete state history -- `--blocks-pruning=archive`: Keeps all block data -- `--prometheus-external`: Exposes metrics externally - -**Security Warning**: The `--unsafe-rpc-external` flag should only be used in development or properly secured environments. For production, use a reverse proxy with authentication. - -### Step 3: Monitor Synchronization - -Monitor the node synchronization status: - -```bash -# Check sync status -curl -H "Content-Type: application/json" \ - -d '{"id":1, "jsonrpc":"2.0", "method": "system_syncState", "params":[]}' \ - http://localhost:9944 -``` - -**Expected Response Format**: - -```json -{ - "jsonrpc":"2.0", - "id":1, - "result":{ - "startingBlock":0, - "currentBlock":3394816, - "highestBlock":3394816 - } -} -``` - -**Synchronization Status**: - -- **In Progress**: `currentBlock` < `highestBlock` -- **Complete**: `currentBlock` = `highestBlock` - -### Step 4: Verify Setup - -Let's verify the Polkadot SDK RPC endpoint is working correctly. - -#### API Endpoint - -**Polkadot SDK RPC (Port 9944)**: - -- WebSocket: `ws://your-server:9944` -- HTTP: `http://your-server:9944` -- Purpose: Full Polkadot SDK API access for parachain data -- Use Cases: Polkadot SDK applications, parachain-specific operations - -#### Polkadot SDK RPC Tests - -**Get Chain Information**: - -```bash -curl -H "Content-Type: application/json" \ - -d '{"id":1, "jsonrpc":"2.0", "method": "system_chain", "params":[]}' \ - http://localhost:9944 -``` - -**Expected Response**: -```json -{ - "jsonrpc":"2.0", - "id":1, - "result":"Polkadot People" -} -``` - -**Get Latest Block**: - -```bash -curl -H "Content-Type: application/json" \ - -d '{"id":1, "jsonrpc":"2.0", "method": "chain_getHeader", "params":[]}' \ - http://localhost:9944 -``` - -**Get Node Health**: - -```bash -curl -H "Content-Type: application/json" \ - -d '{"id":1, "jsonrpc":"2.0", "method": "system_health", "params":[]}' \ - http://localhost:9944 -``` - -**Get Peer Count**: - -```bash -curl -H "Content-Type: application/json" \ - -d '{"id":1, "jsonrpc":"2.0", "method": "system_peers", "params":[]}' \ - http://localhost:9944 -``` - -### Managing Docker Containers - -**View logs**: - -```bash -docker logs -f people-chain-rpc -``` - -**Stop container**: - -```bash -docker stop people-chain-rpc -``` - -**Start container**: - -```bash -docker start people-chain-rpc -``` - -**Remove container**: - -```bash -docker rm people-chain-rpc -``` - -!!! note - - For update procedures, see the [Updates and Upgrades](#updates-and-upgrades) section. - ---- - -## Option 2: Manual/Systemd Setup - -This option provides more control and is recommended for production environments requiring custom configurations. - -### Step 1: Install the Polkadot Parachain Binary - -Download the `polkadot-parachain` binary from the latest stable [Polkadot SDK release](https://github.com/paritytech/polkadot-sdk/releases){target=\_blank}: - -```bash -# Download the latest stable release (check releases page for current version) -wget https://github.com/paritytech/polkadot-sdk/releases/download/polkadot-stable2509-2/polkadot-parachain - -# Make it executable and move to system path -chmod +x polkadot-parachain -sudo mv polkadot-parachain /usr/local/bin/ - -# Verify installation -polkadot-parachain --version -``` - -Check the [Polkadot SDK releases](https://github.com/paritytech/polkadot-sdk/releases){target=\_blank} page for the latest stable version. - -### Step 2: Create User and Directory Structure - -Ensure you have downloaded your parachain's chain specification as described in [Obtaining a Chain Specification](#obtaining-a-chain-specification). - -```bash -# Create dedicated user (skip if already exists) -sudo useradd -r -s /bin/bash polkadot - -# Create data directory -sudo mkdir -p /var/lib/people-chain-rpc - -# Copy chain spec to the directory -sudo cp people-polkadot.json /var/lib/people-chain-rpc/ - -# Set permissions -sudo chown -R polkadot:polkadot /var/lib/people-chain-rpc -``` - -### Step 3: Create Systemd Service - -Create a service file for your parachain RPC node: - -```bash -sudo nano /etc/systemd/system/people-chain-rpc.service -``` - -Add the following configuration: - -```ini -[Unit] -Description=People Chain RPC Node -After=network.target - -[Service] -Type=simple -User=polkadot -Group=polkadot -WorkingDirectory=/var/lib/people-chain-rpc - -ExecStart=/usr/local/bin/polkadot-parachain \ - --name=PeopleChainRPC \ - --chain=/var/lib/people-chain-rpc/people-polkadot.json \ - --base-path=/var/lib/people-chain-rpc \ - --port=30333 \ - --rpc-port=9944 \ - --rpc-external \ - --rpc-cors=all \ - --rpc-methods=safe \ - --rpc-max-connections=1000 \ - --prometheus-port=9615 \ - --prometheus-external \ - --state-pruning=archive \ - --blocks-pruning=archive \ - -- \ - --chain=polkadot \ - --base-path=/var/lib/people-chain-rpc \ - --port=30334 \ - --state-pruning=256 \ - --blocks-pruning=archive-canonical - -Restart=always -RestartSec=10 -LimitNOFILE=65536 - -[Install] -WantedBy=multi-user.target -``` - -### Step 4: Start Service - -```bash -# Reload systemd -sudo systemctl daemon-reload - -# Enable service to start on boot -sudo systemctl enable people-chain-rpc - -# Start the node -sudo systemctl start people-chain-rpc - -# Check status and wait for sync -sudo systemctl status people-chain-rpc -sudo journalctl -u people-chain-rpc -f -``` - -### Step 5: Verify Setup - -Use the same verification tests as in the Docker setup (see Step 4 above). - ---- - -## Monitoring and Maintenance - -### Log Management - -**Docker Setup**: - -```bash -# View node logs -docker logs -f people-chain-rpc -``` - -**Systemd Setup**: - -```bash -# View node logs -sudo journalctl -u people-chain-rpc -f - -# View recent logs -sudo journalctl -u people-chain-rpc -n 100 - -# Filter for errors -sudo journalctl -u people-chain-rpc | grep -i error -``` - -### Performance Monitoring - -Monitor key metrics: - -- **Sync status**: Ensure node stays fully synced -- **Peer connections**: Maintain 30+ peers for good connectivity -- **Resource usage**: Monitor CPU, RAM, and disk I/O -- **RPC request latency**: Track response times for the Polkadot SDK API -- **Connection count**: Monitor active RPC connections - -**Prometheus Metrics**: - -Metrics are available at `http://localhost:9615/metrics` - -Example Prometheus scrape configuration: - -```yaml -scrape_configs: - - job_name: 'people-chain-rpc' - static_configs: - - targets: ['localhost:9615'] -``` - -**Key Metrics to Monitor**: +This guide provides two options for deployment: + +- **Docker-based Setup**: Best for simpler set up and maintenance +- **Manual/systemd Setup**: Best for production environments requiring more control + +Select the best option for your project, then use the steps in the following tabs to complete set up. + +=== "Docker-Based Setup" + + This option uses Docker containers for the Polkadot SDK node, making it easy to set up and manage. Follow these steps to set your RPC node using Docker: + + 1. Download your parachain's chain specification as described in [Obtaining a Chain Specification](#obtaining-a-chain-specification). + + 2. (Optional but recommended) Download database snapshots: + - Using pre-synchronized snapshots significantly reduces initial sync time from several days to just a few hours. You need to download both parachain and relay chain data. + - You can obtain the latest snapshot from the [Snapshot Provider](https://snapshots.polkadot.io/){target=\_blank}. Follow these steps to download and use snapshots: + + !!! note + + Snapshots are available for system parachains and the Polkadot relay chain. For other parachains, check with the parachain team for snapshot availability or sync from genesis. + + 1. Create new directories with the following commands: + ```bash + mkdir -p my-node-data/chains/people-polkadot/db + mkdir -p my-node-data/chains/polkadot/db + ``` + 2. Download the appropriate snapshot using the following commands: + + === "Archive snapshot" + + Contains complete history, recommended for RPC with historical data. + ```bash + # Check https://snapshots.polkadot.io/ for the latest snapshot URL + export SNAPSHOT_URL_PARACHAIN="https://snapshots.polkadot.io/polkadot-people-rocksdb-archive/LATEST" + + rclone copyurl $SNAPSHOT_URL_PARACHAIN/files.txt files.txt + rclone copy --progress --transfers 20 \ + --http-url $SNAPSHOT_URL_PARACHAIN \ + --no-traverse --http-no-head --disable-http2 \ + --inplace --no-gzip-encoding --size-only \ + --retries 6 --retries-sleep 10s \ + --files-from files.txt :http: my-node-data/chains/people-polkadot/db/ + + rm files.txt + ``` + + - `--transfers 20`: Uses 20 parallel transfers for faster download. + - `--retries 6`: Automatically retries failed transfers up to 6 times. + - `--retries-sleep 10s`: Waits 10 seconds between retry attempts. + - `--size-only`: Only transfers if sizes differ (prevents unnecessary re-downloads). + + === "Pruned snapshot" + + Contains recent state for a smaller package size, recommended for RPC nodes. + ```bash + # Check https://snapshots.polkadot.io/ for the latest snapshot URL + export SNAPSHOT_URL_RELAY="https://snapshots.polkadot.io/polkadot-rocksdb-prune/LATEST" + + rclone copyurl $SNAPSHOT_URL_RELAY/files.txt files.txt + rclone copy --progress --transfers 20 \ + --http-url $SNAPSHOT_URL_RELAY \ + --no-traverse --http-no-head --disable-http2 \ + --inplace --no-gzip-encoding --size-only \ + --retries 6 --retries-sleep 10s \ + --files-from files.txt :http: my-node-data/chains/polkadot/db/ + + rm files.txt + ``` + + 3. Launch the parachain node using the official [Parity Docker image](https://hub.docker.com/r/parity/polkadot-parachain){target=\_blank} with the following command: + ```bash + docker run -d --name people-chain-rpc --restart unless-stopped \ + -p 9944:9944 \ + -p 9933:9933 \ + -p 9615:9615 \ + -p 30334:30334 \ + -p 30333:30333 \ + -v $(pwd)/people-polkadot.json:/people-polkadot.json \ + -v $(pwd)/my-node-data:/data \ + parity/polkadot-parachain:stable2509-2 \ + --name=PeopleChainRPC \ + --base-path=/data \ + --chain=/people-polkadot.json \ + --prometheus-external \ + --prometheus-port 9615 \ + --unsafe-rpc-external \ + --rpc-port=9944 \ + --rpc-cors=all \ + --rpc-methods=safe \ + --rpc-max-connections=1000 \ + --state-pruning=archive \ + --blocks-pruning=archive \ + -- \ + --base-path=/data \ + --chain=polkadot \ + --state-pruning=256 \ + --blocks-pruning=archive-canonical + ``` + + !!! note + + The `parity/polkadot-parachain` image works for system parachains and parachains built with standard Cumulus templates. For parachains with custom runtimes, check the parachain's documentation for their specific Docker image or binary. + + Critical configuration parameters include port mappings and node parameters: + + === "Port mappings" + + - `9944`: Polkadot SDK RPC endpoint (WebSocket/HTTP) + - `9933`: Polkadot SDK HTTP RPC endpoint + - `9615`: Prometheus metrics endpoint + - `30333/30334`: P2P networking ports + + === "Node parameters" + + - `--unsafe-rpc-external`: Enables external RPC access + - `--rpc-cors=all`: Allows all origins for CORS + - `--rpc-methods=safe`: Only allows safe RPC methods + - `--state-pruning=archive`: Keeps complete state history + - `--blocks-pruning=archive`: Keeps all block data + - `--prometheus-external`: Exposes metrics externally + + !!! warning + + The `--unsafe-rpc-external` flag should only be used in development or properly secured environments. For production, use a reverse proxy with authentication. + + 4. Monitor the node synchronization status using the following command: + ```bash + curl -H "Content-Type: application/json" \ + -d '{"id":1, "jsonrpc":"2.0", "method": "system_syncState", "params":[]}' \ + http://localhost:9944 + ``` + + You should see a response similar to the following: + + ```json + { + "jsonrpc":"2.0", + "id":1, + "result":{ + "startingBlock":0, + "currentBlock":3394816, + "highestBlock":3394816 + } + } + ``` + + When synchronization is complete, `currentBlock` will be equal to `highestBlock`. + + 5. You can use a few different commands to verify your node is running properly: + + - Get chain information: + ```bash + curl -H "Content-Type: application/json" \ + -d '{"id":1, "jsonrpc":"2.0", "method": "system_chain", "params":[]}' \ + http://localhost:9944 + ``` + - Get the latest block: + ```bash + curl -H "Content-Type: application/json" \ + -d '{"id":1, "jsonrpc":"2.0", "method": "chain_getHeader", "params":[]}' \ + http://localhost:9944 + ``` + - Query node health: + ```bash + curl -H "Content-Type: application/json" \ + -d '{"id":1, "jsonrpc":"2.0", "method": "system_health", "params":[]}' \ + http://localhost:9944 + ``` + + 6. Use the following commands to manage your Docker containers: + + - View node logs: + ```bash + docker logs -f people-chain-rpc + ``` + - Stop container: + ```bash + docker stop people-chain-rpc + ``` + - Start container: + ```bash + docker start people-chain-rpc + ``` + - Remove container: + ```bash + docker rm people-chain-rpc + ``` + +=== "Manual systemd Setup" + + This option provides more control and is recommended for production environments requiring custom configurations. + + 1. Download the `polkadot-parachain` binary from the latest stable [Polkadot SDK release](https://github.com/paritytech/polkadot-sdk/releases){target=\_blank}: + ```bash + # Download the latest stable release (check releases page for current version) + wget https://github.com/paritytech/polkadot-sdk/releases/download/polkadot-stable2509-2/polkadot-parachain + + # Make it executable and move to system path + chmod +x polkadot-parachain + sudo mv polkadot-parachain /usr/local/bin/ + + # Verify installation + polkadot-parachain --version + ``` + + Check the [Polkadot SDK releases](https://github.com/paritytech/polkadot-sdk/releases){target=\_blank} page for the latest stable version. + + 2. Download your parachain's chain specification as described in [Obtaining a Chain Specification](#obtaining-a-chain-specification). + + 3. Create user and directory structures using the following commands: + - Create a dedicated user: + ```bash + sudo useradd -r -s /bin/bash polkadot + ``` + - Create data directory: + ```bash + sudo mkdir -p /var/lib/people-chain-rpc + ``` + - Copy the chain spec to the directory: + ```bash + sudo cp people-polkadot.json /var/lib/people-chain-rpc/ + ``` + - Set permissions: + ```bash + sudo chown -R polkadot:polkadot /var/lib/people-chain-rpc + ``` + + 4. Create a systemd service file for the Polkadot SDK RPC node: + ```bash + sudo nano /etc/systemd/system/people-chain-rpc.service + ``` + + 5. Open the new service file and add the following configuration: + ```ini + [Unit] + Description=People Chain RPC Node + After=network.target + + [Service] + Type=simple + User=polkadot + Group=polkadot + WorkingDirectory=/var/lib/people-chain-rpc + + ExecStart=/usr/local/bin/polkadot-parachain \ + --name=PeopleChainRPC \ + --chain=/var/lib/people-chain-rpc/people-polkadot.json \ + --base-path=/var/lib/people-chain-rpc \ + --port=30333 \ + --rpc-port=9944 \ + --rpc-external \ + --rpc-cors=all \ + --rpc-methods=safe \ + --rpc-max-connections=1000 \ + --prometheus-port=9615 \ + --prometheus-external \ + --state-pruning=archive \ + --blocks-pruning=archive \ + -- \ + --chain=polkadot \ + --base-path=/var/lib/people-chain-rpc \ + --port=30334 \ + --state-pruning=256 \ + --blocks-pruning=archive-canonical + + Restart=always + RestartSec=10 + LimitNOFILE=65536 + + [Install] + WantedBy=multi-user.target + ``` + + 6. Start the service using the following commands: + - Reload systemd: + ```bash + sudo systemctl daemon-reload + ``` + - Enable service to start on boot: + ```bash + sudo systemctl enable people-chain-rpc + ``` + - Start the Polkadot SDK node: + ```bash + sudo systemctl start people-chain-rpc + ``` + - Check status and wait for sync: + ```bash + sudo systemctl status people-chain-rpc + sudo journalctl -u people-chain-rpc -f + ``` + + 7. You can use a few different commands to verify your node is running properly: + - Get chain information: + ```bash + curl -H "Content-Type: application/json" \ + -d '{"id":1, "jsonrpc":"2.0", "method": "system_chain", "params":[]}' \ + http://localhost:9944 + ``` + - Get the latest block: + ```bash + curl -H "Content-Type: application/json" \ + -d '{"id":1, "jsonrpc":"2.0", "method": "chain_getHeader", "params":[]}' \ + http://localhost:9944 + ``` + - Query node health: + ```bash + curl -H "Content-Type: application/json" \ + -d '{"id":1, "jsonrpc":"2.0", "method": "system_health", "params":[]}' \ + http://localhost:9944 + ``` + +## Monitor and Maintain RPC Node + +There are a few key monitoring and maintenance commands that can help you ensure RPC node uptime and performance including log monitoring and metrics. + +To view node logs, use the appropriate command for your setup: + +=== "Docker Setup" + + ```bash + docker logs -f people-chain-rpc + ``` + +=== "systemd Setup" + + - View node logs: + ```bash + sudo journalctl -u people-chain-rpc -f + ``` + - View recent logs: + ```bash + sudo journalctl -u people-chain-rpc -n 100 + ``` + - Filter for errors: + ```bash + sudo journalctl -u people-chain-rpc | grep -i error + ``` + +Monitoring key metrics like the following items can help you stay up to date on the performance and health of your node and allow you to intervene at the first sign of issues: + +- **Sync status**: Ensure node stays fully synced. +- **Peer connections**: Maintain 30+ peers for good connectivity. +- **Resource usage**: Monitor CPU, RAM, and disk I/O. +- **RPC request latency**: Track response times for the Polkadot SDK API. +- **Connection count**: Monitor active RPC connections. + +You can use the following information to configure [Prometheus](https://prometheus.io/docs/introduction/first_steps/){target=\_blank} to monitor, collect, and store your RPC node metrics: + +- **URL**: Metrics are available to view at `http://localhost:9615/metrics` +- **Example Prometheus configuration**: Update your `prometheus.yml` to add the following code: + ```yaml + scrape_configs: + - job_name: 'people-chain-rpc' + static_configs: + - targets: ['localhost:9615'] + ``` + +Key metrics to monitor via Prometheus include: - `substrate_block_height`: Current block height - `substrate_finalized_height`: Finalized block height @@ -519,58 +456,67 @@ scrape_configs: ### Database Maintenance -Check database size periodically: - -```bash -# Docker setup -du -sh my-node-data - -# Systemd setup -du -sh /var/lib/people-chain-rpc -``` - -The node handles pruning automatically based on configuration unless running in archive mode. - -### Updates and Upgrades +Check database size periodically using the commands for your selected setup: -**Docker Setup**: +=== "Docker Setup" -```bash -# Pull latest image -docker pull parity/polkadot-parachain: - -# Restart container -docker stop people-chain-rpc -docker rm people-chain-rpc + ```bash + du -sh my-node-data + ``` -# Start new container (use same command from setup with updated image tag) -``` +=== "systemd Setup" -**Systemd Setup**: + ```bash + du -sh /var/lib/people-chain-rpc + ``` -```bash -# Stop service -sudo systemctl stop people-chain-rpc - -# Backup data -sudo cp -r /var/lib/people-chain-rpc /var/lib/people-chain-rpc.backup +The node handles pruning automatically based on configuration unless running in archive mode. -# Download new binary from GitHub releases -wget https://github.com/paritytech/polkadot-sdk/releases/download//polkadot-parachain -chmod +x polkadot-parachain -sudo mv polkadot-parachain /usr/local/bin/ +### Updates and Upgrades -# Restart service -sudo systemctl start people-chain-rpc -``` +Use the following commands for updating or upgrading your RPC node according to your setup: + +=== "Docker Setup" + + 1. Stop and remove the existing container: + ```bash + docker stop people-chain-rpc + docker rm people-chain-rpc + ``` + 2. Pull the latest image: + ```bash + docker pull parity/polkadot-parachain: + ``` + 3. Start the new container using the same command from the setup section with the updated image tag. + +=== "systemd Setup" + + 1. Stop the service: + ```bash + sudo systemctl stop people-chain-rpc + ``` + 2. Backup data: + ```bash + sudo cp -r /var/lib/people-chain-rpc /var/lib/people-chain-rpc.backup + ``` + 3. Download the new binary from [GitHub releases](https://github.com/paritytech/polkadot-sdk/releases){target=\_blank}: + ```bash + wget https://github.com/paritytech/polkadot-sdk/releases/download//polkadot-parachain + chmod +x polkadot-parachain + sudo mv polkadot-parachain /usr/local/bin/ + ``` + 4. Restart the service: + ```bash + sudo systemctl start people-chain-rpc + ``` ## Conclusion Running a parachain RPC node provides critical infrastructure for accessing Polkadot network services. By following this guide, you have set up a production-ready RPC node that: -- Provides reliable access to parachain functionality for applications and users -- Supports flexible deployment with both Docker and systemd options -- Implements comprehensive monitoring, security, and maintenance practices -- Can be adapted for any parachain by substituting the appropriate chain specification +- Provides reliable access to parachain functionality for applications and users. +- Supports flexible deployment with both Docker and systemd options. +- Implements comprehensive monitoring, security, and maintenance practices. +- Can be adapted for any parachain by substituting the appropriate chain specification. Whether you're running a node for system parachains (People Chain, Bridge Hub, Coretime Chain) or other parachains in the ecosystem, regular maintenance and monitoring will ensure your RPC node continues to provide reliable service. Stay updated with the latest releases and best practices to keep your infrastructure secure and performant. From f645cca760feed0517ddd83f1beb09295571485f Mon Sep 17 00:00:00 2001 From: Bruno Galvao Date: Mon, 24 Nov 2025 17:08:28 +0700 Subject: [PATCH 18/21] Add Docker commands to collator.md monitoring and maintenance sections --- .../run-a-collator/collator.md | 90 ++++++++++++++----- 1 file changed, 66 insertions(+), 24 deletions(-) diff --git a/node-infrastructure/run-a-collator/collator.md b/node-infrastructure/run-a-collator/collator.md index e4f645733..2560584d1 100644 --- a/node-infrastructure/run-a-collator/collator.md +++ b/node-infrastructure/run-a-collator/collator.md @@ -388,10 +388,20 @@ The registration process varies by system parachain. General steps include the f Monitoring the following items will help ensure your collator runs efficiently to avoid service interruptions or down time: -- **Block Production**: monitor for block production using the following command: - ```bash - sudo journalctl -u polkadot-collator | grep -i "prepared block" - ``` +- **Block Production**: Monitor for block production using the appropriate command for your setup: + + === "Docker Setup" + + ```bash + docker logs polkadot-collator 2>&1 | grep -i "prepared block" + ``` + + === "systemd Setup" + + ```bash + sudo journalctl -u polkadot-collator | grep -i "prepared block" + ``` + - **Peer Connections**: Maintain a sufficient amount of peers for good connectivity and set up alerts to notify if peer connections falls below ten. - **Resource Usage**: Monitor CPU, RAM, and disk I/O and set up alerts for unusual or high usage. - **Sync Status**: Ensure both chains stay synced and set up alerts for sync issues. @@ -419,32 +429,64 @@ Key metrics to monitor via Prometheus include the following: ## Log Management -Effecient log management is essential to ensure collator performance and uptime. Use the following commands to help you manage logs to monitor and maintain your collator: +Efficient log management is essential to ensure collator performance and uptime. Use the following commands to help you manage logs to monitor and maintain your collator: + +=== "Docker Setup" + + - View logs: + ```bash + docker logs -f polkadot-collator + ``` + - View recent logs (last 100 lines): + ```bash + docker logs --tail 100 polkadot-collator + ``` + - Filter for errors: + ```bash + docker logs polkadot-collator 2>&1 | grep -i error + ``` + - Filter for block production: + ```bash + docker logs polkadot-collator 2>&1 | grep -i "imported" + ``` + +=== "systemd Setup" + + - View recent logs: + ```bash + sudo journalctl -u polkadot-collator -n 100 + ``` + - Follow logs in real-time: + ```bash + sudo journalctl -u polkadot-collator -f + ``` + - Filter for errors: + ```bash + sudo journalctl -u polkadot-collator | grep -i error + ``` + - Filter for block production: + ```bash + sudo journalctl -u polkadot-collator | grep -i "imported" + ``` + +## Database Maintenance + +Check database size periodically using the commands for your selected setup: + +=== "Docker Setup" -- View recent logs: - ```bash - sudo journalctl -u polkadot-collator -n 100 - ``` -- Follow logs in real-time: - ```bash - sudo journalctl -u polkadot-collator -f - ``` -- Filter for errors: ```bash - sudo journalctl -u polkadot-collator | grep -i error + # Replace with your mounted data directory path + du -sh ./collator-data ``` -- Filter for block production: + +=== "systemd Setup" + ```bash - sudo journalctl -u polkadot-collator | grep -i "imported" + du -sh /var/lib/polkadot-collator ``` -## Database Maintenance - -You can check database size using the following command: -```bash -du -sh /var/lib/polkadot-collator -``` -The collator node handles pruning automatically. +The collator node handles pruning automatically based on configuration. ## Updates and Upgrades From 1a74bf48cc04492ef462510ce139979bdc5243ee Mon Sep 17 00:00:00 2001 From: Bruno Galvao Date: Mon, 24 Nov 2025 17:09:49 +0700 Subject: [PATCH 19/21] Update content --- ...-infrastructure-run-a-collator-collator.md | 117 ++- ...infrastructure-run-a-node-parachain-rpc.md | 892 +++++++++--------- ...rastructure-run-a-node-polkadot-hub-rpc.md | 24 +- .../run-a-node/polkadot-hub-rpc.md | 2 - 4 files changed, 514 insertions(+), 521 deletions(-) diff --git a/.ai/pages/node-infrastructure-run-a-collator-collator.md b/.ai/pages/node-infrastructure-run-a-collator-collator.md index bb8fd588e..cf706b94e 100644 --- a/.ai/pages/node-infrastructure-run-a-collator-collator.md +++ b/.ai/pages/node-infrastructure-run-a-collator-collator.md @@ -97,18 +97,22 @@ This guide provides two deployment options. Select the option that best fits you === "Manual Setup" - Extract the binary from the official Docker image: + Download the `polkadot-parachain` binary from the latest stable [Polkadot SDK release](https://github.com/paritytech/polkadot-sdk/releases){target=\_blank}: ```bash - # Create a temporary container and copy the binary - docker create --name temp-parachain parity/polkadot-parachain:stable2509-2 - sudo docker cp temp-parachain:/usr/local/bin/polkadot-parachain /usr/local/bin/ - docker rm temp-parachain + # Download the latest stable release (check releases page for current version) + wget https://github.com/paritytech/polkadot-sdk/releases/download/polkadot-stable2509-2/polkadot-parachain + + # Make it executable and move to system path + chmod +x polkadot-parachain + sudo mv polkadot-parachain /usr/local/bin/ # Verify installation polkadot-parachain --version ``` + Check the [Polkadot SDK releases](https://github.com/paritytech/polkadot-sdk/releases){target=\_blank} page for the latest stable version. + ## Generate Node Key Generating a stable node key enables a consistent peer ID across the network. Follow these steps to generate a node key: @@ -305,7 +309,9 @@ Sync time depends on: The relay chain uses warp sync for faster synchronization. -**Important**: Do not proceed with registration until both chains are fully synced. Monitor sync progress using the log viewing commands in the [Log Management](#log-management) section. +!!! warning + + Do not proceed with registration until both chains are fully synced. Monitor sync progress using the log viewing commands in the [Log Management](#log-management) section. ## Generate Session Keys @@ -383,10 +389,20 @@ The registration process varies by system parachain. General steps include the f Monitoring the following items will help ensure your collator runs efficiently to avoid service interruptions or down time: -- **Block Production**: monitor for block production using the following command: - ```bash - sudo journalctl -u polkadot-collator | grep -i "prepared block" - ``` +- **Block Production**: Monitor for block production using the appropriate command for your setup: + + === "Docker Setup" + + ```bash + docker logs polkadot-collator 2>&1 | grep -i "prepared block" + ``` + + === "systemd Setup" + + ```bash + sudo journalctl -u polkadot-collator | grep -i "prepared block" + ``` + - **Peer Connections**: Maintain a sufficient amount of peers for good connectivity and set up alerts to notify if peer connections falls below ten. - **Resource Usage**: Monitor CPU, RAM, and disk I/O and set up alerts for unusual or high usage. - **Sync Status**: Ensure both chains stay synced and set up alerts for sync issues. @@ -416,32 +432,64 @@ Key metrics to monitor via Prometheus include the following: ## Log Management -Effecient log management is essential to ensure collator performance and uptime. Use the following commands to help you manage logs to monitor and maintain your collator: +Efficient log management is essential to ensure collator performance and uptime. Use the following commands to help you manage logs to monitor and maintain your collator: + +=== "Docker Setup" + + - View logs: + ```bash + docker logs -f polkadot-collator + ``` + - View recent logs (last 100 lines): + ```bash + docker logs --tail 100 polkadot-collator + ``` + - Filter for errors: + ```bash + docker logs polkadot-collator 2>&1 | grep -i error + ``` + - Filter for block production: + ```bash + docker logs polkadot-collator 2>&1 | grep -i "imported" + ``` + +=== "systemd Setup" + + - View recent logs: + ```bash + sudo journalctl -u polkadot-collator -n 100 + ``` + - Follow logs in real-time: + ```bash + sudo journalctl -u polkadot-collator -f + ``` + - Filter for errors: + ```bash + sudo journalctl -u polkadot-collator | grep -i error + ``` + - Filter for block production: + ```bash + sudo journalctl -u polkadot-collator | grep -i "imported" + ``` + +## Database Maintenance + +Check database size periodically using the commands for your selected setup: + +=== "Docker Setup" -- View recent logs: - ```bash - sudo journalctl -u polkadot-collator -n 100 - ``` -- Follow logs in real-time: - ```bash - sudo journalctl -u polkadot-collator -f - ``` -- Filter for errors: ```bash - sudo journalctl -u polkadot-collator | grep -i error + # Replace with your mounted data directory path + du -sh ./collator-data ``` -- Filter for block production: + +=== "systemd Setup" + ```bash - sudo journalctl -u polkadot-collator | grep -i "imported" + du -sh /var/lib/polkadot-collator ``` -## Database Maintenance - -You can check database size using the following command: -```bash -du -sh /var/lib/polkadot-collator -``` -The collator node handles pruning automatically. +The collator node handles pruning automatically based on configuration. ## Updates and Upgrades @@ -492,12 +540,11 @@ Updates or upgrades can happen on either the runtime or client. Runtime upgrades sudo cp -r /var/lib/polkadot-collator /var/lib/polkadot-collator.backup ``` - 3. Pull the new image and extract the binary: + 3. Download the new binary from [GitHub releases](https://github.com/paritytech/polkadot-sdk/releases){target=\_blank}: ```bash - docker pull parity/polkadot-parachain: - docker create --name temp-parachain parity/polkadot-parachain: - sudo docker cp temp-parachain:/usr/local/bin/polkadot-parachain /usr/local/bin/ - docker rm temp-parachain + wget https://github.com/paritytech/polkadot-sdk/releases/download//polkadot-parachain + chmod +x polkadot-parachain + sudo mv polkadot-parachain /usr/local/bin/ ``` 4. Verify `polkadot-parachain` version to confirm successful update: diff --git a/.ai/pages/node-infrastructure-run-a-node-parachain-rpc.md b/.ai/pages/node-infrastructure-run-a-node-parachain-rpc.md index e38c1a1fd..3d39db163 100644 --- a/.ai/pages/node-infrastructure-run-a-node-parachain-rpc.md +++ b/.ai/pages/node-infrastructure-run-a-node-parachain-rpc.md @@ -9,15 +9,13 @@ url: https://docs.polkadot.com/node-infrastructure/run-a-node/parachain-rpc/ ## Overview -Running an RPC node for a parachain enables applications, wallets, and users to interact with the parachain's functionality. This guide applies to **any parachain** in the Polkadot ecosystem, including: - -- **System parachains**: Bridge Hub, People Chain, Coretime Chain -- **Common good parachains**: Collectives, Encointer -- **Commercial parachains**: Any parachain with a publicly available chain specification +Running an RPC node for a parachain enables applications, wallets, and users to interact with the parachain's functionality. Each parachain RPC node provides access through the Polkadot SDK Node RPC (Port 9944), offering native Polkadot API access via WebSocket and HTTP. This setup enables block explorer indexing and provides full compatibility with Polkadot SDK development tools. -**Important Note**: The parameters and configurations in this guide are provided as illustrative examples. You may need to modify them according to your specific environment, hardware capabilities, and network conditions. +!!! note + + The parameters and configurations in this guide are provided as illustrative examples. You may need to modify them according to your specific environment, hardware capabilities, and network conditions. ## Obtaining a Chain Specification @@ -48,13 +46,11 @@ curl -L https://raw.githubusercontent.com/paritytech/polkadot-sdk/master/cumulus ### Other Parachains -For non-system parachains: +For non-system parachains, check the parachain's documentation for official chain specification files. -- **Check the parachain's documentation** for official chain specification files -- **Contact the parachain team** if no public chain spec is available -- **Export from a running node** using `system_chainSpec` RPC method if you have access to an existing node +!!! note -**Note**: Throughout this guide, we use **People Chain** as the example. To set up a different parachain, substitute the chain spec file, snapshot path, and chain name with values for your target parachain. + Throughout this guide, we use **People Chain** as the example. To set up a different parachain, substitute the chain spec file, snapshot path, and chain name with values for your target parachain. ## Prerequisites @@ -80,7 +76,9 @@ RPC nodes serving production traffic require robust hardware: - 9615 (Prometheus metrics - optional) - Consider DDoS protection and rate limiting for production deployments -**Note**: For development or low-traffic scenarios, you can reduce these requirements proportionally. Consider using a reverse proxy (nginx, Caddy) for production deployments. +!!! note + + For development or low-traffic scenarios, you can reduce these requirements proportionally. Consider using a reverse proxy (nginx, Caddy) for production deployments. ### Software Requirements @@ -92,423 +90,367 @@ Required software: ## Setup Options -This guide provides two deployment options: - -1. **Docker-based Setup**: Simpler to set up and maintain -2. **Manual/Systemd Setup**: For production environments requiring more control - -Choose the option that best fits your needs. - ---- - -## Option 1: Docker-Based Setup - -This option uses Docker containers for the Polkadot SDK node, making it easy to set up and manage. - -### Step 1: Download Database Snapshots (Optional but Recommended) - -Using pre-synchronized snapshots significantly reduces initial sync time from several days to just a few hours. You need to download both parachain and relay chain data. - -**Snapshot Provider**: https://snapshots.polkadot.io/ - -**Note**: Snapshots are available for system parachains and the Polkadot relay chain. For other parachains, check with the parachain team for snapshot availability or sync from genesis. - -#### Create Directories - -```bash -mkdir -p my-node-data/chains/people-polkadot/db -mkdir -p my-node-data/chains/polkadot/db -``` - -#### Download People Chain Snapshot - -Choose between archive (complete history) or pruned (recent state) snapshots. - -**Archive Snapshot** (recommended for RPC with historical data): - -```bash -# Check https://snapshots.polkadot.io/ for the latest snapshot URL -export SNAPSHOT_URL_PARACHAIN="https://snapshots.polkadot.io/polkadot-people-rocksdb-archive/LATEST" - -rclone copyurl $SNAPSHOT_URL_PARACHAIN/files.txt files.txt -rclone copy --progress --transfers 20 \ - --http-url $SNAPSHOT_URL_PARACHAIN \ - --no-traverse --http-no-head --disable-http2 \ - --inplace --no-gzip-encoding --size-only \ - --retries 6 --retries-sleep 10s \ - --files-from files.txt :http: my-node-data/chains/people-polkadot/db/ - -rm files.txt -``` - -**Parameter Explanation**: - -- `--transfers 20`: Uses 20 parallel transfers for faster download -- `--retries 6`: Automatically retries failed transfers up to 6 times -- `--retries-sleep 10s`: Waits 10 seconds between retry attempts -- `--size-only`: Only transfers if sizes differ (prevents unnecessary re-downloads) - -**Note**: If a snapshot is not available, you can sync from scratch (which will take longer) or check https://snapshots.polkadot.io/ for alternative snapshot providers. - -#### Download Polkadot Relay Chain Snapshot - -**Pruned Snapshot** (recommended for RPC nodes): - -```bash -# Check https://snapshots.polkadot.io/ for the latest snapshot URL -export SNAPSHOT_URL_RELAY="https://snapshots.polkadot.io/polkadot-rocksdb-prune/LATEST" - -rclone copyurl $SNAPSHOT_URL_RELAY/files.txt files.txt -rclone copy --progress --transfers 20 \ - --http-url $SNAPSHOT_URL_RELAY \ - --no-traverse --http-no-head --disable-http2 \ - --inplace --no-gzip-encoding --size-only \ - --retries 6 --retries-sleep 10s \ - --files-from files.txt :http: my-node-data/chains/polkadot/db/ - -rm files.txt -``` - -**Alternative Options**: - -- Pruned snapshot: `polkadot-rocksdb-prune` (smaller size, recent state) -- Archive snapshot: `polkadot-rocksdb-archive` (complete history, larger size) - -### Step 2: Start the Parachain Node - -Launch the node using the official Parity Docker image. - -**Docker Image**: https://hub.docker.com/r/parity/polkadot-parachain - -**Note**: The `parity/polkadot-parachain` image works for system parachains and parachains built with standard Cumulus templates. For parachains with custom runtimes, check the parachain's documentation for their specific Docker image or binary. - -```bash -docker run -d --name people-chain-rpc --restart unless-stopped \ - -p 9944:9944 \ - -p 9933:9933 \ - -p 9615:9615 \ - -p 30334:30334 \ - -p 30333:30333 \ - -v $(pwd)/people-polkadot.json:/people-polkadot.json \ - -v $(pwd)/my-node-data:/data \ - parity/polkadot-parachain:stable2509-2 \ - --name=PeopleChainRPC \ - --base-path=/data \ - --chain=/people-polkadot.json \ - --prometheus-external \ - --prometheus-port 9615 \ - --unsafe-rpc-external \ - --rpc-port=9944 \ - --rpc-cors=all \ - --rpc-methods=safe \ - --rpc-max-connections=1000 \ - --state-pruning=archive \ - --blocks-pruning=archive \ - -- \ - --base-path=/data \ - --chain=polkadot \ - --state-pruning=256 \ - --blocks-pruning=archive-canonical -``` - -**Critical Configuration Parameters**: - -**Port Mappings**: - -- `9944`: Polkadot SDK RPC endpoint (WebSocket/HTTP) -- `9933`: Polkadot SDK HTTP RPC endpoint -- `9615`: Prometheus metrics endpoint -- `30333/30334`: P2P networking ports - -**Node Parameters**: - -- `--unsafe-rpc-external`: Enables external RPC access -- `--rpc-cors=all`: Allows all origins for CORS -- `--rpc-methods=safe`: Only allows safe RPC methods -- `--state-pruning=archive`: Keeps complete state history -- `--blocks-pruning=archive`: Keeps all block data -- `--prometheus-external`: Exposes metrics externally - -**Security Warning**: The `--unsafe-rpc-external` flag should only be used in development or properly secured environments. For production, use a reverse proxy with authentication. - -### Step 3: Monitor Synchronization - -Monitor the node synchronization status: - -```bash -# Check sync status -curl -H "Content-Type: application/json" \ - -d '{"id":1, "jsonrpc":"2.0", "method": "system_syncState", "params":[]}' \ - http://localhost:9944 -``` - -**Expected Response Format**: - -```json -{ - "jsonrpc":"2.0", - "id":1, - "result":{ - "startingBlock":0, - "currentBlock":3394816, - "highestBlock":3394816 - } -} -``` - -**Synchronization Status**: - -- **In Progress**: `currentBlock` < `highestBlock` -- **Complete**: `currentBlock` = `highestBlock` - -### Step 4: Verify Setup - -Let's verify the Polkadot SDK RPC endpoint is working correctly. - -#### API Endpoint - -**Polkadot SDK RPC (Port 9944)**: - -- WebSocket: `ws://your-server:9944` -- HTTP: `http://your-server:9944` -- Purpose: Full Polkadot SDK API access for parachain data -- Use Cases: Polkadot SDK applications, parachain-specific operations - -#### Polkadot SDK RPC Tests - -**Get Chain Information**: - -```bash -curl -H "Content-Type: application/json" \ - -d '{"id":1, "jsonrpc":"2.0", "method": "system_chain", "params":[]}' \ - http://localhost:9944 -``` - -**Expected Response**: -```json -{ - "jsonrpc":"2.0", - "id":1, - "result":"Polkadot People" -} -``` - -**Get Latest Block**: - -```bash -curl -H "Content-Type: application/json" \ - -d '{"id":1, "jsonrpc":"2.0", "method": "chain_getHeader", "params":[]}' \ - http://localhost:9944 -``` - -**Get Node Health**: - -```bash -curl -H "Content-Type: application/json" \ - -d '{"id":1, "jsonrpc":"2.0", "method": "system_health", "params":[]}' \ - http://localhost:9944 -``` - -**Get Peer Count**: - -```bash -curl -H "Content-Type: application/json" \ - -d '{"id":1, "jsonrpc":"2.0", "method": "system_peers", "params":[]}' \ - http://localhost:9944 -``` - -### Managing Docker Containers - -**View logs**: - -```bash -docker logs -f people-chain-rpc -``` - -**Stop container**: - -```bash -docker stop people-chain-rpc -``` - -**Start container**: - -```bash -docker start people-chain-rpc -``` - -**Remove container**: - -```bash -docker rm people-chain-rpc -``` - -**Note**: For update procedures, see the [Updates and Upgrades](#updates-and-upgrades) section. - ---- - -## Option 2: Manual/Systemd Setup - -This option provides more control and is recommended for production environments requiring custom configurations. - -### Step 1: Install the Polkadot Parachain Binary - -Extract the binary from the official Docker image: - -```bash -# Pull the image and extract binary -docker pull parity/polkadot-parachain:stable2509-2 -docker create --name temp-parachain parity/polkadot-parachain:stable2509-2 -sudo docker cp temp-parachain:/usr/local/bin/polkadot-parachain /usr/local/bin/ -docker rm temp-parachain - -# Verify installation -polkadot-parachain --version -``` - -Check [Docker Hub](https://hub.docker.com/r/parity/polkadot-parachain/tags) for the latest stable tags. - -### Step 2: Create User and Directory Structure - -Ensure you have downloaded your parachain's chain specification as described in [Obtaining a Chain Specification](#obtaining-a-chain-specification). - -```bash -# Create dedicated user (skip if already exists) -sudo useradd -r -s /bin/bash polkadot - -# Create data directory -sudo mkdir -p /var/lib/people-chain-rpc - -# Copy chain spec to the directory -sudo cp people-polkadot.json /var/lib/people-chain-rpc/ - -# Set permissions -sudo chown -R polkadot:polkadot /var/lib/people-chain-rpc -``` - -### Step 3: Create Systemd Service - -Create a service file for your parachain RPC node: - -```bash -sudo nano /etc/systemd/system/people-chain-rpc.service -``` - -Add the following configuration: - -```ini -[Unit] -Description=People Chain RPC Node -After=network.target - -[Service] -Type=simple -User=polkadot -Group=polkadot -WorkingDirectory=/var/lib/people-chain-rpc - -ExecStart=/usr/local/bin/polkadot-parachain \ - --name=PeopleChainRPC \ - --chain=/var/lib/people-chain-rpc/people-polkadot.json \ - --base-path=/var/lib/people-chain-rpc \ - --port=30333 \ - --rpc-port=9944 \ - --rpc-external \ - --rpc-cors=all \ - --rpc-methods=safe \ - --rpc-max-connections=1000 \ - --prometheus-port=9615 \ - --prometheus-external \ - --state-pruning=archive \ - --blocks-pruning=archive \ - -- \ - --chain=polkadot \ - --base-path=/var/lib/people-chain-rpc \ - --port=30334 \ - --state-pruning=256 \ - --blocks-pruning=archive-canonical - -Restart=always -RestartSec=10 -LimitNOFILE=65536 - -[Install] -WantedBy=multi-user.target -``` - -### Step 4: Start Service - -```bash -# Reload systemd -sudo systemctl daemon-reload - -# Enable service to start on boot -sudo systemctl enable people-chain-rpc - -# Start the node -sudo systemctl start people-chain-rpc - -# Check status and wait for sync -sudo systemctl status people-chain-rpc -sudo journalctl -u people-chain-rpc -f -``` - -### Step 5: Verify Setup - -Use the same verification tests as in the Docker setup (see Step 4 above). - ---- - -## Monitoring and Maintenance - -### Log Management - -**Docker Setup**: - -```bash -# View node logs -docker logs -f people-chain-rpc -``` - -**Systemd Setup**: - -```bash -# View node logs -sudo journalctl -u people-chain-rpc -f - -# View recent logs -sudo journalctl -u people-chain-rpc -n 100 - -# Filter for errors -sudo journalctl -u people-chain-rpc | grep -i error -``` - -### Performance Monitoring - -Monitor key metrics: - -- **Sync status**: Ensure node stays fully synced -- **Peer connections**: Maintain 30+ peers for good connectivity -- **Resource usage**: Monitor CPU, RAM, and disk I/O -- **RPC request latency**: Track response times for the Polkadot SDK API -- **Connection count**: Monitor active RPC connections - -**Prometheus Metrics**: - -Metrics are available at `http://localhost:9615/metrics` - -Example Prometheus scrape configuration: - -{% raw %} -```yaml -scrape_configs: - - job_name: 'people-chain-rpc' - static_configs: - - targets: ['localhost:9615'] -``` -{% endraw %} - -**Key Metrics to Monitor**: +This guide provides two options for deployment: + +- **Docker-based Setup**: Best for simpler set up and maintenance +- **Manual/systemd Setup**: Best for production environments requiring more control + +Select the best option for your project, then use the steps in the following tabs to complete set up. + +=== "Docker-Based Setup" + + This option uses Docker containers for the Polkadot SDK node, making it easy to set up and manage. Follow these steps to set your RPC node using Docker: + + 1. Download your parachain's chain specification as described in [Obtaining a Chain Specification](#obtaining-a-chain-specification). + + 2. (Optional but recommended) Download database snapshots: + - Using pre-synchronized snapshots significantly reduces initial sync time from several days to just a few hours. You need to download both parachain and relay chain data. + - You can obtain the latest snapshot from the [Snapshot Provider](https://snapshots.polkadot.io/){target=\_blank}. Follow these steps to download and use snapshots: + + !!! note + + Snapshots are available for system parachains and the Polkadot relay chain. For other parachains, check with the parachain team for snapshot availability or sync from genesis. + + 1. Create new directories with the following commands: + ```bash + mkdir -p my-node-data/chains/people-polkadot/db + mkdir -p my-node-data/chains/polkadot/db + ``` + 2. Download the appropriate snapshot using the following commands: + + === "Archive snapshot" + + Contains complete history, recommended for RPC with historical data. + ```bash + # Check https://snapshots.polkadot.io/ for the latest snapshot URL + export SNAPSHOT_URL_PARACHAIN="https://snapshots.polkadot.io/polkadot-people-rocksdb-archive/LATEST" + + rclone copyurl $SNAPSHOT_URL_PARACHAIN/files.txt files.txt + rclone copy --progress --transfers 20 \ + --http-url $SNAPSHOT_URL_PARACHAIN \ + --no-traverse --http-no-head --disable-http2 \ + --inplace --no-gzip-encoding --size-only \ + --retries 6 --retries-sleep 10s \ + --files-from files.txt :http: my-node-data/chains/people-polkadot/db/ + + rm files.txt + ``` + + - `--transfers 20`: Uses 20 parallel transfers for faster download. + - `--retries 6`: Automatically retries failed transfers up to 6 times. + - `--retries-sleep 10s`: Waits 10 seconds between retry attempts. + - `--size-only`: Only transfers if sizes differ (prevents unnecessary re-downloads). + + === "Pruned snapshot" + + Contains recent state for a smaller package size, recommended for RPC nodes. + ```bash + # Check https://snapshots.polkadot.io/ for the latest snapshot URL + export SNAPSHOT_URL_RELAY="https://snapshots.polkadot.io/polkadot-rocksdb-prune/LATEST" + + rclone copyurl $SNAPSHOT_URL_RELAY/files.txt files.txt + rclone copy --progress --transfers 20 \ + --http-url $SNAPSHOT_URL_RELAY \ + --no-traverse --http-no-head --disable-http2 \ + --inplace --no-gzip-encoding --size-only \ + --retries 6 --retries-sleep 10s \ + --files-from files.txt :http: my-node-data/chains/polkadot/db/ + + rm files.txt + ``` + + 3. Launch the parachain node using the official [Parity Docker image](https://hub.docker.com/r/parity/polkadot-parachain){target=\_blank} with the following command: + ```bash + docker run -d --name people-chain-rpc --restart unless-stopped \ + -p 9944:9944 \ + -p 9933:9933 \ + -p 9615:9615 \ + -p 30334:30334 \ + -p 30333:30333 \ + -v $(pwd)/people-polkadot.json:/people-polkadot.json \ + -v $(pwd)/my-node-data:/data \ + parity/polkadot-parachain:stable2509-2 \ + --name=PeopleChainRPC \ + --base-path=/data \ + --chain=/people-polkadot.json \ + --prometheus-external \ + --prometheus-port 9615 \ + --unsafe-rpc-external \ + --rpc-port=9944 \ + --rpc-cors=all \ + --rpc-methods=safe \ + --rpc-max-connections=1000 \ + --state-pruning=archive \ + --blocks-pruning=archive \ + -- \ + --base-path=/data \ + --chain=polkadot \ + --state-pruning=256 \ + --blocks-pruning=archive-canonical + ``` + + !!! note + + The `parity/polkadot-parachain` image works for system parachains and parachains built with standard Cumulus templates. For parachains with custom runtimes, check the parachain's documentation for their specific Docker image or binary. + + Critical configuration parameters include port mappings and node parameters: + + === "Port mappings" + + - `9944`: Polkadot SDK RPC endpoint (WebSocket/HTTP) + - `9933`: Polkadot SDK HTTP RPC endpoint + - `9615`: Prometheus metrics endpoint + - `30333/30334`: P2P networking ports + + === "Node parameters" + + - `--unsafe-rpc-external`: Enables external RPC access + - `--rpc-cors=all`: Allows all origins for CORS + - `--rpc-methods=safe`: Only allows safe RPC methods + - `--state-pruning=archive`: Keeps complete state history + - `--blocks-pruning=archive`: Keeps all block data + - `--prometheus-external`: Exposes metrics externally + + !!! warning + + The `--unsafe-rpc-external` flag should only be used in development or properly secured environments. For production, use a reverse proxy with authentication. + + 4. Monitor the node synchronization status using the following command: + ```bash + curl -H "Content-Type: application/json" \ + -d '{"id":1, "jsonrpc":"2.0", "method": "system_syncState", "params":[]}' \ + http://localhost:9944 + ``` + + You should see a response similar to the following: + + ```json + { + "jsonrpc":"2.0", + "id":1, + "result":{ + "startingBlock":0, + "currentBlock":3394816, + "highestBlock":3394816 + } + } + ``` + + When synchronization is complete, `currentBlock` will be equal to `highestBlock`. + + 5. You can use a few different commands to verify your node is running properly: + + - Get chain information: + ```bash + curl -H "Content-Type: application/json" \ + -d '{"id":1, "jsonrpc":"2.0", "method": "system_chain", "params":[]}' \ + http://localhost:9944 + ``` + - Get the latest block: + ```bash + curl -H "Content-Type: application/json" \ + -d '{"id":1, "jsonrpc":"2.0", "method": "chain_getHeader", "params":[]}' \ + http://localhost:9944 + ``` + - Query node health: + ```bash + curl -H "Content-Type: application/json" \ + -d '{"id":1, "jsonrpc":"2.0", "method": "system_health", "params":[]}' \ + http://localhost:9944 + ``` + + 6. Use the following commands to manage your Docker containers: + + - View node logs: + ```bash + docker logs -f people-chain-rpc + ``` + - Stop container: + ```bash + docker stop people-chain-rpc + ``` + - Start container: + ```bash + docker start people-chain-rpc + ``` + - Remove container: + ```bash + docker rm people-chain-rpc + ``` + +=== "Manual systemd Setup" + + This option provides more control and is recommended for production environments requiring custom configurations. + + 1. Download the `polkadot-parachain` binary from the latest stable [Polkadot SDK release](https://github.com/paritytech/polkadot-sdk/releases){target=\_blank}: + ```bash + # Download the latest stable release (check releases page for current version) + wget https://github.com/paritytech/polkadot-sdk/releases/download/polkadot-stable2509-2/polkadot-parachain + + # Make it executable and move to system path + chmod +x polkadot-parachain + sudo mv polkadot-parachain /usr/local/bin/ + + # Verify installation + polkadot-parachain --version + ``` + + Check the [Polkadot SDK releases](https://github.com/paritytech/polkadot-sdk/releases){target=\_blank} page for the latest stable version. + + 2. Download your parachain's chain specification as described in [Obtaining a Chain Specification](#obtaining-a-chain-specification). + + 3. Create user and directory structures using the following commands: + - Create a dedicated user: + ```bash + sudo useradd -r -s /bin/bash polkadot + ``` + - Create data directory: + ```bash + sudo mkdir -p /var/lib/people-chain-rpc + ``` + - Copy the chain spec to the directory: + ```bash + sudo cp people-polkadot.json /var/lib/people-chain-rpc/ + ``` + - Set permissions: + ```bash + sudo chown -R polkadot:polkadot /var/lib/people-chain-rpc + ``` + + 4. Create a systemd service file for the Polkadot SDK RPC node: + ```bash + sudo nano /etc/systemd/system/people-chain-rpc.service + ``` + + 5. Open the new service file and add the following configuration: + ```ini + [Unit] + Description=People Chain RPC Node + After=network.target + + [Service] + Type=simple + User=polkadot + Group=polkadot + WorkingDirectory=/var/lib/people-chain-rpc + + ExecStart=/usr/local/bin/polkadot-parachain \ + --name=PeopleChainRPC \ + --chain=/var/lib/people-chain-rpc/people-polkadot.json \ + --base-path=/var/lib/people-chain-rpc \ + --port=30333 \ + --rpc-port=9944 \ + --rpc-external \ + --rpc-cors=all \ + --rpc-methods=safe \ + --rpc-max-connections=1000 \ + --prometheus-port=9615 \ + --prometheus-external \ + --state-pruning=archive \ + --blocks-pruning=archive \ + -- \ + --chain=polkadot \ + --base-path=/var/lib/people-chain-rpc \ + --port=30334 \ + --state-pruning=256 \ + --blocks-pruning=archive-canonical + + Restart=always + RestartSec=10 + LimitNOFILE=65536 + + [Install] + WantedBy=multi-user.target + ``` + + 6. Start the service using the following commands: + - Reload systemd: + ```bash + sudo systemctl daemon-reload + ``` + - Enable service to start on boot: + ```bash + sudo systemctl enable people-chain-rpc + ``` + - Start the Polkadot SDK node: + ```bash + sudo systemctl start people-chain-rpc + ``` + - Check status and wait for sync: + ```bash + sudo systemctl status people-chain-rpc + sudo journalctl -u people-chain-rpc -f + ``` + + 7. You can use a few different commands to verify your node is running properly: + - Get chain information: + ```bash + curl -H "Content-Type: application/json" \ + -d '{"id":1, "jsonrpc":"2.0", "method": "system_chain", "params":[]}' \ + http://localhost:9944 + ``` + - Get the latest block: + ```bash + curl -H "Content-Type: application/json" \ + -d '{"id":1, "jsonrpc":"2.0", "method": "chain_getHeader", "params":[]}' \ + http://localhost:9944 + ``` + - Query node health: + ```bash + curl -H "Content-Type: application/json" \ + -d '{"id":1, "jsonrpc":"2.0", "method": "system_health", "params":[]}' \ + http://localhost:9944 + ``` + +## Monitor and Maintain RPC Node + +There are a few key monitoring and maintenance commands that can help you ensure RPC node uptime and performance including log monitoring and metrics. + +To view node logs, use the appropriate command for your setup: + +=== "Docker Setup" + + ```bash + docker logs -f people-chain-rpc + ``` + +=== "systemd Setup" + + - View node logs: + ```bash + sudo journalctl -u people-chain-rpc -f + ``` + - View recent logs: + ```bash + sudo journalctl -u people-chain-rpc -n 100 + ``` + - Filter for errors: + ```bash + sudo journalctl -u people-chain-rpc | grep -i error + ``` + +Monitoring key metrics like the following items can help you stay up to date on the performance and health of your node and allow you to intervene at the first sign of issues: + +- **Sync status**: Ensure node stays fully synced. +- **Peer connections**: Maintain 30+ peers for good connectivity. +- **Resource usage**: Monitor CPU, RAM, and disk I/O. +- **RPC request latency**: Track response times for the Polkadot SDK API. +- **Connection count**: Monitor active RPC connections. + +You can use the following information to configure [Prometheus](https://prometheus.io/docs/introduction/first_steps/){target=\_blank} to monitor, collect, and store your RPC node metrics: + +- **URL**: Metrics are available to view at `http://localhost:9615/metrics` +- **Example Prometheus configuration**: Update your `prometheus.yml` to add the following code: + {% raw %} + ```yaml + scrape_configs: + - job_name: 'people-chain-rpc' + static_configs: + - targets: ['localhost:9615'] + ``` + {% endraw %} + +Key metrics to monitor via Prometheus include: - `substrate_block_height`: Current block height - `substrate_finalized_height`: Finalized block height @@ -517,59 +459,67 @@ scrape_configs: ### Database Maintenance -Check database size periodically: +Check database size periodically using the commands for your selected setup: -```bash -# Docker setup -du -sh my-node-data +=== "Docker Setup" -# Systemd setup -du -sh /var/lib/people-chain-rpc -``` + ```bash + du -sh my-node-data + ``` -The node handles pruning automatically based on configuration unless running in archive mode. +=== "systemd Setup" -### Updates and Upgrades + ```bash + du -sh /var/lib/people-chain-rpc + ``` -**Docker Setup**: - -```bash -# Pull latest image -docker pull parity/polkadot-parachain: - -# Restart container -docker stop people-chain-rpc -docker rm people-chain-rpc - -# Start new container (use same command from setup with updated image tag) -``` - -**Systemd Setup**: - -```bash -# Stop service -sudo systemctl stop people-chain-rpc - -# Backup data -sudo cp -r /var/lib/people-chain-rpc /var/lib/people-chain-rpc.backup +The node handles pruning automatically based on configuration unless running in archive mode. -# Pull new image and extract binary -docker pull parity/polkadot-parachain: -docker create --name temp-parachain parity/polkadot-parachain: -sudo docker cp temp-parachain:/usr/local/bin/polkadot-parachain /usr/local/bin/ -docker rm temp-parachain +### Updates and Upgrades -# Restart service -sudo systemctl start people-chain-rpc -``` +Use the following commands for updating or upgrading your RPC node according to your setup: + +=== "Docker Setup" + + 1. Stop and remove the existing container: + ```bash + docker stop people-chain-rpc + docker rm people-chain-rpc + ``` + 2. Pull the latest image: + ```bash + docker pull parity/polkadot-parachain: + ``` + 3. Start the new container using the same command from the setup section with the updated image tag. + +=== "systemd Setup" + + 1. Stop the service: + ```bash + sudo systemctl stop people-chain-rpc + ``` + 2. Backup data: + ```bash + sudo cp -r /var/lib/people-chain-rpc /var/lib/people-chain-rpc.backup + ``` + 3. Download the new binary from [GitHub releases](https://github.com/paritytech/polkadot-sdk/releases){target=\_blank}: + ```bash + wget https://github.com/paritytech/polkadot-sdk/releases/download//polkadot-parachain + chmod +x polkadot-parachain + sudo mv polkadot-parachain /usr/local/bin/ + ``` + 4. Restart the service: + ```bash + sudo systemctl start people-chain-rpc + ``` ## Conclusion Running a parachain RPC node provides critical infrastructure for accessing Polkadot network services. By following this guide, you have set up a production-ready RPC node that: -- Provides reliable access to parachain functionality for applications and users -- Supports flexible deployment with both Docker and systemd options -- Implements comprehensive monitoring, security, and maintenance practices -- Can be adapted for any parachain by substituting the appropriate chain specification +- Provides reliable access to parachain functionality for applications and users. +- Supports flexible deployment with both Docker and systemd options. +- Implements comprehensive monitoring, security, and maintenance practices. +- Can be adapted for any parachain by substituting the appropriate chain specification. Whether you're running a node for system parachains (People Chain, Bridge Hub, Coretime Chain) or other parachains in the ecosystem, regular maintenance and monitoring will ensure your RPC node continues to provide reliable service. Stay updated with the latest releases and best practices to keep your infrastructure secure and performant. diff --git a/.ai/pages/node-infrastructure-run-a-node-polkadot-hub-rpc.md b/.ai/pages/node-infrastructure-run-a-node-polkadot-hub-rpc.md index e1dab7997..6c5880316 100644 --- a/.ai/pages/node-infrastructure-run-a-node-polkadot-hub-rpc.md +++ b/.ai/pages/node-infrastructure-run-a-node-polkadot-hub-rpc.md @@ -249,20 +249,19 @@ Select the best option for your project, then use the steps in the following tab This option provides more control and is recommended for production environments requiring custom configurations. - 1. Install the Polkadot Parachain binary by extracting it from the official Docker image: + 1. Download the `polkadot-parachain` binary from the latest stable [Polkadot SDK release](https://github.com/paritytech/polkadot-sdk/releases){target=\_blank}: ```bash - # Pull the image and extract binary - docker pull parity/polkadot-parachain:stable2509-2 - docker create --name temp-parachain parity/polkadot-parachain:stable2509-2 - sudo docker cp temp-parachain:/usr/local/bin/polkadot-parachain /usr/local/bin/ - docker rm temp-parachain + # Download the latest stable release (check releases page for current version) + wget https://github.com/paritytech/polkadot-sdk/releases/download/polkadot-stable2509-2/polkadot-parachain + + # Make it executable and move to system path + chmod +x polkadot-parachain + sudo mv polkadot-parachain /usr/local/bin/ # Verify installation polkadot-parachain --version ``` - Check [Docker Hub](https://hub.docker.com/r/parity/polkadot-parachain/tags){target=\_blank} for the latest stable tags. - 2. Download the Polkadot Hub chain specification: ```bash curl -L https://raw.githubusercontent.com/paritytech/polkadot-sdk/master/cumulus/parachains/chain-specs/asset-hub-polkadot.json -o asset-hub-polkadot.json @@ -471,12 +470,11 @@ Use the following commands for updating or upgrading your RPC node according to ```bash sudo cp -r /var/lib/polkadot-hub-rpc /var/lib/polkadot-hub-rpc.backup ``` - 3. Pull the new image and extract the binary: + 3. Download the new binary from [GitHub releases](https://github.com/paritytech/polkadot-sdk/releases){target=\_blank}: ```bash - docker pull parity/polkadot-parachain: - docker create --name temp-parachain parity/polkadot-parachain: - sudo docker cp temp-parachain:/usr/local/bin/polkadot-parachain /usr/local/bin/ - docker rm temp-parachain + wget https://github.com/paritytech/polkadot-sdk/releases/download//polkadot-parachain + chmod +x polkadot-parachain + sudo mv polkadot-parachain /usr/local/bin/ ``` 4. Restart the service: ```bash diff --git a/node-infrastructure/run-a-node/polkadot-hub-rpc.md b/node-infrastructure/run-a-node/polkadot-hub-rpc.md index 8104bff42..a7dd6388a 100644 --- a/node-infrastructure/run-a-node/polkadot-hub-rpc.md +++ b/node-infrastructure/run-a-node/polkadot-hub-rpc.md @@ -261,8 +261,6 @@ Select the best option for your project, then use the steps in the following tab polkadot-parachain --version ``` - Check the [Polkadot SDK releases](https://github.com/paritytech/polkadot-sdk/releases){target=\_blank} page for the latest stable version. - 2. Download the Polkadot Hub chain specification: ```bash curl -L https://raw.githubusercontent.com/paritytech/polkadot-sdk/master/cumulus/parachains/chain-specs/asset-hub-polkadot.json -o asset-hub-polkadot.json From 6822d5717b00fcd843c4445a90bc96a43634caa2 Mon Sep 17 00:00:00 2001 From: DAWN KELLY Date: Mon, 24 Nov 2025 11:11:11 -0500 Subject: [PATCH 20/21] wip rpc page cleanup --- node-infrastructure/run-a-node/polkadot-hub-rpc.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/node-infrastructure/run-a-node/polkadot-hub-rpc.md b/node-infrastructure/run-a-node/polkadot-hub-rpc.md index a7dd6388a..9b956861c 100644 --- a/node-infrastructure/run-a-node/polkadot-hub-rpc.md +++ b/node-infrastructure/run-a-node/polkadot-hub-rpc.md @@ -79,7 +79,7 @@ Select the best option for your project, then use the steps in the following tab This chain specification is the official configuration file that defines the network parameters for Polkadot Hub. - 2. (Optional but recommended) Download database snapshots + 2. (Optional but recommended) Download database snapshots: - Using pre-synchronized snapshots significantly reduces initial sync time from several days to just a few hours. You need to download both parachain and relay chain data. - You can obtain the latest snapshot from the [Snapshot Provider](https://snapshots.polkadot.io/){target=\_blank}. Follow these steps to download and use snapshots: 1. Create new directories with the following commands: @@ -189,7 +189,7 @@ Select the best option for your project, then use the steps in the following tab ``` You should see a response similar to the following: - + ```json { "jsonrpc":"2.0", From 65132eda66d751a2cf8aa97db4010b57a435d877 Mon Sep 17 00:00:00 2001 From: Bruno Galvao Date: Wed, 26 Nov 2025 12:39:43 +0700 Subject: [PATCH 21/21] Update content --- .ai/pages/node-infrastructure-overview.md | 15 +- ...-infrastructure-run-a-collator-collator.md | 79 ++-- ...infrastructure-run-a-node-parachain-rpc.md | 405 ++++++++-------- ...rastructure-run-a-node-polkadot-hub-rpc.md | 437 +++++++++--------- node-infrastructure/overview.md | 15 +- .../run-a-collator/collator.md | 77 ++- .../run-a-node/parachain-rpc.md | 403 ++++++++-------- .../run-a-node/polkadot-hub-rpc.md | 433 ++++++++--------- 8 files changed, 945 insertions(+), 919 deletions(-) diff --git a/.ai/pages/node-infrastructure-overview.md b/.ai/pages/node-infrastructure-overview.md index fe7e4f507..15da08c89 100644 --- a/.ai/pages/node-infrastructure-overview.md +++ b/.ai/pages/node-infrastructure-overview.md @@ -24,7 +24,12 @@ RPC nodes provide API access to blockchain data without participating in consens - **Wallets**: Check balances and broadcast transactions - **Development**: Test and debug applications -RPC nodes can be run for both the relay chain and parachains, with varying levels of data retention (pruned vs archive). +RPC nodes can be run for both the relay chain and parachains, with varying levels of data retention: + +- **Pruned Nodes**: Keep recent state and a limited number of finalized blocks. Suitable for most applications that only need current state and recent history. More efficient in terms of storage and sync time. +- **Archive Nodes**: Maintain complete historical state and all blocks since genesis. Required for block explorers, analytics platforms, or applications that need to query historical data at any point in time. + +**Transaction Broadcasting**: RPC nodes play a crucial role in transaction submission and propagation. When a client submits a transaction via RPC methods like `author_submitExtrinsic`, the node validates the transaction format, adds it to its local transaction pool, and broadcasts it across the P2P network. Block producers (collators or validators) then pick up these transactions from their pools for inclusion in blocks. This makes RPC nodes the primary gateway for users and applications to interact with the blockchain. ### Collators @@ -32,17 +37,17 @@ Collators are block producers for parachains. They perform critical functions: - **Collect transactions**: Aggregate user transactions into blocks - **Produce blocks**: Create parachain block candidates -- **Generate proofs**: Produce state transition proofs (Proof-of-Validity) -- **Submit to validators**: Send block candidates to relay chain validators +- **Generate and package PoV**: Generate the Proof-of-Validity containing the state transition proof and necessary witness data for validation +- **Submit to validators**: Send block candidates and PoVs to relay chain validators Unlike validators, collators do not provide security guarantees—that responsibility lies with relay chain validators. However, collators are essential for parachain liveness and censorship resistance. ### Validators -Validators secure the Polkadot relay chain through Nominated Proof of Stake (NPoS). They: +Validators secure the Polkadot relay chain through [Nominated Proof of Stake (NPoS)](https://wiki.polkadot.network/docs/learn-staking){target=\_blank}. They: - **Validate blocks**: Verify parachain blocks and relay chain transactions -- **Participate in consensus**: Run BABE and GRANDPA protocols +- **Participate in consensus**: Run [BABE](https://wiki.polkadot.network/docs/learn-consensus#babe-block-production){target=\_blank} and [GRANDPA](https://wiki.polkadot.network/docs/learn-consensus#grandpa-finality-gadget){target=\_blank} protocols - **Earn rewards**: Receive staking rewards for honest behavior - **Risk slashing**: Face penalties for misbehavior or downtime diff --git a/.ai/pages/node-infrastructure-run-a-collator-collator.md b/.ai/pages/node-infrastructure-run-a-collator-collator.md index cf706b94e..8525e7ee4 100644 --- a/.ai/pages/node-infrastructure-run-a-collator-collator.md +++ b/.ai/pages/node-infrastructure-run-a-collator-collator.md @@ -31,7 +31,7 @@ Block-producing collators perform critical functions: - Send block candidates to relay chain validators. - Enable cross-chain message passing using XCM -Unlike relay chain validators, collators do not provide security guarantees—that responsibility lies with relay chain validators through the ELVES protocol. Rather, collators are essential for network liveness and censorship resistance. +Unlike relay chain validators, collators do not provide security guarantees—that responsibility lies with relay chain validators through the [ELVES protocol](https://wiki.polkadot.com/learn/learn-parachains-protocol/){target=\_blank}. Rather, collators are essential for network liveness and censorship resistance. ## Prerequisites @@ -57,7 +57,7 @@ Uptime is critical. Consider redundancy and monitoring to maintain block product ### Software Requirements -Collators use the **Polkadot Parachain** binary, the standard client for running Polkadot system parachains. +Collators use the Polkadot Parachain binary, the standard client for running Polkadot system parachains. Required software includes the following: @@ -143,6 +143,17 @@ Generate an account for on-chain transactions as follows: docker run -it parity/subkey:latest generate --scheme sr25519 ``` + The output will be similar to the following: + ```bash + Secret phrase: embody rail hour peanut .... badge syrup luggage canvas + Network ID: substrate + Secret seed: 0x6498dd3416c491406e2c8283c76760ce4ca018478888b42315e7718778f2c2e1 + Public key (hex): 0x2202210357e49390d4f8d868da983940fe220a0a0e00bc6feaeda462aa031810 + Account ID: 0x2202210357e49390d4f8d868da983940fe220a0a0e00bc6feaeda462aa031810 + Public key (SS58): 5CqJ7n72GvvF5ZzUT2HMj83KyDje4n8sXR8kuiK8HWtfDktF + SS58 Address: 5CqJ7n72GvvF5ZzUT2HMj83KyDje4n8sXR8kuiK8HWtfDktF + ``` + 2. Save the following items displayed in the output: - Secret phrase (seed) - Keep this secure! - Public key (hex) @@ -161,7 +172,7 @@ Download the chain specification for your target system parachain using one of t Follow these steps to download your specification from the Chainspec Collection: -1. Visit the [Chainspec Collection](https://paritytech.github.io/chainspecs/) website. +1. Visit the [Chainspec Collection](https://paritytech.github.io/chainspecs/){target=\_blank} website. 2. Find your target system parachain. 3. Download the chain specification JSON file. 4. Save it as `chain-spec.json`. @@ -315,6 +326,8 @@ The relay chain uses warp sync for faster synchronization. ## Generate Session Keys +Session keys are cryptographic keys used by your collator node to sign authorship information when producing blocks. They uniquely identify your collator on the network and must be registered on-chain before your collator can participate in block production. + Once your node is fully synced, use the following command to generate session keys via RPC: ```bash @@ -338,7 +351,7 @@ System parachains use different collator selection mechanisms. Explore the follo === "On-chain Selection" - - Some parachains use pallet-collator-selection. + - Some parachains use [pallet-collator-selection](https://paritytech.github.io/polkadot-sdk/master/pallet_collator_selection/index.html){target=\_blank}. - May require bonding tokens. - Automatic selection based on criteria. @@ -349,6 +362,8 @@ System parachains use different collator selection mechanisms. Explore the follo ### Registration Process +Collator registration authorizes your node to produce blocks on the network. The parachain's collator selection mechanism uses this on-chain registration to determine which nodes are eligible to author blocks. + The registration process varies by system parachain. General steps include the following: 1. Check the existing collators for your target parachain: @@ -375,61 +390,19 @@ The registration process varies by system parachain. General steps include the f - `proof`: 0x00 (typically) 6. Submit and sign the transaction. -4. If the parachain requires bonding tokens, use the follow these steps to submit them using Polkadot.js Apps: +4. (Optional - primarily for non-system parachains) If the parachain uses on-chain bonding for collator selection, register as a candidate using Polkadot.js Apps: + + !!! note + Most system parachains use invulnerables lists exclusively and do not require this step. Skip to step 5 if you're running a collator for a system parachain. + 1. Locate **Developer > Extrinsics**. 2. Select `collatorSelection.registerAsCandidate`. - 3. Submit the transaction with the required bond amount. - 4. Sign the transaction. + 3. Submit and sign the transaction. The required bond amount will be automatically reserved from your account based on the pallet's configured `CandidacyBond`. -5. If applying to join the invulnerables list, you must now await governance approval for your proposal. Monitor the [Polkadot Forum](https://forum.polkadot.network){target=\_blank} governance channels and announcements. Once approved, your collator is added to the invulnerables list and will begin producing blocks in the next session or era. +5. For system parachains using invulnerables lists, await governance approval for your proposal. Monitor the [Polkadot Forum](https://forum.polkadot.network){target=\_blank} governance channels and announcements. Once approved, your collator is added to the invulnerables list and will begin producing blocks in the next session or era. 6. Verify your collator is active by monitoring logs for block production messages like "Prepared block for proposing" and "Imported #123". See the [Log Management](#log-management) section for log viewing commands. -## Monitor and Maintain Your Collator - -Monitoring the following items will help ensure your collator runs efficiently to avoid service interruptions or down time: - -- **Block Production**: Monitor for block production using the appropriate command for your setup: - - === "Docker Setup" - - ```bash - docker logs polkadot-collator 2>&1 | grep -i "prepared block" - ``` - - === "systemd Setup" - - ```bash - sudo journalctl -u polkadot-collator | grep -i "prepared block" - ``` - -- **Peer Connections**: Maintain a sufficient amount of peers for good connectivity and set up alerts to notify if peer connections falls below ten. -- **Resource Usage**: Monitor CPU, RAM, and disk I/O and set up alerts for unusual or high usage. -- **Sync Status**: Ensure both chains stay synced and set up alerts for sync issues. - -### Prometheus Metrics - -You can use the following information to configure [Prometheus](https://prometheus.io/docs/introduction/first_steps/){target=\_blank} to monitor, collect, and store your collator node metrics: - -- **URL**: Metrics are available to view at `http://localhost:9615/metrics` -- **Example Prometheus configuration**: Update your `prometheus.yml` to add the following code: - {% raw %} - ```yaml - scrape_configs: - - job_name: 'polkadot-collator' - static_configs: - - targets: ['localhost:9615'] - ``` - {% endraw %} - -Key metrics to monitor via Prometheus include the following: - -- `substrate_block_height`: Current block height -- `substrate_finalized_height`: Finalized block height -- `substrate_peers_count`: Peer connections -- `substrate_ready_transactions_number`: Transaction queue - - ## Log Management Efficient log management is essential to ensure collator performance and uptime. Use the following commands to help you manage logs to monitor and maintain your collator: diff --git a/.ai/pages/node-infrastructure-run-a-node-parachain-rpc.md b/.ai/pages/node-infrastructure-run-a-node-parachain-rpc.md index 3d39db163..b5b394b3a 100644 --- a/.ai/pages/node-infrastructure-run-a-node-parachain-rpc.md +++ b/.ai/pages/node-infrastructure-run-a-node-parachain-rpc.md @@ -61,7 +61,7 @@ RPC nodes serving production traffic require robust hardware: - **CPU**: 8+ cores (16+ cores for high traffic) - **Memory**: 64 GB RAM minimum (128 GB recommended for high traffic) - **Storage**: - - 500 GB+ NVMe SSD for parachain state (archive nodes require 2-4 TB+) + - Storage requirements vary by parachain. System parachains: Asset Hub (~600-800 GB), Bridge Hub (~500-600 GB), Collectives (~400-500 GB), People Chain (~300-400 GB), Coretime (~300-400 GB). For non-system parachains, check the [snapshot sizes](https://snapshots.polkadot.io/){target=\_blank} if available. - Additional 200+ GB for relay chain pruned database - Fast disk I/O is critical for query performance - **Network**: @@ -116,11 +116,13 @@ Select the best option for your project, then use the steps in the following tab mkdir -p my-node-data/chains/people-polkadot/db mkdir -p my-node-data/chains/polkadot/db ``` - 2. Download the appropriate snapshot using the following commands: + 2. Download the appropriate snapshots using the following commands: - === "Archive snapshot" + === "Archive Node" - Contains complete history, recommended for RPC with historical data. + Archive node setup maintains complete parachain history. Download both parachain archive and relay chain pruned snapshots: + + **Parachain archive snapshot** (People Chain example): ```bash # Check https://snapshots.polkadot.io/ for the latest snapshot URL export SNAPSHOT_URL_PARACHAIN="https://snapshots.polkadot.io/polkadot-people-rocksdb-archive/LATEST" @@ -136,14 +138,50 @@ Select the best option for your project, then use the steps in the following tab rm files.txt ``` - - `--transfers 20`: Uses 20 parallel transfers for faster download. - - `--retries 6`: Automatically retries failed transfers up to 6 times. - - `--retries-sleep 10s`: Waits 10 seconds between retry attempts. - - `--size-only`: Only transfers if sizes differ (prevents unnecessary re-downloads). + **Relay chain pruned snapshot** (~200 GB): + ```bash + # Check https://snapshots.polkadot.io/ for the latest snapshot URL + export SNAPSHOT_URL_RELAY="https://snapshots.polkadot.io/polkadot-rocksdb-prune/LATEST" + + rclone copyurl $SNAPSHOT_URL_RELAY/files.txt files.txt + rclone copy --progress --transfers 20 \ + --http-url $SNAPSHOT_URL_RELAY \ + --no-traverse --http-no-head --disable-http2 \ + --inplace --no-gzip-encoding --size-only \ + --retries 6 --retries-sleep 10s \ + --files-from files.txt :http: my-node-data/chains/polkadot/db/ + + rm files.txt + ``` + + **rclone parameters:** + + - `--transfers 20`: Uses 20 parallel transfers for faster download + - `--retries 6`: Automatically retries failed transfers up to 6 times + - `--retries-sleep 10s`: Waits 10 seconds between retry attempts + - `--size-only`: Only transfers if sizes differ (prevents unnecessary re-downloads) + + === "Pruned Node" + + Pruned node setup keeps recent state for smaller storage. Download both parachain pruned and relay chain pruned snapshots: + + **Parachain pruned snapshot** (People Chain example): + ```bash + # Check https://snapshots.polkadot.io/ for the latest snapshot URL + export SNAPSHOT_URL_PARACHAIN="https://snapshots.polkadot.io/polkadot-people-rocksdb-prune/LATEST" + + rclone copyurl $SNAPSHOT_URL_PARACHAIN/files.txt files.txt + rclone copy --progress --transfers 20 \ + --http-url $SNAPSHOT_URL_PARACHAIN \ + --no-traverse --http-no-head --disable-http2 \ + --inplace --no-gzip-encoding --size-only \ + --retries 6 --retries-sleep 10s \ + --files-from files.txt :http: my-node-data/chains/people-polkadot/db/ - === "Pruned snapshot" + rm files.txt + ``` - Contains recent state for a smaller package size, recommended for RPC nodes. + **Relay chain pruned snapshot** (~200 GB): ```bash # Check https://snapshots.polkadot.io/ for the latest snapshot URL export SNAPSHOT_URL_RELAY="https://snapshots.polkadot.io/polkadot-rocksdb-prune/LATEST" @@ -159,35 +197,80 @@ Select the best option for your project, then use the steps in the following tab rm files.txt ``` - 3. Launch the parachain node using the official [Parity Docker image](https://hub.docker.com/r/parity/polkadot-parachain){target=\_blank} with the following command: - ```bash - docker run -d --name people-chain-rpc --restart unless-stopped \ - -p 9944:9944 \ - -p 9933:9933 \ - -p 9615:9615 \ - -p 30334:30334 \ - -p 30333:30333 \ - -v $(pwd)/people-polkadot.json:/people-polkadot.json \ - -v $(pwd)/my-node-data:/data \ - parity/polkadot-parachain:stable2509-2 \ - --name=PeopleChainRPC \ - --base-path=/data \ - --chain=/people-polkadot.json \ - --prometheus-external \ - --prometheus-port 9615 \ - --unsafe-rpc-external \ - --rpc-port=9944 \ - --rpc-cors=all \ - --rpc-methods=safe \ - --rpc-max-connections=1000 \ - --state-pruning=archive \ - --blocks-pruning=archive \ - -- \ - --base-path=/data \ - --chain=polkadot \ - --state-pruning=256 \ - --blocks-pruning=archive-canonical - ``` + **rclone parameters:** + + - `--transfers 20`: Uses 20 parallel transfers for faster download + - `--retries 6`: Automatically retries failed transfers up to 6 times + - `--retries-sleep 10s`: Waits 10 seconds between retry attempts + - `--size-only`: Only transfers if sizes differ (prevents unnecessary re-downloads) + + 3. Launch the parachain node using the official [Parity Docker image](https://hub.docker.com/r/parity/polkadot-parachain){target=\_blank}: + + === "Archive Node" + + Archive node configuration maintains complete parachain history for historical queries: + + ```bash + docker run -d --name people-chain-rpc --restart unless-stopped \ + -p 9944:9944 \ + -p 9933:9933 \ + -p 9615:9615 \ + -p 30334:30334 \ + -p 30333:30333 \ + -v $(pwd)/people-polkadot.json:/people-polkadot.json \ + -v $(pwd)/my-node-data:/data \ + parity/polkadot-parachain:stable2509-2 \ + --name=PeopleChainRPC \ + --base-path=/data \ + --chain=/people-polkadot.json \ + --prometheus-external \ + --prometheus-port 9615 \ + --unsafe-rpc-external \ + --rpc-port=9944 \ + --rpc-cors=all \ + --rpc-methods=safe \ + --rpc-max-connections=1000 \ + --state-pruning=archive \ + --blocks-pruning=archive \ + -- \ + --base-path=/data \ + --chain=polkadot \ + --state-pruning=256 \ + --blocks-pruning=archive-canonical + ``` + + === "Pruned Node" + + Pruned node configuration keeps recent state for smaller storage requirements: + + ```bash + docker run -d --name people-chain-rpc --restart unless-stopped \ + -p 9944:9944 \ + -p 9933:9933 \ + -p 9615:9615 \ + -p 30334:30334 \ + -p 30333:30333 \ + -v $(pwd)/people-polkadot.json:/people-polkadot.json \ + -v $(pwd)/my-node-data:/data \ + parity/polkadot-parachain:stable2509-2 \ + --name=PeopleChainRPC \ + --base-path=/data \ + --chain=/people-polkadot.json \ + --prometheus-external \ + --prometheus-port 9615 \ + --unsafe-rpc-external \ + --rpc-port=9944 \ + --rpc-cors=all \ + --rpc-methods=safe \ + --rpc-max-connections=1000 \ + --state-pruning=1000 \ + --blocks-pruning=256 \ + -- \ + --base-path=/data \ + --chain=polkadot \ + --state-pruning=256 \ + --blocks-pruning=archive-canonical + ``` !!! note @@ -207,8 +290,8 @@ Select the best option for your project, then use the steps in the following tab - `--unsafe-rpc-external`: Enables external RPC access - `--rpc-cors=all`: Allows all origins for CORS - `--rpc-methods=safe`: Only allows safe RPC methods - - `--state-pruning=archive`: Keeps complete state history - - `--blocks-pruning=archive`: Keeps all block data + - `--state-pruning=archive` or `--state-pruning=1000`: Archive keeps complete state history, pruned keeps last 1000 blocks + - `--blocks-pruning=archive` or `--blocks-pruning=256`: Archive keeps all blocks, pruned keeps last 256 finalized blocks - `--prometheus-external`: Exposes metrics externally !!! warning @@ -322,46 +405,95 @@ Select the best option for your project, then use the steps in the following tab sudo nano /etc/systemd/system/people-chain-rpc.service ``` - 5. Open the new service file and add the following configuration: - ```ini - [Unit] - Description=People Chain RPC Node - After=network.target - - [Service] - Type=simple - User=polkadot - Group=polkadot - WorkingDirectory=/var/lib/people-chain-rpc - - ExecStart=/usr/local/bin/polkadot-parachain \ - --name=PeopleChainRPC \ - --chain=/var/lib/people-chain-rpc/people-polkadot.json \ - --base-path=/var/lib/people-chain-rpc \ - --port=30333 \ - --rpc-port=9944 \ - --rpc-external \ - --rpc-cors=all \ - --rpc-methods=safe \ - --rpc-max-connections=1000 \ - --prometheus-port=9615 \ - --prometheus-external \ - --state-pruning=archive \ - --blocks-pruning=archive \ - -- \ - --chain=polkadot \ - --base-path=/var/lib/people-chain-rpc \ - --port=30334 \ - --state-pruning=256 \ - --blocks-pruning=archive-canonical - - Restart=always - RestartSec=10 - LimitNOFILE=65536 - - [Install] - WantedBy=multi-user.target - ``` + 5. Open the new service file and add the configuration for your chosen node type: + + === "Archive Node" + + Archive node configuration maintains complete parachain history for historical queries: + + ```ini + [Unit] + Description=People Chain RPC Node + After=network.target + + [Service] + Type=simple + User=polkadot + Group=polkadot + WorkingDirectory=/var/lib/people-chain-rpc + + ExecStart=/usr/local/bin/polkadot-parachain \ + --name=PeopleChainRPC \ + --chain=/var/lib/people-chain-rpc/people-polkadot.json \ + --base-path=/var/lib/people-chain-rpc \ + --port=30333 \ + --rpc-port=9944 \ + --rpc-external \ + --rpc-cors=all \ + --rpc-methods=safe \ + --rpc-max-connections=1000 \ + --prometheus-port=9615 \ + --prometheus-external \ + --state-pruning=archive \ + --blocks-pruning=archive \ + -- \ + --chain=polkadot \ + --base-path=/var/lib/people-chain-rpc \ + --port=30334 \ + --state-pruning=256 \ + --blocks-pruning=archive-canonical + + Restart=always + RestartSec=10 + LimitNOFILE=65536 + + [Install] + WantedBy=multi-user.target + ``` + + === "Pruned Node" + + Pruned node configuration keeps recent state for smaller storage requirements: + + ```ini + [Unit] + Description=People Chain RPC Node + After=network.target + + [Service] + Type=simple + User=polkadot + Group=polkadot + WorkingDirectory=/var/lib/people-chain-rpc + + ExecStart=/usr/local/bin/polkadot-parachain \ + --name=PeopleChainRPC \ + --chain=/var/lib/people-chain-rpc/people-polkadot.json \ + --base-path=/var/lib/people-chain-rpc \ + --port=30333 \ + --rpc-port=9944 \ + --rpc-external \ + --rpc-cors=all \ + --rpc-methods=safe \ + --rpc-max-connections=1000 \ + --prometheus-port=9615 \ + --prometheus-external \ + --state-pruning=1000 \ + --blocks-pruning=256 \ + -- \ + --chain=polkadot \ + --base-path=/var/lib/people-chain-rpc \ + --port=30334 \ + --state-pruning=256 \ + --blocks-pruning=archive-canonical + + Restart=always + RestartSec=10 + LimitNOFILE=65536 + + [Install] + WantedBy=multi-user.target + ``` 6. Start the service using the following commands: - Reload systemd: @@ -402,117 +534,6 @@ Select the best option for your project, then use the steps in the following tab http://localhost:9944 ``` -## Monitor and Maintain RPC Node - -There are a few key monitoring and maintenance commands that can help you ensure RPC node uptime and performance including log monitoring and metrics. - -To view node logs, use the appropriate command for your setup: - -=== "Docker Setup" - - ```bash - docker logs -f people-chain-rpc - ``` - -=== "systemd Setup" - - - View node logs: - ```bash - sudo journalctl -u people-chain-rpc -f - ``` - - View recent logs: - ```bash - sudo journalctl -u people-chain-rpc -n 100 - ``` - - Filter for errors: - ```bash - sudo journalctl -u people-chain-rpc | grep -i error - ``` - -Monitoring key metrics like the following items can help you stay up to date on the performance and health of your node and allow you to intervene at the first sign of issues: - -- **Sync status**: Ensure node stays fully synced. -- **Peer connections**: Maintain 30+ peers for good connectivity. -- **Resource usage**: Monitor CPU, RAM, and disk I/O. -- **RPC request latency**: Track response times for the Polkadot SDK API. -- **Connection count**: Monitor active RPC connections. - -You can use the following information to configure [Prometheus](https://prometheus.io/docs/introduction/first_steps/){target=\_blank} to monitor, collect, and store your RPC node metrics: - -- **URL**: Metrics are available to view at `http://localhost:9615/metrics` -- **Example Prometheus configuration**: Update your `prometheus.yml` to add the following code: - {% raw %} - ```yaml - scrape_configs: - - job_name: 'people-chain-rpc' - static_configs: - - targets: ['localhost:9615'] - ``` - {% endraw %} - -Key metrics to monitor via Prometheus include: - -- `substrate_block_height`: Current block height -- `substrate_finalized_height`: Finalized block height -- `substrate_peers_count`: Number of connected peers -- `substrate_ready_transactions_number`: Transaction queue size - -### Database Maintenance - -Check database size periodically using the commands for your selected setup: - -=== "Docker Setup" - - ```bash - du -sh my-node-data - ``` - -=== "systemd Setup" - - ```bash - du -sh /var/lib/people-chain-rpc - ``` - -The node handles pruning automatically based on configuration unless running in archive mode. - -### Updates and Upgrades - -Use the following commands for updating or upgrading your RPC node according to your setup: - -=== "Docker Setup" - - 1. Stop and remove the existing container: - ```bash - docker stop people-chain-rpc - docker rm people-chain-rpc - ``` - 2. Pull the latest image: - ```bash - docker pull parity/polkadot-parachain: - ``` - 3. Start the new container using the same command from the setup section with the updated image tag. - -=== "systemd Setup" - - 1. Stop the service: - ```bash - sudo systemctl stop people-chain-rpc - ``` - 2. Backup data: - ```bash - sudo cp -r /var/lib/people-chain-rpc /var/lib/people-chain-rpc.backup - ``` - 3. Download the new binary from [GitHub releases](https://github.com/paritytech/polkadot-sdk/releases){target=\_blank}: - ```bash - wget https://github.com/paritytech/polkadot-sdk/releases/download//polkadot-parachain - chmod +x polkadot-parachain - sudo mv polkadot-parachain /usr/local/bin/ - ``` - 4. Restart the service: - ```bash - sudo systemctl start people-chain-rpc - ``` - ## Conclusion Running a parachain RPC node provides critical infrastructure for accessing Polkadot network services. By following this guide, you have set up a production-ready RPC node that: diff --git a/.ai/pages/node-infrastructure-run-a-node-polkadot-hub-rpc.md b/.ai/pages/node-infrastructure-run-a-node-polkadot-hub-rpc.md index 6c5880316..70fdd2192 100644 --- a/.ai/pages/node-infrastructure-run-a-node-polkadot-hub-rpc.md +++ b/.ai/pages/node-infrastructure-run-a-node-polkadot-hub-rpc.md @@ -16,6 +16,7 @@ url: https://docs.polkadot.com/node-infrastructure/run-a-node/polkadot-hub-rpc/ - **Cross-chain Communication**: XCM message handling Running an RPC node for Polkadot Hub enables applications, wallets, and users to interact with the parachain through: + - **Polkadot SDK Node RPC** (Port 9944): Native Polkadot API (WebSocket and HTTP) This setup enables block explorer indexing and provides full compatibility with Polkadot SDK development tools. @@ -33,7 +34,7 @@ RPC nodes serving production traffic require robust hardware. The following shou - **CPU**: 8+ cores (16+ cores for high traffic) - **Memory**: 64 GB RAM minimum (128 GB recommended for high traffic) - **Storage**: - - 500 GB+ NVMe SSD for parachain state (archive nodes require 2-4 TB+) + - 500 GB+ NVMe SSD for parachain state (archive nodes require 600-800 GB+) - Additional 200+ GB for relay chain pruned database - Fast disk I/O is critical for query performance - **Network**: @@ -80,7 +81,7 @@ Select the best option for your project, then use the steps in the following tab This chain specification is the official configuration file that defines the network parameters for Polkadot Hub. - 2. (Optional but recommended) Download database snapshots + 2. (Optional but recommended) Download database snapshots: - Using pre-synchronized snapshots significantly reduces initial sync time from several days to just a few hours. You need to download both parachain and relay chain data. - You can obtain the latest snapshot from the [Snapshot Provider](https://snapshots.polkadot.io/){target=\_blank}. Follow these steps to download and use snapshots: 1. Create new directories with the following commands: @@ -88,34 +89,72 @@ Select the best option for your project, then use the steps in the following tab mkdir -p my-node-data/chains/asset-hub-polkadot/db mkdir -p my-node-data/chains/polkadot/db ``` - 2. Download the appropriate snapshot using the following commands: + 2. Download the appropriate snapshots using the following commands: + + === "Archive Node" - === "Archive snapshot" + Archive node setup maintains complete parachain history (~600-800 GB total). Download both Asset Hub archive and Relay chain pruned snapshots: - Contains complete history, recommended for RPC with historical data. + **Asset Hub archive snapshot** (~400 GB): ```bash # Check https://snapshots.polkadot.io/ for the latest snapshot URL export SNAPSHOT_URL_ASSET_HUB="https://snapshots.polkadot.io/polkadot-asset-hub-rocksdb-archive/LATEST" rclone copyurl $SNAPSHOT_URL_ASSET_HUB/files.txt files.txt rclone copy --progress --transfers 20 \ - --http-url $SNAPSHOT_URL_ASSET_HUB \ - --no-traverse --http-no-head --disable-http2 \ - --inplace --no-gzip-encoding --size-only \ - --retries 6 --retries-sleep 10s \ - --files-from files.txt :http: my-node-data/chains/asset-hub-polkadot/db/ + --http-url $SNAPSHOT_URL_ASSET_HUB \ + --no-traverse --http-no-head --disable-http2 \ + --inplace --no-gzip-encoding --size-only \ + --retries 6 --retries-sleep 10s \ + --files-from files.txt :http: my-node-data/chains/asset-hub-polkadot/db/ rm files.txt ``` - - `--transfers 20`: Uses 20 parallel transfers for faster download. - - `--retries 6`: Automatically retries failed transfers up to 6 times. - - `--retries-sleep 10s`: Waits 10 seconds between retry attempts. - - `--size-only`: Only transfers if sizes differ (prevents unnecessary re-downloads). + **Relay chain pruned snapshot** (~200 GB): + ```bash + # Check https://snapshots.polkadot.io/ for the latest snapshot URL + export SNAPSHOT_URL_RELAY="https://snapshots.polkadot.io/polkadot-rocksdb-prune/LATEST" - === "Pruned snapshot" + rclone copyurl $SNAPSHOT_URL_RELAY/files.txt files.txt + rclone copy --progress --transfers 20 \ + --http-url $SNAPSHOT_URL_RELAY \ + --no-traverse --http-no-head --disable-http2 \ + --inplace --no-gzip-encoding --size-only \ + --retries 6 --retries-sleep 10s \ + --files-from files.txt :http: my-node-data/chains/polkadot/db/ - Contains recent state for a smaller package size, recommended for RPC nodes. + rm files.txt + ``` + + **rclone parameters:** + + - `--transfers 20`: Uses 20 parallel transfers for faster download + - `--retries 6`: Automatically retries failed transfers up to 6 times + - `--retries-sleep 10s`: Waits 10 seconds between retry attempts + - `--size-only`: Only transfers if sizes differ (prevents unnecessary re-downloads) + + === "Pruned Node" + + Pruned node setup keeps recent state for smaller storage (~500 GB total). Download both Asset Hub pruned and Relay chain pruned snapshots: + + **Asset Hub pruned snapshot**: + ```bash + # Check https://snapshots.polkadot.io/ for the latest snapshot URL + export SNAPSHOT_URL_ASSET_HUB="https://snapshots.polkadot.io/polkadot-asset-hub-rocksdb-prune/LATEST" + + rclone copyurl $SNAPSHOT_URL_ASSET_HUB/files.txt files.txt + rclone copy --progress --transfers 20 \ + --http-url $SNAPSHOT_URL_ASSET_HUB \ + --no-traverse --http-no-head --disable-http2 \ + --inplace --no-gzip-encoding --size-only \ + --retries 6 --retries-sleep 10s \ + --files-from files.txt :http: my-node-data/chains/asset-hub-polkadot/db/ + + rm files.txt + ``` + + **Relay chain pruned snapshot** (~200 GB): ```bash # Check https://snapshots.polkadot.io/ for the latest snapshot URL export SNAPSHOT_URL_RELAY="https://snapshots.polkadot.io/polkadot-rocksdb-prune/LATEST" @@ -128,37 +167,82 @@ Select the best option for your project, then use the steps in the following tab --retries 6 --retries-sleep 10s \ --files-from files.txt :http: my-node-data/chains/polkadot/db/ - rm files.txt + rm files.txt ``` - 3. Launch Polkadot Hub Node using the official [Parity Docker image](https://hub.docker.com/r/parity/polkadot-parachain){target=\_blank} with the following command: - ```bash - docker run -d --name polkadot-hub-rpc --restart unless-stopped \ - -p 9944:9944 \ - -p 9933:9933 \ - -p 9615:9615 \ - -p 30334:30334 \ - -p 30333:30333 \ - -v $(pwd)/asset-hub-polkadot.json:/asset-hub-polkadot.json \ - -v $(pwd)/my-node-data:/data \ - parity/polkadot-parachain:stable2509-2 \ - --name=PolkadotHubRPC \ - --base-path=/data \ - --chain=/asset-hub-polkadot.json \ - --prometheus-external \ - --prometheus-port 9615 \ - --unsafe-rpc-external \ - --rpc-port=9944 \ - --rpc-cors=all \ - --rpc-methods=safe \ - --rpc-max-connections=1000 \ - --state-pruning=archive \ - --blocks-pruning=archive \ - -- \ - --base-path=/data \ - --chain=polkadot \ - --state-pruning=256 \ - --blocks-pruning=archive-canonical - ``` + + **rclone parameters:** + + - `--transfers 20`: Uses 20 parallel transfers for faster download + - `--retries 6`: Automatically retries failed transfers up to 6 times + - `--retries-sleep 10s`: Waits 10 seconds between retry attempts + - `--size-only`: Only transfers if sizes differ (prevents unnecessary re-downloads) + 3. Launch Polkadot Hub Node using the official [Parity Docker image](https://hub.docker.com/r/parity/polkadot-parachain){target=\_blank}: + + === "Archive Node" + + Archive node configuration maintains complete parachain history for historical queries: + + ```bash + docker run -d --name polkadot-hub-rpc --restart unless-stopped \ + -p 9944:9944 \ + -p 9933:9933 \ + -p 9615:9615 \ + -p 30334:30334 \ + -p 30333:30333 \ + -v $(pwd)/asset-hub-polkadot.json:/asset-hub-polkadot.json \ + -v $(pwd)/my-node-data:/data \ + parity/polkadot-parachain:stable2509-2 \ + --name=PolkadotHubRPC \ + --base-path=/data \ + --chain=/asset-hub-polkadot.json \ + --prometheus-external \ + --prometheus-port 9615 \ + --unsafe-rpc-external \ + --rpc-port=9944 \ + --rpc-cors=all \ + --rpc-methods=safe \ + --rpc-max-connections=1000 \ + --state-pruning=archive \ + --blocks-pruning=archive \ + -- \ + --base-path=/data \ + --chain=polkadot \ + --state-pruning=256 \ + --blocks-pruning=archive-canonical + ``` + + === "Pruned Node" + + Pruned node configuration keeps recent state for smaller storage requirements: + + ```bash + docker run -d --name polkadot-hub-rpc --restart unless-stopped \ + -p 9944:9944 \ + -p 9933:9933 \ + -p 9615:9615 \ + -p 30334:30334 \ + -p 30333:30333 \ + -v $(pwd)/asset-hub-polkadot.json:/asset-hub-polkadot.json \ + -v $(pwd)/my-node-data:/data \ + parity/polkadot-parachain:stable2509-2 \ + --name=PolkadotHubRPC \ + --base-path=/data \ + --chain=/asset-hub-polkadot.json \ + --prometheus-external \ + --prometheus-port 9615 \ + --unsafe-rpc-external \ + --rpc-port=9944 \ + --rpc-cors=all \ + --rpc-methods=safe \ + --rpc-max-connections=1000 \ + --state-pruning=1000 \ + --blocks-pruning=256 \ + -- \ + --base-path=/data \ + --chain=polkadot \ + --state-pruning=256 \ + --blocks-pruning=archive-canonical + ``` Critical configuration parameters include port mappings and node parameters: @@ -174,8 +258,8 @@ Select the best option for your project, then use the steps in the following tab - `--unsafe-rpc-external`: Enables external RPC access - `--rpc-cors=all`: Allows all origins for CORS - `--rpc-methods=safe`: Only allows safe RPC methods - - `--state-pruning=archive`: Keeps complete state history - - `--blocks-pruning=archive`: Keeps all block data + - `--state-pruning=archive` or `--state-pruning=1000`: Archive keeps complete state history, pruned keeps last 1000 blocks + - `--blocks-pruning=archive` or `--blocks-pruning=256`: Archive keeps all blocks, pruned keeps last 256 finalized blocks - `--prometheus-external`: Exposes metrics externally !!! warning @@ -183,6 +267,7 @@ Select the best option for your project, then use the steps in the following tab The `--unsafe-rpc-external` flag should only be used in development or properly secured environments. For production, use a reverse proxy with authentication. 4. Monitor the node synchronization status using the following command: + ```bash curl -H "Content-Type: application/json" \ -d '{"id":1, "jsonrpc":"2.0", "method": "system_syncState", "params":[]}' \ @@ -190,7 +275,7 @@ Select the best option for your project, then use the steps in the following tab ``` You should see a response similar to the following: - + ```json { "jsonrpc":"2.0", @@ -205,20 +290,8 @@ Select the best option for your project, then use the steps in the following tab When synchronization is complete, `currentBlock` will be equal to `highestBlock`. - 5. You can use a few different commands to verify your node is running properly: + 5. You can use the `system_health` command to verify your node is running properly: - - Get chain information: - ```bash - curl -H "Content-Type: application/json" \ - -d '{"id":1, "jsonrpc":"2.0", "method": "system_chain", "params":[]}' \ - http://localhost:9944 - ``` - - Get the latest block: - ```bash - curl -H "Content-Type: application/json" \ - -d '{"id":1, "jsonrpc":"2.0", "method": "chain_getHeader", "params":[]}' \ - http://localhost:9944 - ``` - Query node health: ```bash curl -H "Content-Type: application/json" \ @@ -290,46 +363,95 @@ Select the best option for your project, then use the steps in the following tab sudo nano /etc/systemd/system/polkadot-hub-rpc.service ``` - 5. Open the new service file and add the following configuration: - ```ini - [Unit] - Description=Polkadot Hub RPC Node - After=network.target - - [Service] - Type=simple - User=polkadot - Group=polkadot - WorkingDirectory=/var/lib/polkadot-hub-rpc - - ExecStart=/usr/local/bin/polkadot-parachain \ - --name=PolkadotHubRPC \ - --chain=/var/lib/polkadot-hub-rpc/asset-hub-polkadot.json \ - --base-path=/var/lib/polkadot-hub-rpc \ - --port=30333 \ - --rpc-port=9944 \ - --rpc-external \ - --rpc-cors=all \ - --rpc-methods=safe \ - --rpc-max-connections=1000 \ - --prometheus-port=9615 \ - --prometheus-external \ - --state-pruning=archive \ - --blocks-pruning=archive \ - -- \ - --chain=polkadot \ - --base-path=/var/lib/polkadot-hub-rpc \ - --port=30334 \ - --state-pruning=256 \ - --blocks-pruning=archive-canonical - - Restart=always - RestartSec=10 - LimitNOFILE=65536 - - [Install] - WantedBy=multi-user.target - ``` + 5. Open the new service file and add the configuration for your chosen node type: + + === "Archive Node" + + Archive node configuration maintains complete parachain history for historical queries: + + ```ini + [Unit] + Description=Polkadot Hub RPC Node + After=network.target + + [Service] + Type=simple + User=polkadot + Group=polkadot + WorkingDirectory=/var/lib/polkadot-hub-rpc + + ExecStart=/usr/local/bin/polkadot-parachain \ + --name=PolkadotHubRPC \ + --chain=/var/lib/polkadot-hub-rpc/asset-hub-polkadot.json \ + --base-path=/var/lib/polkadot-hub-rpc \ + --port=30333 \ + --rpc-port=9944 \ + --rpc-external \ + --rpc-cors=all \ + --rpc-methods=safe \ + --rpc-max-connections=1000 \ + --prometheus-port=9615 \ + --prometheus-external \ + --state-pruning=archive \ + --blocks-pruning=archive \ + -- \ + --chain=polkadot \ + --base-path=/var/lib/polkadot-hub-rpc \ + --port=30334 \ + --state-pruning=256 \ + --blocks-pruning=archive-canonical + + Restart=always + RestartSec=10 + LimitNOFILE=65536 + + [Install] + WantedBy=multi-user.target + ``` + + === "Pruned Node" + + Pruned node configuration keeps recent state for smaller storage requirements: + + ```ini + [Unit] + Description=Polkadot Hub RPC Node + After=network.target + + [Service] + Type=simple + User=polkadot + Group=polkadot + WorkingDirectory=/var/lib/polkadot-hub-rpc + + ExecStart=/usr/local/bin/polkadot-parachain \ + --name=PolkadotHubRPC \ + --chain=/var/lib/polkadot-hub-rpc/asset-hub-polkadot.json \ + --base-path=/var/lib/polkadot-hub-rpc \ + --port=30333 \ + --rpc-port=9944 \ + --rpc-external \ + --rpc-cors=all \ + --rpc-methods=safe \ + --rpc-max-connections=1000 \ + --prometheus-port=9615 \ + --prometheus-external \ + --state-pruning=1000 \ + --blocks-pruning=256 \ + -- \ + --chain=polkadot \ + --base-path=/var/lib/polkadot-hub-rpc \ + --port=30334 \ + --state-pruning=256 \ + --blocks-pruning=archive-canonical + + Restart=always + RestartSec=10 + LimitNOFILE=65536 + + [Install] + WantedBy=multi-user.target + ``` 6. Start the service using the following commands: - Reload systemd: @@ -370,117 +492,6 @@ Select the best option for your project, then use the steps in the following tab http://localhost:9944 ``` -## Monitor and Maintain RPC Node - -There are a few key monitoring and maintenance commands that can help you ensure RPC node uptime and performance including log monitoring and metrics. - -To view node logs, use the appropriate command for your setup: - -=== "Docker Setup" - - ```bash - docker logs -f polkadot-hub-rpc - ``` - -=== "systemd Setup" - - - View node logs: - ```bash - sudo journalctl -u polkadot-hub-rpc -f - ``` - - View recent logs: - ```bash - sudo journalctl -u polkadot-hub-rpc -n 100 - ``` - - Filter for errors: - ```bash - sudo journalctl -u polkadot-hub-rpc | grep -i error - ``` - -Monitoring key metrics like the following items can help you stay up to date on the performance and health of your node and allow you to intervene at the first sign of issues: - -- **Sync status**: Ensure node stays fully synced. -- **Peer connections**: Maintain 30+ peers for good connectivity. -- **Resource usage**: Monitor CPU, RAM, and disk I/O. -- **RPC request latency**: Track response times for the Polkadot SDK API. -- **Connection count**: Monitor active RPC connections. - -You can use the following information to configure [Prometheus](https://prometheus.io/docs/introduction/first_steps/){target=\_blank} to monitor, collect, and store your RPC node metrics: - -- **URL**: Metrics are available to view at `http://localhost:9615/metrics` -- **Example Prometheus configuration**: Update your `prometheus.yml` to add the following code: - {% raw %} - ```yaml - scrape_configs: - - job_name: 'polkadot-hub-rpc' - static_configs: - - targets: ['localhost:9615'] - ``` - {% endraw %} - -Key metrics to monitor via Prometheus include: - -- `substrate_block_height`: Current block height -- `substrate_finalized_height`: Finalized block height -- `substrate_peers_count`: Number of connected peers -- `substrate_ready_transactions_number`: Transaction queue size - -### Database Maintenance - -Check database size periodically using the commands for your selected setup: - -=== "Docker Setup" - - ```bash - du -sh my-node-data - ``` - -=== "systemd Setup" - - ```bash - du -sh /var/lib/polkadot-hub-rpc - ``` - -The node handles pruning automatically based on configuration unless running in archive mode. - -### Updates and Upgrades - -Use the following commands for updating or upgrading your RPC node according to your setup: - -=== "Docker Setup" - - 1. Stop and remove the existing container: - ```bash - docker stop polkadot-hub-rpc - docker rm polkadot-hub-rpc - ``` - 2. Pull the latest image: - ```bash - docker pull parity/polkadot-parachain: - ``` - 3. Start the new container using the same command from the setup section with the updated image tag. - -=== "systemd Setup" - - 1. Stop the service: - ```bash - sudo systemctl stop polkadot-hub-rpc - ``` - 2. Backup data: - ```bash - sudo cp -r /var/lib/polkadot-hub-rpc /var/lib/polkadot-hub-rpc.backup - ``` - 3. Download the new binary from [GitHub releases](https://github.com/paritytech/polkadot-sdk/releases){target=\_blank}: - ```bash - wget https://github.com/paritytech/polkadot-sdk/releases/download//polkadot-parachain - chmod +x polkadot-parachain - sudo mv polkadot-parachain /usr/local/bin/ - ``` - 4. Restart the service: - ```bash - sudo systemctl start polkadot-hub-rpc - ``` - ## Conclusion Running an RPC node for Polkadot Hub provides essential infrastructure for applications and users to interact with the network. By following this guide, you have set up a production-ready RPC node that: diff --git a/node-infrastructure/overview.md b/node-infrastructure/overview.md index 13098d5ad..4b06c00ee 100644 --- a/node-infrastructure/overview.md +++ b/node-infrastructure/overview.md @@ -23,7 +23,12 @@ RPC nodes provide API access to blockchain data without participating in consens - **Wallets**: Check balances and broadcast transactions - **Development**: Test and debug applications -RPC nodes can be run for both the relay chain and parachains, with varying levels of data retention (pruned vs archive). +RPC nodes can be run for both the relay chain and parachains, with varying levels of data retention: + +- **Pruned Nodes**: Keep recent state and a limited number of finalized blocks. Suitable for most applications that only need current state and recent history. More efficient in terms of storage and sync time. +- **Archive Nodes**: Maintain complete historical state and all blocks since genesis. Required for block explorers, analytics platforms, or applications that need to query historical data at any point in time. + +**Transaction Broadcasting**: RPC nodes play a crucial role in transaction submission and propagation. When a client submits a transaction via RPC methods like `author_submitExtrinsic`, the node validates the transaction format, adds it to its local transaction pool, and broadcasts it across the P2P network. Block producers (collators or validators) then pick up these transactions from their pools for inclusion in blocks. This makes RPC nodes the primary gateway for users and applications to interact with the blockchain. ### Collators @@ -31,17 +36,17 @@ Collators are block producers for parachains. They perform critical functions: - **Collect transactions**: Aggregate user transactions into blocks - **Produce blocks**: Create parachain block candidates -- **Generate proofs**: Produce state transition proofs (Proof-of-Validity) -- **Submit to validators**: Send block candidates to relay chain validators +- **Generate and package PoV**: Generate the Proof-of-Validity containing the state transition proof and necessary witness data for validation +- **Submit to validators**: Send block candidates and PoVs to relay chain validators Unlike validators, collators do not provide security guarantees—that responsibility lies with relay chain validators. However, collators are essential for parachain liveness and censorship resistance. ### Validators -Validators secure the Polkadot relay chain through Nominated Proof of Stake (NPoS). They: +Validators secure the Polkadot relay chain through [Nominated Proof of Stake (NPoS)](https://wiki.polkadot.network/docs/learn-staking){target=\_blank}. They: - **Validate blocks**: Verify parachain blocks and relay chain transactions -- **Participate in consensus**: Run BABE and GRANDPA protocols +- **Participate in consensus**: Run [BABE](https://wiki.polkadot.network/docs/learn-consensus#babe-block-production){target=\_blank} and [GRANDPA](https://wiki.polkadot.network/docs/learn-consensus#grandpa-finality-gadget){target=\_blank} protocols - **Earn rewards**: Receive staking rewards for honest behavior - **Risk slashing**: Face penalties for misbehavior or downtime diff --git a/node-infrastructure/run-a-collator/collator.md b/node-infrastructure/run-a-collator/collator.md index 2560584d1..778d6aea2 100644 --- a/node-infrastructure/run-a-collator/collator.md +++ b/node-infrastructure/run-a-collator/collator.md @@ -30,7 +30,7 @@ Block-producing collators perform critical functions: - Send block candidates to relay chain validators. - Enable cross-chain message passing using XCM -Unlike relay chain validators, collators do not provide security guarantees—that responsibility lies with relay chain validators through the ELVES protocol. Rather, collators are essential for network liveness and censorship resistance. +Unlike relay chain validators, collators do not provide security guarantees—that responsibility lies with relay chain validators through the [ELVES protocol](https://wiki.polkadot.com/learn/learn-parachains-protocol/){target=\_blank}. Rather, collators are essential for network liveness and censorship resistance. ## Prerequisites @@ -56,7 +56,7 @@ Uptime is critical. Consider redundancy and monitoring to maintain block product ### Software Requirements -Collators use the **Polkadot Parachain** binary, the standard client for running Polkadot system parachains. +Collators use the Polkadot Parachain binary, the standard client for running Polkadot system parachains. Required software includes the following: @@ -142,6 +142,17 @@ Generate an account for on-chain transactions as follows: docker run -it parity/subkey:latest generate --scheme sr25519 ``` + The output will be similar to the following: + ```bash + Secret phrase: embody rail hour peanut .... badge syrup luggage canvas + Network ID: substrate + Secret seed: 0x6498dd3416c491406e2c8283c76760ce4ca018478888b42315e7718778f2c2e1 + Public key (hex): 0x2202210357e49390d4f8d868da983940fe220a0a0e00bc6feaeda462aa031810 + Account ID: 0x2202210357e49390d4f8d868da983940fe220a0a0e00bc6feaeda462aa031810 + Public key (SS58): 5CqJ7n72GvvF5ZzUT2HMj83KyDje4n8sXR8kuiK8HWtfDktF + SS58 Address: 5CqJ7n72GvvF5ZzUT2HMj83KyDje4n8sXR8kuiK8HWtfDktF + ``` + 2. Save the following items displayed in the output: - Secret phrase (seed) - Keep this secure! - Public key (hex) @@ -160,7 +171,7 @@ Download the chain specification for your target system parachain using one of t Follow these steps to download your specification from the Chainspec Collection: -1. Visit the [Chainspec Collection](https://paritytech.github.io/chainspecs/) website. +1. Visit the [Chainspec Collection](https://paritytech.github.io/chainspecs/){target=\_blank} website. 2. Find your target system parachain. 3. Download the chain specification JSON file. 4. Save it as `chain-spec.json`. @@ -314,6 +325,8 @@ The relay chain uses warp sync for faster synchronization. ## Generate Session Keys +Session keys are cryptographic keys used by your collator node to sign authorship information when producing blocks. They uniquely identify your collator on the network and must be registered on-chain before your collator can participate in block production. + Once your node is fully synced, use the following command to generate session keys via RPC: ```bash @@ -337,7 +350,7 @@ System parachains use different collator selection mechanisms. Explore the follo === "On-chain Selection" - - Some parachains use pallet-collator-selection. + - Some parachains use [pallet-collator-selection](https://paritytech.github.io/polkadot-sdk/master/pallet_collator_selection/index.html){target=\_blank}. - May require bonding tokens. - Automatic selection based on criteria. @@ -348,6 +361,8 @@ System parachains use different collator selection mechanisms. Explore the follo ### Registration Process +Collator registration authorizes your node to produce blocks on the network. The parachain's collator selection mechanism uses this on-chain registration to determine which nodes are eligible to author blocks. + The registration process varies by system parachain. General steps include the following: 1. Check the existing collators for your target parachain: @@ -374,59 +389,19 @@ The registration process varies by system parachain. General steps include the f - `proof`: 0x00 (typically) 6. Submit and sign the transaction. -4. If the parachain requires bonding tokens, use the follow these steps to submit them using Polkadot.js Apps: +4. (Optional - primarily for non-system parachains) If the parachain uses on-chain bonding for collator selection, register as a candidate using Polkadot.js Apps: + + !!! note + Most system parachains use invulnerables lists exclusively and do not require this step. Skip to step 5 if you're running a collator for a system parachain. + 1. Locate **Developer > Extrinsics**. 2. Select `collatorSelection.registerAsCandidate`. - 3. Submit the transaction with the required bond amount. - 4. Sign the transaction. + 3. Submit and sign the transaction. The required bond amount will be automatically reserved from your account based on the pallet's configured `CandidacyBond`. -5. If applying to join the invulnerables list, you must now await governance approval for your proposal. Monitor the [Polkadot Forum](https://forum.polkadot.network){target=\_blank} governance channels and announcements. Once approved, your collator is added to the invulnerables list and will begin producing blocks in the next session or era. +5. For system parachains using invulnerables lists, await governance approval for your proposal. Monitor the [Polkadot Forum](https://forum.polkadot.network){target=\_blank} governance channels and announcements. Once approved, your collator is added to the invulnerables list and will begin producing blocks in the next session or era. 6. Verify your collator is active by monitoring logs for block production messages like "Prepared block for proposing" and "Imported #123". See the [Log Management](#log-management) section for log viewing commands. -## Monitor and Maintain Your Collator - -Monitoring the following items will help ensure your collator runs efficiently to avoid service interruptions or down time: - -- **Block Production**: Monitor for block production using the appropriate command for your setup: - - === "Docker Setup" - - ```bash - docker logs polkadot-collator 2>&1 | grep -i "prepared block" - ``` - - === "systemd Setup" - - ```bash - sudo journalctl -u polkadot-collator | grep -i "prepared block" - ``` - -- **Peer Connections**: Maintain a sufficient amount of peers for good connectivity and set up alerts to notify if peer connections falls below ten. -- **Resource Usage**: Monitor CPU, RAM, and disk I/O and set up alerts for unusual or high usage. -- **Sync Status**: Ensure both chains stay synced and set up alerts for sync issues. - -### Prometheus Metrics - -You can use the following information to configure [Prometheus](https://prometheus.io/docs/introduction/first_steps/){target=\_blank} to monitor, collect, and store your collator node metrics: - -- **URL**: Metrics are available to view at `http://localhost:9615/metrics` -- **Example Prometheus configuration**: Update your `prometheus.yml` to add the following code: - ```yaml - scrape_configs: - - job_name: 'polkadot-collator' - static_configs: - - targets: ['localhost:9615'] - ``` - -Key metrics to monitor via Prometheus include the following: - -- `substrate_block_height`: Current block height -- `substrate_finalized_height`: Finalized block height -- `substrate_peers_count`: Peer connections -- `substrate_ready_transactions_number`: Transaction queue - - ## Log Management Efficient log management is essential to ensure collator performance and uptime. Use the following commands to help you manage logs to monitor and maintain your collator: diff --git a/node-infrastructure/run-a-node/parachain-rpc.md b/node-infrastructure/run-a-node/parachain-rpc.md index e857f6ec6..3dbbcf988 100644 --- a/node-infrastructure/run-a-node/parachain-rpc.md +++ b/node-infrastructure/run-a-node/parachain-rpc.md @@ -60,7 +60,7 @@ RPC nodes serving production traffic require robust hardware: - **CPU**: 8+ cores (16+ cores for high traffic) - **Memory**: 64 GB RAM minimum (128 GB recommended for high traffic) - **Storage**: - - 500 GB+ NVMe SSD for parachain state (archive nodes require 2-4 TB+) + - Storage requirements vary by parachain. System parachains: Asset Hub (~600-800 GB), Bridge Hub (~500-600 GB), Collectives (~400-500 GB), People Chain (~300-400 GB), Coretime (~300-400 GB). For non-system parachains, check the [snapshot sizes](https://snapshots.polkadot.io/){target=\_blank} if available. - Additional 200+ GB for relay chain pruned database - Fast disk I/O is critical for query performance - **Network**: @@ -115,11 +115,13 @@ Select the best option for your project, then use the steps in the following tab mkdir -p my-node-data/chains/people-polkadot/db mkdir -p my-node-data/chains/polkadot/db ``` - 2. Download the appropriate snapshot using the following commands: + 2. Download the appropriate snapshots using the following commands: - === "Archive snapshot" + === "Archive Node" - Contains complete history, recommended for RPC with historical data. + Archive node setup maintains complete parachain history. Download both parachain archive and relay chain pruned snapshots: + + **Parachain archive snapshot** (People Chain example): ```bash # Check https://snapshots.polkadot.io/ for the latest snapshot URL export SNAPSHOT_URL_PARACHAIN="https://snapshots.polkadot.io/polkadot-people-rocksdb-archive/LATEST" @@ -135,14 +137,50 @@ Select the best option for your project, then use the steps in the following tab rm files.txt ``` - - `--transfers 20`: Uses 20 parallel transfers for faster download. - - `--retries 6`: Automatically retries failed transfers up to 6 times. - - `--retries-sleep 10s`: Waits 10 seconds between retry attempts. - - `--size-only`: Only transfers if sizes differ (prevents unnecessary re-downloads). + **Relay chain pruned snapshot** (~200 GB): + ```bash + # Check https://snapshots.polkadot.io/ for the latest snapshot URL + export SNAPSHOT_URL_RELAY="https://snapshots.polkadot.io/polkadot-rocksdb-prune/LATEST" + + rclone copyurl $SNAPSHOT_URL_RELAY/files.txt files.txt + rclone copy --progress --transfers 20 \ + --http-url $SNAPSHOT_URL_RELAY \ + --no-traverse --http-no-head --disable-http2 \ + --inplace --no-gzip-encoding --size-only \ + --retries 6 --retries-sleep 10s \ + --files-from files.txt :http: my-node-data/chains/polkadot/db/ + + rm files.txt + ``` + + **rclone parameters:** + + - `--transfers 20`: Uses 20 parallel transfers for faster download + - `--retries 6`: Automatically retries failed transfers up to 6 times + - `--retries-sleep 10s`: Waits 10 seconds between retry attempts + - `--size-only`: Only transfers if sizes differ (prevents unnecessary re-downloads) + + === "Pruned Node" + + Pruned node setup keeps recent state for smaller storage. Download both parachain pruned and relay chain pruned snapshots: + + **Parachain pruned snapshot** (People Chain example): + ```bash + # Check https://snapshots.polkadot.io/ for the latest snapshot URL + export SNAPSHOT_URL_PARACHAIN="https://snapshots.polkadot.io/polkadot-people-rocksdb-prune/LATEST" + + rclone copyurl $SNAPSHOT_URL_PARACHAIN/files.txt files.txt + rclone copy --progress --transfers 20 \ + --http-url $SNAPSHOT_URL_PARACHAIN \ + --no-traverse --http-no-head --disable-http2 \ + --inplace --no-gzip-encoding --size-only \ + --retries 6 --retries-sleep 10s \ + --files-from files.txt :http: my-node-data/chains/people-polkadot/db/ - === "Pruned snapshot" + rm files.txt + ``` - Contains recent state for a smaller package size, recommended for RPC nodes. + **Relay chain pruned snapshot** (~200 GB): ```bash # Check https://snapshots.polkadot.io/ for the latest snapshot URL export SNAPSHOT_URL_RELAY="https://snapshots.polkadot.io/polkadot-rocksdb-prune/LATEST" @@ -158,35 +196,80 @@ Select the best option for your project, then use the steps in the following tab rm files.txt ``` - 3. Launch the parachain node using the official [Parity Docker image](https://hub.docker.com/r/parity/polkadot-parachain){target=\_blank} with the following command: - ```bash - docker run -d --name people-chain-rpc --restart unless-stopped \ - -p 9944:9944 \ - -p 9933:9933 \ - -p 9615:9615 \ - -p 30334:30334 \ - -p 30333:30333 \ - -v $(pwd)/people-polkadot.json:/people-polkadot.json \ - -v $(pwd)/my-node-data:/data \ - parity/polkadot-parachain:stable2509-2 \ - --name=PeopleChainRPC \ - --base-path=/data \ - --chain=/people-polkadot.json \ - --prometheus-external \ - --prometheus-port 9615 \ - --unsafe-rpc-external \ - --rpc-port=9944 \ - --rpc-cors=all \ - --rpc-methods=safe \ - --rpc-max-connections=1000 \ - --state-pruning=archive \ - --blocks-pruning=archive \ - -- \ - --base-path=/data \ - --chain=polkadot \ - --state-pruning=256 \ - --blocks-pruning=archive-canonical - ``` + **rclone parameters:** + + - `--transfers 20`: Uses 20 parallel transfers for faster download + - `--retries 6`: Automatically retries failed transfers up to 6 times + - `--retries-sleep 10s`: Waits 10 seconds between retry attempts + - `--size-only`: Only transfers if sizes differ (prevents unnecessary re-downloads) + + 3. Launch the parachain node using the official [Parity Docker image](https://hub.docker.com/r/parity/polkadot-parachain){target=\_blank}: + + === "Archive Node" + + Archive node configuration maintains complete parachain history for historical queries: + + ```bash + docker run -d --name people-chain-rpc --restart unless-stopped \ + -p 9944:9944 \ + -p 9933:9933 \ + -p 9615:9615 \ + -p 30334:30334 \ + -p 30333:30333 \ + -v $(pwd)/people-polkadot.json:/people-polkadot.json \ + -v $(pwd)/my-node-data:/data \ + parity/polkadot-parachain:stable2509-2 \ + --name=PeopleChainRPC \ + --base-path=/data \ + --chain=/people-polkadot.json \ + --prometheus-external \ + --prometheus-port 9615 \ + --unsafe-rpc-external \ + --rpc-port=9944 \ + --rpc-cors=all \ + --rpc-methods=safe \ + --rpc-max-connections=1000 \ + --state-pruning=archive \ + --blocks-pruning=archive \ + -- \ + --base-path=/data \ + --chain=polkadot \ + --state-pruning=256 \ + --blocks-pruning=archive-canonical + ``` + + === "Pruned Node" + + Pruned node configuration keeps recent state for smaller storage requirements: + + ```bash + docker run -d --name people-chain-rpc --restart unless-stopped \ + -p 9944:9944 \ + -p 9933:9933 \ + -p 9615:9615 \ + -p 30334:30334 \ + -p 30333:30333 \ + -v $(pwd)/people-polkadot.json:/people-polkadot.json \ + -v $(pwd)/my-node-data:/data \ + parity/polkadot-parachain:stable2509-2 \ + --name=PeopleChainRPC \ + --base-path=/data \ + --chain=/people-polkadot.json \ + --prometheus-external \ + --prometheus-port 9615 \ + --unsafe-rpc-external \ + --rpc-port=9944 \ + --rpc-cors=all \ + --rpc-methods=safe \ + --rpc-max-connections=1000 \ + --state-pruning=1000 \ + --blocks-pruning=256 \ + -- \ + --base-path=/data \ + --chain=polkadot \ + --state-pruning=256 \ + --blocks-pruning=archive-canonical + ``` !!! note @@ -206,8 +289,8 @@ Select the best option for your project, then use the steps in the following tab - `--unsafe-rpc-external`: Enables external RPC access - `--rpc-cors=all`: Allows all origins for CORS - `--rpc-methods=safe`: Only allows safe RPC methods - - `--state-pruning=archive`: Keeps complete state history - - `--blocks-pruning=archive`: Keeps all block data + - `--state-pruning=archive` or `--state-pruning=1000`: Archive keeps complete state history, pruned keeps last 1000 blocks + - `--blocks-pruning=archive` or `--blocks-pruning=256`: Archive keeps all blocks, pruned keeps last 256 finalized blocks - `--prometheus-external`: Exposes metrics externally !!! warning @@ -321,46 +404,95 @@ Select the best option for your project, then use the steps in the following tab sudo nano /etc/systemd/system/people-chain-rpc.service ``` - 5. Open the new service file and add the following configuration: - ```ini - [Unit] - Description=People Chain RPC Node - After=network.target - - [Service] - Type=simple - User=polkadot - Group=polkadot - WorkingDirectory=/var/lib/people-chain-rpc - - ExecStart=/usr/local/bin/polkadot-parachain \ - --name=PeopleChainRPC \ - --chain=/var/lib/people-chain-rpc/people-polkadot.json \ - --base-path=/var/lib/people-chain-rpc \ - --port=30333 \ - --rpc-port=9944 \ - --rpc-external \ - --rpc-cors=all \ - --rpc-methods=safe \ - --rpc-max-connections=1000 \ - --prometheus-port=9615 \ - --prometheus-external \ - --state-pruning=archive \ - --blocks-pruning=archive \ - -- \ - --chain=polkadot \ - --base-path=/var/lib/people-chain-rpc \ - --port=30334 \ - --state-pruning=256 \ - --blocks-pruning=archive-canonical - - Restart=always - RestartSec=10 - LimitNOFILE=65536 - - [Install] - WantedBy=multi-user.target - ``` + 5. Open the new service file and add the configuration for your chosen node type: + + === "Archive Node" + + Archive node configuration maintains complete parachain history for historical queries: + + ```ini + [Unit] + Description=People Chain RPC Node + After=network.target + + [Service] + Type=simple + User=polkadot + Group=polkadot + WorkingDirectory=/var/lib/people-chain-rpc + + ExecStart=/usr/local/bin/polkadot-parachain \ + --name=PeopleChainRPC \ + --chain=/var/lib/people-chain-rpc/people-polkadot.json \ + --base-path=/var/lib/people-chain-rpc \ + --port=30333 \ + --rpc-port=9944 \ + --rpc-external \ + --rpc-cors=all \ + --rpc-methods=safe \ + --rpc-max-connections=1000 \ + --prometheus-port=9615 \ + --prometheus-external \ + --state-pruning=archive \ + --blocks-pruning=archive \ + -- \ + --chain=polkadot \ + --base-path=/var/lib/people-chain-rpc \ + --port=30334 \ + --state-pruning=256 \ + --blocks-pruning=archive-canonical + + Restart=always + RestartSec=10 + LimitNOFILE=65536 + + [Install] + WantedBy=multi-user.target + ``` + + === "Pruned Node" + + Pruned node configuration keeps recent state for smaller storage requirements: + + ```ini + [Unit] + Description=People Chain RPC Node + After=network.target + + [Service] + Type=simple + User=polkadot + Group=polkadot + WorkingDirectory=/var/lib/people-chain-rpc + + ExecStart=/usr/local/bin/polkadot-parachain \ + --name=PeopleChainRPC \ + --chain=/var/lib/people-chain-rpc/people-polkadot.json \ + --base-path=/var/lib/people-chain-rpc \ + --port=30333 \ + --rpc-port=9944 \ + --rpc-external \ + --rpc-cors=all \ + --rpc-methods=safe \ + --rpc-max-connections=1000 \ + --prometheus-port=9615 \ + --prometheus-external \ + --state-pruning=1000 \ + --blocks-pruning=256 \ + -- \ + --chain=polkadot \ + --base-path=/var/lib/people-chain-rpc \ + --port=30334 \ + --state-pruning=256 \ + --blocks-pruning=archive-canonical + + Restart=always + RestartSec=10 + LimitNOFILE=65536 + + [Install] + WantedBy=multi-user.target + ``` 6. Start the service using the following commands: - Reload systemd: @@ -401,115 +533,6 @@ Select the best option for your project, then use the steps in the following tab http://localhost:9944 ``` -## Monitor and Maintain RPC Node - -There are a few key monitoring and maintenance commands that can help you ensure RPC node uptime and performance including log monitoring and metrics. - -To view node logs, use the appropriate command for your setup: - -=== "Docker Setup" - - ```bash - docker logs -f people-chain-rpc - ``` - -=== "systemd Setup" - - - View node logs: - ```bash - sudo journalctl -u people-chain-rpc -f - ``` - - View recent logs: - ```bash - sudo journalctl -u people-chain-rpc -n 100 - ``` - - Filter for errors: - ```bash - sudo journalctl -u people-chain-rpc | grep -i error - ``` - -Monitoring key metrics like the following items can help you stay up to date on the performance and health of your node and allow you to intervene at the first sign of issues: - -- **Sync status**: Ensure node stays fully synced. -- **Peer connections**: Maintain 30+ peers for good connectivity. -- **Resource usage**: Monitor CPU, RAM, and disk I/O. -- **RPC request latency**: Track response times for the Polkadot SDK API. -- **Connection count**: Monitor active RPC connections. - -You can use the following information to configure [Prometheus](https://prometheus.io/docs/introduction/first_steps/){target=\_blank} to monitor, collect, and store your RPC node metrics: - -- **URL**: Metrics are available to view at `http://localhost:9615/metrics` -- **Example Prometheus configuration**: Update your `prometheus.yml` to add the following code: - ```yaml - scrape_configs: - - job_name: 'people-chain-rpc' - static_configs: - - targets: ['localhost:9615'] - ``` - -Key metrics to monitor via Prometheus include: - -- `substrate_block_height`: Current block height -- `substrate_finalized_height`: Finalized block height -- `substrate_peers_count`: Number of connected peers -- `substrate_ready_transactions_number`: Transaction queue size - -### Database Maintenance - -Check database size periodically using the commands for your selected setup: - -=== "Docker Setup" - - ```bash - du -sh my-node-data - ``` - -=== "systemd Setup" - - ```bash - du -sh /var/lib/people-chain-rpc - ``` - -The node handles pruning automatically based on configuration unless running in archive mode. - -### Updates and Upgrades - -Use the following commands for updating or upgrading your RPC node according to your setup: - -=== "Docker Setup" - - 1. Stop and remove the existing container: - ```bash - docker stop people-chain-rpc - docker rm people-chain-rpc - ``` - 2. Pull the latest image: - ```bash - docker pull parity/polkadot-parachain: - ``` - 3. Start the new container using the same command from the setup section with the updated image tag. - -=== "systemd Setup" - - 1. Stop the service: - ```bash - sudo systemctl stop people-chain-rpc - ``` - 2. Backup data: - ```bash - sudo cp -r /var/lib/people-chain-rpc /var/lib/people-chain-rpc.backup - ``` - 3. Download the new binary from [GitHub releases](https://github.com/paritytech/polkadot-sdk/releases){target=\_blank}: - ```bash - wget https://github.com/paritytech/polkadot-sdk/releases/download//polkadot-parachain - chmod +x polkadot-parachain - sudo mv polkadot-parachain /usr/local/bin/ - ``` - 4. Restart the service: - ```bash - sudo systemctl start people-chain-rpc - ``` - ## Conclusion Running a parachain RPC node provides critical infrastructure for accessing Polkadot network services. By following this guide, you have set up a production-ready RPC node that: diff --git a/node-infrastructure/run-a-node/polkadot-hub-rpc.md b/node-infrastructure/run-a-node/polkadot-hub-rpc.md index 9b956861c..cb2048b23 100644 --- a/node-infrastructure/run-a-node/polkadot-hub-rpc.md +++ b/node-infrastructure/run-a-node/polkadot-hub-rpc.md @@ -15,6 +15,7 @@ categories: Infrastructure - **Cross-chain Communication**: XCM message handling Running an RPC node for Polkadot Hub enables applications, wallets, and users to interact with the parachain through: + - **Polkadot SDK Node RPC** (Port 9944): Native Polkadot API (WebSocket and HTTP) This setup enables block explorer indexing and provides full compatibility with Polkadot SDK development tools. @@ -32,7 +33,7 @@ RPC nodes serving production traffic require robust hardware. The following shou - **CPU**: 8+ cores (16+ cores for high traffic) - **Memory**: 64 GB RAM minimum (128 GB recommended for high traffic) - **Storage**: - - 500 GB+ NVMe SSD for parachain state (archive nodes require 2-4 TB+) + - 500 GB+ NVMe SSD for parachain state (archive nodes require 600-800 GB+) - Additional 200+ GB for relay chain pruned database - Fast disk I/O is critical for query performance - **Network**: @@ -87,34 +88,72 @@ Select the best option for your project, then use the steps in the following tab mkdir -p my-node-data/chains/asset-hub-polkadot/db mkdir -p my-node-data/chains/polkadot/db ``` - 2. Download the appropriate snapshot using the following commands: + 2. Download the appropriate snapshots using the following commands: + + === "Archive Node" - === "Archive snapshot" + Archive node setup maintains complete parachain history (~600-800 GB total). Download both Asset Hub archive and Relay chain pruned snapshots: - Contains complete history, recommended for RPC with historical data. + **Asset Hub archive snapshot** (~400 GB): ```bash # Check https://snapshots.polkadot.io/ for the latest snapshot URL export SNAPSHOT_URL_ASSET_HUB="https://snapshots.polkadot.io/polkadot-asset-hub-rocksdb-archive/LATEST" rclone copyurl $SNAPSHOT_URL_ASSET_HUB/files.txt files.txt rclone copy --progress --transfers 20 \ - --http-url $SNAPSHOT_URL_ASSET_HUB \ - --no-traverse --http-no-head --disable-http2 \ - --inplace --no-gzip-encoding --size-only \ - --retries 6 --retries-sleep 10s \ - --files-from files.txt :http: my-node-data/chains/asset-hub-polkadot/db/ + --http-url $SNAPSHOT_URL_ASSET_HUB \ + --no-traverse --http-no-head --disable-http2 \ + --inplace --no-gzip-encoding --size-only \ + --retries 6 --retries-sleep 10s \ + --files-from files.txt :http: my-node-data/chains/asset-hub-polkadot/db/ + + rm files.txt + ``` + + **Relay chain pruned snapshot** (~200 GB): + ```bash + # Check https://snapshots.polkadot.io/ for the latest snapshot URL + export SNAPSHOT_URL_RELAY="https://snapshots.polkadot.io/polkadot-rocksdb-prune/LATEST" + + rclone copyurl $SNAPSHOT_URL_RELAY/files.txt files.txt + rclone copy --progress --transfers 20 \ + --http-url $SNAPSHOT_URL_RELAY \ + --no-traverse --http-no-head --disable-http2 \ + --inplace --no-gzip-encoding --size-only \ + --retries 6 --retries-sleep 10s \ + --files-from files.txt :http: my-node-data/chains/polkadot/db/ rm files.txt ``` - - `--transfers 20`: Uses 20 parallel transfers for faster download. - - `--retries 6`: Automatically retries failed transfers up to 6 times. - - `--retries-sleep 10s`: Waits 10 seconds between retry attempts. - - `--size-only`: Only transfers if sizes differ (prevents unnecessary re-downloads). + **rclone parameters:** + + - `--transfers 20`: Uses 20 parallel transfers for faster download + - `--retries 6`: Automatically retries failed transfers up to 6 times + - `--retries-sleep 10s`: Waits 10 seconds between retry attempts + - `--size-only`: Only transfers if sizes differ (prevents unnecessary re-downloads) + + === "Pruned Node" + + Pruned node setup keeps recent state for smaller storage (~500 GB total). Download both Asset Hub pruned and Relay chain pruned snapshots: + + **Asset Hub pruned snapshot**: + ```bash + # Check https://snapshots.polkadot.io/ for the latest snapshot URL + export SNAPSHOT_URL_ASSET_HUB="https://snapshots.polkadot.io/polkadot-asset-hub-rocksdb-prune/LATEST" + + rclone copyurl $SNAPSHOT_URL_ASSET_HUB/files.txt files.txt + rclone copy --progress --transfers 20 \ + --http-url $SNAPSHOT_URL_ASSET_HUB \ + --no-traverse --http-no-head --disable-http2 \ + --inplace --no-gzip-encoding --size-only \ + --retries 6 --retries-sleep 10s \ + --files-from files.txt :http: my-node-data/chains/asset-hub-polkadot/db/ - === "Pruned snapshot" + rm files.txt + ``` - Contains recent state for a smaller package size, recommended for RPC nodes. + **Relay chain pruned snapshot** (~200 GB): ```bash # Check https://snapshots.polkadot.io/ for the latest snapshot URL export SNAPSHOT_URL_RELAY="https://snapshots.polkadot.io/polkadot-rocksdb-prune/LATEST" @@ -127,37 +166,82 @@ Select the best option for your project, then use the steps in the following tab --retries 6 --retries-sleep 10s \ --files-from files.txt :http: my-node-data/chains/polkadot/db/ - rm files.txt + rm files.txt ``` - 3. Launch Polkadot Hub Node using the official [Parity Docker image](https://hub.docker.com/r/parity/polkadot-parachain){target=\_blank} with the following command: - ```bash - docker run -d --name polkadot-hub-rpc --restart unless-stopped \ - -p 9944:9944 \ - -p 9933:9933 \ - -p 9615:9615 \ - -p 30334:30334 \ - -p 30333:30333 \ - -v $(pwd)/asset-hub-polkadot.json:/asset-hub-polkadot.json \ - -v $(pwd)/my-node-data:/data \ - parity/polkadot-parachain:stable2509-2 \ - --name=PolkadotHubRPC \ - --base-path=/data \ - --chain=/asset-hub-polkadot.json \ - --prometheus-external \ - --prometheus-port 9615 \ - --unsafe-rpc-external \ - --rpc-port=9944 \ - --rpc-cors=all \ - --rpc-methods=safe \ - --rpc-max-connections=1000 \ - --state-pruning=archive \ - --blocks-pruning=archive \ - -- \ - --base-path=/data \ - --chain=polkadot \ - --state-pruning=256 \ - --blocks-pruning=archive-canonical - ``` + + **rclone parameters:** + + - `--transfers 20`: Uses 20 parallel transfers for faster download + - `--retries 6`: Automatically retries failed transfers up to 6 times + - `--retries-sleep 10s`: Waits 10 seconds between retry attempts + - `--size-only`: Only transfers if sizes differ (prevents unnecessary re-downloads) + 3. Launch Polkadot Hub Node using the official [Parity Docker image](https://hub.docker.com/r/parity/polkadot-parachain){target=\_blank}: + + === "Archive Node" + + Archive node configuration maintains complete parachain history for historical queries: + + ```bash + docker run -d --name polkadot-hub-rpc --restart unless-stopped \ + -p 9944:9944 \ + -p 9933:9933 \ + -p 9615:9615 \ + -p 30334:30334 \ + -p 30333:30333 \ + -v $(pwd)/asset-hub-polkadot.json:/asset-hub-polkadot.json \ + -v $(pwd)/my-node-data:/data \ + parity/polkadot-parachain:stable2509-2 \ + --name=PolkadotHubRPC \ + --base-path=/data \ + --chain=/asset-hub-polkadot.json \ + --prometheus-external \ + --prometheus-port 9615 \ + --unsafe-rpc-external \ + --rpc-port=9944 \ + --rpc-cors=all \ + --rpc-methods=safe \ + --rpc-max-connections=1000 \ + --state-pruning=archive \ + --blocks-pruning=archive \ + -- \ + --base-path=/data \ + --chain=polkadot \ + --state-pruning=256 \ + --blocks-pruning=archive-canonical + ``` + + === "Pruned Node" + + Pruned node configuration keeps recent state for smaller storage requirements: + + ```bash + docker run -d --name polkadot-hub-rpc --restart unless-stopped \ + -p 9944:9944 \ + -p 9933:9933 \ + -p 9615:9615 \ + -p 30334:30334 \ + -p 30333:30333 \ + -v $(pwd)/asset-hub-polkadot.json:/asset-hub-polkadot.json \ + -v $(pwd)/my-node-data:/data \ + parity/polkadot-parachain:stable2509-2 \ + --name=PolkadotHubRPC \ + --base-path=/data \ + --chain=/asset-hub-polkadot.json \ + --prometheus-external \ + --prometheus-port 9615 \ + --unsafe-rpc-external \ + --rpc-port=9944 \ + --rpc-cors=all \ + --rpc-methods=safe \ + --rpc-max-connections=1000 \ + --state-pruning=1000 \ + --blocks-pruning=256 \ + -- \ + --base-path=/data \ + --chain=polkadot \ + --state-pruning=256 \ + --blocks-pruning=archive-canonical + ``` Critical configuration parameters include port mappings and node parameters: @@ -173,8 +257,8 @@ Select the best option for your project, then use the steps in the following tab - `--unsafe-rpc-external`: Enables external RPC access - `--rpc-cors=all`: Allows all origins for CORS - `--rpc-methods=safe`: Only allows safe RPC methods - - `--state-pruning=archive`: Keeps complete state history - - `--blocks-pruning=archive`: Keeps all block data + - `--state-pruning=archive` or `--state-pruning=1000`: Archive keeps complete state history, pruned keeps last 1000 blocks + - `--blocks-pruning=archive` or `--blocks-pruning=256`: Archive keeps all blocks, pruned keeps last 256 finalized blocks - `--prometheus-external`: Exposes metrics externally !!! warning @@ -182,6 +266,7 @@ Select the best option for your project, then use the steps in the following tab The `--unsafe-rpc-external` flag should only be used in development or properly secured environments. For production, use a reverse proxy with authentication. 4. Monitor the node synchronization status using the following command: + ```bash curl -H "Content-Type: application/json" \ -d '{"id":1, "jsonrpc":"2.0", "method": "system_syncState", "params":[]}' \ @@ -189,7 +274,7 @@ Select the best option for your project, then use the steps in the following tab ``` You should see a response similar to the following: - + ```json { "jsonrpc":"2.0", @@ -204,20 +289,8 @@ Select the best option for your project, then use the steps in the following tab When synchronization is complete, `currentBlock` will be equal to `highestBlock`. - 5. You can use a few different commands to verify your node is running properly: + 5. You can use the `system_health` command to verify your node is running properly: - - Get chain information: - ```bash - curl -H "Content-Type: application/json" \ - -d '{"id":1, "jsonrpc":"2.0", "method": "system_chain", "params":[]}' \ - http://localhost:9944 - ``` - - Get the latest block: - ```bash - curl -H "Content-Type: application/json" \ - -d '{"id":1, "jsonrpc":"2.0", "method": "chain_getHeader", "params":[]}' \ - http://localhost:9944 - ``` - Query node health: ```bash curl -H "Content-Type: application/json" \ @@ -289,46 +362,95 @@ Select the best option for your project, then use the steps in the following tab sudo nano /etc/systemd/system/polkadot-hub-rpc.service ``` - 5. Open the new service file and add the following configuration: - ```ini - [Unit] - Description=Polkadot Hub RPC Node - After=network.target - - [Service] - Type=simple - User=polkadot - Group=polkadot - WorkingDirectory=/var/lib/polkadot-hub-rpc - - ExecStart=/usr/local/bin/polkadot-parachain \ - --name=PolkadotHubRPC \ - --chain=/var/lib/polkadot-hub-rpc/asset-hub-polkadot.json \ - --base-path=/var/lib/polkadot-hub-rpc \ - --port=30333 \ - --rpc-port=9944 \ - --rpc-external \ - --rpc-cors=all \ - --rpc-methods=safe \ - --rpc-max-connections=1000 \ - --prometheus-port=9615 \ - --prometheus-external \ - --state-pruning=archive \ - --blocks-pruning=archive \ - -- \ - --chain=polkadot \ - --base-path=/var/lib/polkadot-hub-rpc \ - --port=30334 \ - --state-pruning=256 \ - --blocks-pruning=archive-canonical - - Restart=always - RestartSec=10 - LimitNOFILE=65536 - - [Install] - WantedBy=multi-user.target - ``` + 5. Open the new service file and add the configuration for your chosen node type: + + === "Archive Node" + + Archive node configuration maintains complete parachain history for historical queries: + + ```ini + [Unit] + Description=Polkadot Hub RPC Node + After=network.target + + [Service] + Type=simple + User=polkadot + Group=polkadot + WorkingDirectory=/var/lib/polkadot-hub-rpc + + ExecStart=/usr/local/bin/polkadot-parachain \ + --name=PolkadotHubRPC \ + --chain=/var/lib/polkadot-hub-rpc/asset-hub-polkadot.json \ + --base-path=/var/lib/polkadot-hub-rpc \ + --port=30333 \ + --rpc-port=9944 \ + --rpc-external \ + --rpc-cors=all \ + --rpc-methods=safe \ + --rpc-max-connections=1000 \ + --prometheus-port=9615 \ + --prometheus-external \ + --state-pruning=archive \ + --blocks-pruning=archive \ + -- \ + --chain=polkadot \ + --base-path=/var/lib/polkadot-hub-rpc \ + --port=30334 \ + --state-pruning=256 \ + --blocks-pruning=archive-canonical + + Restart=always + RestartSec=10 + LimitNOFILE=65536 + + [Install] + WantedBy=multi-user.target + ``` + + === "Pruned Node" + + Pruned node configuration keeps recent state for smaller storage requirements: + + ```ini + [Unit] + Description=Polkadot Hub RPC Node + After=network.target + + [Service] + Type=simple + User=polkadot + Group=polkadot + WorkingDirectory=/var/lib/polkadot-hub-rpc + + ExecStart=/usr/local/bin/polkadot-parachain \ + --name=PolkadotHubRPC \ + --chain=/var/lib/polkadot-hub-rpc/asset-hub-polkadot.json \ + --base-path=/var/lib/polkadot-hub-rpc \ + --port=30333 \ + --rpc-port=9944 \ + --rpc-external \ + --rpc-cors=all \ + --rpc-methods=safe \ + --rpc-max-connections=1000 \ + --prometheus-port=9615 \ + --prometheus-external \ + --state-pruning=1000 \ + --blocks-pruning=256 \ + -- \ + --chain=polkadot \ + --base-path=/var/lib/polkadot-hub-rpc \ + --port=30334 \ + --state-pruning=256 \ + --blocks-pruning=archive-canonical + + Restart=always + RestartSec=10 + LimitNOFILE=65536 + + [Install] + WantedBy=multi-user.target + ``` 6. Start the service using the following commands: - Reload systemd: @@ -369,115 +491,6 @@ Select the best option for your project, then use the steps in the following tab http://localhost:9944 ``` -## Monitor and Maintain RPC Node - -There are a few key monitoring and maintenance commands that can help you ensure RPC node uptime and performance including log monitoring and metrics. - -To view node logs, use the appropriate command for your setup: - -=== "Docker Setup" - - ```bash - docker logs -f polkadot-hub-rpc - ``` - -=== "systemd Setup" - - - View node logs: - ```bash - sudo journalctl -u polkadot-hub-rpc -f - ``` - - View recent logs: - ```bash - sudo journalctl -u polkadot-hub-rpc -n 100 - ``` - - Filter for errors: - ```bash - sudo journalctl -u polkadot-hub-rpc | grep -i error - ``` - -Monitoring key metrics like the following items can help you stay up to date on the performance and health of your node and allow you to intervene at the first sign of issues: - -- **Sync status**: Ensure node stays fully synced. -- **Peer connections**: Maintain 30+ peers for good connectivity. -- **Resource usage**: Monitor CPU, RAM, and disk I/O. -- **RPC request latency**: Track response times for the Polkadot SDK API. -- **Connection count**: Monitor active RPC connections. - -You can use the following information to configure [Prometheus](https://prometheus.io/docs/introduction/first_steps/){target=\_blank} to monitor, collect, and store your RPC node metrics: - -- **URL**: Metrics are available to view at `http://localhost:9615/metrics` -- **Example Prometheus configuration**: Update your `prometheus.yml` to add the following code: - ```yaml - scrape_configs: - - job_name: 'polkadot-hub-rpc' - static_configs: - - targets: ['localhost:9615'] - ``` - -Key metrics to monitor via Prometheus include: - -- `substrate_block_height`: Current block height -- `substrate_finalized_height`: Finalized block height -- `substrate_peers_count`: Number of connected peers -- `substrate_ready_transactions_number`: Transaction queue size - -### Database Maintenance - -Check database size periodically using the commands for your selected setup: - -=== "Docker Setup" - - ```bash - du -sh my-node-data - ``` - -=== "systemd Setup" - - ```bash - du -sh /var/lib/polkadot-hub-rpc - ``` - -The node handles pruning automatically based on configuration unless running in archive mode. - -### Updates and Upgrades - -Use the following commands for updating or upgrading your RPC node according to your setup: - -=== "Docker Setup" - - 1. Stop and remove the existing container: - ```bash - docker stop polkadot-hub-rpc - docker rm polkadot-hub-rpc - ``` - 2. Pull the latest image: - ```bash - docker pull parity/polkadot-parachain: - ``` - 3. Start the new container using the same command from the setup section with the updated image tag. - -=== "systemd Setup" - - 1. Stop the service: - ```bash - sudo systemctl stop polkadot-hub-rpc - ``` - 2. Backup data: - ```bash - sudo cp -r /var/lib/polkadot-hub-rpc /var/lib/polkadot-hub-rpc.backup - ``` - 3. Download the new binary from [GitHub releases](https://github.com/paritytech/polkadot-sdk/releases){target=\_blank}: - ```bash - wget https://github.com/paritytech/polkadot-sdk/releases/download//polkadot-parachain - chmod +x polkadot-parachain - sudo mv polkadot-parachain /usr/local/bin/ - ``` - 4. Restart the service: - ```bash - sudo systemctl start polkadot-hub-rpc - ``` - ## Conclusion Running an RPC node for Polkadot Hub provides essential infrastructure for applications and users to interact with the network. By following this guide, you have set up a production-ready RPC node that: