Skip to content

Commit 17b892b

Browse files
Jim8yNeo Bot
Jim8y
authored and
Neo Bot
committed
Add telemetry documentation and monitoring configs
1 parent b563f1f commit 17b892b

File tree

4 files changed

+624
-0
lines changed

4 files changed

+624
-0
lines changed

‎src/Plugins/Telemetry/README.md‎

Lines changed: 238 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,238 @@
1+
# Neo N3 Telemetry Plugin
2+
3+
A comprehensive telemetry and metrics collection plugin for Neo N3 full nodes, providing Prometheus-compatible metrics for monitoring node health, performance, and operational status.
4+
5+
## Features
6+
7+
- **Blockchain Metrics**: Block height, sync status, block processing time, transaction counts
8+
- **Network Metrics**: Peer connections, message statistics, bandwidth usage
9+
- **Mempool Metrics**: Transaction pool size, utilization, add/remove rates
10+
- **System Metrics**: CPU usage, memory consumption, GC statistics, thread pool
11+
- **Plugin Metrics**: Loaded plugins count and status
12+
- **Prometheus Export**: Native Prometheus metrics endpoint
13+
14+
## Installation
15+
16+
1. Build the plugin:
17+
18+
```bash
19+
dotnet build src/Plugins/Telemetry/Telemetry.csproj
20+
```
21+
22+
2. Copy the output to your Neo node's `Plugins/Telemetry/` directory:
23+
24+
```bash
25+
cp -r bin/Debug/net9.0/* /path/to/neo-node/Plugins/Telemetry/
26+
```
27+
28+
3. Configure the plugin by editing `config.json` in the plugin directory.
29+
30+
## Configuration
31+
32+
Create or edit `config.json` in the `Plugins/Telemetry/` directory:
33+
34+
```json
35+
{
36+
"PluginConfiguration": {
37+
"Enabled": true,
38+
"ExceptionPolicy": "StopPlugin",
39+
"PrometheusPort": 9100,
40+
"PrometheusHost": "localhost",
41+
"PrometheusPath": "/metrics",
42+
"HealthPort": 9101,
43+
"SystemMetricsIntervalMs": 5000,
44+
"CollectBlockchainMetrics": true,
45+
"CollectNetworkMetrics": true,
46+
"CollectMempoolMetrics": true,
47+
"CollectSystemMetrics": true,
48+
"NodeId": "my-neo-node",
49+
"NetworkName": "mainnet"
50+
}
51+
}
52+
```
53+
54+
### Configuration Options
55+
56+
| Option | Type | Default | Description |
57+
| -------------------------- | ------ | -------------- | ---------------------------------------------------------------------------- |
58+
| `Enabled` | bool | `true` | Enable/disable the telemetry plugin |
59+
| `ExceptionPolicy` | string | `StopPlugin` | Exception handling policy (`StopNode`, `StopPlugin`, `Ignore`) |
60+
| `PrometheusPort` | int | `9100` | Port for the Prometheus metrics endpoint |
61+
| `PrometheusHost` | string | `localhost` | Host address for the metrics endpoint |
62+
| `PrometheusPath` | string | `/metrics` | URL path for the metrics endpoint |
63+
| `HealthPort` | int? | PrometheusPort | Port for `/health`, `/ready`, `/live` endpoints (defaults to PrometheusPort) |
64+
| `SystemMetricsIntervalMs` | int | `5000` | Interval for collecting system metrics (ms) |
65+
| `CollectBlockchainMetrics` | bool | `true` | Enable blockchain metrics collection |
66+
| `CollectNetworkMetrics` | bool | `true` | Enable network metrics collection |
67+
| `CollectMempoolMetrics` | bool | `true` | Enable mempool metrics collection |
68+
| `CollectSystemMetrics` | bool | `true` | Enable system resource metrics collection |
69+
| `NodeId` | string | hostname | Unique identifier for this node |
70+
| `NetworkName` | string | auto-detect | Network name label (mainnet, testnet, etc.) |
71+
72+
## Metrics Reference
73+
74+
### Blockchain Metrics
75+
76+
| Metric | Type | Description |
77+
| ---------------------------------------------------- | --------- | --------------------------------- |
78+
| `neo_blockchain_height` | Gauge | Current block height |
79+
| `neo_blockchain_header_height` | Gauge | Current header height |
80+
| `neo_blockchain_blocks_persisted_total` | Counter | Total blocks persisted |
81+
| `neo_blockchain_block_persist_duration_milliseconds` | Histogram | Block persist duration |
82+
| `neo_blockchain_block_transactions` | Gauge | Transactions in last block |
83+
| `neo_blockchain_transactions_processed_total` | Counter | Total transactions processed |
84+
| `neo_blockchain_sync_status` | Gauge | Sync status (1=synced, 0=syncing) |
85+
| `neo_blockchain_blocks_behind` | Gauge | Blocks behind network |
86+
| `neo_blockchain_time_since_last_block_seconds` | Gauge | Time since last block |
87+
88+
### Network Metrics
89+
90+
| Metric | Type | Description |
91+
| --------------------------------------- | ------- | ------------------------- |
92+
| `neo_network_peers_connected` | Gauge | Connected peer count |
93+
| `neo_network_peers_unconnected` | Gauge | Unconnected peer count |
94+
| `neo_network_peer_connections_total` | Counter | Total peer connections |
95+
| `neo_network_peer_disconnections_total` | Counter | Total peer disconnections |
96+
| `neo_network_messages_received_total` | Counter | Messages received by type |
97+
| `neo_network_messages_sent_total` | Counter | Messages sent by type |
98+
| `neo_network_bytes_received_total` | Counter | Total bytes received |
99+
| `neo_network_bytes_sent_total` | Counter | Total bytes sent |
100+
101+
### Mempool Metrics
102+
103+
| Metric | Type | Description |
104+
| ---------------------------------------- | ------- | ---------------------------- |
105+
| `neo_mempool_transactions` | Gauge | Current mempool size |
106+
| `neo_mempool_verified_transactions` | Gauge | Verified transaction count |
107+
| `neo_mempool_unverified_transactions` | Gauge | Unverified transaction count |
108+
| `neo_mempool_capacity` | Gauge | Mempool capacity |
109+
| `neo_mempool_utilization_ratio` | Gauge | Mempool utilization (0-1) |
110+
| `neo_mempool_transactions_added_total` | Counter | Total transactions added |
111+
| `neo_mempool_transactions_removed_total` | Counter | Total transactions removed |
112+
113+
### System Metrics
114+
115+
| Metric | Type | Description |
116+
| ----------------------------------------------- | ----- | ------------------------------ |
117+
| `neo_system_cpu_usage_ratio` | Gauge | CPU usage ratio (0-1) |
118+
| `neo_system_memory_usage_bytes` | Gauge | Memory usage by type |
119+
| `neo_system_gc_collection_count` | Gauge | GC collections by generation |
120+
| `neo_system_threadpool_worker_threads` | Gauge | Active worker threads |
121+
| `neo_system_threadpool_completion_port_threads` | Gauge | Active completion port threads |
122+
| `neo_system_process_uptime_seconds` | Gauge | Process uptime |
123+
124+
### Node Info Metrics
125+
126+
| Metric | Type | Description |
127+
| ----------------------------- | ----- | --------------------------------- |
128+
| `neo_node_info` | Gauge | Node information (version labels) |
129+
| `neo_node_start_time_seconds` | Gauge | Node start timestamp |
130+
| `neo_plugins_loaded` | Gauge | Number of loaded plugins |
131+
| `neo_plugin_status` | Gauge | Individual plugin status |
132+
133+
## Prometheus Integration
134+
135+
### Scrape Configuration
136+
137+
Add to your `prometheus.yml`:
138+
139+
```yaml
140+
scrape_configs:
141+
- job_name: "neo-node"
142+
static_configs:
143+
- targets: ["localhost:9100"]
144+
scrape_interval: 15s
145+
```
146+
147+
### Example Queries
148+
149+
```promql
150+
# Block height
151+
neo_blockchain_height{node_id="my-node"}
152+
153+
# Sync progress
154+
1 - (neo_blockchain_blocks_behind / neo_blockchain_header_height)
155+
156+
# Mempool utilization
157+
neo_mempool_utilization_ratio{node_id="my-node"}
158+
159+
# Transaction rate (per minute)
160+
rate(neo_blockchain_transactions_processed_total[1m])
161+
162+
# Connected peers
163+
neo_network_peers_connected{node_id="my-node"}
164+
165+
# Memory usage
166+
neo_system_memory_usage_bytes{type="working_set"}
167+
```
168+
169+
## Grafana Dashboard
170+
171+
Import the provided Grafana dashboard JSON for a pre-configured monitoring view:
172+
173+
1. Open Grafana
174+
2. Go to Dashboards → Import
175+
3. Upload `grafana-dashboard.json` or paste the JSON content
176+
4. Select your Prometheus data source
177+
5. Click Import
178+
179+
## Alerting Examples
180+
181+
### Prometheus Alerting Rules
182+
183+
```yaml
184+
groups:
185+
- name: neo-node-alerts
186+
rules:
187+
- alert: NeoNodeOutOfSync
188+
expr: neo_blockchain_blocks_behind > 10
189+
for: 5m
190+
labels:
191+
severity: warning
192+
annotations:
193+
summary: "Neo node is out of sync"
194+
description: "Node {{ $labels.node_id }} is {{ $value }} blocks behind"
195+
196+
- alert: NeoNodeNoPeers
197+
expr: neo_network_peers_connected == 0
198+
for: 2m
199+
labels:
200+
severity: critical
201+
annotations:
202+
summary: "Neo node has no peers"
203+
description: "Node {{ $labels.node_id }} has no connected peers"
204+
205+
- alert: NeoMempoolFull
206+
expr: neo_mempool_utilization_ratio > 0.9
207+
for: 5m
208+
labels:
209+
severity: warning
210+
annotations:
211+
summary: "Neo mempool is nearly full"
212+
description: "Mempool utilization is {{ $value | humanizePercentage }}"
213+
```
214+
215+
## Troubleshooting
216+
217+
### Metrics endpoint not accessible
218+
219+
1. Check if the plugin is enabled in `config.json`
220+
2. Verify the port is not in use: `netstat -tlnp | grep 9100`
221+
3. Check firewall rules allow the port
222+
4. Review node logs for plugin startup errors
223+
224+
### Missing metrics
225+
226+
1. Ensure the corresponding `Collect*Metrics` option is enabled
227+
2. Check that the node has fully started
228+
3. Verify the metric collection interval is appropriate
229+
230+
### High resource usage
231+
232+
1. Increase `SystemMetricsIntervalMs` to reduce collection frequency
233+
2. Disable unnecessary metric categories
234+
3. Consider using a dedicated metrics aggregation service
235+
236+
## License
237+
238+
This plugin is part of the Neo project and is distributed under the MIT license.

‎src/Plugins/Telemetry/alerting/neo-alerts.yml‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
# Neo N3 Node Prometheus Alerting Rules
2+
groups:
3+
- name: neo-node-critical
4+
rules:
5+
- alert: NeoNodeOutOfSync
6+
expr: neo_blockchain_sync_status == 0
7+
for: 5m
8+
labels:
9+
severity: warning
10+
annotations:
11+
summary: "Neo node is out of sync"
12+
description: "Node {{ $labels.node_id }} on {{ $labels.network }} is not synced"
13+
14+
- alert: NeoNodeNoPeers
15+
expr: neo_network_peers_connected == 0
16+
for: 2m
17+
labels:
18+
severity: critical
19+
annotations:
20+
summary: "Neo node has no peers"
21+
description: "Node {{ $labels.node_id }} has no connected peers"
22+
23+
- alert: NeoNodeFarBehind
24+
expr: neo_blockchain_blocks_behind > 100
25+
for: 10m
26+
labels:
27+
severity: critical
28+
annotations:
29+
summary: "Neo node is far behind"
30+
description: "Node {{ $labels.node_id }} is {{ $value }} blocks behind"
31+
32+
- name: neo-node-performance
33+
rules:
34+
- alert: NeoNodeHighCPU
35+
expr: neo_system_cpu_usage_ratio > 0.9
36+
for: 10m
37+
labels:
38+
severity: warning
39+
annotations:
40+
summary: "Neo node high CPU usage"
41+
description: "CPU usage is {{ $value | humanizePercentage }}"
42+
43+
- alert: NeoMempoolNearlyFull
44+
expr: neo_mempool_utilization_ratio > 0.9
45+
for: 5m
46+
labels:
47+
severity: warning
48+
annotations:
49+
summary: "Neo mempool is nearly full"
50+
description: "Mempool utilization is {{ $value | humanizePercentage }}"
51+
52+
- alert: NeoNodeLowPeers
53+
expr: neo_network_peers_connected < 3
54+
for: 5m
55+
labels:
56+
severity: warning
57+
annotations:
58+
summary: "Neo node has low peer count"
59+
description: "Only {{ $value }} connected peers"

‎src/Plugins/Telemetry/alerting/prometheus-scrape.yml‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
# Prometheus scrape configuration for Neo N3 nodes
2+
# Add this to your prometheus.yml scrape_configs section
3+
4+
scrape_configs:
5+
- job_name: 'neo-node'
6+
static_configs:
7+
- targets: ['localhost:9100']
8+
labels:
9+
instance: 'neo-node-1'
10+
scrape_interval: 15s
11+
scrape_timeout: 10s
12+
metrics_path: /metrics
13+
14+
# For multiple nodes:
15+
# scrape_configs:
16+
# - job_name: 'neo-nodes'
17+
# static_configs:
18+
# - targets:
19+
# - 'node1.example.com:9100'
20+
# - 'node2.example.com:9100'
21+
# - 'node3.example.com:9100'

0 commit comments

Comments
 (0)