Release (#293)

* Examples of alerts

* Remove special forks annotations (#271)

* Remove special forks annotations

* take addresses from envs

* Add env

* Variables in README

* Move to other section

* Holesky only

* fix check balance test

* Actualize readme

* remove deprecated entrypoints

* Add account label to balance

* fix xfail unit tests

* Update minor deps + remove package-mode

* add balance check

* remove tests

* Remove mellow

* Remove test

* Fix test

* Refactor strategy

* Update alerts

* Empty whitelist (#282)

* Empty whitelist

* Change default

* Refactor

---------

Co-authored-by: Kirill <[email protected]>
Co-authored-by: Raman Siamionau <[email protected]>

* Track recommended fee despite checks (#281)

* Fetch prefix one time for a batch (#279)

* Fetch prefix one time

* default arg

* function arg

* Filters on request

* Fix tests

* refactor

* Fix unvetter tests

* Fix bug

* fix signatures in test

* Fix

* Fix

* Fix

* Fix

* Fix comments

* Gas strategy with addendum (#285)

* Gas strategy with addendum

* Precommit hooks

* Env variable

* Update README

* Gwei default

* Module iteration until first healthy module (#286)

* Cache quorum

* Gather quorum in advance

* Update tests

* Fix types

* fix var

* fix

* Fix comments

* Fix sorting

* Remove unit tests

* Move minutes to a variable

* Add more logs

* Unit tests for iteration order

* Comment

* Fix comments

* Update curl (#287)

* Update curl

* Pin version

* Update aiohttp (#289)

* Max operators per unvetting check (#290)

* Max operators per unvetting check

* Upgrade test

* Cache chain-id (#288)

* Cache chain-id

* middleware to cache id

* Cache gnosis

* Add metrics to gnosis

* Remove i

* Fix recursion (#292)

* Fix recursion

* Remove metrics from onchain web3

* Reorder middlewares (#294)

---------

Co-authored-by: F4ever <[email protected]>
Co-authored-by: Kirill <[email protected]>

Author: 3 people

.env.example

@@ -38,7 +38,3 @@ DEPOSIT_MODULES_WHITELIST=1
 
 # Prefix for the Prometheus metrics(depositor_bot,pauser_bot,unvetter_bot)
 PROMETHEUS_PREFIX=depositor_bot
-
-# Mellow strategy address for direct deposits
-# Holesky: 0x182Cb3A76B0EFaCb25255F9594B5807460882fa4
-MELLOW_CONTRACT_ADDRESS=0x182Cb3A76B0EFaCb25255F9594B5807460882fa4

.github/workflows/integration-tests.yml

@@ -29,34 +29,15 @@ jobs:
       - name: Install Foundry
         uses: foundry-rs/foundry-toolchain@v1
 
-      - name: Integration tests with pytest, holesky fork
-        run: |
-          poetry run pytest tests -m integration_holesky
-        env:
-          WEB3_RPC_ENDPOINTS: ${{ secrets.HOLESKY_WEB3_RPC_ENDPOINT }}
-          DEPOSIT_CONTRACT: "0x4242424242424242424242424242424242424242"
-          LIDO_LOCATOR: "0x28FAB2059C713A7F9D8c86Db49f9bb0e96Af1ef8"
-          MELLOW_CONTRACT_ADDRESS: "0x182Cb3A76B0EFaCb25255F9594B5807460882fa4"
-          ANVIL_PATH: ""
-
-      - name: Integration tests with pytest, chiado testnet
-        run: |
-          poetry run pytest tests -m integration_chiado
-        env:
-          WEB3_RPC_ENDPOINTS: "https://gnosis-chiado-rpc.publicnode.com"
-          ONCHAIN_TRANSPORT_RPC_ENDPOINTS: "https://gnosis-chiado-rpc.publicnode.com"
-          ONCHAIN_TRANSPORT_ADDRESS: "0x42E1DEfC18388E3AA1fCADa851499A11405cf37f"
-          DEPOSIT_CONTRACT: "0x4242424242424242424242424242424242424242"
-          LIDO_LOCATOR: "0x28FAB2059C713A7F9D8c86Db49f9bb0e96Af1ef8"
-          MELLOW_CONTRACT_ADDRESS: "0x182Cb3A76B0EFaCb25255F9594B5807460882fa4"
-          ANVIL_PATH: ""
-
-      - name: Integration tests with pytest, mainnet fork
+      - name: Integration tests with pytest
         if: success() || failure()
         run: |
           poetry run pytest tests -m integration
         env:
           WEB3_RPC_ENDPOINTS: ${{ secrets.WEB3_RPC_ENDPOINT }}
+          TESTNET_WEB3_RPC_ENDPOINTS: 'https://ethereum-holesky-rpc.publicnode.com'
           DEPOSIT_CONTRACT: "0x00000000219ab540356cBB839Cbe05303d7705Fa"
           LIDO_LOCATOR: "0xC1d0b3DE6792Bf6b4b37EccdcC24e45978Cfd2Eb"
+          ONCHAIN_TRANSPORT_RPC_ENDPOINTS: "https://gnosis-chiado-rpc.publicnode.com"
+          ONCHAIN_TRANSPORT_ADDRESS: "0x42E1DEfC18388E3AA1fCADa851499A11405cf37f"
           ANVIL_PATH: ""

.pre-commit-config.yaml

@@ -1,4 +1,21 @@
 repos:
+- repo: https://github.com/pre-commit/pre-commit-hooks
+  rev: v4.5.0  # Use the latest stable version
+  hooks:
+    - id: trailing-whitespace
+    - id: end-of-file-fixer
+    - id: check-yaml
+
+- repo: local
+  hooks:
+    - id: poetry-pytest-unit
+      name: Run pytest unit tests with Poetry
+      entry: poetry run pytest -m unit
+      language: system
+      pass_filenames: false
+      always_run: true
+      stages: [commit]
+
 - repo: https://github.com/astral-sh/ruff-pre-commit
   # Ruff version.
   rev: v0.4.4

Dockerfile

@@ -4,7 +4,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends -qq \
     gcc=4:12.2.0-3 \
     libffi-dev=3.4.4-1 \
     g++=4:12.2.0-3 \
-    curl=7.88.1-10+deb12u7 \
+    curl=7.88.1-10+deb12u8 \
  && apt-get clean \
  && rm -rf /var/lib/apt/lists/*
 

README.md

@@ -34,13 +34,13 @@ Unvetting is the proces of decreasing approved depositable signing keys.
 
 ## Running Daemon
 
-1. Create `.env` file 
+1. Create `.env` file
 2. Setup variables
-   - Set WEB3_RPC_ENDPOINTS
-   - Set WALLET_PRIVATE_KEY
-   - Set CREATE_TRANSACTIONS to true
-   - Set MESSAGE_TRANSPORTS to rabbit
-   - Set RABBIT_MQ_URL, RABBIT_MQ_USERNAME and RABBIT_MQ_PASSWORD
+    - Set WEB3_RPC_ENDPOINTS
+    - Set WALLET_PRIVATE_KEY
+    - Set CREATE_TRANSACTIONS to true
+    - Set MESSAGE_TRANSPORTS to rabbit
+    - Set RABBIT_MQ_URL, RABBIT_MQ_USERNAME and RABBIT_MQ_PASSWORD
 3. ```docker-compose up```
 4. Send metrics and logs to grafana
 5. Setup alerts
@@ -76,13 +76,14 @@ Unvetting is the proces of decreasing approved depositable signing keys.
 | GAS_FEE_PERCENTILE_1              | 20            | Percentile for first recommended fee calculation                                                                         |
 | GAS_FEE_PERCENTILE_DAYS_HISTORY_1 | 1             | Percentile for first recommended calculates from N days of the fee history                                               |
 | GAS_PRIORITY_FEE_PERCENTILE       | 25            | Priority transaction will be N percentile from priority fees in last block (min MIN_PRIORITY_FEE - max MAX_PRIORITY_FEE) |
+| GAS_ADDENDUM                      | 6 gwei        | Addendum to the GAS_PRIORITY_FEE_PERCENTILE percentile value                                                             |
 | MAX_BUFFERED_ETHERS               | 5000 ether    | Maximum amount of ETH in the buffer, after which the bot deposits at any gas                                             |
 | PROMETHEUS_PORT                   | 9000          | Port with metrics server                                                                                                 |
 | PROMETHEUS_PREFIX                 | depositor_bot | Prefix for the metrics                                                                                                   |
 | HEALTHCHECK_SERVER_PORT           | 9010          | Port with bot`s status server                                                                                            |
 | MAX_CYCLE_LIFETIME_IN_SECONDS     | 1200          | Max lifetime of usual cycle. If cycle will not end in this time, bot will crush                                          |
-| MELLOW_CONTRACT_ADDRESS           | None          | If variable is set then deposit can go to predifined module                                                              |
-| VAULT_DIRECT_DEPOSIT_THRESHOLD    | 1 ether       | If mellow vault has VAULT_DIRECT_DEPOSIT_THRESHOLD ethers then direct deposit will be sent                               |
+| ONCHAIN_TRANSPORT_RPC_ENDPOINTS   | -             | RPC endpoint for the databus RPC, Gnosis at the moment                                                                   |
+| QUORUM_RETENTION_MINUTES          | 5             | TTL of quorum collection for marking module as healthy                                                                   |
 
 ## Metrics and logs
 
@@ -96,17 +97,18 @@ Prometheus server hosted on `http://localhost:${{PROMETHEUS_PORT}}/`.
 ```bash
 git clone [email protected]:lidofinance/depositor-bot.git
 cd depositor-bot
-poetry install
+poetry install --with dev
+poetry run pre-commit install
 ```
 
 To run bot
 
 ```bash
-poetry run python main depositor
+poetry run python src/main.py depositor
 
-poetry run python main pauser
+poetry run python src/main.py pauser
 
-poetry run python main unvetter
+poetry run python src/main.py unvetter
 ```
 
 ### Tests
@@ -119,6 +121,8 @@ poetry run pytest tests -m unit
 
 #### Run integration tests.
 
+TESTNET_WEB3_RPC_ENDPOINTS - set this variable for the Ethereum EL testnet RPC, Holesky only.
+
 Install Anvil
 
 ```bash

alerts/alerts.yml

@@ -0,0 +1,57 @@
+groups:
+  - name: Depositor bot. No metrics
+    rules:
+      - alert: DepositorBotNoMetrics
+        annotations:
+          info: "Depositor has not reported metrics for 5m."
+        expr: absent_over_time(depositor_bot_build_info{service="lido-depositor-bot"}[5m])
+        labels:
+          severity: critical
+
+  - name: Pauser bot. No metrics
+    rules:
+      - alert: PauserBotNoMetrics
+        annotations:
+          info: "Pauser has not reported metrics for 5m."
+        expr: absent_over_time(pauser_bot_build_info{service="lido-pause-bot"}[5m])
+        labels:
+          severity: critical
+
+  - name: Unvetter bot. No metrics
+    rules:
+      - alert: UnvetterBotNoMetrics
+        annotations:
+          info: "Unvetter has not reported metrics for 5m."
+        expr: absent_over_time(unvetter_bot_build_info{service="lido-unvetter-bot"}[5m])
+        labels:
+          severity: critical
+
+  - name: No transport heartbeat
+    rules:
+      - alert: NoTransportConnection
+        annotations:
+          info: "None of the counters (deposit, pause, unvet, ping) have increased in the last 6 hours for bot {{ $labels.service }}"
+        expr: sum by (service) (changes(label_replace({__name__=~"depositor_bot_ping_messages|depositor_bot_deposit_messages|pauser_bot_ping_messages|pauser_bot_pause_messages|unvetter_bot_ping_messages|unvetter_bot_unvet_messages"},"name_label","$1","__name__", "(.+)")[6h:])) == 0
+        labels:
+          severity: critical
+  - name: Depositor bot. Account balance
+    rules:
+      - alert: DepositorBotAccountBalanceLow
+        annotations:
+          info: "Account balance is {{ $value }} ETH. Less than 3 ETH."
+        expr:
+          depositor_bot_account_balance{service="lido-depositor-bot"}/10^18 < 3
+          UNLESS ON(service)
+          depositor_bot_build_info{service="lido-depositor-bot",CREATE_TRANSACTIONS="False"}
+        labels:
+          severity: warning
+
+      - alert: DepositorBotAccountBalanceCriticalLow
+        annotations:
+          info: "Account balance is {{ $value }} ETH."
+        expr:
+          depositor_bot_account_balance{service="lido-depositor-bot"}/10^18 < 1
+          UNLESS ON(service)
+          depositor_bot_build_info{service="lido-depositor-bot",CREATE_TRANSACTIONS="False"}
+        labels:
+          severity: major