Observe settlement of rails + alert

[bajtos]

• Figure out how to get time-series for "the oldest rail with non-zero usage that was not settled"
• Visualise that time-series in Grafana and create an alert

This is important to allow us to safely ignore errors when trying to settle individual payment rails, because we have confidence that either the rail will eventually be settled by the future invocation of the payment-settler worker or we will be notified about the problem.

In particular, we don't want alerts for individual errors, because such alerts are not actionable.

Related discussion on Slack:

• https://space-meridian.slack.com/archives/C08TC8NMM17/p1768991215474129

Tasks:

• Goldsky subgraph & pipeline implementation
• Webhook implementation
• Webhook deployed
• Goldsky subgraph & pipeline deployed
• Grafana visualisations
• Fix the way how we calculate the lag
• Grafana alerts
• Modify observer to ignore datasets from sanctioned wallets

[bajtos]

Figure out how to get time-series for "the oldest rail with non-zero usage that was not settled"

What I'd like to do:

• Create a new subgraph observing FilBeam contract events CDNSettlement (see e.g. https://calibration.filfox.info/en/message/0x5b86327efbcdaac65ba71cfdbcb5402eccf7bcf023ff7b9048e9fd2f745da863?t=3), and a pipeline to call our indexer webhook.
• Modify our indexer webhook to process these events and update data_sets.cdn_settled_until column (we will need to add that column to our DB)
• In the scheduled indexer handler, add a new step to run a SQL query to find the oldest rail where cdn_settled_until < usage_reported_until and write a new data point to CF Worker Analytics.
  • This assumes that we update usage_reported_until only when there was a non-zero usage since the last report. If that's not the case, then we need to find a different approach.

Visualise that time-series in Grafana and create an alert

The Grafana panel will query CF Worker Analytics data-set created by the code above and visualise the lag in settlements over time. We will add an alert based on the panel data.

[bajtos]

This assumes that we update usage_reported_until only when there was a non-zero usage since the last report. If that's not the case, then we need to find a different approach.

That's a correct assumption, we have a test for the case when a dataset does not have any new usage to report:

worker/usage-reporter/test/usage-report.test.js

Line 199 in f4223b4

it('should filter out datasets with zero cdn and cache-miss bytes', async () => {

The SQL query:

worker/usage-reporter/lib/usage-report.js

Line 35 in f4223b4

HAVING (cdn_bytes > 0 OR cache_miss_bytes > 0)

[bajtos]

Blocked by #609

[bajtos]

For some reason, the indexer worker is getting a different result from the SQL query that I get when using the Dashboard Console.

The query:

worker/indexer/bin/indexer.js

Lines 404 to 408 in 9062759

	SELECT id, cdn_payments_settled_until
	FROM data_sets
	WHERE usage_reported_until > cdn_payments_settled_until
	ORDER BY cdn_payments_settled_until ASC
	LIMIT 1

Result when I run it manually:

Log from the indexer:

2026-02-05 15:19:58:578 CET
Oldest unsettled CDN usage: data_set=75, cdn_payments_settled_until=2025-11-13T13:47:00.000Z, lag_ms=7259578578

(...)


2026-02-05 15:18:57:908 CET
Oldest unsettled CDN usage: data_set=11322, cdn_payments_settled_until=1970-01-01T00:00:00.000Z, lag_ms=1770301137908

[bajtos]

I think I see the problem now. We look at cdn_payments_settled_until to calculate the lag, but that's not correct. We need to calculate the lag as Date.now() - usage_reported_until.

[bajtos]

Cross-posting from #619:

Hmm, this new approach is not correct either.

Consider the following sequence of events:

• 00:01 - usage reported
• 00:02 - payment settled
• 00:03 - usage reported
• 00:04 - usage reported
• 00:05 - usage reported

At 00:06, my code would report a lag of 1 minute (00:06 - 00:05).

However, the actual lag is 3 minutes (00:06 - 00:03) - that's the time since some new usage was reported for the first time after the last settlement.

Here is what I think we need to do:

1. Add a new column usage_unsettled_since (timestamp with timezone default NULL)
2. Whenever we report new egress usage on the chain, if usage_unsettled_since is NULL, set it to the epoch time when the usage was reported
3. Whenever we settle payment on the chain, reset usage_unsettled_since to NULL
4. Calculate the settlement lag as now() - usage_unsettled_since. Skip rows where usage_unsettled_since is NULL.

[BigLep]

@juliangruber : can you take this over?