fix(grafana): use percentage for hummock read metrics

[kwannoel]

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

Artifacts generated in #23676

These metrics are using rate and so the unit can't be per second, it should be a percentage. They reflect the percentage of time spent per interval reading records, either via iteration or point gets.

sum(rate({table_metric('state_store_get_duration_bucket')
sum(rate({table_metric('state_store_iter_init_duration_bucket')
sum(rate({table_metric('state_store_iter_scan_duration_bucket')

Further, these metrics are all at the second level granularity. Since rate is:

rate(v range-vector) calculates the per-second average rate of increase of the time series in the range vector.

Therefore we don't need to apply normalization as we would for other metrics tracked as sub-second level (e.g. ns, ms, etc...).

Checklist

• I have written necessary rustdoc comments.
• I have added necessary unit tests and integration tests.
• I have added test labels as necessary.
• I have added fuzzing tests or opened an issue to track them.
• My PR contains breaking changes.
• My PR changes performance-critical code, so I will run (micro) benchmarks and present the results.
• I have checked the Release Timeline and Currently Supported Versions to determine which release branches I need to cherry-pick this PR into.

Documentation

• My PR needs documentation updates.

Release note

[kwannoel]

• fix(grafana): use percentage for hummock read metrics #23675 : 2 dependent PRs (#23607 , #23676 ) 👈 (View in Graphite)
• feat(stream,dashboard): improve actor input buffer blocking metrics and add actor busy time panel #23612 : 1 other dependent PR (#23730 )
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[hzxa21]

These metrics are using rate and so the unit can't be per second, it should be a percentage. They reflect the percentage of time spent per interval reading records, either via iteration or point gets.

Can you explain more on this? I think it returns the tail duration in time unit instead of percentage.

For example, this returns the duration (in seconds) that 99% of your API calls completed within, calculated over the 5-minute window

histogram_quantile(0.99, sum(rate(api_call_duration_seconds_bucket[5m])) by (le, path))

[kwannoel]

These metrics are using rate and so the unit can't be per second, it should be a percentage. They reflect the percentage of time spent per interval reading records, either via iteration or point gets.

Can you explain more on this? I think it returns the tail duration in time unit instead of percentage.

For example, this returns the duration (in seconds) that 99% of your API calls completed within, calculated over the 5-minute window

histogram_quantile(0.99, sum(rate(api_call_duration_seconds_bucket[5m])) by (le, path))

Nevermind ignore me. I misinterpreted the metric: api_call_duration_seconds_bucket to be measuring seconds, rather than counts in duration buckets.