Monitoring mixin: dashboard + alerts

[clementduveau]

Hello,

As I suspected, one of my disk was failing and so I took time to build a dashboard and a set of alerts to proactively monitor this. Instead of having a separate unknown repo somewhere, I hereby submit this mixin to share and develop with the community.

The dashboard and alerts are pre-compiled in the output folder.

BTW, it's my first mixin. I tried to follow some examples from other mixins but I am gueninely not sure what are the best practices there. It works but happy to tweak anything to comply with a different way of doing that.

Dashboard

Variables:

• Prometheus datasource
• Instance (multi-values)

3 rows:

• General overview:
  • state of the probes (smartctl exit status)
  • number of disk failing
  • list of disks (id, instance, serial numbers, model, etc)
  • temperature graph
  • list of prefailures checks met
• HDD / SDD
  • Count of failing disks (all signals aggregated)
  • SSD wearout
  • Offline sectors
  • Pending sectors
  • Reallocated sectors (graph as steep increase is a clearer signal than a steady low number)
  • Error log count (graph, same reason)
• NVMe
  • Lifetime
  • Media errors

Alerts

• Temperature warning (60°C)
• Temperature critical (70°C)
• Temperature near trip value
• Temperature over trip value
• SMART status (smartctl exit status)
• Media errors
• Critical warning
• Wearout indicator
• SSD wearout indicator
• SMART Error logs increase