This change introduces using prometheus exporter integration

[manoj-freyr]

We have device-metrics-exporter as publicly available package to have prometheus exporter capability. Adding this to nodes to export device metrics which can be collected by central node and also have grafna integration to the same with said prometheus data source. Tested locally for deployemnt using

./utils/deploy_monitoring_stack.sh
which deploys these three utilities to nodes.
After this can run any other cvs tests.

Device Metrics Exporter runs 24/7 exposing GPU metrics
Prometheus scrapes metrics every 15 seconds automatically
Tests run independently - metrics collected in background
After test, you can query Prometheus to see what happened during test

./utils/deploy_monitoring_stack.sh
============================================
CVS Monitoring Stack Deployment
============================================

Working Directory: /etc/cvs
Cluster File: ./input/cluster_file/local_test_cluster.json
Monitoring Config: ./input/config_file/monitoring/monitoring_config.json
Prometheus Version: 2.55.0
Grafana Version: 10.4.1
Exporter Version: v1.4.0

post deployment success we can query either curl or UI based:

curl -s 'http://localhost:9090/api/v1/query?query=gpu_edge_temperature' | jq -r '.data.result[] | "GPU (.metric.gpu_id) on (.metric.hostname): (.value[1])°C"'

or via browsing at http://localhost:9090/

Running gives

./utils/deploy_monitoring_stack.sh input/cluster_file/sample_monitor_cluster.json input/config_file/monitoring/monitoring_config.json

and

pytest tests/monitoring/install_device_metrics_exporter.py --cluster_file=input/cluster_file/sample_monitor_cluster.json --config_file=input/config_file/monitoring/monitoring_config.json v

[solaiys]

its same as clustor.json template right? User need to fill in the nodes and key details.

May be we can rename it as sample_monitor_cluster.json

[solaiys]

I think you added this file for testing with the local host. which may not be needed in actual case.

May be we can rename it as "sample_localhost_monitor_clustor.json"