feat(slurm): Add native Prometheus OpenMetrics telemetry for Slurm >= 25.11

[AdarshK15]

Summary

This PR introduces native OpenMetrics (Prometheus) telemetry support to the Slurm controller module (schedmd-slurm-gcp-v6-controller), exporting Slurm metrics directly to Google Cloud Monitoring.

For Slurm clusters running version >= 25.11, this implementation leverages Slurm's native metrics/openmetrics plugin for Prometheus exporter. These metrics are exported directly from the slurmctld daemon natively, they are collected and exported from the controller instance only.

List of supported endpoints:

$ curl localhost:6818/metrics
slurmctld index of metrics endpoints:
  '/metrics/jobs': get job metrics
  '/metrics/nodes': get node metrics
  '/metrics/partitions': get partition metrics
  '/metrics/jobs-users-accts': get user and account jobs metrics
  '/metrics/scheduler': get scheduler metrics

To know more about these endpoints, visit the SchedMD Metrics Documentation.

Targeted Scraping

Because the root /metrics endpoint returns a plain-text index instead of raw OpenMetrics payloads, the Google Cloud Ops Agent cannot scrape it directly. To handle this, we explicitly configure distinct scrape jobs to target the sub-endpoints individually.
Additionally, we intentionally omit scraping the /metrics/jobs-users-accts endpoint. This endpoint duplicates job metrics broken down by every user and account, which introduces excessively high metric cardinality and bloat without adding significant value to cluster-level observability.

Key Changes

• Terraform Variables: Added the enable_openmetrics boolean variable to the controller and slurm_files modules.
• Slurm Configuration: Updated conf_v2511.py to conditionally inject MetricsType=metrics/openmetrics into the slurm.conf generation logic.
• Ops Agent Pipeline: Updated setup.py to dynamically inject the Prometheus receiver into /etc/google-cloud-ops-agent/config.yaml, configuring the 4 targeted scrape jobs.
• Unit Tests: Added parameterized tests in tests/test_conf.py to ensure these flags only ever get applied to Slurm 25.11 clusters.

Testing & Cluster Validation

Tested manually with a live cluster running Slurm 25.11.
1. Configuration Check:
Verified that the OpenMetrics plugin was correctly injected into the Slurm configuration:

[adarsh2511-controller ~]$ scontrol show config | grep MetricsType
MetricsType             = metrics/openmetrics

2. OpenMetrics Endpoint: Verified that the controller natively exposes Prometheus parseable metrics on the control host port (6818) by querying one of the targeted sub-paths.

[adarsh2511-controller ~]$ curl -s localhost:6818/metrics/nodes | head -n 10
# HELP slurm_node_cpus Total number of cpus in the node
# TYPE slurm_node_cpus gauge
slurm_node_cpus{node="adarsh2511-dynamicnodeset-0"} 1
slurm_node_cpus{node="adarsh2511-dynamicnodeset-1"} 1
slurm_node_cpus{node="adarsh2511-dynamicnodeset-2"} 1
slurm_node_cpus{node="adarsh2511-dynamicnodeset-3"} 1
# HELP slurm_node_cpus_alloc Allocated cpus in the node
# TYPE slurm_node_cpus_alloc gauge
slurm_node_cpus_alloc{node="adarsh2511-dynamicnodeset-0"} 0
slurm_node_cpus_alloc{node="adarsh2511-dynamicnodeset-1"} 0

3. Ops Agent Pipeline: Verified that the Python setup scripts successfully injected the safe receiver and pipeline definitions into the Ops Agent without overwriting existing logging pipelines.

[adarsh2511-controller ~]$ cat /etc/google-cloud-ops-agent/config.yaml
# ... (logging pipelines preserved) ...
metrics:
  receivers:
    slurm_prometheus:
      type: prometheus
      config:
        scrape_configs:
        - job_name: slurm-controller-jobs
          metrics_path: /metrics/jobs
          scrape_interval: 60s
          static_configs:
          - targets:
            - localhost:6818
# ... (other endpoints) ...
  service:
    pipelines:
      prometheus_pipeline:
        receivers:
        - slurm_prometheus

4. Cloud Monitoring: Verified that metrics are exposed in GCP Metric Explorer under Prometheus/Slurm.

Backward Compatibility

Verified backward compatibility by setting enable_openmetrics = true in the blueprint for a cluster with Slurm version 25.05. The version checking logic successfully caught the unsupported version, ensuring that the Ops Agent safely skipped configuring the Prometheus receivers and pipelines, and that MetricsType=metrics/openmetrics was not injected into the Slurm config.

Usage Example

To enable this feature, set enable_openmetrics to true on the controller module in your blueprint. Note that this requires a Slurm version of 25.11 or higher.

- id: slurm_controller
    source: community/modules/scheduler/schedmd-slurm-gcp-v6-controller
    settings:
      enable_openmetrics: true

[gemini-code-assist]

Code Review

This pull request refactors the Slurm configuration generation logic into an object-oriented structure to support multiple Slurm versions, introducing specific generators for Slurm 24.11, 25.05, and 25.11. It also adds support for native Prometheus OpenMetrics telemetry via Slurm and Google Cloud Ops Agent, and introduces an experimental enable_async_reply setting. The review feedback highlights a potential KeyError in setup.py when initializing nested dictionaries for the Ops Agent configuration, suggesting a safer initialization approach.

[AdarshK15]

/gemini summary

[AdarshK15]

/gemini review

[gemini-code-assist]

Summary of Changes

This pull request enables native Prometheus OpenMetrics telemetry for Slurm controllers on GCP. By leveraging Slurm's built-in metrics plugin and configuring the Google Cloud Ops Agent to scrape targeted endpoints, the changes provide improved cluster-level observability. The implementation includes version-aware logic to ensure compatibility and safe configuration injection, with comprehensive testing to validate the feature across different Slurm versions.

Highlights

• Native OpenMetrics Support: Introduced native Prometheus OpenMetrics telemetry for Slurm clusters running version 25.11 or higher by enabling the metrics/openmetrics plugin.
• Ops Agent Integration: Configured the Google Cloud Ops Agent to scrape specific Slurm sub-endpoints, ensuring robust observability while avoiding high-cardinality data from user-account metrics.
• Configuration Management: Added Terraform variables to toggle telemetry and updated Python setup scripts to dynamically inject scrape configurations into the Ops Agent.
• Validation and Testing: Implemented parameterized unit tests to verify that telemetry features are correctly applied only to supported Slurm versions.

Activity

• The author requested a summary and review via automated commands.
• A review comment regarding safe dictionary initialization in the Ops Agent configuration was addressed by the author.
• The author confirmed the implementation of safe dictionary initialization to prevent potential KeyErrors.

[gemini-code-assist]

Code Review

This pull request introduces support for native Prometheus OpenMetrics telemetry via Slurm and the Google Cloud Ops Agent by adding the enable_openmetrics configuration option. Feedback on the changes highlights two key areas of improvement in the Ops Agent setup script: first, ensuring that the prometheus_pipeline is initialized with the required exporters list (e.g., ["google"]) to prevent configuration validation failures; and second, implementing a cleanup mechanism to remove the slurm_prometheus receiver and pipeline references if enable_openmetrics is subsequently disabled.

[AdarshK15]

/gemini review

[gemini-code-assist]

Code Review

This pull request adds support for native Prometheus OpenMetrics telemetry via Slurm and the Google Cloud Ops Agent by introducing the enable_openmetrics configuration option. Feedback on the changes highlights two main areas for improvement: first, the cleanup logic in setup.py should recursively remove empty parent configurations (such as prometheus_pipeline and metrics) when slurm_prometheus is removed, preventing Google Cloud Ops Agent validation failures; second, retrieving slurm_control_host_port from lkp.cfg should use a defensive .get() call with a default value to avoid potential AttributeErrors.

[AdarshK15]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces the enable_openmetrics variable to enable native Prometheus OpenMetrics telemetry via Slurm and the Google Cloud Ops Agent on Slurm version 25.11 and above. Feedback on the changes suggests using getattr instead of .get() on the configuration object to prevent AttributeError during unit tests, and raising an explicit ValueError if the user enables OpenMetrics on an unsupported Slurm version rather than silently skipping the configuration.