Is it possible to obtain kubernetes (node/pod level) metrics and forward them to a sink?

[hickersonj]

I see that vector can obtain host metrics (CPU/Mem/Disk), but is there anyway to obtain node level metrics and pod level metrics as a source and forward them via a sink? I looked at internal metrics, but that doesn't seem to be what I am looking for. Thanks!

[pront]

Yes. Vector's prometheus_scrape source can collect these metrics and forward them to a metrics-capable sink. The endpoint depends on what you need:

• Pod/container CPU and memory usage: the kubelet's /metrics/cadvisor endpoint on each node.
• Node hardware and OS metrics: node-exporter.
• Kubernetes object state, such as pod phase and deployment replicas: kube-state-metrics. This does not provide actual CPU or memory usage.

For example, with kube-state-metrics installed, this prints the collected metrics as JSON. Use your service's actual address:

sources:
  kubernetes_metrics:
    type: prometheus_scrape
    endpoints:
      - http://kube-state-metrics.monitoring.svc:8080/metrics
    scrape_interval_secs: 15

sinks:
  inspect:
    type: console
    inputs: [kubernetes_metrics]
    encoding:
      codec: json

A forwarding sink can consume the same kubernetes_metrics input. For kubelet or node-exporter metrics, configure each node's endpoint and any required authentication/TLS settings. internal_metrics describes Vector itself.