Gather pod/container/project based metrics

[mwringe]

Since Heapster is being deprecated, we probably want to gather the same metrics that Heapster is currently gathering and store those into Hawkular Metrics.

In essence the agent would be reading the metrics directly from the /metrics endpoint of the node, gathering the metrics for each individual pod/container, fetching the metadata associated for those pod/containers/projects and writing them to the proper tenant in Hawkular Metrics.

[jmazzitelli]

@burmanm @mwringe I'm going to try to see if I can code something up quickly for this. Can you fill me in on the secret sauce I'm going to need to know in order to collect these node metrics? Do I go through the api master? Is there some other endpoint that exposes these metrics? Are these metrics exposed as a prometheus endpoint? anything like that.

[mwringe]

The endpoint where these metrics reside is ${NODE_IP}/metrics

You will probably need to get those metrics and parse out what each metric means and where it should go.

Part of this will likely require getting additional information from the api master (eg things like pod id, labels, etc)

[jmazzitelli]

FYI: the /metrics endpoint is exposing its data as a Prometheus endpoint. So the agent has the ability to read it. I would just need to know what we want to scrape and how we want to store it in h-metrics. @mwringe or @burmanm - can you give me some insight into what kinds of node metrics we want to collect and store?

NOTE ALSO: the only way for me to authenticate was to pass in the OpenShift bearer token (I used the one I am given in the UI Console). It there some more permanent authentication credentials to use? Because these tokens have a short shelf-life so we can't put them in the agent config. We would need something else to authenticate with.

[mwringe]

----- Original Message -----

From: "John Mazzitelli" ***@***.***> To: "hawkular/hawkular-openshift-agent" ***@***.***> Cc: "Matthew Wringe" ***@***.***>, "Mention" ***@***.***> Sent: Friday, 24 March, 2017 8:14:51 PM Subject: Re: [hawkular/hawkular-openshift-agent] Gather pod/container/project based metrics (#110) FYI: the /metrics endpoint is exposing its data as a Prometheus endpoint. So the agent has the ability to read it. I would just need to know what we want to scrape and how we want to store it in h-metrics. @mwringe or @burmanm - can you give me some insight into what kinds of node metrics we want to collect and store?

What I would be doing would be to install Heapster and figure out what metrics it currently collects and start from there. Or even look at the summary api endpoint and see what that exposes. We can get the pod information from the master api and use that to parse out the information returned under /metrics (and this is also needed to get the tag information) I would start with one metric type, say memory usage, and determine from /metrics what pattern is being used there.

NOTE ALSO: the only way for me to authenticate was to pass in the OpenShift bearer token (I used the one I am given in the UI Console). It there some more permanent authentication credentials to use? Because these tokens have a short shelf-life so we can't put them in the agent config. We would need something else to authenticate with.

The agent already uses a service account which has a long lived token (it only expires when the pod is destroyed). This is the token we are using when connecting to the Kubernetes API server. We need to use the same one when accessing any other Kubernetes/OpenShift endpoints (eg /metrics) We may need to grant it extra permission to read the /metrics endpoint, but other than that we need to reuse the same service account token we have always been using.

…

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #110 (comment)