Instance labels missing from pmseries, part of the API and downstream consumers (proxy, pcp-grafana)

[turbo]

I'm writing a PMDA, and it works as expected, but instance label propagation seems inconsistent in PCP. This affects my PMDA, but also standard ones. It seems like some logic leads to the REST API not returning instance labels, which in turn causes errors in grafana-pcp and other downstream consumers.

System info:

• PCP 7.0.3 (installed from packagecloud repo)
• OS: ubuntu/noble
• pmproxy: Running as systemd service on port 44322

Can't share the PMDA code (it's platform specific anyway), but I'll show the behavior and also how to repro with a bundled PMDA.

The gist is that the following tools show all expected instance labels:

• pminfo --labels
• dbpmda with the label subcommand
• /metrics endpoint

But they're missing from:

• REST API (PMAPI)
• pmseries
  • and downstream pcp-grafana

Working through this with my own PMDA, which is exposing low-level AMD EPYC metrics:

static int
esmi_labelCallBack(pmInDom indom, unsigned int inst, pmLabelSet **lp)
{
    int serial;

    if (indom == PM_INDOM_NULL)
        return 0;

    serial = pmInDom_serial(indom);

    /* Add disp_instance label for socket indom */
    if (serial == SOCKET_INDOM) {
        if (inst < num_sockets && socket_names[inst] != NULL) {
            return pmdaAddLabels(lp, "{\"disp_instance\":\"%s\"}",
                                 socket_names[inst]);
        }
    }

    /* Add disp_instance, die_id, and socket_id labels for core indom */
    if (serial == CORE_INDOM) {
        if (inst < num_cores && core_names[inst] != NULL) {
            return pmdaAddLabels(lp, "{\"disp_instance\":\"%s\",\"die_id\":%d,\"socket_id\":%d}",
                                 core_names[inst], core_die_id[inst], core_socket_id[inst]);
        }
    }

    return 0;
}

All core-scope metrics get three instance labels to localize them on the chiplets and sockets. The PMDA installs and runs without warnings or errors.

Poking via dbpmda:

echo 'open pipe /var/lib/pcp/pmdas/esmi/pmdaesmi -d 470
label instances 470.1' | sudo dbpmda

output:

Start pmdaesmi PMDA: /var/lib/pcp/pmdas/esmi/pmdaesmi -d 470
Instances of pmInDom: 470.1
[  0] Labels inst: 0
    die_id=0
    disp_instance="core0"
    socket_id=0
[  1] Labels inst: 1
    die_id=0
    disp_instance="core1"
    socket_id=0
[  2] Labels inst: 2
    die_id=0
    disp_instance="core2"
    socket_id=0
    ...

Then pminfo:

pminfo --labels esmi.energy.core | head -n5

output:

esmi.energy.core
    labels {"agent":"esmi","device_type":"cpu_core","domainname":"localdomain","groupid":1000,"hostname":"aitop","indom_name":"per core","machineid":"4542127d93e1480e823cf51ba57d25a3","userid":1000}
    inst [0 or "core0"] labels {"agent":"esmi","device_type":"cpu_core","die_id":0,"disp_instance":"core0","domainname":"localdomain","groupid":1000,"hostname":"aitop","indom_name":"per core","machineid":"4542127d93e1480e823cf51ba57d25a3","socket_id":0,"userid":1000}
    inst [1 or "core1"] labels {"agent":"esmi","device_type":"cpu_core","die_id":0,"disp_instance":"core1","domainname":"localdomain","groupid":1000,"hostname":"aitop","indom_name":"per core","machineid":"4542127d93e1480e823cf51ba57d25a3","socket_id":0,"userid":1000}

Note: die_id, disp_instance, and socket_id are present in instance labels.

Metrics endpoint:

curl -s http://localhost:44322/metrics | grep esmi_energy_core | head -5

output:

# HELP esmi_energy_core Cumulative core energy consumption in Joules
# TYPE esmi_energy_core counter
esmi_energy_core{disp_instance="core0",agent="esmi",indom_name="per core",hostname="aitop",instid="0",instname="core0",machineid="4542127d93e1480e823cf51ba57d25a3",domainname="localdomain",die_id="0",socket_id="0",device_type="cpu_core"} 383191.321868
esmi_energy_core{disp_instance="core1",agent="esmi",indom_name="per core",hostname="aitop",instid="1",instname="core1",machineid="4542127d93e1480e823cf51ba57d25a3",domainname="localdomain",die_id="0",socket_id="0",device_type="cpu_core"} 339390.282028
esmi_energy_core{disp_instance="core2",agent="esmi",indom_name="per core",hostname="aitop",instid="2",instname="core2",machineid="4542127d93e1480e823cf51ba57d25a3",domainname="localdomain",die_id="0",socket_id="0",device_type="cpu_core"} 75834.41345199999

Still there ...

Now the instance domain REST API:

curl -s 'http://localhost:44322/pmapi/indom?indom=470.1' | jq | head -n30

Output:

{
  "context": 184337792,
  "indom": "470.1",
  "labels": {
    "device_type": "cpu_core",
    "domainname": "localdomain",
    "hostname": "aitop",
    "indom_name": "per core",
    "machineid": "4542127d93e1480e823cf51ba57d25a3"
  },
  "text-oneline": "Instance domain \"core\" for ESMI PMDA",
  "text-help": "One instance per physical CPU core detected by the ESMI library.\nInstances are named \"core0\", \"core1\", etc.\nNote: With SMT enabled, sibling threads share the same core.",
  "instances": [
    {
      "instance": 11,
      "name": "core11",
      "labels": {
        "domainname": "localdomain",
        "hostname": "aitop",
        "machineid": "4542127d93e1480e823cf51ba57d25a3"
      }
    },
    {
      "instance": 23,
      "name": "core23",
      "labels": {
        "domainname": "localdomain",
        "hostname": "aitop",
        "machineid": "4542127d93e1480e823cf51ba57d25a3"
      }

Instance labels are missing.

Now the metric endpoint:

curl -s 'http://localhost:44322/pmapi/metric?names=esmi.energy.core' | jq

Output:

{
  "context": 1243538900,
  "metrics": [
    {
      "name": "esmi.energy.core",
      "series": "a3f7748a41b1aca855f561e934ab092b2a78428f",
      "pmid": "470.5.0",
      "indom": "470.1",
      "type": "double",
      "sem": "counter",
      "units": "none",
      "labels": {
        "agent": "esmi",
        "device_type": "cpu_core",
        "domainname": "localdomain",
        "hostname": "aitop",
        "indom_name": "per core",
        "machineid": "4542127d93e1480e823cf51ba57d25a3"
      },
      "text-oneline": "Cumulative core energy consumption in Joules",
      "text-help": "The cumulative energy consumption of each physical core in Joules.[...]"
    }
  ]
}

Now this isn't specific to my PMDA, for example I noticed missing labels for a NVIDIA, disk, and others, so dashboards/pmrpoxy queries lack important context.

For example, there's a device_name here:

pminfo --labels disk.dev.read 2>/dev/null | grep device_name | head -n1
#     inst [0 or "nvme3n1"] labels {"agent":"linux","device_name":"nvme3n1","device_type":"block","domainname":"localdomain","groupid":1000,"hostname":"aitop","indom_name":"per disk","machineid":"4542127d93e1480e823cf51ba57d25a3","userid":1000}

But not here:

pmseries -l `pmseries 'disk.dev.read'` 2>&1 | grep nvme3n1
#    inst [0 or "nvme3n1"] labels {"agent":"linux","device_type":"block","domainname":"localdomain","groupid":986,"hostname":"aitop","indom_name":"per disk","machineid":"4542127d93e1480e823cf51ba57d25a3","userid":997}

nor here:

curl -s 'http://localhost:44322/pmapi/indom?indom=60.1' | jq '.instances[] | select(.name == "nvme3n1")'
# {
#   "instance": 0,
#   "name": "nvme3n1",
#   "labels": {
#     "domainname": "localdomain",
#     "hostname": "aitop",
#     "machineid": "4542127d93e1480e823cf51ba57d25a3"
#   }
# }

And another example from NVIDIA:

pminfo --labels nvidia.power

# nvidia.power
#    labels {"agent":"nvidia","device_type":"gpu","domainname":"localdomain","groupid":1000,"hostname":"aitop","indom_name":"per gpu","machineid":"4542127d93e1480e823cf51ba57d25a3","units":"milliwatts","userid":1000}
#    inst [0 or "gpu0"] labels {"agent":"nvidia","device_type":"gpu","domainname":"localdomain","gpu":0,"groupid":1000,"hostname":"aitop","indom_name":"per gpu","machineid":"4542127d93e1480e823cf51ba57d25a3","units":"milliwatts","userid":1000,"uuid":"GPU-7235b8ec-cfc6-c44b-967f-c404e4564320"}
#    inst [1 or "gpu1"] labels {"agent":"nvidia","device_type":"gpu","domainname":"localdomain","gpu":1,"groupid":1000,"hostname":"aitop","indom_name":"per gpu","machineid":"4542127d93e1480e823cf51ba57d25a3","units":"milliwatts","userid":1000,"uuid":"GPU-4db1918f-1bb1-7200-983c-0ceb1ae0c7b3"}
#    inst [2 or "gpu2"] labels {"agent":"nvidia","device_type":"gpu","domainname":"localdomain","gpu":2,"groupid":1000,"hostname":"aitop","indom_name":"per gpu","machineid":"4542127d93e1480e823cf51ba57d25a3","units":"milliwatts","userid":1000,"uuid":"GPU-bb8efb89-eacc-8ef4-fd19-4ca522115940"}

gpu exists as a label here, but not the in API, the proxy, Redis or others.

[turbo]

Hypothesis (I am yet to test as I work on getting PCP to build locally). I think the hashing logic causes this.

It's first called here:

pcp/src/libpcp_web/src/util.c

Lines 979 to 995 in 6c5f084

	struct instance *
	pmwebapi_new_instance(indom_t *indom, int inst, sds name)
	{
	struct instance *instance;
	if (name == NULL || sdslen(name) == 0)
	return NULL;
	if ((instance = calloc(1, sizeof(instance_t))) == NULL)
	return NULL;
	instance->inst = inst;
	instance->name.sds = name;
	pmwebapi_string_hash(instance->name.id, name, sdslen(name));
	pmwebapi_instance_hash(indom, instance);
	dictAdd(indom->insts, &inst, (void *)instance);
	return instance;
	}

As pmwebapi_instance_hash(indom, instance);. At this point instance->labels is guaranteed to contain data because of the logic in pmwebapi_instance_hash (see below).

Then in this block:

pcp/src/libpcp_web/src/util.c

Lines 911 to 950 in 6c5f084

	if (indom->updated == 0) {
	sts = nsets = pmGetInstancesLabels(indom->indom, &labelsets);
	if (sts == PM_ERR_IPC)
	context->setup = 0;
	for (i = 0; i < nsets; i++) {
	labels = &labelsets[i];
	if ((length = labelsetlen(labels)) == 0)
	continue;
	inst = labelsets[i].inst;
	if ((instance = dictFetchValue(indom->insts, &inst)) == NULL)
	continue;
	if ((labels = pmwebapi_labelsetdup(labels)) == NULL) {
	if (pmDebugOptions.series)
	fprintf(stderr, "failed to dup %s instance labels: %s\n",
	pmInDomStr_r(indom->indom, buffer, sizeof(buffer)),
	pmErrStr_r(sts, errmsg, sizeof(errmsg)));
	continue;
	}
	if (instance->labelset)
	pmFreeLabelSets(instance->labelset, 1);
	instance->labelset = labels;
	pmwebapi_instance_hash(indom, instance);
	if (pmDebugOptions.series) {
	pmwebapi_hash_str(instance->name.hash, buffer, sizeof(buffer));
	fprintf(stderr, "pmwebapi instance %s", instance->name.sds);
	fprintf(stderr, "\nSHA1=%s\n", buffer);
	}
	}
	if (sts >= 0)
	indom->updated = 1;
	else if (pmDebugOptions.series)
	fprintf(stderr, "failed to get indom (%s) instance labels: %s\n",
	pmInDomStr_r(indom->indom, buffer, sizeof(buffer)),
	pmErrStr_r(sts, errmsg, sizeof(errmsg)));
	} else {
	assert(sts >= 0);
	indom->updated = 1;
	}

This happens:

1. Labels are retrieved via sts = nsets = pmGetInstancesLabels(indom->indom, &labelsets);

2. Then they're swapped in:

   if (instance->labelset)
      pmFreeLabelSets(instance->labelset, 1);
   instance->labelset = labels;
   

3. pmwebapi_instance_hash(indom, instance); is called again

Then this check is re-run pmwebapi_instance_hash:

pcp/src/libpcp_web/src/util.c

Lines 519 to 524 in 6c5f084

	if (instance->labels == NULL) {
	len = instance_labelsets(ip, instance, buf, sizeof(buf), labels, NULL);
	if (len <= 0)
	len = pmsprintf(buf, sizeof(buf), "null");
	instance->labels = sdsnewlen(buf, len);
	}

However, since pmwebapi_new_instance already triggered the update instance->labels is not NULL, this one is skipped, and the data from pmGetInstancesLabels is discarded.

---

In contrast, the working metrics endpoint seems to avoid double-hashing:

pcp/src/libpcp_web/src/webgroup.c

Lines 1829 to 1846 in 235e7e2

	instance = dictFetchValue(indom->insts, &value->inst);
	if (instance == NULL)
	continue;
	v = webgroup_encode_value(v, type, &value->atom);
	series = pmwebapi_hash_sds(series, instance->name.hash);
	scrape.value.series = series;
	scrape.value.inst = value->inst;
	scrape.value.value = v;
	scrape.instance.inst = instance->inst;
	scrape.instance.name = instance->name.sds;
	if (instance->labels == NULL)
	pmwebapi_instance_hash(indom, instance);
	scrape_instance_labelsets(metric, indom, instance, &labels);
	if (settings->callbacks.on_scrape_labels)
	settings->callbacks.on_scrape_labels(
	cp->origin, &labels, arg);
	scrape.instance.labels = labels.buffer;

[turbo]

Alright, here's a fix: https://github.com/turbo/pcp/compare/6c5f0844f83927be023e6af8679426d59ff43357..e1b1c2fe7b4bd07ad5d2ea4d174756eedbc191f7

This works with the current PCP built from main on ubuntu/noble. Metrics now appear:

curl -s "http://localhost:44322/pmapi/indom?name=esmi.energy.core" | python3 -m json.tool 2>/dev/null | head -40
{
    "context": 1497977127,
    "indom": "470.1",
    "labels": {
        "device_type": "cpu_core",
        "domainname": "localdomain",
        "hostname": "aitop",
        "indom_name": "per core",
        "machineid": "4542127d93e1480e823cf51ba57d25a3"
    },
    "text-oneline": "Instance domain \"core\" for ESMI PMDA",
    "text-help": "One instance per physical CPU core detected by the ESMI library.\nInstances are named \"core0\", \"core1\", etc.\nNote: With SMT enabled, sibling threads share the same core.",
    "instances": [
        {
            "instance": 11,
            "name": "core11",
            "labels": {
                "device_type": "cpu_core",
                "die_id": 3,
                "disp_instance": "core11",
                "domainname": "localdomain",
                "hostname": "aitop",
                "indom_name": "per core",
                "machineid": "4542127d93e1480e823cf51ba57d25a3",
                "socket_id": 0
            }
        },

pmseries -l `pmseries 'esmi.energy.core'` | head -n10

a3f7748a41b1aca855f561e934ab092b2a78428f
    inst [0 or "core0"] labels {"agent":"esmi","device_type":"cpu_core","die_id":0,"disp_instance":"core0","domainname":"localdomain","groupid":986,"hostname":"aitop","indom_name":"per core","machineid":"4542127d93e1480e823cf51ba57d25a3","socket_id":0,"userid":997}
    inst [1 or "core1"] labels {"agent":"esmi","device_type":"cpu_core","die_id":0,"disp_instance":"core1","domainname":"localdomain","groupid":986,"hostname":"aitop","indom_name":"per core","machineid":"4542127d93e1480e823cf51ba57d25a3","socket_id":0,"userid":997}
    inst [2 or "core2"] labels {"agent":"esmi","device_type":"cpu_core","die_id":0,"disp_instance":"core2","domainname":"localdomain","groupid":986,"hostname":"aitop","indom_name":"per core","machineid":"4542127d93e1480e823cf51ba57d25a3","socket_id":0,"userid":997}
    inst [3 or "core3"] labels {"agent":"esmi","device_type":"cpu_core","die_id":1,"disp_instance":"core3","domainname":"localdomain","groupid":986,"hostname":"aitop","indom_name":"per core","machineid":"4542127d93e1480e823cf51ba57d25a3","socket_id":0,"userid":997}
    inst [4 or "core4"] labels {"agent":"esmi","device_type":"cpu_core","die_id":1,"disp_instance":"core4","domainname":"localdomain","groupid":986,"hostname":"aitop","indom_name":"per core","machineid":"4542127d93e1480e823cf51ba57d25a3","socket_id":0,"userid":997}
    inst [5 or "core5"] labels {"agent":"esmi","device_type":"cpu_core","die_id":1,"disp_instance":"core5","domainname":"localdomain","groupid":986,"hostname":"aitop","indom_name":"per core","machineid":"4542127d93e1480e823cf51ba57d25a3","socket_id":0,"userid":997}
    inst [6 or "core6"] labels {"agent":"esmi","device_type":"cpu_core","die_id":2,"disp_instance":"core6","domainname":"localdomain","groupid":986,"hostname":"aitop","indom_name":"per core","machineid":"4542127d93e1480e823cf51ba57d25a3","socket_id":0,"userid":997}
    inst [7 or "core7"] labels {"agent":"esmi","device_type":"cpu_core","die_id":2,"disp_instance":"core7","domainname":"localdomain","groupid":986,"hostname":"aitop","indom_name":"per core","machineid":"4542127d93e1480e823cf51ba57d25a3","socket_id":0,"userid":997}

And in PCP-Grafana as well:

[natoscott]

@turbo fantastic! Could you open a PR? On very quick review, I notice some comments mentioning "Redis" - we support either Redis/Valkey now, and so code was changed to generically refer to a "key server"(if you could tweak those references, that'd be great).

I wonder is this issue is reproducible with the sample PMDA as well, so that we could add a regression test - possibly with the mirage metric?

pminfo -ftT sample.mirage sample.controller.mirage

[kmcdonell]

@turbo I'm borderline-competent in most parts of PCP, but libpcp_web is not one of them.

But I wanted to take the time to thank you. I've been dealing with customer-reported software (and hardware) issues for close to 50 years, and this issue is one of the best documented and best triaged ones I've seen in that time. And for extra points you've proposed a fix.

So thank you once again.