[RFE] Option to preserve file name using pmlogpush

[eberbesi]

Is it possible to add an option to keep the archive file name after is pushed to the pmproxy server? I am creating some archives locally with a specific info in the file name and when is pushed with pmlogpush the file name changes.

[kmcdonell]

Hi @eberbesi.

At the moment the directory and filename at the pmproxy end are sort of hard-wired to match the directory structure and date-and-time naming convention that pmlogger_daily is expecting for regular archive merging and culling. I'm guessing that does not match you use case. Do you also need control over the directory where the archive ends up?

I am assuming this is in some environment where you can make an http connection but cannot copy the file directly using scp(1) or rsync(1) or similar.

Perhaps a simple example from you would help confirm what you'd like to happen.

Changing this would require a protocol change for pmproxy, but introduces a potential problem with respect to the ownership of the file and containing directory, as pmproxy runs as the user "pcp".

[eberbesi]

Hi @kmcdonell,

Thanks for your response. I'll put some more context in my implementation.

I am trying to modify the current implementation of the performance analysis in computing clusters XDMOD(https://supremm.xdmod.org/11.0/supremm-compute-pcp.html) as their setup is using and older PCP version (4.3). My challenge is that I need somehow relate the collected metrics with the Slurm Job ID running in the compute nodes.
In their implementation, they add the Slurm Job ID in the archive file name and save it in a shared location (NFS, GPFS, etc).

Now with PCP offering a new Centralized logging (pmlogger push) on version 7, I am looking to adapt their setup to avoid the use of a shared location (for security reasons), and push the archives directly using pmlogger.
On compute nodes, there is a primary pmlogger always pushing archives to a pmproxy, a second pmlogger is manually launched after a new Slurm Job starts and ends in the node, which I need to tag with the Slurm Job ID.
Ideally, I would love to have the option of a custom archive label where I can save a file description including the Slurm Job ID that I desperate need.

I hope this explanation helps to clarify my request. Is there anything I can do that maybe I am missing??

Thanks,

[natoscott]

@eberbesi thanks for the details! One possible option that might help you with this aspect (I recommend this whether your filename contains Slurm job details or not) ...

Ideally, I would love to have the option of a custom archive label where I can save a file description including the Slurm Job ID that I desperate need.

... would be to write the Slurm job ID into /etc/pcp/labels/optional/slurm (uses JSON content, e.g. {"slurm-job-id":"xxx"}) - this will be automatically recorded in every archive from that host as a "context label", and is then available as search term in Grafana and so on.

[eberbesi]

Hi @natoscott,

Thanks for pitching in some ideas. This was my first thought but unfortunately it doesn't work in my implementation. A compute node may have more than one Slurm job running at the same time and the label needs to change dynamically every time a job starts. PMCD service needs to be restarted to take the new labels as well.

I found a solution that might work at the moment: Modify the hostname in the archive label to "'hostname'/'slurm job id'" using pmlogger -H option.
pmproxy will use the "new" hostname label to create the directory where the archives will be store: $PCP_ARCHIVE_DIR/'hostname'/'slurm job id'.
I tested it and worked as expected. I still need to modify the XDMOD SUPREMM application to parse the slurm job id from the archive directory, but that's another story...

Thanks again.

[natoscott]

| This was my first thought but unfortunately it doesn't work in my implementation.

Oh well, no problem.

| PMCD service needs to be restarted to take the new labels as well.

FWIW, pmcd will automatically detect changes in labels without restart ...

root@fedora:~# echo '{"a":1}' > /etc/pcp/labels/optional/a1
root@fedora:~# pminfo -l hinv.ncpu

hinv.ncpu
    labels {"a":1,"agent":"linux","domainname":"localdomain","groupid":0,"hostname":"fedora","latitude":-37.8140,"longitude":144.9633,"machineid":"7fe1ddf88e584d78b1a0069af7ce3e61","userid":0}
root@fedora:~# rm  /etc/pcp/labels/optional/a1
rm: remove regular file '/etc/pcp/labels/optional/a1'? y
root@fedora:~# pminfo -l hinv.ncpu

hinv.ncpu
    labels {"agent":"linux","domainname":"localdomain","groupid":0,"hostname":"fedora","latitude":-37.8140,"longitude":144.9633,"machineid":"7fe1ddf88e584d78b1a0069af7ce3e61","userid":0}
root@fedora:~#