Document how to debug performance of prow job

[serathius]

What would you like to be added?

In robustness tests we noticed a series of flakes and we wanted to eliminate performance as a factor. @ah8ad3 has reached out to Kubernetes prow and found interesting dashboards:
https://monitoring-gke.prow.k8s.io/d/96Q8oOOZk/builds?orgId=1&refresh=30s&var-org=etcd-io&var-repo=etcd&var-job=ci-etcd-robustness-main-amd64&var-build=All&from=now-7d&to=now

We should preserve this knowledge and document for future how etcd contributors can navigate performance of prow jobs. Possibly a new section in etcd contributor documentation about prow that either links to the dashboard or some prow documentation.

Why is this needed?

Make it easier for others to debug etcd job performance issues.

[serathius]

cc @ah8ad3

[serathius]

cc @jmhbnz @ivanvc

[ivanvc]

I agree. There's also an EKS dashboard, but they are not documented anywhere in the infra testing repo, I also found about it by asking on Slack.

• GKE: https://monitoring-gke.prow.k8s.io/d/96Q8oOOZk/builds?orgId=1&refresh=30s&var-org=etcd-io&var-repo=etcd&var-build=All&from=now-7d&to=now
• EKS: https://monitoring-eks.prow.k8s.io/d/96Q8oOOZk/builds?orgId=1&refresh=30s&var-org=etcd-io&var-repo=etcd&var-build=All&from=now-7d&to=now

[CFly17]

I am interested in learning more about and contributing to etcd!

Are you thinking of a new doc (like etcd/Documentation/contributor-guide/prow_jobs.md) with sections like:

• Introduction to Prow
• How Prow is Used for etcd Testing
• Creating/Navigating Performance Dashboard (Grafana)
• Interpreting Metrics (link to prow documentation)
• Troubleshooting Common Issues (e.g.?)
• etc.

Please let me know if I can help! :)

[ivanvc]

Hi, @CFly17, thanks for volunteering!

Are you thinking of a new doc (like etcd/Documentation/contributor-guide/prow_jobs.md) with sections like:

Yes, we need a new document, and your suggestion sounds good. I think we can start with:

• Introduction to Prow
• How Prow is Used for etcd Testing - And where the jobs are located, i.e., test-infra repository, in the config/jobs/etcd directory
• Navigating Performance Dashboard (Grafana) (just navigating. I don't think we have an intention to create dashboards at this moment
• Selecting our jobs, explaining the different kinds of workloads/jobs, i.e., robustness, integration, static check, etc. (at a high level).
• Interpreting Metrics

Feel free to open a pull request (or draft pull request). Would you like to be assigned? If so, just send a comment with /assign (in plain text, w/o code backticks), and the k8s bot will assign it to you.

[CFly17]

/assign

[ivanvc]

We reviewed this task in the triage meeting. @CFly17, are you still working on this one, or can we reassign it? Thanks.

[CFly17]

Please forgive me @ivanvc -- I tried reading up on the Raft algorithm and got lost in the sauce. I hope to contribute to etcd sometime, but I fear I may be in over my head with this one.

[CFly17]

/unassign

[joshjms]

/assign