Is there a way to identify pods that were rescheduled due to Karpenter disruption?

[kumo-rn5s]

Summary

When a pod gets recreated in our cluster, we need to know if it was caused by Karpenter node termination or something else (application crash, normal deployment, etc). Currently there's no straightforward way to identify pods that were evicted due to Karpenter disruption.

Background

We run staging/sandbox environments with many single-replica pods to save costs. Karpenter frequently consolidates nodes due to low utilization, causing pods to be evicted and recreated. Some pods experience downtime during rescheduling.

When investigating a recreated pod, we cannot tell if:

• The application crashed
• Karpenter terminated the node it was on
• It was a normal deployment

Right now we use this manual investigation process:

1. Find pod history in Datadog/Grafana and identify which node it was on
2. Search Karpenter logs for that node and record timestamps
3. Compare pod termination/recreation times with node termination to determine if Karpenter caused it

We do this multiple times a week. It's time consuming and not scalable.

What we need

A way to identify which pods were evicted by Karpenter. This would help us:

• Understand which pods are repeatedly affected by consolidation
• Decide which pods need PDBs or replica increases
• Distinguish between Karpenter disruptions and application issues

Ideally, when a pod is recreated after Karpenter eviction, there would be some indicator (annotation, label, event, etc) that shows it was evacuated due to Karpenter disruption.

Questions

How do others track pods evacuated by Karpenter? Is there a simpler approach than manual log correlation?

We tried Kubernetes Events but they're node-focused and expire after an hour. Are we missing something?

Should Karpenter provide this visibility, or should users build it themselves? I don't think workarounds like SQS → Lambda → CloudWatch is a good idea.

[FernandoMiguel]

karpenter events show what pods are being moved due to consolidation

[kumo-rn5s]

Thanks for the reply!

karpenter events show what pods are being moved due to consolidation

Are those events on the Karpenter pod itself? I don't see any recent events on my Karpenter pods. Should they normally appear there?

I sometimes see "disruption blocked" events on nodes, but is that different from what you're referring to?

[FernandoMiguel]

you can't use PDB with single replica.. that would prevent the pod from ever being replaced

[kumo-rn5s]

You are right. PDB with single replica would block eviction.