Descheduler triggers infinite eviction/reschedule loop with Cluster Autoscaler and spot nodes

[AlexGurtoff]

What version of descheduler are you using?

descheduler version: chart version is 0.31.0

Does this issue reproduce with the latest release?

Yes

Which descheduler CLI options are you using?

--policy-config-file=/policy-dir/policy.yaml --descheduling-interval=60m --v=3

Please provide a copy of your descheduler policy config file

  policy.yaml: |
    apiVersion: "descheduler/v1alpha2"
    kind: "DeschedulerPolicy"
    profiles:
    - name: default
      pluginConfig:
      - args:
          evictLocalStoragePods: true
          ignorePvcPods: true
        name: DefaultEvictor
      - args:
          nodeAffinityType:
          - preferredDuringSchedulingIgnoredDuringExecution
        name: RemovePodsViolatingNodeAffinity
      plugins:
        deschedule:
          enabled:
          - RemovePodsViolatingNodeAffinity

What k8s version are you using (kubectl version)?

kubectl version Output

$ kubectl version
Server Version: v1.31.8-gke.1045000

What did you do?

We are using the RemovePodsViolatingNodeAffinity strategy to evict pods from on-demand nodes and schedule them on spot ones

Scenario:

1. Some pods were initially scheduled on on-demand nodes because spot nodes were temporarily unavailable.

2. The descheduler evicted those pods due to node affinity preferences.

3. New pods were created and marked as pending.

4. Cluster Autoscaler saw that pending pods require spot nodes and began scaling up the corresponding spot node pools.

5. However, the Kubernetes scheduler, not seeing the (yet-to-be-created) spot nodes, scheduled the pending pods back on available on-demand nodes.

6. As a result, pods end up again on the regular nodes, and the spot nodes that eventually come up remain unused.

7. Then Cluster Autoscaler detects the spot nodes are unused and scales them down.

8. This cycle repeats endlessly, causing unnecessary evictions and scaling churn.

What did you expect to see?
I expected the descheduler to somehow avoid creating such a tight eviction loop, potentially by:

• Pre-scaling of target node pools: Descheduler could have an option to trigger pre-scaling of node pools that match the preferred node affinity (e.g., spot pools), similar to how it’s done in estafette-gke-node-pool-shifter. This would ensure that suitable nodes are already provisioning before evictions occur.

Or perhaps smarter coordination between descheduler and cluster autoscaler behavior.

Environment Details:
Cloud Provider: GCP
Nodes: mix of on-demand and spot
Cluster Autoscaler: enabled

[a7i]

If you use preferredDuringSchedulingIgnoredDuringExecution , then I recommend enabling nodeFit check so that Descheduler only evicts them if there's another node available.

As a result, pods end up again on the regular nodes, and the spot nodes that eventually come up remain unused.
Then Cluster Autoscaler detects the spot nodes are unused and scales them down.

Why is this the case? What is your Descheduler interval, and what is your Cluster Autoscaler scale down unneeded threshold? I would expect Descheduler to evict and put them on the Spot nodes, preventing the scale down.

[AlexGurtoff]

@a7i

If you use preferredDuringSchedulingIgnoredDuringExecution , then I recommend enabling nodeFit check so that Descheduler only evicts them if there's another node available.

As a result, pods end up again on the regular nodes, and the spot nodes that eventually come up remain unused.
Then Cluster Autoscaler detects the spot nodes are unused and scales them down.

Why is this the case? What is your Descheduler interval, and what is your Cluster Autoscaler scale down unneeded threshold? I would expect Descheduler to evict and put them on the Spot nodes, preventing the scale down.

1. Regarding nodeFit - yes, I understand that with it enabled, Descheduler will only evict pods when matching nodes are already available. However, in our case, this would prevent any eviction from happening unless there’s already a spot node up and running. This defeats our goal, which is to move workloads to spot nodes as soon as they become available, not just during redeployments or HPA scale-ups. So we intentionally keep nodeFit disabled to proactively trigger spot node provisioning via pending pods.

2. Our Descheduler interval is set to 30 minutes. We chose this value to avoid overly frequent evictions when spot nodes are not available. The issue is that if Descheduler runs too often, it keeps evicting pods even when no spot nodes are present - leading to unnecessary pod churn and wasting autoscaler cycles. That’s why we try to balance this with a slower interval

Hope that clarifies the reasoning behind our setup

[a7i]

If a pod has preferredDuringSchedulingIgnoredDuringExecution, CA is not going to force a scale-up as Scheduler will put it back on any available nodes (even if less desirable - which in this case is on-demand)

I think this is less of a Descheduler issue, but rather your setup. Descheduler offers nodeFit to prevent loops when there is no nodes available.

[AlexGurtoff]

If a pod has preferredDuringSchedulingIgnoredDuringExecution, CA is not going to force a scale-up as Scheduler will put it back on any available nodes (even if less desirable - which in this case is on-demand)

I think this is less of a Descheduler issue, but rather your setup. Descheduler offers nodeFit to prevent loops when there is no nodes available.

Just to clarify - as I mentioned earlier:

Some pods were initially scheduled on on-demand nodes because spot nodes were temporarily unavailable.
The descheduler evicted those pods due to node affinity preferences.
New pods were created and marked as pending.
Cluster Autoscaler saw that these pending pods require spot nodes and started scaling up the spot node pool.

So in our case, Cluster Autoscaler does trigger a scale-up for spot nodes, even if there are still free on-demand nodes. This happens because the pending pods have node affinity that prefers spot nodes, and CA takes that into account. So I don’t think this part of the setup is an issue.

The real challenge is that by the time spot nodes come up, the scheduler has already placed the pods back onto the on-demand nodes, simply because no suitable spot nodes were ready yet. This creates a loop: descheduler evicts → pods are recreated → CA scales spot pool → scheduler reschedules to on-demand → spot nodes become ready too late → repeat.

That’s why I believe this could still be addressed within Descheduler — perhaps by supporting a pre-scaling mechanism

[a7i]

It sounds to me that the CA scale-down-unneeded-timeneeds to be longer than your Descheduler interval.

You can also increase the frequency of your descheduling cycle, but add more strict limits around per namespace and per node eviction limits.

[AlexGurtoff]

It sounds to me that the CA scale-down-unneeded-timeneeds to be longer than your Descheduler interval.

You can also increase the frequency of your descheduling cycle, but add more strict limits around per namespace and per node eviction limits.

Unfortunately, this still doesn’t solve the core problem.

The issue is that Descheduler may trigger evictions even when spot nodes don’t exist at all at the cloud provider level - meaning there’s no chance the pods will land on spot nodes. This leads to unnecessary evictions (descale operations), which we want to avoid.

That’s why I’m proposing a mechanism where Descheduler would only proceed with evictions after we’ve explicitly triggered provisioning of the desired node types (e.g., spot), similar to the behavior in estafette-gke-node-pool-shifter.

I totally understand that Descheduler wasn’t designed for this use case - and we’ve already found alternative tool that better match our needs. But I still think it’s worth highlighting that in the current behavior, using preferredDuringSchedulingIgnoredDuringExecution with spot nodes isn't ideal.

You’re forced to choose between:
1.Excessive, wasteful descale operations
or
2.Sticking to nodeFit, which delays moving workloads to spot nodes until an HPA event or redeploy happens - which may not occur for days or weeks.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale