Add retry mechanism for AdmissionChecks in Kueue

[PBundyra]

What would you like to be added:
Retry mechanism for AdmissionChecks.

Why is this needed:
Currently there is no retry mechanism for AdmissionChecks. There is a deprecated field admissionCheckSpec.retryDelayMinutes which was never supported and will be remove in API v1beta2. In the AdmissionCheck API there is the Retry, however the workload controller evicts the Workload without retrying it. Hence, there is a lot of improvement potential for the retry mechanism.

Completion requirements:

This enhancement requires the following artifacts:

• Design doc
• API change
• Docs update

The artifacts should be linked in subsequent comments.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[PBundyra]

/remove-lifecycle stale

[Horiodino]

/assign @PBundyra is this issue still opened ?

[Horiodino]

@PBundyra can you specify in AdmissionCheck API where is Retry ?

[mimowo]

Retry is one of the states an AdmissionCheck could be in.

We set in some scenarios, such as ProvisioningRequest Retry (see here. However, currently the retry mechanism is coupled to ProvisioningRequest. It means if there are some external ACs they need to retry on their own.

The issue will require designing the API for a generic mechanism for retrying an AC. However, I believe the issue was created as one of the alternatives to retry ProvisioningRequests (see Support for configurable ProvisioningRequest retries). In other words: the actual business need of being able to retry ProvisioningRequests had these two competing approaches. Since we started to support retries at the ProvisioningRequest-level I think the relative priority of this issue has decreased.

While still the mechanism could be useful for other AdmissionChecks, I would like to hear first in the community how much of a need there remains, before investing time into development and reviews.

[leipanhz]

Retry is one of the states an AdmissionCheck could be in.

We set in some scenarios, such as ProvisioningRequest Retry (see here. However, currently the retry mechanism is coupled to ProvisioningRequest. It means if there are some external ACs they need to retry on their own.

The issue will require designing the API for a generic mechanism for retrying an AC. However, I believe the issue was created as one of the alternatives to retry ProvisioningRequests (see Support for configurable ProvisioningRequest retries). In other words: the actual business need of being able to retry ProvisioningRequests had these two competing approaches. Since we started to support retries at the ProvisioningRequest-level I think the relative priority of this issue has decreased.

While still the mechanism could be useful for other AdmissionChecks, I would like to hear first in the community how much of a need there remains, before investing time into development and reviews.

Hi @mimowo , thanks for documenting this. Retry serves an important role for external admission check to work properly. Imho, the fact it only works with internal provision requests would limit Kueue's applicabilty.

[mimowo]

the fact it only works with internal provision requests would limit Kueue's applicabilty.

Right, thanks for emphasizing the need for the feature to support external AdmissionChecks. I believe the retry delay still could be hacked as in ProvisioningRequest AdmissionCheck, but I'm aware it means duplicating and maintaining complex code.

cc @mwielgus @mwysokin

[dhenkel92]

We are currently in the process of introducing Kueue to our infrastructure. One reason why we picked it is the ability to run custom admission checks to have more control over whether a workload can/should be run in a cluster or not.

For example: We've created a check that detects if the Kubernetes control-plane of that cluster is overloaded. In this case, the workload should be delayed until the control-plane is healthy again.

Unfortunately, without retries, Kueue will constantly reconcile the workload, which does not help the control-plane recover.
Right now, we've built a workaround for the missing capability by misusing the .status.requeueState in our admission checks.
However, this is not maintainable, so it would be great to have a proper solution long term.

In case you have higher priorities, we would love to help.