-
Notifications
You must be signed in to change notification settings - Fork 451
Description
What would you like to be added:
We would like to prevent Kueue from un-suspending any Job object (from the supported job types that Kueue can manage) that has some agreed upon label e.g. kueue.x-k8s.io/ignore: "true"
While this label is present and its value is true, Kueue should skip the Job - similar to Jobs that implement JobWithSkip when they return Skip()=True. When the label is removed, or its value is set to "false" Kueue, should handle this Job like it normally would.
Why this is needed:
We are developing Kubernetes controllers that modify fields in Jobs and we'd like our changes to apply before Kueue decides whether to schedule them or not. For example, we're automatically setting the resource requirements of Jobs. To avoid race conditions and ensure correctness, we need a reliable but also lightweight way to prevent Kueue from un-suspending a Job while our controllers are still modifying it. The proposed feature will also enable users to reliably keep a Job suspended on demand, since Kueue currently controls spec.suspend for Jobs it manages.
This suggestion:
- Eliminates race conditions between our controllers and Kueue’s reconciler
- Has a small surface area and is thus unlikely to cause any problems
- It is completely opt‑in and explicit thereby easy to document
Alternatives we considered:
-
Do nothing
Our Controllers may modify the Job object several times which will trigger Kueue to process the Job and modify the associated Workload object several times. In the worst case scenario, Kueue will also schedule a Job before our controllers finish updating it and the Job may behave in an unexpected way. -
Omit
kueue.x-k8s.io/queue-name
Doesn’t work where a default queue is configured, or where policy enforces that all Jobs must carry a queue name. -
External AdmissionCheck controller that sets its admission check state to Retry while some label or equivalent is set in the Job object
Heavyweight for this use case, requires spinning up extra Pod and having RBAC to add an AdmissionCheckController to one or more ClusterQueues. -
Set an invalid queue name first, then swap to the real one
Hacky and error‑prone. Also, to an external observer the Jobs will appear like they are malformed. This could introduce problems in environments where Jobs are expected to have correct names (e.g. namespaces with Kyverno policies or equivalent). -
Use mutating webhooks instead of controllers to ensure Jobs are updated before Kueue's reconciler sees the object
Guarantees ordering but is not a lightweight approach. This would add latency to the create path and would require additional RBAC to deploy. -
The external controllers also monitors and updates the associated Workload object by setting
Workload.spec.status.active=false, then back totrueafter it finishes modifying the Job object
Prevents admission, but requires writing to a Workload owned by Kueue. It also, requires additional RBAC to external controllers. It has a large surface area and is twice as resource hungry as the external controller.
Completion requirements:
This enhancement requires the following artifacts:
- Design doc
- API change
- Docs update
The artifacts should be linked in subsequent comments.