Skip to content

Make KubernetesStartKueueJobOperator wait through Kueue preemption - #75658

Open
raphaelauv wants to merge 1 commit into
apache:mainfrom
raphaelauv:fix-kueue-job-wait
Open

raphaelauv wants to merge 1 commit into
apache:mainfrom
raphaelauv:fix-kueue-job-wait

Conversation

@raphaelauv

Copy link
Copy Markdown
Contributor

KubernetesStartKueueJobOperator now watches the Job itself, so it keeps working while Kueue queues it, preempts it or reruns it. Job operators also stop reading jobs/status, so get on jobs is enough.

Was generative AI tooling used to co-author this PR?
  • Yes (oppus 5.5)

@boring-cyborg boring-cyborg Bot added area:providers kind:documentation provider:cncf-kubernetes Kubernetes (k8s) provider related issues labels Oct 26, 2026
@raphaelauv
raphaelauv force-pushed the fix-kueue-job-wait branch 5 times, most recently from 354bbea to fe03f6f Compare October 27, 2026 15:58
@shahar1
shahar1 marked this pull request as ready for review October 27, 2026 18:57
@shahar1 shahar1 changed the title fix kueue job wait Make KubernetesStartKueueJobOperator wait through Kueue preemption Oct 27, 2026
@raphaelauv
raphaelauv force-pushed the fix-kueue-job-wait branch 3 times, most recently from d1d52d5 to 97eb217 Compare October 29, 2026 13:33
KubernetesStartKueueJobOperator inherited the KubernetesJobOperator wait,
which looks for the Job's pods right after creating it. Kueue keeps the
Job suspended without pods until it admits it, and may preempt it (suspend
it again and delete its pods) then admit it again, so the task failed to
find pods before admission, and after a preemption waited on, or read logs
and XCom from, pods that had been deleted.

Least-privilege Roles that grant get on jobs but not on jobs/status also
got a 403 when job operators and triggers waited for a Job: Kubernetes
RBAC checks subresources separately, although the Job object carries the
same status.

closes: apache#74373

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

KubernetesStartKueueJobOperator: completion wait ignores Kueue preemption and requires the jobs/status subresource

1 participant