DRAFT: Add watcher to controller and webhook

.github/ISSUE_TEMPLATE/NEW_RELEASE.md

@@ -14,13 +14,16 @@ Please do not remove items from the checklist
   At least two for minor or major releases. At least one for a patch release.
 - [ ] Verify that the changelog in this issue and the CHANGELOG folder is up-to-date
   - [ ] Use https://github.com/kubernetes/release/tree/master/cmd/release-notes to gather notes.
-    Example: `release-notes --org kubernetes-sigs --repo kueue --branch release-0.3 --start-sha 4a0ebe7a3c5f2775cdf5fc7d60c23225660f8702 --end-sha a51cf138afe65677f5f5c97f8f8b1bc4887f73d2`
+    Example: `release-notes --org kubernetes-sigs --repo kueue --branch release-0.3 --start-sha 4a0ebe7a3c5f2775cdf5fc7d60c23225660f8702 --end-sha a51cf138afe65677f5f5c97f8f8b1bc4887f73d2 --dependencies=false --required-author=""`
 - [ ] For major or minor releases (v$MAJ.$MIN.0), create a new release branch.
-  - [ ] an OWNER creates a vanilla release branch with
+  - [ ] An OWNER creates a vanilla release branch with
         `git branch release-$MAJ.$MIN main`
   - [ ] An OWNER pushes the new release branch with
         `git push release-$MAJ.$MIN`
-- [ ] Update `README.md`, `CHANGELOG`, `charts/kueue/Chart.yaml` (`appVersion`) and `charts/kueue/values.yaml` (`controllerManager.manager.image.tag`) in the release branch: <!-- example #211 #214 -->
+- [ ] Update the release branch:
+  - [ ] Update `RELEASE_BRANCH` and `RELEASE_VERSION` in `Makefile` and run `make prepare-release-branch`
+  - [ ] Update the `CHANGELOG`
+  - [ ] Submit a pull request with the changes: <!-- example #211 #214 -->
 - [ ] An OWNER [prepares a draft release](https://github.com/kubernetes-sigs/kueue/releases)
   - [ ] Write the change log into the draft release.
   - [ ] Run

.github/workflows/odh-release.yml

@@ -0,0 +1,54 @@
+# This workflow will compile e2e tests and release them
+
+name: ODH Release
+on:
+  workflow_dispatch:
+    inputs:
+      version:
+        description: 'Tag to be used for release, i.e.: v0.0.1'
+        required: true
+  push:
+    tags:
+      - '*'
+
+env:
+  KUEUE_RELEASE_VERSION: ${{ github.event.inputs.version || github.ref_name }}
+
+jobs:
+  release-odh:
+    runs-on: ubuntu-latest
+
+    # Permission required to create a release
+    permissions:
+      contents: write
+
+    steps:
+    - uses: actions/checkout@v4
+
+    - name: Set Go
+      uses: actions/setup-go@v5
+      with:
+        go-version: v1.21
+
+    - name: Verify that release doesn't exist yet
+      shell: bash {0}
+      run: |
+        gh release view $KUEUE_RELEASE_VERSION
+        status=$?
+        if [[ $status -eq 0 ]]; then
+          echo "Release $KUEUE_RELEASE_VERSION already exists."
+          exit 1
+        fi
+      env:
+        GITHUB_TOKEN: ${{ github.TOKEN }}
+
+    - name: Compile tests
+      run: |
+        go test -c -o compiled-tests/e2e-singlecluster ./test/e2e/singlecluster/
+
+    - name: Creates a release in GitHub
+      run: |
+        gh release create $KUEUE_RELEASE_VERSION --target $GITHUB_SHA compiled-tests/*
+      env:
+        GITHUB_TOKEN: ${{ secrets.CODEFLARE_MACHINE_ACCOUNT_TOKEN }}
+      shell: bash

.github/workflows/openvex.yaml

@@ -24,9 +24,9 @@ jobs:
         name: Run vexctl
         with:
           product: pkg:golang/sigs.k8s.io/kueue@${{ env.TAG }}
-          file: /tmp/kueve.vex.json
+          file: /tmp/kueue.vex.json
       - name: Upload openvex data
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
         run: |
-          gh release upload $TAG /tmp/kueve.vex.json
+          gh release upload $TAG /tmp/kueue.vex.json

CHANGELOG/CHANGELOG-0.6.md

@@ -0,0 +1,119 @@
+## v0.6.2
+
+Changes since `v0.6.1`:
+
+### Bug or Regression
+
+- Avoid unnecessary preemptions when there are multiple candidates for preemption with the same admission timestamp (#1880, @alculquicondor)
+- Fix Pods in Pod groups stuck with finalizers when deleted immediately after Succeeded (#1916, @alculquicondor)
+- Fix preemption to reclaim quota that is blocked by an earlier pending Workload from another ClusterQueue in the same cohort. (#1868, @alculquicondor)
+- Reduce number of Workload reconciliations due to wrong equality check. (#1917, @gabesaba)
+
+### Other (Cleanup or Flake)
+
+- Improve pod integration performance (#1953, @gabesaba)
+
+## v0.6.1
+
+Changes Since `v0.6.0`:
+
+### Feature
+
+- Added MultiKueue worker connection monitoring and reconnect. (#1809, @trasc)
+- The Failed pods in a pod-group are finalized once a replacement pods are created. (#1801, @trasc)
+
+### Bug or Regression
+
+- Exclude Pod labels, preemptionPolicy and container images when determining whether pods in a pod group have the same shape. (#1760, @alculquicondor)
+- Fix incorrect quota management when lendingLimit enabled in preemption (#1826, @kerthcet, @B1F030)
+- Fix the configuration for the number of reconcilers for the Pod integration. It was only reconciling one group at a time. (#1837, @alculquicondor)
+- Kueue visibility API is no longer installed by default. Users can install it via helm or applying the visibility-api.yaml artifact. (#1764, @trasc)
+- WaitForPodsReady: Fix a bug that the requeueState isn't reset. (#1843, @tenzen-y)
+
+### Other (Cleanup or Flake)
+
+- Avoid API calls for admission attempts when Workload already has condition Admitted=false (#1845, @alculquicondor)
+- Skip requeueing of Workloads when there is a status update for a ClusterQueue, saving on API calls for Workloads that were already attempted for admission. (#1832, @alculquicondor)
+
+## v0.6.0
+
+Changes since `v0.5.0`:
+
+### API Change
+
+- A `stopPolicy` field in the ClusterQueue allows to hold or drain a ClusterQueue (#1299, @trasc)
+- Add a lendingLimit field in ClusterQueue's quotas, to allow restricting how much of the unused resources by the ClusterQueue can be borrowed by other ClusterQueues in the cohort.
+  In other words, this allows a quota equal to `nominal-lendingLimit` to be exclusively used by the ClusterQueue. (#1385, @B1F030)
+- Add validation for clusterQueue: when cohort is empty, borrowingLimit must be nil. (#1525, @B1F030)
+- Allow decrease reclaimable pods to 0 for suspended job (#1277, @yaroslava-serdiuk)
+- MultiKueue: Add Path location type for cluster KubeConfigs. (#1640, @trasc)
+- MultiKueue: Add garbage collection of deleted Workloads. (#1643, @trasc)
+- MultiKueue: Multi cluster job dispatching for k8s Job. This doesn't include support for live status updates. (#1313, @trasc)
+- Support for a mechanism to suspend a running Job without requeueing (#1252, @vicentefb)
+- Support for preemption while borrowing (#1397, @mimowo)
+- The leaderElection field in the Configuration API is now defaulted.
+  Leader election is now enabled by default. (#1598, @astefanutti)
+- Visibility API: Add an endpoint that allows a user to fetch information about pending workloads and their position in LocalQueue. (#1365, @PBundyra)
+- Visibility API: Introduce an on-demand API endpoint for fetching pending workloads in a ClusterQueue. (#1251, @PBundyra)
+- Visibility API: extend the information returned for the pending workloads in a ClusterQueue, including the workload position in the queue. (#1362, @PBundyra)
+- WaitForPodsReady: Add a config field to allow admins to configure the timestamp used when sorting workloads that were evicted due to their Pods not becoming ready on time. (#1542, @nstogner)
+- WaitForPodsReady: Support a backoff re-queueing mechanism with configurable limit. (#1709, @tenzen-y)
+
+### Feature
+
+- Add Prebuilt Workload support for JobSets. (#1575, @trasc)
+- Add events for transitions of the provisioning AdmissionCheck (#1271, @stuton)
+- Add prebuilt workload support for batch/job. (#1358, @trasc)
+- Add support for groups of plain Pods. (#1319, @achernevskii)
+- Allow configuring featureGates on helm charts. (#1314, @B1F030)
+- At log level 6, the usage of ClusterQueues and cohorts is included in logs.
+
+  The status of the internal cache and queues is also logged on demand when a SIGUSR2 is sent to kueue, regardless of the log level. (#1528, @alculquicondor)
+- Changing tolerations in an inadmissible job triggers an admission retry with the updated tolerations. (#1304, @stuton)
+- Increase the default number of reconcilers for Pod and Workload objects to 5, each. (#1589, @alculquicondor)
+- Jobs preserve their position in the queue if the number of pods change before being admitted (#1223, @yaroslava-serdiuk)
+- Make the image build setting CGO_ENABLED configurable (#1391, @anishasthana)
+- MultiKueue: Add live status updates for multikueue JobSets (#1668, @trasc)
+- MultiKueue: Support for JobSets. (#1606, @trasc)
+- Support RayCluster as a queue-able workload in Kueue (#1520, @vicentefb)
+- Support for retry of provisioning request.
+
+  When `ProvisioningACC` is enabled, and there are existing ProvisioningRequests, they are going to be recreated.
+  This may cause job evictions for some long-running jobs which were using the ProvisioningRequests. (#1351, @mimowo)
+- The image gcr.io/k8s-staging-kueue/debug:main, along with the script ./hack/dump_cache.sh can be used to trigger a dump of the internal cache into the logs. (#1541, @alculquicondor)
+- The priority sorting within the cohort could be disabled by setting the feature gate PrioritySortingWithinCohort to false (#1406, @yaroslava-serdiuk)
+- Visibility API: Add HA support. (#1554, @astefanutti)
+
+### Bug or Regression
+
+- Add Missing RBAC on finalizer sub-resources for job integrations. (#1486, @astefanutti)
+- Add Mutating WebhookConfigurations for the AdmissionCheck, RayJob, and JobSet to helm charts (#1567, @B1F030)
+- Add Validating/Mutating WebhookConfigurations for the KubeflowJobs like PyTorchJob (#1460, @tenzen-y)
+- Added event for QuotaReserved and fixed event for Admitted to trigger when admission checks complete (#1436, @trasc)
+- Avoid finished Workloads from blocking quota after a Kueue restart (#1689, @trasc)
+- Avoid recreating a Workload for a finished Job and finalize a job when the workload is declared finished. (#1383, @achernevskii)
+- Do not (re)create ProvReq if the state of admission check is Ready (#1617, @mimowo)
+- Fix Kueue crashing at the log level 6 when re-admitting workloads (#1644, @mimowo)
+- Fix a bug in the pod integration that unexpected errors will occur when the pod isn't found (#1512, @achernevskii)
+- Fix a bug that plain pods managed by kueue will remain in a terminating state, due to a finalizer (#1342, @tenzen-y)
+- Fix client-go libraries bug that can not operate clusterScoped resources like ClusterQueue and ResourceFlavor. (#1294, @tenzen-y)
+- Fix fungibility policy `Preempt` where it was not able to utilize the next flavor if preemption was not possible. (#1366, @alculquicondor)
+- Fix handling of preemption within a cohort when there is no borrowingLimit. In that case,
+  during preemption, the permitted resources to borrow were calculated as if borrowingLimit=0, instead of unlimited.
+
+  As a consequence, when using `reclaimWithinCohort`, it was possible that a workload, scheduled to ClusterQueue with no borrowingLimit, would preempt more workloads than needed, even though it could fit by borrowing. (#1561, @mimowo)
+- Fix the synchronization of the admission check state based on recreated ProvisioningRequest (#1585, @mimowo)
+- Fixed fungibility policy `whenCanPreempt: Preempt`. The admission should happen in the flavor for which preemptions were issued. (#1332, @alculquicondor)
+- Kueue replicas are advertised as Ready only once the webhooks are functional.
+
+  This allows users to wait with the first requests until the Kueue deployment is available, so that the 
+  early requests don't fail. (#1676, @mimowo)
+- Pending workload from StrictFIFO ClusterQueue doesn't block borrowing from other ClusterQueues (#1399, @yaroslava-serdiuk)
+- Remove deleted pending workloads from the cache (#1679, @astefanutti)
+- Remove finalizer from Workloads that are orphaned (have no owners). (#1523, @achernevskii)
+- Trigger an eviction for an admitted Job after an admission check changed state to Rejected. (#1562, @trasc)
+- Webhooks are served in non-leading replicas (#1509, @astefanutti)
+
+### Other (Cleanup or Flake)
+
+- Expose utilization functions to setup jobframework reconcilers and webhooks (#1630, @tenzen-y)

Dockerfile

@@ -3,7 +3,7 @@ ARG BASE_IMAGE
 # Build the manager binary
 FROM --platform=${BUILDPLATFORM} ${BUILDER_IMAGE} as builder
 
-ARG CGO_ENABLED
+ARG CGO_ENABLED=1
 ARG TARGETARCH
 
 WORKDIR /workspace

Dockerfile.rhoai

@@ -0,0 +1,28 @@
+# Build the manager binary
+FROM registry.access.redhat.com/ubi9/go-toolset:1.21 as builder
+
+WORKDIR /workspace
+
+RUN git config --global --add safe.directory /workspace
+
+# Copy the Go Modules manifests
+COPY go.mod go.mod
+COPY go.sum go.sum
+# cache deps before building and copying source so that we don't need to re-download as much
+# and so that source changes don't invalidate our downloaded layer
+RUN go mod download
+
+# Copy the go source
+COPY . .
+
+# Build
+RUN make build GO_BUILD_ENV='CGO_ENABLED=1 GOOS=linux'
+
+FROM registry.access.redhat.com/ubi9/ubi:latest
+
+WORKDIR /
+USER root
+COPY --from=builder /workspace/bin/manager /manager
+USER 65532:65532
+
+ENTRYPOINT ["/manager"]

Makefile

@@ -71,6 +71,11 @@ E2E_K8S_VERSIONS ?= 1.26.12 1.27.9 1.28.5 1.29.0
 # Default will delete default kind cluster
 KIND_CLUSTER_NAME ?= kind
 
+# Number of processes to use during integration tests to run specs within a
+# suite in parallel. Suites still run sequentially. User may set this value to 1
+# to run without parallelism.
+INTEGRATION_NPROCS ?= 4
+
 # Setting SHELL to bash allows bash commands to be executed by recipes.
 # This is a requirement for 'setup-envtest.sh' in the test target.
 # Options are set to exit when a recipe line exits non-zero or a piped command fails.
@@ -81,6 +86,11 @@ version_pkg = sigs.k8s.io/kueue/pkg/version
 LD_FLAGS += -X '$(version_pkg).GitVersion=$(GIT_TAG)'
 LD_FLAGS += -X '$(version_pkg).GitCommit=$(shell git rev-parse HEAD)'
 
+# Update these variables when preparing a new release or a release branch.
+# Then run `make prepare-release-branch`
+RELEASE_VERSION=v0.6.2
+RELEASE_BRANCH=release-0.6
+
 .PHONY: all
 all: generate fmt vet build
 
@@ -162,7 +172,7 @@ test: gotestsum ## Run tests.
 .PHONY: test-integration
 test-integration: gomod-download envtest ginkgo mpi-operator-crd ray-operator-crd jobset-operator-crd kf-training-operator-crd  cluster-autoscaler-crd ## Run tests.
 	KUBEBUILDER_ASSETS="$(shell $(ENVTEST) use $(ENVTEST_K8S_VERSION) -p path)" \
-	$(GINKGO) $(GINKGO_ARGS) --junit-report=junit.xml --output-dir=$(ARTIFACTS) -v $(INTEGRATION_TARGET)
+	$(GINKGO) $(GINKGO_ARGS) -procs=$(INTEGRATION_NPROCS) --junit-report=junit.xml --output-dir=$(ARTIFACTS) -v $(INTEGRATION_TARGET)
 
 CREATE_KIND_CLUSTER ?= true
 .PHONY: test-e2e
@@ -192,8 +202,8 @@ ci-lint: golangci-lint
 	$(GOLANGCI_LINT) run --timeout 15m0s
 
 .PHONY: verify
-verify: gomod-verify vet ci-lint fmt-verify toc-verify manifests generate update-helm generate-apiref
-	git --no-pager diff --exit-code config/components apis charts/kueue/templates client-go site/content/en/docs/reference
+verify: gomod-verify vet ci-lint fmt-verify toc-verify manifests generate update-helm generate-apiref prepare-release-branch
+	git --no-pager diff --exit-code config/components apis charts/kueue/templates client-go site/
 
 ##@ Build
 
@@ -243,7 +253,7 @@ ifndef ignore-not-found
   ignore-not-found = false
 endif
 
-clean-manifests = (cd config/components/manager && $(KUSTOMIZE) edit set image controller=gcr.io/k8s-staging-kueue/kueue:main)
+clean-manifests = (cd config/components/manager && $(KUSTOMIZE) edit set image controller=gcr.io/k8s-staging-kueue/kueue:$(RELEASE_BRANCH))
 
 .PHONY: install
 install: manifests kustomize ## Install CRDs into the K8s cluster specified in ~/.kube/config.
@@ -277,6 +287,7 @@ artifacts: kustomize yq helm
 	$(KUSTOMIZE) build config/dev -o artifacts/manifests-dev.yaml
 	$(KUSTOMIZE) build config/alpha-enabled -o artifacts/manifests-alpha-enabled.yaml
 	$(KUSTOMIZE) build config/prometheus -o artifacts/prometheus.yaml
+	$(KUSTOMIZE) build config/visibility -o artifacts/visibility-api.yaml
 	@$(call clean-manifests)
 	# Update the image tag and policy
 	$(YQ)  e  '.controllerManager.manager.image.repository = "$(IMAGE_REPO)" | .controllerManager.manager.image.tag = "$(GIT_TAG)" | .controllerManager.manager.image.pullPolicy = "IfNotPresent"' -i charts/kueue/values.yaml
@@ -286,6 +297,12 @@ artifacts: kustomize yq helm
 	# Revert the image changes
 	$(YQ)  e  '.controllerManager.manager.image.repository = "$(STAGING_IMAGE_REGISTRY)/$(IMAGE_NAME)" | .controllerManager.manager.image.tag = "main" | .controllerManager.manager.image.pullPolicy = "Always"' -i charts/kueue/values.yaml
 
+.PHONY: prepare-release-branch
+prepare-release-branch: yq kustomize
+	sed -r 's/v[0-9]+\.[0-9]+\.[0-9]+/$(RELEASE_VERSION)/g' -i README.md -i site/config.toml
+	$(YQ) e '.appVersion = "$(RELEASE_VERSION)"' -i charts/kueue/Chart.yaml
+	@$(call clean-manifests)
+
 ##@ Tools
 
 # Build an image that can be used with kubectl debug
@@ -315,7 +332,7 @@ kustomize: ## Download kustomize locally if necessary.
 ENVTEST = $(PROJECT_DIR)/bin/setup-envtest
 .PHONY: envtest
 envtest: ## Download envtest-setup locally if necessary.
-	@GOBIN=$(PROJECT_DIR)/bin GO111MODULE=on $(GO_CMD) install sigs.k8s.io/controller-runtime/tools/setup-envtest@latest
+	@GOBIN=$(PROJECT_DIR)/bin GO111MODULE=on $(GO_CMD) install sigs.k8s.io/controller-runtime/tools/setup-envtest@v0.0.0-20240320141353-395cfc7486e6
 
 GINKGO = $(PROJECT_DIR)/bin/ginkgo
 .PHONY: ginkgo
@@ -372,7 +389,7 @@ jobset-operator-crd:
 	cp -f $(JOBSETROOT)/config/components/crd/bases/* $(PROJECT_DIR)/dep-crds/jobset-operator/
 
 
-CAROOT = $(shell $(GO_CMD) list -m -f "{{.Dir}}" k8s.io/autoscaler/cluster-autoscaler)
+CAROOT = $(shell $(GO_CMD) list -m -f "{{.Dir}}" k8s.io/autoscaler/cluster-autoscaler/apis)
 .PHONY: cluster-autoscaler-crd
 cluster-autoscaler-crd:
 	mkdir -p $(PROJECT_DIR)/dep-crds/cluster-autoscaler/

OWNERS

@@ -1,17 +1,23 @@
 # See the OWNERS docs at https://go.k8s.io/owners
 
 approvers:
-- alculquicondor
-- tenzen-y
+- anishasthana
+- astefanutti
+- jbusche
+- kpostoffice
+- maxusmusti
+- sutaakar
+- tedhtchang
 
 reviewers:
-- alculquicondor
-- denkensk
-- kerthcet
-- mimowo
-- tenzen-y
-- trasc
-
-emeritus_approvers:
-- ahg-g
-- denkensk
+- anishasthana
+- astefanutti
+- ChristianZaccaria
+- dimakis
+- Fiona-Waters
+- jbusche
+- kpostoffice
+- maxusmusti
+- MichaelClifford
+- sutaakar
+- tedhtchang

README.md

@@ -57,7 +57,7 @@ Read the [overview](https://kueue.sigs.k8s.io/docs/overview/) to learn more.
 To install the latest release of Kueue in your cluster, run the following command:
 
 ```shell
-kubectl apply --server-side -f https://github.com/kubernetes-sigs/kueue/releases/download/v0.5.3/manifests.yaml
+kubectl apply --server-side -f https://github.com/kubernetes-sigs/kueue/releases/download/v0.6.2/manifests.yaml
 ```
 
 The controller runs in the `kueue-system` namespace.

charts/kueue/Chart.yaml

@@ -18,4 +18,4 @@ version: 0.1.0
 # incremented each time you make changes to the application. Versions are not expected to
 # follow Semantic Versioning. They should reflect the version the application is using.
 # It is recommended to use it with quotes.
-appVersion: "0.5.3"
+appVersion: "v0.6.2"