From 971b113b520a0e4153abe2714a692c0a461ac808 Mon Sep 17 00:00:00 2001 From: Yu-Lin Chen Date: Mon, 9 Sep 2024 21:59:21 +0800 Subject: [PATCH 1/2] [YUNIKORN-2847] Documentation: Update new label/annotation changes before 1.6.0 release --- .../labels_and_annotations_in_yunikorn.md | 48 ++++++++++--------- 1 file changed, 26 insertions(+), 22 deletions(-) diff --git a/docs/user_guide/labels_and_annotations_in_yunikorn.md b/docs/user_guide/labels_and_annotations_in_yunikorn.md index 2069d29c353..40f7b316150 100644 --- a/docs/user_guide/labels_and_annotations_in_yunikorn.md +++ b/docs/user_guide/labels_and_annotations_in_yunikorn.md @@ -22,33 +22,37 @@ under the License. --> ## Labels and Annotations in YuniKorn -YuniKorn utilizes several Kubernetes labels and annotations to support various features: +YuniKorn utilizes several Kubernetes labels and annotations to support various features. + +Since YuniKorn 1.6.0, we have provided canonical representations for defining applicationID and queue in labels under the namespace `yunikorn.apache.org`. Using these labels can benefit from the Kubernetes filtering and grouping capabilities. +The non-canonical(Classic) representation will not be deprecated and will continue to be supported in future releases. ### Labels in YuniKorn Label values should comply with [Kubernetes Syntax and character set](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#syntax-and-character-set). -| Name | Description | -|-----------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `applicationId` | Associates this pod with an application. | -| `queue` | Selects the YuniKorn queue this application should be scheduled in.
Queue name should comply with [Kubernetes Syntax and character set](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#syntax-and-character-set) and also with [Partition and Queue Configuration](queue_config#queues).
If the queue doesn't comply, annotations could be used as described below. This may be ignored if a placement policy is in effect. | -| `spark-app-selector` | Alternative method of specifying `applicationId` used by Spark Operator if the label `applicationId` and annotation `yunikorn.apache.org/app-id` are unset. | +| Name | Description | +|-------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `yunikorn.apache.org/app-id` | Associates this pod with an application. This is the canonical representation and it is recommended to use it.

The priority of applicationID is determined in the following order:
  1. Label `yunikorn.apache.org/app-id`
  2. Annotation `yunikorn.apache.org/app-id`
  3. Label `applicationId`
  4. Label `spark-app-selector`
**Note**: Pod has inconsistent application metadata will be rejected in 1.7.0. | | +| `yunikorn.apache.org/queue` | Selects the YuniKorn queue this application should be scheduled in. This is the canonical representation and it is recommended to use it.

Queue name should comply with [Kubernetes Syntax and character set](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#syntax-and-character-set) and also with [Partition and Queue Configuration](queue_config#queues). This may be ignored if a placement policy is in effect.

The priority of queue name is determined in the following order:
  1. Label `yunikorn.apache.org/queue`
  2. Annotation `yunikorn.apache.org/queue`
  3. Label `queue`
**Note**: Pod has inconsistent queue metadata will be rejected in 1.7.0. | +| [Classic]
`applicationId` | It is equivalent to the label `yunikorn.apache.org/app-id`, and we recommend using it. | +| [Classic]
`queue` | It is equivalent to the label `yunikorn.apache.org/queue`, and we recommend using it. | +| `spark-app-selector` | It is equivalent to the label `yunikorn.apache.org/app-id`.

It is automatically attached by third-party Spark jobs triggered by the [Spark Operator](https://github.com/kubeflow/spark-operator). YuniKorn internally supports this label as one of the sources of applicationID. | ### Annotations in YuniKorn -All annotations are under the namespace `yunikorn.apache.org`. For example `yunikorn.apache.org/app-id`. - -| Name | Description | -|-------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `app-id` | Assoiates this pod with an application.
The priority of applicationID is determined by: annotation `yunikorn.apache.org/app-id` > label `applicationId` > label `spark-app-selector`. | -| `queue` | Selects the YuniKorn queue this application should be scheduled in.
Queue name should comply with rules described in [Partition and Queue Configuration](queue_config#queues).
The priority of queue is determined by: label `queue` > annotation `yunikorn.apache.org/queue` > default. | -| `task-group-name` | Sets the task group name this pod belongs to for the purposes of gang scheduling. It must be listed within `task-groups`. | -| `task-groups` | Defines the set of task groups for this application for gang scheduling. Each pod within an application must define all task groups. | -| `schedulingPolicyParameters` | Arbitrary key-value pairs used to customize scheduling policies such as gang scheduling. | -| `allow-preemption` | The `allow-preemption` annotation can be set on the Pod or PriorityClass object. The annotation in Pod takes priority over PriorityClass. It will trigger opt out of preemption for pods. Further details can be found in the [DaemonSet Scheduling using Simple Preemptor](./../design/simple_preemptor) documentation. | -| `parentqueue` | Define a parent queue for a set of K8s namespaces. Further details can be found in the [ Resource Quota Management](resource_quota_management#parent-queue-mapping-for-namespaces) documentation. | -| `namespace.quota` | Set the maximum capacity of the queue mapped to this namespace. Further details can be found in the [ Resource Quota Management](resource_quota_management#namespace-quota) documentation. | -| [DEPRECATED] `namespace.max.cpu` | Replaced with ``namespace.quota`` since version 1.2.0 | -| [DEPRECATED] `namespace.max.memory` | Replaced with `namespace.quota` since version 1.2.0 | -| `namespace.enableYuniKorn` | Controls which namespaces will have pods forwarded to Yunikorn for scheduling. Further details can be found in the [Service Configuration #admission-controller-filtering-settings](service_config#admission-controller-filtering-settings)documentation. | -| `namespace.generateAppId` | Controls which namespaces will have pods labeled with an `applicationId`. Further details can be found in the [Service Configuration #admission-controller-filtering-settings](service_config#admission-controller-filtering-settings) documentation. | + +| Name | Description | +|-------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| [Classic]
`yunikorn.apache.org/app-id` | It is equivalent to the label `yunikorn.apache.org/app-id`, and we recommend using it. | +| [Classic]
`yunikorn.apache.org/queue` | It is equivalent to the label `yunikorn.apache.org/queue`, and we recommend using it. | +| `yunikorn.apache.org/task-group-name` | Sets the task group name this pod belongs to for the purposes of gang scheduling. It must be listed within `yunikorn.apache.org/task-groups`. | +| `yunikorn.apache.org/task-groups` | Defines the set of task groups for this application for gang scheduling. Each pod within an application must define all task groups. | +| `yunikorn.apache.org/schedulingPolicyParameters` | Arbitrary key-value pairs used to customize scheduling policies such as gang scheduling. | +| `yunikorn.apache.org/allow-preemption` | This annotation can be set on the Pod or PriorityClass object. The annotation in Pod takes priority over PriorityClass. It will trigger opt out of preemption for pods. Further details can be found in the [DaemonSet Scheduling using Simple Preemptor](./../design/simple_preemptor) documentation. | +| `yunikorn.apache.org/parentqueue` | Define a parent queue for a set of K8s namespaces. Further details can be found in the [ Resource Quota Management](resource_quota_management#parent-queue-mapping-for-namespaces) documentation. | +| `yunikorn.apache.org/namespace.quota` | Set the maximum capacity of the queue mapped to this namespace. Further details can be found in the [ Resource Quota Management](resource_quota_management#namespace-quota) documentation. | +| [DEPRECATED]
`yunikorn.apache.org/namespace.max.cpu` | Replaced with `yunikorn.apache.org/namespace.quota` since version 1.2.0 | +| [DEPRECATED]
`yunikorn.apache.org/namespace.max.memory` | Replaced with `yunikorn.apache.org/namespace.quota` since version 1.2.0 | +| `yunikorn.apache.org/namespace.enableYuniKorn` | Controls which namespaces will have pods forwarded to Yunikorn for scheduling. Further details can be found in the [Service Configuration #admission-controller-filtering-settings](service_config#admission-controller-filtering-settings)documentation. | +| `yunikorn.apache.org/namespace.generateAppId` | Controls which namespaces will have pods labeled with an `yunikorn.apache.org/app-id`. Further details can be found in the [Service Configuration #admission-controller-filtering-settings](service_config#admission-controller-filtering-settings) documentation. | For more details surrounding gang-scheduling labels and annotations, please refer to the documentation on [gang scheduling](user_guide/gang_scheduling.md). From 0b6cf49c75e5c2612d917f9c21852ea74d4580d9 Mon Sep 17 00:00:00 2001 From: Yu-Lin Chen Date: Tue, 10 Sep 2024 17:14:20 +0800 Subject: [PATCH 2/2] Address the code reviews. --- .../labels_and_annotations_in_yunikorn.md | 22 +++++++++---------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/docs/user_guide/labels_and_annotations_in_yunikorn.md b/docs/user_guide/labels_and_annotations_in_yunikorn.md index 40f7b316150..bbe4e084bbf 100644 --- a/docs/user_guide/labels_and_annotations_in_yunikorn.md +++ b/docs/user_guide/labels_and_annotations_in_yunikorn.md @@ -24,26 +24,26 @@ under the License. ## Labels and Annotations in YuniKorn YuniKorn utilizes several Kubernetes labels and annotations to support various features. -Since YuniKorn 1.6.0, we have provided canonical representations for defining applicationID and queue in labels under the namespace `yunikorn.apache.org`. Using these labels can benefit from the Kubernetes filtering and grouping capabilities. -The non-canonical(Classic) representation will not be deprecated and will continue to be supported in future releases. +Since YuniKorn 1.6.0, we have provided canonical representations for defining application ID and queue in labels under the namespace `yunikorn.apache.org`. Using these labels can benefit from the Kubernetes filtering and grouping capabilities. +The non-canonical (legacy) representation will not be deprecated and will continue to be supported in future releases. ### Labels in YuniKorn Label values should comply with [Kubernetes Syntax and character set](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#syntax-and-character-set). -| Name | Description | -|-------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `yunikorn.apache.org/app-id` | Associates this pod with an application. This is the canonical representation and it is recommended to use it.

The priority of applicationID is determined in the following order:
  1. Label `yunikorn.apache.org/app-id`
  2. Annotation `yunikorn.apache.org/app-id`
  3. Label `applicationId`
  4. Label `spark-app-selector`
**Note**: Pod has inconsistent application metadata will be rejected in 1.7.0. | | -| `yunikorn.apache.org/queue` | Selects the YuniKorn queue this application should be scheduled in. This is the canonical representation and it is recommended to use it.

Queue name should comply with [Kubernetes Syntax and character set](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#syntax-and-character-set) and also with [Partition and Queue Configuration](queue_config#queues). This may be ignored if a placement policy is in effect.

The priority of queue name is determined in the following order:
  1. Label `yunikorn.apache.org/queue`
  2. Annotation `yunikorn.apache.org/queue`
  3. Label `queue`
**Note**: Pod has inconsistent queue metadata will be rejected in 1.7.0. | -| [Classic]
`applicationId` | It is equivalent to the label `yunikorn.apache.org/app-id`, and we recommend using it. | -| [Classic]
`queue` | It is equivalent to the label `yunikorn.apache.org/queue`, and we recommend using it. | -| `spark-app-selector` | It is equivalent to the label `yunikorn.apache.org/app-id`.

It is automatically attached by third-party Spark jobs triggered by the [Spark Operator](https://github.com/kubeflow/spark-operator). YuniKorn internally supports this label as one of the sources of applicationID. | +| Name | Description | +|-------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `yunikorn.apache.org/app-id` | Associates this pod with an application. This is the canonical representation and it is recommended to use it.

The priority of application ID is determined in the following order:
  1. Label `yunikorn.apache.org/app-id`
  2. Annotation `yunikorn.apache.org/app-id`
  3. Label `applicationId`
  4. Label `spark-app-selector`
**Note**: Pods with inconsistent application metadata will be rejected in 1.7.0. | | +| `yunikorn.apache.org/queue` | Selects the YuniKorn queue this application should be scheduled in. This is the canonical representation and it is recommended to use it.

Queue name should comply with [Kubernetes Syntax and character set](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#syntax-and-character-set) and also with [Partition and Queue Configuration](queue_config#queues). This may be ignored if a placement policy is in effect.

The priority of queue name is determined in the following order:
  1. Label `yunikorn.apache.org/queue`
  2. Annotation `yunikorn.apache.org/queue`
  3. Label `queue`
**Note**: Pods with inconsistent queue metadata will be rejected in 1.7.0. | +| [LEGACY]
`applicationId` | Equivalent to the preferred label `yunikorn.apache.org/app-id`. | +| [LEGACY]
`queue` | Equivalent to the preferred label `yunikorn.apache.org/queue`. | +| `spark-app-selector` | Equivalent to the preferred label `yunikorn.apache.org/app-id`.

It is automatically attached by third-party Spark jobs triggered by the [Spark Operator](https://github.com/kubeflow/spark-operator). YuniKorn uses this label to define application ID if no other metadata exists. | ### Annotations in YuniKorn | Name | Description | |-------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| [Classic]
`yunikorn.apache.org/app-id` | It is equivalent to the label `yunikorn.apache.org/app-id`, and we recommend using it. | -| [Classic]
`yunikorn.apache.org/queue` | It is equivalent to the label `yunikorn.apache.org/queue`, and we recommend using it. | +| [LEGACY]
`yunikorn.apache.org/app-id` | Equivalent to the preferred label `yunikorn.apache.org/app-id`. | +| [LEGACY]
`yunikorn.apache.org/queue` | Equivalent to the preferred label `yunikorn.apache.org/queue`. | | `yunikorn.apache.org/task-group-name` | Sets the task group name this pod belongs to for the purposes of gang scheduling. It must be listed within `yunikorn.apache.org/task-groups`. | | `yunikorn.apache.org/task-groups` | Defines the set of task groups for this application for gang scheduling. Each pod within an application must define all task groups. | | `yunikorn.apache.org/schedulingPolicyParameters` | Arbitrary key-value pairs used to customize scheduling policies such as gang scheduling. |