From cb64fe554bf3f4d1d5c0611628d6b9996b0501e9 Mon Sep 17 00:00:00 2001 From: Tara Charter Date: Mon, 23 Sep 2024 13:01:14 -0500 Subject: [PATCH] docs: Describe task config policies --- docs/manage/_index.rst | 6 + docs/manage/task-config-policies.rst | 122 ++++++++++++++++++ docs/reference/_index.rst | 7 + .../task-config-policies-reference.rst | 122 ++++++++++++++++++ 4 files changed, 257 insertions(+) create mode 100644 docs/manage/task-config-policies.rst create mode 100644 docs/reference/task-config-policies-reference.rst diff --git a/docs/manage/_index.rst b/docs/manage/_index.rst index 601749f784eb..29d3fa70a82d 100644 --- a/docs/manage/_index.rst +++ b/docs/manage/_index.rst @@ -29,6 +29,12 @@

Instructions for configuring security features on your cluster such as RBAC and OSS.

+
+ +

Task Config Policies

+

Instructions for configuring task config policies on your cluster.

+ +

Troubleshooting

diff --git a/docs/manage/task-config-policies.rst b/docs/manage/task-config-policies.rst new file mode 100644 index 000000000000..caa9671b6637 --- /dev/null +++ b/docs/manage/task-config-policies.rst @@ -0,0 +1,122 @@ +.. _task-config-policies: + +###################### + Task Config Policies +###################### + +Task Config Policies allow administrators to set limits on how users can define workloads (e.g., experiments and notebooks, tensorboards, shells, and commands). This feature enables enterprises to govern user behavior more closely at various levels within the Determined cluster. + +************** + Key Features +************** + +- Set policies via WebUI or CLI +- Define limits for resource usage, environment settings, and more +- Apply policies at different levels (cluster-wide, workspace, or project) +- Override capabilities for specific user roles or groups +- Set different priority limits for experiments and NTSC (notebooks, tensorboards, shells, and commands) tasks + +******************* + Setting Policies +******************* + +Administrators can set Task Config Policies using either the WebUI or the CLI. + +WebUI +===== + +1. Log in to the Determined WebUI as an administrator. +2. Navigate to the "Admin" section. +3. Select "Task Config Policies" from the menu. +4. Click "Create New Policy" or edit an existing policy. +5. Define the policy settings and scope (cluster-wide, workspace, or project). +6. Save the policy. + +CLI +=== + +Use the following commands to manage Task Config Policies via CLI: + +.. code:: bash + + # List existing policies + det policy list + + # Create a new policy + det policy create --config + + # Update an existing policy + det policy update --config + + # Delete a policy + det policy delete + +***************** + Policy Examples +***************** + +Here are some example policies: + +1. Limit maximum GPU usage per experiment: + +.. code:: yaml + + name: max-gpu-limit + scope: cluster + config: + resources: + max_gpus: 4 + +2. Set different priority limits for experiments and NTSC tasks: + +.. code:: yaml + + name: priority-limits + scope: workspace + workspace: research + config: + scheduling: + max_priority: + experiments: 80 + ntsc: 60 + +3. Set default priority for all tasks: + +.. code:: yaml + + name: default-priority + scope: cluster + config: + scheduling: + default_priority: 5 + +******************* + Policy Hierarchy +******************* + +Policies are applied in the following order of precedence: + +1. Project-level policies +2. Workspace-level policies +3. Cluster-wide policies + +More specific policies (e.g., project-level) override more general policies (e.g., cluster-wide) when conflicts occur. + +******************* + Priority Override +******************* + +If a given scope has the priority field of a default invariant config set, this priority can still be overridden using the SetJobPriority API endpoint. However, the new priority must not violate any constraints placed on the scope. + +***************** + Best Practices +***************** + +- Start with cluster-wide policies for general governance. +- Use workspace and project-level policies for more granular control. +- Set different priority limits for experiments and NTSC tasks to better manage resource allocation. +- Regularly review and update policies as organizational needs change. +- Communicate policy changes to users to ensure smooth adoption. +- Use the SetJobPriority API endpoint cautiously, ensuring that priority changes do not violate existing policies. + +For more detailed information on configuring and managing Task Config Policies, refer to the :ref:`Task Config Policies API Reference `. \ No newline at end of file diff --git a/docs/reference/_index.rst b/docs/reference/_index.rst index 3475640054d1..edc6ff315c42 100644 --- a/docs/reference/_index.rst +++ b/docs/reference/_index.rst @@ -45,6 +45,12 @@

Reference documentation for the experiment config file.
+
+ +

Task Config Policies Reference

+

Reference documentation for task config policies.

+ +

CLI Reference

@@ -68,6 +74,7 @@ Job Configuration Reference Deployment Reference Training Reference + Task Config Policies Reference Python SDK REST API Determined CLI Reference diff --git a/docs/reference/task-config-policies-reference.rst b/docs/reference/task-config-policies-reference.rst new file mode 100644 index 000000000000..f2e25d07d578 --- /dev/null +++ b/docs/reference/task-config-policies-reference.rst @@ -0,0 +1,122 @@ +.. _task-config-policies-reference: + +################################# + Task Config Policies Reference +################################# + +Add a description of the task config policies reference. + +DRAFT CONTENT: + +- Task Config Policies API Reference +- Task Config Policies CLI Reference +- Task Config Policies UI Reference +- Task Config Policies Best Practices +- Task Config Policies Examples +- Task Config Policies FAQ + +This reference guide provides detailed information about Task Config Policies in Determined AI, including API endpoints, CLI commands, UI interactions, best practices, examples, and frequently asked questions. + +Task Config Policies API Reference +================================== + +The Task Config Policies API allows administrators to programmatically manage policies. Here are the main endpoints: + +- ``GET /api/v1/task-policies``: List all task policies +- ``POST /api/v1/task-policies``: Create a new task policy +- ``GET /api/v1/task-policies/{policy_id}``: Get details of a specific policy +- ``PATCH /api/v1/task-policies/{policy_id}``: Update an existing policy +- ``DELETE /api/v1/task-policies/{policy_id}``: Delete a policy + +For detailed request and response formats, refer to the API documentation. + +Task Config Policies CLI Reference +================================== + +Determined CLI provides commands to manage task policies: + +.. code-block:: bash + + # List all policies + det policy list + + # Create a new policy + det policy create --config + + # Show details of a policy + det policy show + + # Update an existing policy + det policy update --config + + # Delete a policy + det policy delete + +Task Config Policies UI Reference +================================= + +The Determined WebUI provides a graphical interface for managing task policies: + +1. Navigate to the "Admin" section +2. Select "Task Config Policies" from the menu +3. Use the interface to create, view, edit, or delete policies + +Task Config Policies Best Practices +=================================== + +1. Start with cluster-wide policies for general governance +2. Use workspace and project-level policies for more granular control +3. Set different priority limits for experiments and NTSC tasks +4. Regularly review and update policies as organizational needs change +5. Communicate policy changes to users to ensure smooth adoption +6. Use the SetJobPriority API endpoint cautiously + +Task Config Policies Examples +============================= + +1. Limit maximum GPU usage per experiment: + + .. code-block:: yaml + + name: max-gpu-limit + scope: cluster + config: + resources: + max_gpus: 4 + +2. Set different priority limits for experiments and NTSC tasks: + + .. code-block:: yaml + + name: priority-limits + scope: workspace + workspace: research + config: + scheduling: + max_priority: + experiments: 80 + ntsc: 60 + +3. Set default priority for all tasks: + + .. code-block:: yaml + + name: default-priority + scope: cluster + config: + scheduling: + default_priority: 5 + +Task Config Policies FAQ +======================== + +Q: How are conflicting policies resolved? +A: Policies are applied in order of specificity: project-level, workspace-level, then cluster-wide. More specific policies override more general ones. + +Q: Can users override policy-set priorities? +A: Users can override priorities using the SetJobPriority API, but the new priority must not violate any existing policy constraints. + +Q: How often should policies be reviewed? +A: It's recommended to review policies quarterly or when there are significant changes in organizational needs or cluster usage patterns. + +For more information on Task Config Policies, refer to the :ref:`Task Config Policies ` documentation.