diff --git a/docs/assets/images/webui-runs-metadata-filter.png b/docs/assets/images/webui-runs-metadata-filter.png new file mode 100644 index 00000000000..1009ebc8948 Binary files /dev/null and b/docs/assets/images/webui-runs-metadata-filter.png differ diff --git a/docs/get-started/webui-qs.rst b/docs/get-started/webui-qs.rst index d26b63c882d..8fdd4fcd507 100644 --- a/docs/get-started/webui-qs.rst +++ b/docs/get-started/webui-qs.rst @@ -20,6 +20,8 @@ You must have a running Determined cluster with the CLI installed. - To set up a remote cluster, visit the :ref:`Installation Guide ` where you'll find options for On Prem, AWS, GCP, Kubernetes, and Slurm. +.. _qs-webui-concepts: + ********** Concepts ********** diff --git a/docs/reference/experiment-config-reference.rst b/docs/reference/experiment-config-reference.rst index fbcf04270c5..8b59c2624b7 100644 --- a/docs/reference/experiment-config-reference.rst +++ b/docs/reference/experiment-config-reference.rst @@ -1004,12 +1004,12 @@ Optional. The maximum number of trials that can be worked on simultaneously. The Optional. If specified, the weights of *every* trial in the search will be initialized to the most recent checkpoint of the given trial ID. This will fail if the source trial's model architecture is -incompatible with the model architecture of any of the trials in this experiment. +inconsistent with the model architecture of any of the trials in this experiment. ``source_checkpoint_uuid`` -------------------------- -Optional. Like ``source_trial_id`` but specifies an arbitrary checkpoint from which to initialize +Optional. Like ``source_trial_id``, but specifies an arbitrary checkpoint from which to initialize weights. At most one of ``source_trial_id`` or ``source_checkpoint_uuid`` should be set. Grid @@ -1654,3 +1654,11 @@ If :ref:`gres_supported ` is set to ``false``, spec to ensure that ``slots_per_node`` GPUs will be available on the nodes selected for the job using other configurations such as targeting a specific resource pool with only ``slots_per_node`` GPU nodes or specifying a PBS constraint in the experiment configuration. + +****************** + Metadata Logging +****************** + +Determined supports logging arbitrary metadata for experiments. This feature allows users to store +additional context and information about their runs. To learn how to log custom metadata, visit +:ref:`the tutorial `. diff --git a/docs/tools/webui-if.rst b/docs/tools/webui-if.rst index 733180a123e..d6d4f39043f 100644 --- a/docs/tools/webui-if.rst +++ b/docs/tools/webui-if.rst @@ -241,3 +241,26 @@ Clear the message with the following command: .. code:: bash det master cluster-message clear + +******************************** + Viewing and Filtering Metadata +******************************** + +You can use the WebUI to view and filter experiment runs based on logged metadata. For a tutorial on +how to log metadata, visit :ref:`metadata-logging-tutorial`. + +- In the Overview tab of the experiment, you can filter and sort runs based on metadata values + using the filter menu. +- In the experiment's Runs view, metadata columns are displayed alongside other experiment + information. +- On the Run details page, you'll find the "Metadata" section under the "Overview" tab, displaying + all logged metadata for that run. +- To download the metadata in JSON format, click the "Download" button. + +To filter runs based on metadata: + +#. In the Runs view, click on the filter icon. +#. Select a metadata field from the dropdown menu. +#. Choose a condition (is, is not, or contains) and enter a value. + +Note: Array-type metadata can be viewed but cannot be used for sorting or filtering. diff --git a/docs/tutorials/_index.rst b/docs/tutorials/_index.rst index 0f28123198c..0a32e26b766 100644 --- a/docs/tutorials/_index.rst +++ b/docs/tutorials/_index.rst @@ -46,6 +46,7 @@ Examples let you build off of an existing model that already runs on Determined. :hidden: Quickstart for Model Developers + Logging Arbitrary Metadata Porting Your PyTorch Model to Determined Get Started with Detached Mode Viewing Epoch-Based Metrics in the WebUI diff --git a/docs/tutorials/metadata-logging.rst b/docs/tutorials/metadata-logging.rst new file mode 100644 index 00000000000..5849f6e6b35 --- /dev/null +++ b/docs/tutorials/metadata-logging.rst @@ -0,0 +1,83 @@ +.. _metadata-logging-tutorial: + +############################ + Arbitrary Metadata Logging +############################ + +Arbitrary Metadata Logging enhances your experiment tracking capabilities by allowing you to: + +#. Log custom metadata specific to your experiments. +#. View logged metadata in the WebUI for each trial. +#. Filter and sort experiment runs based on custom metadata. +#. Compare and analyze experiments using custom metadata fields. + +By leveraging this feature, you can capture and analyze experiment-specific information beyond +standard metrics, leading to more insightful comparisons and better experiment management within the +Determined platform. + +****************** + Example Use Case +****************** + +This example creates an arbitrary metadata field ``effectiveness`` and then (does something else). + +**Section Title** + +#. Run an experiment to create a :ref:`single-trial run `. +#. Note the Run ID, e.g., Run 110. +#. Navigate to the cluster address for your training environment, e.g., **http://localhost:8080/**. +#. In the WebUI, click **API(Beta)** in the left navigation pane. +#. Execute the following PostRunMetadata ``/Internal/PostRunMetadata`` + +.. code:: bash + + { + "runId": 110, + "metadata": { + "effectiveness": 20 + } + } + +Next, we'll filter our runs by a specific metadata condition. + +**Filter by Metadata** + +#. In the WebUI, select your experiment to view the Runs table. +#. Select the **Filter**. +#. In **Show runs...**, select your metadata field from the dropdown menu. +#. Choose a condition (e.g., is, is not, or contains) and enter a value. + +.. image:: /assets/images/webui-runs-metadata-filter.png + :alt: Determined AI metadata filter for runs for an experiment + +Finally, let's view the logged metadata for our run. + +**View Metadata** + +To view the logged metadata: + +#. In the WebUI, navigate to your experiment. +#. Click on the run you want to inspect. +#. In the Run details page, find the "Metadata" section under the "Overview" tab. + +**************************** + Performance Considerations +**************************** + +When using Arbitrary Metadata Logging, consider the following: + +- Metadata is stored efficiently for fast retrieval and filtering. +- Avoid logging very large metadata objects, as this may impact performance. +- Use consistent naming conventions for keys to make filtering and sorting easier. +- For deeply nested JSON structures, filtering and sorting are supported at the top level. + +************ + Next Steps +************ + +- Experiment with logging different types of metadata in your trials. +- Use the filtering and sorting capabilities in the WebUI to analyze your experiments. +- Integrate metadata logging into your existing Determined AI workflows to enhance your experiment + tracking. + +For more tutorials and guides, visit the :ref:`tutorials-index`. diff --git a/docs/tutorials/quickstart-mdldev.rst b/docs/tutorials/quickstart-mdldev.rst index 0bdaaba2da3..88bc7cb20a3 100644 --- a/docs/tutorials/quickstart-mdldev.rst +++ b/docs/tutorials/quickstart-mdldev.rst @@ -352,6 +352,16 @@ This example uses a fixed batch size and searches on dropout size, filters, and one trial performing at about 98 percent validation accuracy. The hyperparameter search halts poorly performing trials. +************************* + Logging Custom Metadata +************************* + +Determined also supports logging custom metadata during a trial run. This feature allows you to +capture additional context and information about your experiments beyond standard metrics. + +To learn more about how to use metadata logging in your experiments, please refer to the +:ref:`metadata-logging-tutorial`. + ************ Learn More ************