docs: Describe arbitrary metadata logging (#10073)

docs/get-started/webui-qs.rst

@@ -20,6 +20,8 @@ You must have a running Determined cluster with the CLI installed.
 -  To set up a remote cluster, visit the :ref:`Installation Guide <installation-guide>` where you'll
    find options for On Prem, AWS, GCP, Kubernetes, and Slurm.
 
+.. _qs-webui-concepts:
+
 **********
  Concepts
 **********

docs/reference/experiment-config-reference.rst

@@ -1004,12 +1004,12 @@ Optional. The maximum number of trials that can be worked on simultaneously. The
 
 Optional. If specified, the weights of *every* trial in the search will be initialized to the most
 recent checkpoint of the given trial ID. This will fail if the source trial's model architecture is
-incompatible with the model architecture of any of the trials in this experiment.
+inconsistent with the model architecture of any of the trials in this experiment.
 
 ``source_checkpoint_uuid``
 --------------------------
 
-Optional. Like ``source_trial_id`` but specifies an arbitrary checkpoint from which to initialize
+Optional. Like ``source_trial_id``, but specifies an arbitrary checkpoint from which to initialize
 weights. At most one of ``source_trial_id`` or ``source_checkpoint_uuid`` should be set.
 
 Grid
@@ -1654,3 +1654,11 @@ If :ref:`gres_supported <cluster-configuration-slurm>` is set to ``false``, spec
 to ensure that ``slots_per_node`` GPUs will be available on the nodes selected for the job using
 other configurations such as targeting a specific resource pool with only ``slots_per_node`` GPU
 nodes or specifying a PBS constraint in the experiment configuration.
+
+******************
+ Metadata Logging
+******************
+
+Determined supports logging arbitrary metadata for experiments. This feature allows users to store
+additional context and information about their runs. To learn how to log custom metadata, visit
+:ref:`the tutorial <metadata-logging-tutorial>`.

docs/tools/webui-if.rst

@@ -241,3 +241,26 @@ Clear the message with the following command:
    .. code:: bash
 
       det master cluster-message clear
+
+********************************
+ Viewing and Filtering Metadata
+********************************
+
+You can use the WebUI to view and filter experiment runs based on logged metadata. For a tutorial on
+how to log metadata, visit :ref:`metadata-logging-tutorial`.
+
+-  In the Overview tab of the experiment, you can filter and sort runs based on metadata values
+   using the filter menu.
+-  In the experiment's Runs view, metadata columns are displayed alongside other experiment
+   information.
+-  On the Run details page, you'll find the "Metadata" section under the "Overview" tab, displaying
+   all logged metadata for that run.
+-  To download the metadata in JSON format, click the "Download" button.
+
+To filter runs based on metadata:
+
+#. In the Runs view, click on the filter icon.
+#. Select a metadata field from the dropdown menu.
+#. Choose a condition (is, is not, or contains) and enter a value.
+
+Note: Array-type metadata can be viewed but cannot be used for sorting or filtering.

docs/tutorials/_index.rst

@@ -46,6 +46,7 @@ Examples let you build off of an existing model that already runs on Determined.
    :hidden:
 
    Quickstart for Model Developers <quickstart-mdldev>
+   Logging Arbitrary Metadata <metadata-logging>
    Porting Your PyTorch Model to Determined <pytorch-mnist-tutorial>
    Get Started with Detached Mode <detached-mode/_index>
    Viewing Epoch-Based Metrics in the WebUI <viewing-epoch-based-metrics>

docs/tutorials/metadata-logging.rst

@@ -0,0 +1,83 @@
+.. _metadata-logging-tutorial:
+
+############################
+ Arbitrary Metadata Logging
+############################
+
+Arbitrary Metadata Logging enhances your experiment tracking capabilities by allowing you to:
+
+#. Log custom metadata specific to your experiments.
+#. View logged metadata in the WebUI for each trial.
+#. Filter and sort experiment runs based on custom metadata.
+#. Compare and analyze experiments using custom metadata fields.
+
+By leveraging this feature, you can capture and analyze experiment-specific information beyond
+standard metrics, leading to more insightful comparisons and better experiment management within the
+Determined platform.
+
+******************
+ Example Use Case
+******************
+
+This example creates an arbitrary metadata field ``effectiveness`` and then (does something else).
+
+**Section Title**
+
+#. Run an experiment to create a :ref:`single-trial run <qs-webui-concepts>`.
+#. Note the Run ID, e.g., Run 110.
+#. Navigate to the cluster address for your training environment, e.g., **http://localhost:8080/**.
+#. In the WebUI, click **API(Beta)** in the left navigation pane.
+#. Execute the following PostRunMetadata ``/Internal/PostRunMetadata``
+
+.. code:: bash
+
+   {
+       "runId": 110,
+       "metadata": {
+        "effectiveness": 20
+   }
+   }
+
+Next, we'll filter our runs by a specific metadata condition.
+
+**Filter by Metadata**
+
+#. In the WebUI, select your experiment to view the Runs table.
+#. Select the **Filter**.
+#. In **Show runs...**, select your metadata field from the dropdown menu.
+#. Choose a condition (e.g., is, is not, or contains) and enter a value.
+
+.. image:: /assets/images/webui-runs-metadata-filter.png
+   :alt: Determined AI metadata filter for runs for an experiment
+
+Finally, let's view the logged metadata for our run.
+
+**View Metadata**
+
+To view the logged metadata:
+
+#. In the WebUI, navigate to your experiment.
+#. Click on the run you want to inspect.
+#. In the Run details page, find the "Metadata" section under the "Overview" tab.
+
+****************************
+ Performance Considerations
+****************************
+
+When using Arbitrary Metadata Logging, consider the following:
+
+-  Metadata is stored efficiently for fast retrieval and filtering.
+-  Avoid logging very large metadata objects, as this may impact performance.
+-  Use consistent naming conventions for keys to make filtering and sorting easier.
+-  For deeply nested JSON structures, filtering and sorting are supported at the top level.
+
+************
+ Next Steps
+************
+
+-  Experiment with logging different types of metadata in your trials.
+-  Use the filtering and sorting capabilities in the WebUI to analyze your experiments.
+-  Integrate metadata logging into your existing Determined AI workflows to enhance your experiment
+   tracking.
+
+For more tutorials and guides, visit the :ref:`tutorials-index`.

docs/tutorials/quickstart-mdldev.rst

@@ -352,6 +352,16 @@ This example uses a fixed batch size and searches on dropout size, filters, and
    one trial performing at about 98 percent validation accuracy. The hyperparameter search halts
    poorly performing trials.
 
+*************************
+ Logging Custom Metadata
+*************************
+
+Determined also supports logging custom metadata during a trial run. This feature allows you to
+capture additional context and information about your experiments beyond standard metrics.
+
+To learn more about how to use metadata logging in your experiments, please refer to the
+:ref:`metadata-logging-tutorial`.
+
 ************
  Learn More
 ************