Merge branch 'main' into modernize_carbon_flux

.github/pull_request_template.md

@@ -1,69 +1,3 @@
-
 <!--
-Jason, Isaiah, please remove the `<-` `->` tags at the start and end of this template when you open a PR. Then, make sure you have gone through the checklist. Thanks!
-
-# Modernizing an example checklist
-
-## Preliminary checks
-
-- [ ] Look for [open PRs](https://github.com/holoviz-topics/examples/pulls) and [issues](https://github.com/holoviz-topics/examples/issues) that reference the project you are updating. It is possible previous unmerged work in PR could be re-used to modernize the project. Comment on these PRs and issues when appropriate, hopefully we should be able to close some of them after your modernizing work.
-
-## Change ‘anaconda-project.yml’ to use the latest workable version of packages
-
-- [ ] Pin python=3.11
-- [ ] Remove the upper pin (e.g. `hvplot<0.9` to `hvplot`, `panel>=0.12,<1.0` to `panel>=0.12`) of all other dependencies. Removing the upper pins of dependencies could necessitate code revisions in the notebooks to address any errors encountered in the updated environment. Should complexities or extensive time requirements arise, document issues for team discussion on whether to re-pin specific packages or explore other solutions.
-- [ ] Add/update the lower pin of all other dependencies (e.g. `hvplot` to `hvplot>=0.9.2`, `hvplot>=0.8` to `hvplot>=0.9.2`). Usually, the new/updated lower pin of a dependency will be the version resolved after `anaconda prepare` has been run. Execute `!conda list` in a notebook, or `anaconda run conda list` in the terminal, to display the version of each dependency installed in the environment. Adjusting the lower pin helps ensure that the locks produced for each platform (linux-64, win-64, osx-64, osx-arm64) rely on the tested dependencies and not on some older versions.
-- [ ] If one of the channels include conda-forge or pyviz, ask Maxime if it can be removed
-
-## Plot API updates (discussed on a per-example basis)
-
-- [ ] Generally, try to replace HoloViews usage with hvPlot. At a certain point of complexity, such as with the use of ‘.select’, it might be better to stick with HoloViews. Additional examples of ‘complexity boundaries’ should be documented in [this document](https://docs.google.com/document/d/1LYkChbVZzqq5S1S4Mkj1rM3PNPJDDGriks8YTc4EQc4/edit?usp=sharing).
-- [ ] Almost always, try to replace the use of `datashade` with `rasterize` (read [this page](https://holoviews.org/user_guide/Large_Data.html)). Essentially, `rasterize` allows Bokeh to handle the colormapping instead of Datashader.
-
-## Interactivity API updates (discussed on a per-example basis)
-
-- [ ] Remove all `pn.interact` usage
-- [ ] Avoid `.param.watch()` usage. This is [pretty low-level and verbose approach](https://param.holoviz.org/user_guide/Dependencies_and_Watchers.html#watchers) and should not be used in Examples unless required, or an Example is specifically trying to demo its usage in an advanced workflow.
-- [ ] Prefer using `pn.bind()`. Read [this page](https://panel.holoviz.org/explanation/api/reactive.html) for explanation.
-- [ ] For apps built using a class approach, when they create a `view()` method and call it directly, update the class by inheriting from `pn.viewable.Viewer` and replace `view()` by `__panel__()`. Here is an [example](https://panel.holoviz.org/gallery/nyc_deckgl.html). 
-
-## Panel App updates (discussed on a per-example basis)
-
-- [ ] If the project doesn’t at any point create a Panel app at all, consider creating one. It can be as simple as wrapping a plot in `pn.Column`, or more complicated to incorporate widgets, etc. Make the final app  `.servable()`.
-- [ ] If the project creates an app in a notebook but doesn’t deploy it (i.e. there is no `command: dashboard` declaration in the `anaconda-project.yml` file), try adding it.
-- [ ] If the project already deploys an app but doesn’t wrap it in a nice template, consider wrapping it in a [template](https://panel.holoviz.org/reference/index.html#templates).
-- [ ] If the project deploys an app wrapped in a template, customize the template a little so all the apps don’t look similar (e.g. change the header background color). This doesn’t need to be discussed.
-- [ ] Comment start If you are building the application in a single cell, you can construct a template explicitly, like `template = pn.template.BootstrampTemplate`, but if building up an app across multiple cells, it is probably cleaner to declare the template at the top with `pn.extension(template='bootstrap')`. See [how to guide on setting a template](https://panel.holoviz.org/how_to/templates/template_set.html).
-
-## General code quality updates
-
-- [ ] If the notebook disables warnings (e.g. with `warnings.simplefilter(‘ignore’)` somewhere at the start of the notebook, remove this line. Try to update the code to remove the warnings, if any. If updating the code to remove the warnings is taking significant amount of time and effort, bring it up for discussion and we may decide to disable warnings again.
-
-## Text content
-
-- [ ] Edit the text content anywhere and everywhere that it can be improved for clarity.
-- [ ] Check the links are valid, and update old links (e.g. http -> https, xyz.pyviz.org -> xyz.holoviz.org)
-- [ ] Remove instructions to install packages inside an example
-
-## Visual appearance - Example
-
-- [ ] Check that the titles/headings make sense and are succinct.
-- [ ] Check that the text content blocks are easily readable; revise into additional paragraphs if needed.
-- [ ] Check that the code blocks are easily readable; revise as needed. (e.g. add spaces after commas in a list if there are none, wrap long lines, etc.)
-- [ ] Check image and plot sizes. If possible, making them responsive is highly recommended.
-- [ ] Check the appearance on a smartphone (check Google to see how to adapt the appearance of your browser to display pages as if they were seen from a smartphone, this is usually done via the web developer tools). This is not a top priority for all examples, but if there are a few easy and straightforward changes to make that can improve the experience, let’s do it.
-- [ ] Check the updated notebook with the original notebook
-
-## Visual appearance - Gallery
-
-- [ ] Check the thumbnail is visually appealing
-- [ ] Check the project title is well formatted (e.g. `Ml Annotators` to `ML Annotators`), if not, add/update the `examples_config.title` field in `anaconda-project.yml`
-- [ ] Check the project description is appropriate, if not, update the `description` field in `anaconda-project.yml`
-
-## Workflow (after you have made the changes above)
-
-- [ ] Run successfully `doit validate:<projectname>`
-- [ ] Run successfully `doit test:<projectname>`
-- [ ] Run successfully `doit doc_one –name <projectname>`. It’s better if the project notebook(s) is saved with its outputs (but be sure to clear outputs before committing to the examples repo!) when building the docs. Then open this file in your browser `./builtdocs/index.html` and check how the site looks.
-- [ ] If you’re happy with all the above, open a PR. Reminder, clear notebook outputs before pushing to the PR.
+If you are opening a PR to modernize/update an existing example, refer to the [modernization checklist](https://github.com/holoviz-topics/examples/wiki/Example-Modernization-Checklist-(2024)) in the Wiki of this repo.
 -->

.github/workflows/build.yml

@@ -58,10 +58,10 @@ jobs:
     outputs:
       projects: ${{ steps.set-projects.outputs.projects }}
     steps:
-    - uses: actions/checkout@v3
-    - uses: actions/setup-python@v4
+    - uses: actions/checkout@v4
+    - uses: actions/setup-python@v5
       with:
-        python-version: '3.9'
+        python-version: '3.12'
     - name: install deps
       # Minimal required deps to run the following script
       run: pip install doit pyyaml

.github/workflows/build_one.yml

@@ -33,10 +33,10 @@ jobs:
     outputs:
       runner: ${{ steps.set-runner.outputs.runner }}
     steps:
-    - uses: actions/checkout@v3
-    - uses: actions/setup-python@v4
+    - uses: actions/checkout@v4
+    - uses: actions/setup-python@v5
       with:
-        python-version: '3.9'
+        python-version: '3.12'
     - name: install deps
       # Minimal required deps to run the following script
       run: pip install doit pyyaml
@@ -55,19 +55,14 @@ jobs:
         shell: bash -l {0}
     timeout-minutes: 60
     steps:
-    - uses: actions/checkout@v3
-    - uses: conda-incubator/setup-miniconda@v2
+    - uses: actions/checkout@v4
+    - uses: conda-incubator/setup-miniconda@v3
       with:
         miniconda-version: "latest"
         auto-update-conda: true
         auto-activate-base: false
         activate-environment: examples-gallery-manage
         environment-file: envs/environment-${{ needs.infer_runner.outputs.runner }}.lock
-    - name: enable libmamba
-      run: |
-        conda activate base
-        conda install conda-libmamba-solver
-        conda config --set solver libmamba
     - name: list files
       run: doit build_list_existing_files:${{ inputs.project }}
     - name: prepare project

.github/workflows/deploy.yml

@@ -50,11 +50,11 @@ jobs:
       changedprojects: ${{ steps.set-projects.outputs.changedprojects }}
       removedprojects: ${{ steps.set-projects.outputs.removedprojects }}
     steps:
-    - uses: actions/checkout@v3
+    - uses: actions/checkout@v4
       with:
         # Needed to compute the diff between the head and the latest commit on main
         fetch-depth: 2
-    - uses: conda-incubator/setup-miniconda@v2
+    - uses: conda-incubator/setup-miniconda@v3
       with:
         miniconda-version: "latest"
         auto-activate-base: false
@@ -71,7 +71,7 @@ jobs:
 
         if [ "$EVENTNAME" == "pull_request" ]; then
 
-          CHANGES=$(doit util_list_changed_dirs_with_last_commit | tail -n -1)
+          CHANGES=$(doit util_list_changed_dirs_with_last_commit --exclude-website-metadata --exclude-test-data | tail -n -1)
 
           CHANGEDPROJECTS=$(echo $CHANGES | jq -c -r '.changed')
           REMOVEDPROJECTS=$(echo $CHANGES | jq -c -r '.removed')

.github/workflows/deploy_one.yml

@@ -32,8 +32,8 @@ jobs:
         shell: bash -l {0}
     timeout-minutes: 60
     steps:
-    - uses: actions/checkout@v3
-    - uses: conda-incubator/setup-miniconda@v2
+    - uses: actions/checkout@v4
+    - uses: conda-incubator/setup-miniconda@v3
       with:
         miniconda-version: "latest"
         auto-update-conda: false
@@ -57,8 +57,8 @@ jobs:
         shell: bash -l {0}
     timeout-minutes: 15
     steps:
-    - uses: actions/checkout@v3
-    - uses: conda-incubator/setup-miniconda@v2
+    - uses: actions/checkout@v4
+    - uses: conda-incubator/setup-miniconda@v3
       with:
         miniconda-version: "latest"
         auto-update-conda: false

.github/workflows/docs.yml

@@ -79,19 +79,19 @@ jobs:
       run:
         shell: bash -el {0}
     steps:
-    - uses: actions/checkout@v3
+    - uses: actions/checkout@v4
       with:
         # Needed to compute the diff between the head and the latest commit on main (so at least 2)
         # AND needed to compute last_updated
         fetch-depth: 0
-    - uses: hmarr/debug-action@v2
+    # - uses: hmarr/debug-action@v2
     - name: debug
       run: |
         echo ${{ github.ref_name }}
         echo ${{ github.event_name }}
         echo ${{ inputs.type }}
         echo ${{ inputs.branch }}
-    - uses: conda-incubator/setup-miniconda@v2
+    - uses: conda-incubator/setup-miniconda@v3
       with:
         miniconda-version: "latest"
         auto-activate-base: false
@@ -142,7 +142,7 @@ jobs:
         git config user.name "github-actions"
 
         # git fetch and git diff
-        CHANGES=$(doit util_list_changed_dirs_with_last_commit | tail -n -1)
+        CHANGES=$(doit util_list_changed_dirs_with_last_commit --exclude-website-metadata --exclude-deployments-metadata --exclude-test-data | tail -n -1)
 
         CHANGEDPROJECTS=$(echo $CHANGES | jq -c -r '.changed')
         REMOVEDPROJECTS=$(echo $CHANGES | jq -c -r '.removed')
@@ -294,8 +294,7 @@ jobs:
           ls
         fi
     - name: archive projects
-      run: |
-        doit doc_archive_projects
+      run: doit doc_archive_projects
     - name: move content
       run: doit doc_move_content
     - name: "temp: remove non evaluated projects"
@@ -313,18 +312,18 @@ jobs:
       # ZIP and upload the built site:
       # Only when called from pr_flow.yml. Done as multiple PRs can update the dev website
       # concurrently, this offers a way to download the site and see it locally.
-    - name: tar built site
-      if: inputs.type == 'workflow_call'
-      run: tar czf builtdocs.tar.gz builtdocs/
-    - uses: actions/upload-artifact@v3
-      if: inputs.type == 'workflow_call'
-      with:
-        name: website
-        path: builtdocs.tar.gz
-        retention-days: 3
-    - name: delete zip
-      if: inputs.type == 'workflow_call'
-      run: rm builtdocs.tar.gz
+    # - name: tar built site
+    #   if: inputs.type == 'workflow_call'
+    #   run: tar czf builtdocs.tar.gz builtdocs/
+    # - uses: actions/upload-artifact@v3
+    #   if: inputs.type == 'workflow_call'
+    #   with:
+    #     name: website
+    #     path: builtdocs.tar.gz
+    #     retention-days: 3
+    # - name: delete zip
+    #   if: inputs.type == 'workflow_call'
+    #   run: rm builtdocs.tar.gz
     - name: Deploy dev
       # workflow_call, by pr_flow.yml
       # workflow_dispatch and dev target
@@ -347,9 +346,6 @@ jobs:
         force_orphan: true
     - name: Clean up
       run: doit clean --clean-dep doc_full
-    - name: debug
-      run: |
-        ls -hla
       # TODO: re-enable git diff --quiet --exit-code
     # - name: Check clean up
     #   run: git diff
@@ -358,10 +354,6 @@ jobs:
     - name: Comment PR
       # Only display the comment in a pr_flow context
       if: inputs.type == 'workflow_call'
-      uses: thollander/actions-comment-pull-request@v2
+      uses: thollander/actions-comment-pull-request@v3
       with:
-        message: |
-          Your changes were successfully integrated in the dev site, make sure to review
-          the pages of the projects you touched before merging this PR: https://holoviz-dev.github.io/examples/.
-          You can also download an archive of the site from the workflow summary page which comes in handy
-          when your dev site built was overriden by another PR (we have a single dev site!).
+        message: 'Your changes were successfully integrated in the <a href="https://holoviz-dev.github.io/examples/" target="_blank">dev site</a>, make sure to review the pages of the projects you touched before merging this PR.'

.github/workflows/pr_flow.yml

@@ -24,6 +24,12 @@ permissions:
   # To allow the docs workflow to submit a comment using thollander/actions-comment-pull-request
   pull-requests: write
 
+# Project file changes:
+# - Always validate it
+# - Website only metadata -> Skip test/build
+# - Deployment metadata (examples_config.deployments, commands) -> We should redeploy, but skip test/build/doc
+# - Docs is special, it's just to pull the `evaluated` branch, and this is only useful to get the built notebooks
+
 jobs:
   setup:
     runs-on: ubuntu-latest
@@ -34,22 +40,22 @@ jobs:
     outputs:
       changedprojects: ${{ steps.set-list.outputs.changedprojects }}
       removedprojects: ${{ steps.set-list.outputs.removedprojects }}
+      changedprojectsfile: ${{ steps.set-list.outputs.changedprojectsfile }}
     steps:
-    - uses: actions/checkout@v3
+    - uses: actions/checkout@v4
       with:
         fetch-depth: 100
-    - uses: hmarr/debug-action@v2
-    - uses: actions/setup-python@v4
+    # - uses: hmarr/debug-action@v2
+    - uses: actions/setup-python@v5
       with:
-        python-version: '3.9'
+        python-version: '3.12'
     - name: install deps
       # Minimal required deps to run the following script
       run: pip install doit pyyaml
     - name: infer project list
       id: set-list
       run: |
-        CHANGES=$(doit util_list_changed_dirs_with_main | tail -n -1)
-
+        CHANGES=$(doit util_list_changed_dirs_with_main --exclude-website-metadata --exclude-deployments-metadata | tail -n -1)
         CHANGEDPROJECTS=$(echo $CHANGES | jq -c -r '.changed')
         REMOVEDPROJECTS=$(echo $CHANGES | jq -c -r '.removed')
 
@@ -64,6 +70,19 @@ jobs:
 
         echo "CHANGEDPROJECTS=$CHANGEDPROJECTS" >> $GITHUB_OUTPUT
         echo "REMOVEDPROJECTS=$REMOVEDPROJECTS" >> $GITHUB_OUTPUT
+    # There's logic to skip testing/building when only some metadata in the
+    # project file changes. In these cases, we still want to validate the project file.
+    - name: validate project files
+      run: |
+        CHANGESPROJECTSFILE=$(doit util_list_changed_dirs_with_main --only-project-file | tail -n -1)
+        CHANGEDPROJECTSFILE=$(echo $CHANGESPROJECTSFILE | jq -c -r '.changed')
+        echo "Changed projects (project file only): $CHANGEDPROJECTSFILE"
+        if [ "$CHANGEDPROJECTSFILE" != "[]" ]; then
+          items=$(echo $CHANGEDPROJECTSFILE | jq -c -r '.[]')
+          for item in ${items[@]}; do
+            doit validate_project_file:$item
+          done
+        fi
   test:
     needs: setup
     uses: ./.github/workflows/test.yml

.github/workflows/test.yml

@@ -44,10 +44,10 @@ jobs:
     outputs:
       projects: ${{ steps.set-projects.outputs.projects }}
     steps:
-    - uses: actions/checkout@v3
-    - uses: actions/setup-python@v4
+    - uses: actions/checkout@v4
+    - uses: actions/setup-python@v5
       with:
-        python-version: '3.9'
+        python-version: '3.12'
     - name: install deps
       # Minimal required deps to run the following script
       run: pip install doit pyyaml

.github/workflows/test_one.yml

@@ -32,10 +32,10 @@ jobs:
     outputs:
       runner: ${{ steps.set-runner.outputs.runner }}
     steps:
-    - uses: actions/checkout@v3
-    - uses: actions/setup-python@v4
+    - uses: actions/checkout@v4
+    - uses: actions/setup-python@v5
       with:
-        python-version: '3.9'
+        python-version: '3.12'
     - name: install deps
       # Minimal required deps to run the following script
       run: pip install doit pyyaml
@@ -57,23 +57,14 @@ jobs:
         shell: bash -l {0}
     timeout-minutes: 60
     steps:
-    - uses: actions/checkout@v3
-    - uses: conda-incubator/setup-miniconda@v2
+    - uses: actions/checkout@v4
+    - uses: conda-incubator/setup-miniconda@v3
       with:
         miniconda-version: "latest"
         auto-update-conda: false
         auto-activate-base: false
         activate-environment: examples-gallery-manage
         environment-file: envs/environment-${{ needs.infer_runner.outputs.runner }}.lock
-    - name: enable libmamba
-      run: |
-        conda activate base
-        conda install conda-libmamba-solver
-        conda config --set solver libmamba
-    - name: install nbval
-      # TODO: latest nbval not yet available on conda-forge
-      # when it is, remove this and add it to the lock file
-      run: pip install "nbval >= 0.10.0"
     - name: validate anaconda-project file
       run: doit validate_project_file:${{ inputs.project }}
     - name: validate anaconda-project lock file