Documentation fails to build

[ns-rse]

Currently the documentation fails to build.

Failed build

Current run is with Sphinx 7.3.7 fails with...

  Extension error (autoapi.extension):
  Handler <function doctree_read at 0x7fd4c6e8d3a0> for event 'doctree-read' threw an exception (exception: nodes.Node.traverse() is obsoleted by Node.findall().)
  Traceback (most recent call last):
    File "/opt/hostedtoolcache/Python/3.9.19/x64/bin/sphinx-multiversion", line 8, in <module>
      sys.exit(main())
    File "/opt/hostedtoolcache/Python/3.9.19/x64/lib/python3.9/site-packages/sphinx_multiversion/main.py", line 338, in main
      subprocess.check_call(cmd, cwd=current_cwd)
    File "/opt/hostedtoolcache/Python/3.9.19/x64/lib/python3.9/subprocess.py", line 373, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '('/opt/hostedtoolcache/Python/3.9.19/x64/bin/python', '-R', '-m', 'sphinx', '-D', 'smv_metadata_path=/tmp/tmpw3r5vhpw/versions.json', '-D', 'smv_current_version=main', '-c', '/home/runner/work/TopoStats/TopoStats/docs', '/tmp/tmpw3r5vhpw/b732f524d9acb1b3fbf04e20d69a300e3cf07d67/docs', '/home/runner/work/TopoStats/TopoStats/docs/_build/html/main')' returned non-zero exit status 2.

The installed packages and versions are listed in
failed_packages.txt

Working build

An older run with Sphinx 7.2.6 built correctly.

The installed packages and versions are listed in

worked_packages.txt

Differences

❱ diff failed_packages.txt worked_packages.txt 
2,4c2,4
< Pygments==2.18.0
< Sphinx==7.3.7
< accessible-pygments==0.0.5
---
> Pygments==2.17.2
> Sphinx==7.2.6
> accessible-pygments==0.0.4
7c7
< astroid==3.2.0
---
> astroid==3.1.0
9c9
< babel==2.15.0
---
> babel==2.14.0
14c14,15
< contourpy==1.2.1
---
> contextlib2==21.6.0
> contourpy==1.2.0
18c19
< exceptiongroup==1.2.1
---
> exceptiongroup==1.2.0
20,24c21,25
< fonttools==4.51.0
< h5py==3.11.0
< idna==3.7
< igor2==0.5.6
< imageio==2.34.1
---
> fonttools==4.50.0
> h5py==3.10.0
> idna==3.6
> igor2==0.5.5
> imageio==2.34.0
30,31c31,32
< jinja2==3.1.4
< joblib==1.4.2
---
> jinja2==3.1.3
> joblib==1.3.2
33c34
< lazy_loader==0.4
---
> lazy_loader==0.3
36,38c37,39
< matplotlib==3.8.4
< matplotlib-inline==0.1.7
< mdit-py-plugins==0.4.1
---
> matplotlib==3.8.3
> matplotlib-inline==0.1.6
> mdit-py-plugins==0.4.0
40c41
< myst-parser==3.0.1
---
> myst-parser==2.0.0
43c44
< numpydoc==1.7.0
---
> numpydoc==1.6.0
45,46c46,47
< pandas==2.2.2
< parso==0.8.4
---
> pandas==2.2.1
> parso==0.8.3
48c49
< pillow==10.3.0
---
> pillow==10.2.0
64c65
< schema==0.7.7
---
> schema==0.7.5
66,67c67,68
< scikit-learn==1.4.2
< scipy==1.13.0
---
> scikit-learn==1.4.1.post1
> scipy==1.12.0
73c74
< sphinx==7.3.7
---
> sphinx==7.2.6
75c76
< sphinx-autodoc-typehints==2.1.0
---
> sphinx-autodoc-typehints==2.0.0
90,91c91,92
< threadpoolctl==3.5.0
< tifffile==2024.5.10
---
> threadpoolctl==3.4.0
> tifffile==2024.2.12
94,97c95,98
< # topostats==2.2.post1.dev174+g2cf47670
< tqdm==4.66.4
< traitlets==5.14.3
< typing-extensions==4.11.0
---
> # topostats==2.2.post1.dev113+g3099098e
> tqdm==4.66.2
> traitlets==5.14.2
> typing-extensions==4.10.0
101c102
< zipp==3.18.1
\ No newline at end of file
---
> zipp==3.18.1

Investigations

I can recreate the error locally in a fresh environment and its not and error with sphinx-multiversion as the same occurs when using the Makefile to build a single version of the currently checked out commit.

With -P/--pdb Sphinx option enabled we see the following output pointing towards sphinx-autoapi being the culprit (its the penultimate call in the traceback and calls list(doctree.traverse(toctree)) (NB this is not the autoapi package at fault here).

g.
Exception occurred while building, starting debugger:
Traceback (most recent call last):
  File "/home/neil/.virtualenvs/topostats/lib/python3.12/site-packages/sphinx/cmd/build.py", line 298, in build_main
    app.build(args.force_all, args.filenames)
  File "/home/neil/.virtualenvs/topostats/lib/python3.12/site-packages/sphinx/application.py", line 355, in build
    self.builder.build_update()
  File "/home/neil/.virtualenvs/topostats/lib/python3.12/site-packages/sphinx/builders/__init__.py", line 293, in build_update
    self.build(to_build,
  File "/home/neil/.virtualenvs/topostats/lib/python3.12/site-packages/sphinx/builders/__init__.py", line 313, in build
    updated_docnames = set(self.read())
                           ^^^^^^^^^^^
  File "/home/neil/.virtualenvs/topostats/lib/python3.12/site-packages/sphinx/builders/__init__.py", line 420, in read
    self._read_serial(docnames)
  File "/home/neil/.virtualenvs/topostats/lib/python3.12/site-packages/sphinx/builders/__init__.py", line 441, in _read_serial
    self.read_doc(docname)
  File "/home/neil/.virtualenvs/topostats/lib/python3.12/site-packages/sphinx/builders/__init__.py", line 498, in read_doc
    publisher.publish()
  File "/home/neil/.virtualenvs/topostats/lib/python3.12/site-packages/docutils/core.py", line 236, in publish
    self.apply_transforms()
  File "/home/neil/.virtualenvs/topostats/lib/python3.12/site-packages/docutils/core.py", line 216, in apply_transforms
    self.document.transformer.apply_transforms()
  File "/home/neil/.virtualenvs/topostats/lib/python3.12/site-packages/sphinx/transforms/__init__.py", line 83, in apply_transforms
    super().apply_transforms()
  File "/home/neil/.virtualenvs/topostats/lib/python3.12/site-packages/docutils/transforms/__init__.py", line 182, in apply_transforms
    transform.apply(**kwargs)
  File "/home/neil/.virtualenvs/topostats/lib/python3.12/site-packages/sphinx/transforms/__init__.py", line 383, in apply
    self.app.emit('doctree-read', self.document)
  File "/home/neil/.virtualenvs/topostats/lib/python3.12/site-packages/sphinx/application.py", line 475, in emit
    return self.events.emit(event, *args, allowed_exceptions=allowed_exceptions)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/neil/.virtualenvs/topostats/lib/python3.12/site-packages/sphinx/events.py", line 97, in emit
    results.append(listener.handler(self.app, *args))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/neil/.virtualenvs/topostats/lib/python3.12/site-packages/autoapi/extension.py", line 158, in doctree_read
    nodes = list(doctree.traverse(toctree))
                 ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/neil/.virtualenvs/topostats/lib/python3.12/site-packages/docutils/nodes.py", line 225, in traverse
    warnings.warn('nodes.Node.traverse() is obsoleted by Node.findall().',
PendingDeprecationWarning: nodes.Node.traverse() is obsoleted by Node.findall().
> /home/neil/.virtualenvs/topostats/lib/python3.12/site-packages/docutils/nodes.py(225)traverse()
-> warnings.warn('nodes.Node.traverse() is obsoleted by Node.findall().',

If we look at the source code in GitHub and check the blame we see this was corrected 2 months ago and is in the 3.1.0b0 pre-release and if we install the pre-release locally make html successfully builds the documentation 🙌 🎉

Unfortunately sphinx-multiversion . _build doesn't build all the version 😿 one of the older versions (not main or v2.0.0 fails).

Solution

Part 1

Temporarily peg the version of sphinx-autoapi in pyproject.toml

docs = [
  ...
  "sphinx-autoapi==3.1.0b0",
  ...
]

...and remember to remove this once sphinx-autoapi-3.1.0 is released.

Check and see what get built in GitHub Workflow.

Part 2

It may be desirable to restrict building of documentation to only tagged releases, there are some comments in .github/workflows/sphinx_docs_to_gh_pages.yaml on how to achieve this.

[ns-rse]

Partial success.

Documentation is back online but only main version works all others fail to be found (403) because whilst the main branch builds in CI it fails for other branches.

I see the same error locally...

Extension error (sphinx_autodoc_typehints):
Handler <function process_docstring at 0x7d94184b4860> for event 'autodoc-process-docstring' threw an exception (exception: The frontend.OptionParser class will be replaced by a subclass of argparse.ArgumentParser in Docutils 0.21 or later.)
Traceback (most recent call last):
  File "/home/neil/.virtualenvs/topostats/bin/sphinx-multiversion", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/neil/.virtualenvs/topostats/lib/python3.12/site-packages/sphinx_multiversion/main.py", line 338, in main
    subprocess.check_call(cmd, cwd=current_cwd)
  File "/usr/lib/python3.12/subprocess.py", line 413, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '('/home/neil/.virtualenvs/topostats/bin/python', '-R', '-m', 'sphinx', '-D', 'smv_metadata_path=/tmp/tmpiodbjbom/versions.json', '-D', 'smv_current_version=v2.0.0', '-c', '/home/neil/work/git/hub/AFM-SPM/TopoStats/docs', '/tmp/tmpiodbjbom/d819d1f3d0bed977cdfa926fb309a76a8e394462/docs', '/home/neil/work/git/hub/AFM-SPM/TopoStats/docs/_build/v2.0.0')' returned non-zero exit status 2.

This is because sphinx-autodoc-typehints uses OptionParser imports docutils.frontend.OptionParser which as noted will become deprecated in favour of argparse.ArgumentParser subclass.

Why this happens on other versions I'm unsure, but I've opened an issue in sphinx-autodocs-typehints about this.

[MaxGamill-Sheffield]

Closing as the versioned docs look back online :)