Merge pull request #220 from lsst-sqre/tickets/DM-43485

DM-43485: Integrate MyST-NB for technotes and user guides

CHANGELOG.md

@@ -2,6 +2,13 @@
 
 <!-- scriv-insert-here -->
 
+<a id='changelog-1.2.0'></a>
+## 1.2.0 (2024-03-26)
+
+### New features
+
+- Rubin user guides (`documenteer.conf.guide`) and technotes (`documenteer.conf.technote`) now include [MyST-NB](https://myst-nb.readthedocs.io/en/latest/) to support Jupyter Notebooks in Sphinx documentation. The MyST-NB extension also supersedes MyST-Parser for Markdown syntax support. For guides, Jupyter Notebook files can be intermixed with Markdown (`.md`) and reStructuredText (`.rst`) files. An `ipynb` file is considered as a page in the documentation. For technotes, the Jupyter Notebook must be named `index.ipynb`. By default, these configurations use MyST-NB's "auto" mode for notebook execution: only if a notebook is missing outputs will it be executed.
+
 <a id='changelog-1.1.1'></a>
 ## 1.1.1 (2024-02-21)
 

demo/ipynb-technote/.gitignore

@@ -0,0 +1,2 @@
+_build
+.technote

demo/ipynb-technote/Makefile

@@ -0,0 +1,3 @@
+.PHONY:
+clean:
+	rm -rf _build

demo/ipynb-technote/conf.py

@@ -0,0 +1 @@
+from documenteer.conf.technote import *  # noqa F401 F403

demo/ipynb-technote/diagram.py

@@ -0,0 +1,14 @@
+from diagrams.k8s.clusterconfig import HPA
+from diagrams.k8s.compute import Deployment, Pod, ReplicaSet
+from diagrams.k8s.network import Ingress, Service
+from sphinx_diagrams import SphinxDiagram
+
+with SphinxDiagram(title="GKE"):
+    net = Ingress("domain.com") >> Service("svc")
+    (
+        net
+        >> [Pod("pod1"), Pod("pod2"), Pod("pod3")]
+        << ReplicaSet("rs")
+        << Deployment("dp")
+        << HPA("hpa")
+    )

demo/ipynb-technote/index.ipynb

@@ -0,0 +1,97 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Demo technote\n",
+    "\n",
+    "```{abstract}\n",
+    "A *technote* is a web-native single page document that enables rapid technical communication within and across teams.\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Introduction\n",
+    "\n",
+    "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Proin facilisis pharetra neque, at semper nulla mattis auctor. Proin semper mollis enim eget interdum. Mauris eleifend eget diam vitae bibendum. Praesent ut aliquet odio, sodales imperdiet nisi. Nam interdum imperdiet tortor sed fringilla. Maecenas efficitur mi sodales nulla commodo rutrum. Ut ornare diam quam, sed commodo turpis aliquam et. In nec enim consequat, suscipit tortor sit amet, luctus ante. Integer dictum augue diam, non pulvinar massa euismod in. Morbi viverra condimentum auctor. Nullam et metus mauris. Cras risus ex, porta sit amet nibh et, dapibus auctor leo.\n",
+    "\n",
+    "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Proin facilisis pharetra neque, at semper nulla mattis auctor. Proin semper mollis enim eget interdum. Mauris eleifend eget diam vitae bibendum. Praesent ut aliquet odio, sodales imperdiet nisi. Nam interdum imperdiet tortor sed fringilla. Maecenas efficitur mi sodales nulla commodo rutrum. Ut ornare diam quam, sed commodo turpis aliquam et. In nec enim consequat, suscipit tortor sit amet, luctus ante. Integer dictum augue diam, non pulvinar massa euismod in. Morbi viverra condimentum auctor. Nullam et metus mauris. Cras risus ex, porta sit amet nibh et, dapibus auctor leo.\n",
+    "\n",
+    "A parenthetical citation {cite:p}`2017arXiv170309824V`. And a textual citation {cite:t}`2017arXiv170309824V`."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Method\n",
+    "\n",
+    "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Proin facilisis pharetra neque, at semper nulla mattis auctor. Proin semper mollis enim eget interdum. Mauris eleifend eget diam vitae bibendum. Praesent ut aliquet odio, sodales imperdiet nisi. Nam interdum imperdiet tortor sed fringilla. Maecenas efficitur mi sodales nulla commodo rutrum. Ut ornare diam quam, sed commodo turpis aliquam et. In nec enim consequat, suscipit tortor sit amet, luctus ante. Integer dictum augue diam, non pulvinar massa euismod in. Morbi viverra condimentum auctor. Nullam et metus mauris. Cras risus ex, porta sit amet nibh et, dapibus auctor leo.\n",
+    "\n",
+    "A list:\n",
+    "\n",
+    "- First item\n",
+    "- Second item\n",
+    "- Third item\n",
+    "\n",
+    "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Proin facilisis pharetra neque, at semper nulla mattis auctor. Proin semper mollis enim eget interdum. Mauris eleifend eget diam vitae bibendum. Praesent ut aliquet odio, sodales imperdiet nisi. Nam interdum imperdiet tortor sed fringilla. Maecenas efficitur mi sodales nulla commodo rutrum. Ut ornare diam quam, sed commodo turpis aliquam et. In nec enim consequat, suscipit tortor sit amet, luctus ante. Integer dictum augue diam, non pulvinar massa euismod in. Morbi viverra condimentum auctor. Nullam et metus mauris. Cras risus ex, porta sit amet nibh et, dapibus auctor leo."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "print(\"Hello world\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Analysis\n",
+    "\n",
+    "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Proin facilisis pharetra neque, at semper nulla mattis auctor. Proin semper mollis enim eget interdum. Mauris eleifend eget diam vitae bibendum. Praesent ut aliquet odio, sodales imperdiet nisi. Nam interdum imperdiet tortor sed fringilla. Maecenas efficitur mi sodales nulla commodo rutrum. Ut ornare diam quam, sed commodo turpis aliquam et. In nec enim consequat, suscipit tortor sit amet, luctus ante. Integer dictum augue diam, non pulvinar massa euismod in. Morbi viverra condimentum auctor. Nullam et metus mauris. Cras risus ex, porta sit amet nibh et, dapibus auctor leo.\n",
+    "\n",
+    "```{prompt} bash\n",
+    "git add index.rst\n",
+    "```\n",
+    "\n",
+    "Some following text.\n",
+    "\n",
+    "```{mermaid}\n",
+    "graph TD\n",
+    "    A[Square Rect] -- Link text --> B((Circle))\n",
+    "    A --> C(Round Rect)\n",
+    "    B --> D{Rhombus}\n",
+    "    C --> D\n",
+    "```\n",
+    "\n",
+    "```{diagrams} diagram.py\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## References\n",
+    "\n",
+    "```{bibliography}\n",
+    "```"
+   ]
+  }
+ ],
+ "metadata": {
+  "language_info": {
+   "name": "python"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

demo/ipynb-technote/technote.toml

@@ -0,0 +1,25 @@
+[technote]
+id = "SQR-000"
+series_id = "SQR"
+canonical_url = "https://sqr-000.lsst.io"
+github_url = "https://github.com/lsst-sqre/sqr-000"
+github_default_branch = "master"
+date_created = 2015-11-18
+date_updated = 2015-11-23
+version = "1.0.0"
+
+[[technote.authors]]
+name.given = "Jonathan"
+name.family = "Sick"
+orcid = "https://orcid.org/0000-0003-3001-676X"
+affiliations = [
+    { name = "Rubin Observatory", ror = "https://ror.org/048g3cy84" }
+]
+
+[[technote.authors]]
+name.given = "Frossie"
+name.family = "Economou"
+
+[[technote.authors]]
+name.given = "Russ"
+name.family = "Allbery"

docs/_rst_epilog.rst

@@ -77,7 +77,7 @@
 
 .. links to sphinx directives
 
-.. |toctree| replace:: :external+sphinx:rst:dir:`toctree`
+.. |toctree| replace:: `toctree <https://www.sphinx-doc.org/en/master/usage/restructuredtext/directives.html#directive-toctree>`__
 
 .. Badges
 

docs/guides/including-notebooks.ipynb

@@ -0,0 +1,90 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Including Jupyter notebooks in documentation\n",
+    "\n",
+    "Rubin Observatory user guides can include pages written as Jupyter notebooks. Jupyter notebooks are ideal for including tutorials and examples in your documentation content. Internally, Documenteer uses [MyST-NB](https://myst-nb.readthedocs.io/en/latest/) to parse Jupyter Notebooks into Sphinx content. Prose cells in Jupyter notebooks use the MyST markdown syntax so that they can use Sphinx directives and roles (see {doc}`markdown-primer`).\n",
+    "\n",
+    "## Using ipynb files as content pages\n",
+    "\n",
+    "To include a Jupyter notebook as a page in the documentation, add and commit its ipynb file to the documentation repository like you would a reStructuredText (`.rst`) or markdown (`.md`) file. Then include that `.ipynb` notebook file in a `toctree` directive. Like regular pages, notebook pages are listed in `toctree` with their file names, but without the file extension:\n",
+    "\n",
+    "```{code-block} rst\n",
+    ".. toctree::\n",
+    "   :maxdepth: 1\n",
+    "\n",
+    "   my-notebook\n",
+    "``` "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Computation\n",
+    "\n",
+    "Unlike regular markdown and reStructuredText pages, Jupyter notebooks can contain executable code cells. By default, Documenteer uses MyST-NB's `auto` setting so that notebooks with missing outputs are executed when the documentation is built. Alternatively, if the notebook is present with all outputs it will not be executed.\n",
+    "\n",
+    "### Pre-executing notebooks that rely on the RSP\n",
+    "\n",
+    "If your notebook relies on running in the Notebook Aspect of the Rubin Science Platform, you currently cannot execute that notebook in the GitHub Actions environment that Rubin documentation is built in. In this case, you must execute the notebook on the Rubin Science Platform, and commit that executed notebook to your documentation repository. By default, the notebook *will not* be re-executed during the Sphinx build when outputs are already present."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Examples\n",
+    "\n",
+    "Here is a Python code cell with its output computed during the documentation build:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "for i in range(5):\n",
+    "    print(i * 2)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Further reading\n",
+    "\n",
+    "- {doc}`markdown-primer`\n",
+    "- The [MyST-NB documentation](https://myst-nb.readthedocs.io/en/latest/)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": ".venv",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.5"
+  },
+  "mystnb": {
+   "code_prompt_show": "source|outputs"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

docs/guides/index.rst

@@ -37,6 +37,14 @@ Documenteer provides a configuration profile for creating branded user guides wi
    badges
    tabsets
 
+.. toctree::
+   :maxdepth: 2
+   :caption: Markdown & Jupyter Notebooks
+   :name: toc-guides-md-ipynb
+
+   markdown-primer
+   including-notebooks
+
 .. toctree::
    :maxdepth: 2
    :caption: Science Pipelines

docs/guides/markdown-primer.md

@@ -0,0 +1,57 @@
+# Using Markdown syntax in Rubin Observatory documentation
+
+Rubin Observatory user guides use [MyST-Parser](https://myst-parser.readthedocs.io/en/latest/) to support Markdown syntax in addition to reStructuredText.
+This page explains the specific Markdown syntax that is supported in Rubin Observatory's Sphinx documentation and how it relates to the reStructuredText syntax you may already be familiar with for Sphinx documentation.
+If you are {doc}`including Jupyter Notebooks in your documentation <including-notebooks>`, you can also use this Markdown syntax in your prose cells.
+
+## Mixing Markdown and reStructuredText
+
+The default content markup for Rubin Observatory documentation is reStructuredText (rst).
+You can also elect to use Markdown syntax for your documentation content.
+In the same user guide, you have both Markdown (``.md``) and reStructuredText (``.rst``) files.
+
+```{note}
+It's a good idea to check with your team to see if they have a preference for using Markdown or reStructuredText.
+Having a consistent style across your documentation will make it easier for your team to maintain and update the content.
+```
+
+## Roles
+
+Roles add semantic meaning to text.
+Roles are *inline*, so they work at the word or phrase level (as opposed to directives, which apply to paragraph blocks).
+
+In Markdown, roles are indicated by curly braces around the role name, followed by the content in single back ticks.
+Examples of `ref` and `doc` roles:
+
+```{code-block} markdown
+{ref}`text <link-target>`
+
+{doc}`index`
+```
+
+## Directives
+
+Directives are block-level formatting elements.
+Examples include admonitions like `note`, content blocks like `code-block`, and content-generators like `toctree`.
+In Markdown, directives use the triple-backtick *code fence* syntax, with the directive name following the opening backticks in curly braces:
+
+````{code-block} markdown
+```{note}
+Content of the note.
+```
+````
+
+Fields in directives are indicated by colons around in the field name, followed by the value (just like in reStructuredText):
+
+````{code-block} markdown
+```{toctree}
+:maxdepth: 2
+
+page-one
+page-two
+```
+````
+
+## More resources
+
+For more information and examples of the Markdown syntax supported in Rubin Observatory documentation, see the [MyST-Parser documentation](https://myst-parser.readthedocs.io/en/latest/).

pyproject.toml

@@ -67,6 +67,7 @@ guide = [
     "sphinx-prompt",
     "sphinx-jinja>=2", # namespace changed in 2.0
     "myst-parser",
+    "myst-nb",
     "markdown-it-py[linkify]",
     "sphinxcontrib-mermaid",
     "sphinxext-opengraph",
@@ -104,6 +105,7 @@ technote = [
     "sphinxcontrib-mermaid",
     "sphinx-diagrams",
     "sphinx_design",
+    "myst-nb"
 ]
 
 [project.urls]

src/documenteer/conf/guide.py

@@ -128,7 +128,7 @@
 
 extensions = [
     "sphinxcontrib.jquery",
-    "myst_parser",
+    "myst_nb",  # enables myst-parser as well
     "sphinx_copybutton",
     "sphinx_design",
     "sphinxcontrib.mermaid",
@@ -167,10 +167,10 @@
 version = _conf.version
 release = version
 
+# The source file suffixes for .md and .ipynb are automatically managed by
+# myst-nb.
 source_suffix = {
     ".rst": "restructuredtext",
-    ".txt": "markdown",
-    ".md": "markdown",
 }
 
 root_doc = "index"

src/documenteer/conf/technote.py

@@ -17,10 +17,18 @@
 except ValueError:
     pass
 
+try:
+    # Remove myst-parser if added by technote.sphinxconf so we can
+    # add myst-nb.
+    extensions.remove("myst_parser")  # noqa: F405
+except ValueError:
+    pass
+
 # Add the GitHub bibfile cache extension before sphinxcontrib-bibtex so
 # that it can add bibfiles to the sphinxcontrib-bibtex configuration.
 extensions.extend(  # noqa: F405
     [
+        "myst_nb",  # enables MyST markdown and Jupyter Notebook parsing
         "documenteer.ext.jira",
         "documenteer.ext.lsstdocushare",
         "documenteer.ext.mockcoderefs",

tox.ini

@@ -84,3 +84,5 @@ allowlist_externals =
 commands =
     rm -rf demo/rst-technote/_build
     sphinx-build --keep-going -n -W -T -b html -d {envtmpdir}/doctrees demo/rst-technote demo/rst-technote/_build/html
+    rm -rf demo/ipynb-technote/_build
+    sphinx-build --keep-going -n -W -T -b html -d {envtmpdir}/doctrees demo/ipynb-technote demo/ipynb-technote/_build/html