Content Comparison

About Sphinx

sphinx Sphinx is a popular tool for documenting Python projects, including the ability to automatically generate automatic documentation using docstrings in your source code.

...

reStructuredText (RST) vs. Markdown (MD)

Because there are never enough markup languages out there, restructuredtext reStructuredText was created for documenting Python, but Sphinx can also support the easier and more popular, markdown Markdown format with a couple of plugins.

...

Code Block

language	python
linenumbers	false

cat docs/requirements.txt
# bring in requirements for my app (excepting the optional database):
-r../requirements-django.txt
# stuff needed for sphinx documentation:
Sphinx==1.8.2
sphinx-markdown-tables==0.0.9
sphinx-rtd-theme==0.4.2
sphinxcontrib-apidoc==0.3.0
sphinxcontrib-confluencebuilder==0.9
sphinxcontrib-django==0.4
sphinxcontrib-websupport==1.1.0
recommonmark==0.4.0

Then run the quickstart:

Code Block

language	bash
linenumbers	false


sphinx-quickstart

This creates a conf.py which is the core configuration file for Sphinx. And, since it’s Python code, you can do all kinds of cool stuff. Here are a few of my changes after the quickstart, which notably includes some django-specific stuff, autmatic automatic API documentation and support for Markdown and Markdown Tables:

Code Block

language	diff
linenumbers	false


diff --git b/docs/conf.py a/docs/conf.py
index 55c2351..dc4c7a4 100644
--- b/docs/conf.py
+++ a/docs/conf.py
@@ -12,22 +12,37 @@
 # add these directories to sys.path here. If the directory is relative to the
 # documentation root, use os.path.abspath to make it absolute, like shown here.
 #
-# import os
-# import sys
-# sys.path.insert(0, os.path.abspath('.'))
+import os
+import sys
+import datetime
+import django
+from recommonmark.parser import CommonMarkParser
 
+django_version = ".".join(map(str, django.VERSION[0:2]))
+python_version = ".".join(map(str, sys.version_info[0:2]))
+
+sys.path.insert(0, os.path.abspath('..'))
+
+os.environ['DJANGO_SETTINGS_MODULE'] = 'training.settings'
+django.setup()
 
 # -- Project information -----------------------------------------------------
 
+# See https://pypi.org/project/sphinxcontrib-django/
 project = 'Django {json:api} training'
-copyright = '2018, Alan Crosswell'
+year = datetime.date.today().year
+copyright = '{}, The Trustees of Columbia University in the City of New York'.format(year)
 author = 'Alan Crosswell'
 
 # The short X.Y version
-version = ''
+from myapp import VERSION
+version = VERSION
 # The full version, including alpha/beta/rc tags
-release = ''
+release = VERSION
 
+# Auto-generate API documentation.
+#os.environ['SPHINX_APIDOC_OPTIONS'] = "members,undoc-members,show-inheritance"
+os.environ['SPHINX_APIDOC_OPTIONS'] = "members,show-inheritance"
 
 # -- General configuration ---------------------------------------------------
 
@@ -39,23 +54,30 @@ release = ''
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
 extensions = [
-    'sphinx.ext.autodoc',
-    'sphinx.ext.intersphinx',
-    'sphinx.ext.todo',
-    'sphinx.ext.viewcode',
+    'sphinxcontrib.apidoc',   # runs sphinx-apidoc automatically as part of sphinx-build
+    'sphinx.ext.autodoc',     # the autodoc extensions uses files generated by apidoc
+    'sphinxcontrib_django',   # does some nicer django autodoc formatting, but:
+                              # https://github.com/edoburu/sphinxcontrib-django/issues/12
+    'sphinx.ext.viewcode',    # enable viewing autodoc'd code
+    'sphinx.ext.intersphinx', # make links between different sphinx-documented packages
+    'sphinx.ext.todo',        # TODO: figure out how to use this;-)
+    'sphinx_markdown_tables', # CommonMark doesn't do tables: This extensions does!
+    'sphinxcontrib.confluencebuilder', # supposedly installs docs on Confluence
 ]
-
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['_templates']
 
+source_parsers = {
+    '.md': CommonMarkParser,
+}
+
 # The suffix(es) of source filenames.
 # You can specify multiple suffix as a list of string:
 #
-source_suffix = '.rst'
+source_suffix = ['.rst', '.md']
 
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.
@@ -67,7 +89,7 @@ language = None
 # List of patterns, relative to source directory, that match files and
 # directories to ignore when looking for source files.
 # This pattern also affects html_static_path and html_extra_path.
-exclude_patterns = []
+exclude_patterns = ['build']
 
 # The name of the Pygments (syntax highlighting) style to use.
 pygments_style = None
@@ -78,13 +100,23 @@ pygments_style = None
 # The theme to use for HTML and HTML Help pages.  See the documentation for
 # a list of builtin themes.
 #
-html_theme = 'alabaster'
+# html_theme = 'alabaster'
+# html_theme = 'default'
+html_theme = 'sphinx_rtd_theme'
 
 # Theme options are theme-specific and customize the look and feel of a theme
 # further.  For a list of options available for each theme, see the
 # documentation.
 #
-# html_theme_options = {}
+html_theme_options = {
+    # these are for sphinx_rtd_theme:
+    'prev_next_buttons_location': 'both',
+    'collapse_navigation': True,
+    # these are for alabaster:
+    # 'show_relbars': True,
+    # 'fixed_sidebar': True,
+    # 'sidebar_collapse': True,
+}
 
 # Add any paths that contain custom static files (such as style sheets) here,
 # relative to this directory. They are copied after the builtin static files,
@@ -99,8 +131,16 @@ html_static_path = ['_static']
 # default: ``['localtoc.html', 'relations.html', 'sourcelink.html',
 # 'searchbox.html']``.
 #
-# html_sidebars = {}
-
+# also for alabaster theme:
+# html_sidebars = {
+#     '**': [
+#         'about.html',
+#         'navigation.html',
+#         'relations.html',
+#         'searchbox.html',
+#         'donate.html',
+#     ]
+# }
 
 # -- Extension configuration -------------------------------------------------
 
+autodoc_member_order = 'bysource'
+autodoc_inherit_docstrings = False
+
+apidoc_module_dir = '../myapp'
+apidoc_output_dir = 'apidoc'
+apidoc_excluded_paths = ['../myapp/migrations']
+apidoc_separate_modules = True
+apidoc_toc_file = False
+apidoc_module_first = True
+apidoc_extra_args = ['-f']
+
+confluence_publish = False  # for now until we figure out how to make it work.
True
+confluence_spaceserver_nameurl = os.environ.get('CONFLUENCE_SPACESERVER', None"https://confluence.columbia.edu")
+confluence_parentspace_pagename = os.environ.get('CONFLUENCE_PARENTSPACE', None)
+confluence_serverparent_urlpage = "https://confluence.columbia.edu"os.environ.get('CONFLUENCE_PARENT', None)
+confluence_server_user = os.environ.get('CONFLUENCE_USER', None)
+confluence_ask_password = True
+# confluence_server_pass = os.environ[.get('CONFLUENCE_PASS'], None)
+
 # -- Options for intersphinx extension ---------------------------------------
 
-intersphinx_mapping = {'https://docs.python.org/': None}
+intersphinx_mapping = {
+    'python': ('https://docs.python.org/{}'.format(python_version), None),
+    'django': ('https://docs.djangoproject.com/en/{}/'.format(django_version), 
+               'https://docs.djangoproject.com/en/{}/_objects/'.format(django_version)),
+    # not sure why but the default lookup of objects.inv fails with None
+    'djangorestframework-jsonapi': ('https://django-rest-framework-json-api.readthedocs.io/en/stable/',
+                                    'https://django-rest-framework-json-api.readthedocs.io/en/stable/objects.inv'),
+    # DRF doesn't use sphinx but rather mkdocs:-(
+    #'djangorestframework': ('https://django-rest-framework.readthedocs.io/en/stable/org/', None),
+}

Viewing Sphinx-generated content locally

...

Code Block

language	python
linenumbers	false


(env) django-training$ cd docs
(env) docs$ make html
Running Sphinx v1.8.2
loading pickled environment... done
Creating file /Users/alan/src/django-training/docs/apidoc/myapp.admin.rst.
Creating file /Users/alan/src/django-training/docs/apidoc/myapp.models.rst.
Creating file /Users/alan/src/django-training/docs/apidoc/myapp.serializers.rst.
Creating file /Users/alan/src/django-training/docs/apidoc/myapp.views.rst.
Creating file /Users/alan/src/django-training/docs/apidoc/myapp.rst.
Creating file /Users/alan/src/django-training/docs/apidoc/myapp.tests.test_models.rst.
Creating file /Users/alan/src/django-training/docs/apidoc/myapp.tests.test_views.rst.
Creating file /Users/alan/src/django-training/docs/apidoc/myapp.tests.rst.
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 1 source files that are out of date
updating environment: 25 added, 0 changed, 0 removed
reading sources... [100%] welcome                                                                                                                                   
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [100%] welcome                                                                                                                                    
generating indices... genindex py-modindex
highlighting module code... [100%] myapp.views                                                                                                                      
writing additional pages... search
copying images... [100%] ./media/image2.png                                                                                                                         
copying static files... done
copying extra files... done
dumping search index in English (code: en) ... done
dumping object inventory... done
build succeeded.

The HTML pages are in build/html.
(env) docs$

Publishing to Confluence

We use confluence for an internal documentation repository.

TODO: Document how to publish to confluencePublishing to Confluence is not recommended due to the limitations described below

We use Confluence for an internal documentation repository and would like to host our sphinx-generated documentation there.

Configuring Confluencebuilder

You have to get a non-CAS guest user and password in order to bypass SAML login.

I use a shell script, confluence.sh to set these environment variables:

Code Block

language	bash
linenumbers	false

export CONFLUENCE_SERVER=https://confluence.columbia.edu/confluence
export CONFLUENCE_USER=mysphinx
export CONFLUENCE_PASS=PASSWORD
export CONFLUENCE_SPACE="~mysphinx"
export CONFLUENCE_PARENT="API"
$*

Confluencebuilder shortcomings

The sphinxcontrib-confluencebuilder attempts to generate Confluence content but suffers from several shortcomings:

Several common code languages are not recognized, yielding these errors:

Code Block

language	python
linenumbers	false

WARNING: unknown code language: console
WARNING: unknown code language: ini
WARNING: unknown code language: json
WARNING: unknown code language: text
WARNING: unknown code language: tsql
WARNING: unknown code language: yaml

Some of these can be easily worked-around (e.g. substitute sql for tsql) but lack of text is pretty basic stuff.

Certain instances of curly braces are not properly quoted, leading to 500 macro unknown errors like this:

Code Block

language	python
linenumbers	false

An unsupported Confluence API call has been made.

REQ: POST
RSP: 500
URL: https://confluence.columbia.edu/confluence/rest/api
API: contentbody/convert/storage
MSG: com.atlassian.confluence.content.render.xhtml.migration.exceptions.UnknownMacroMigrationException: The macro 'json' is unknown.
---

Curly braces appear to be OK for normal body text but break down in:

browser link titles such as [See {json:api}](https://jsonapi.org)

autodoc-generated code blocks:

Code Block

language	diff
linenumbers	false

diff --git a/myapp/serializers.py b/myapp/serializers.py
index beb962c..e8e15ac 100644
--- a/myapp/serializers.py
+++ b/myapp/serializers.py
@@ -70,8 +70,8 @@ class CourseSerializer(HyperlinkedModelSerializer):
         related_link_view_name='course-related',
     )

-    #: `{json:api} compound document <https://jsonapi.org/format/#document-compound-documents>`_
-    #: (also used for `related_serializers` for DJA 2.6.0)
+    # `JSON:API compound document <https://jsonapi.org/format/#document-compound-documents>`_
+    # (also used for `related_serializers` for DJA 2.6.0)
     included_serializers = {
         'course_terms': 'myapp.serializers.CourseTermSerializer',
     }
@@ -111,8 +111,8 @@ class CourseTermSerializer(HyperlinkedModelSerializer):
         related_link_view_name='course_term-related',
     )

These can be worked around by excluding undocumented members and removing docstrings or #: comments (which sphinx treats like docstrings). This was supposedly fixed but is apparently not (or this is a new way to trigger the issue).

There’s no way to put the ToC in the sidebar so navigation sucks.
No search.
Poor formatting of autodocs.

My conclusion: Just find a way to locally host the HTML tree generated by sphinx-build rather than trying to force this into Confluence. For example, this works:

Publishing to a static web site

Code Block

language	bash
linenumbers	false

(env) django-training$ rsync -av -e ssh docs/build/html/ alan@cunix:public_html/django-jsonapi-training

You can see the pages at http://www.columbia.edu/~alan/django-jsonapi-training/

Or use this if you want to secure the content:

Code Block

language	bash
linenumbers	false

(env) django-training$ rsync -av -e ssh docs/build/html/ alan@cunix:secure_html/django-jsonapi-training

After adding an appropriate .htaccess you can see these, if you are a CUIT staff member, at https://www1.columbia.edu/~alan/django-jsonapi-training/

Publishing to RTD

https://readthedocs.io (RTD) is where most open-source projects host their documentation. While your project is probably internal, here’s how to do it if you are open-sourcing it.

Once we’ve got sphinx working locally, and the project hosted on github, getting it working with RTD is pretty straightforward. See the sphinx getting started guide.

On the rtd RTD dashboard import a new project and make sure to:

Pick a name. I’ve chosen columbia-it-django-jsonapi-training
Provide the github repository URL: https://github.com/columbia-it/django-jsonapi-training
In advanced settings configure the PIP requirements file: docs/requirements.txt and make sure to select CPython 3.x as the Python interpreter.

Fine print: pyodbc breakage

...

Code Block

language	python
linenumbers	false


# requirements for our app:
-rrequirements-django.txt
# optional sqlserver requirements:
-rrequirements-sqlserver.txt

with the main stuff in requirements-django.txt and the additional SQL Server stuff in requirements-sqlserver.txt.

...

Code Block

language	python
linenumbers	false


# bring in requirements for my app (excepting the optional database):
-r../requirements-django.txt
# stuff needed for sphinx documentation:
Sphinx==1.8.2
sphinx-markdown-tables==0.0.9
sphinx-rtd-theme==0.4.2
sphinxcontrib-apidoc==0.3.0
sphinxcontrib-confluencebuilder==0.9
sphinxcontrib-django==0.4
sphinxcontrib-websupport==1.1.0
recommonmark==0.4.0

In anticipation of adding travis support on github, I also changed tox.ini to have a separate section for local sphinx builds: tox -e sphinx.

Version	Old Version 8	New Version 9
Changes made by	Alan Sphinx Service Account	Alan Sphinx Service Account
Saved on	Dec 11, 2018	Jan 12, 2019

Versions Compared

Key

About Sphinx

reStructuredText (RST) vs. Markdown (MD)

Viewing Sphinx-generated content locally

Publishing to Confluence

Configuring Confluencebuilder

Confluencebuilder shortcomings

Publishing to a static web site

Publishing to RTD

Fine print: pyodbc breakage