Documenting code is notoriously hard to do well.

Many years ago now, Donald Knuth introduced the idea of literate programming, where code essentially exists as marked up snippets inside an essay that expounded the algorithms being developed by the programmer.

Knuth’s emphasis was on the essay part of the puzzle. Very fitting for a distinguished author such as himself with numerous seminal and beautifully produced textbooks to his name.

However, most programmers aren’t all that good at writing a small number of lines of code, let alone a long form exposition of some sort!

Nevertheless, the idea of code as an essay lives on and flourishes these days in many popular environments like Jupyter notebooks, Mathematica notebooks, Maple sheets etc. There are even collaborative tools that allow a group of authors to work on a common notebook without clobbering each other’s work. However, Knuth’s idea of one source both getting tangled into efficient code for optimal execution and also woven into beautiful documentation gets lost in those systems. In the hands of good authors, notebooks can be great at exposition but they generally do not compose well into larger systems.

Code libraries of one form or another remain the main tools that are deployed to build those larger systems. And libraries are really only widely used if they are properly documented.

Most coders do learn to occasionally plop a comment line or two into their code. There have been various attempts to standardize the format of those code comments with the aim of easily extracting them into a form of readable documentation.

At first blush, this seems like a reasonable goal. The library author concentrates for the most part on writing code but she also learns a little extra syntax which is used to slightly enrich and, more importantly, standardize the way comments are written—particularly those at, or near, the start of major code blocks like classes, methods, functions, etc. The compiler ignores all comments anyway so a separate tool extracts the specially formatted comments and uses those, perhaps along with other information from the compiler, to automatically generate documentation.

The emphasis here is the opposite of Knuth’s. You mainly write code and hopefully embed enough comments to document the intent and use of that code. The main advantage is that the comments live right in the source itself so have a meaningful chance of remaining up to date and relevant as the code evolves.

One advantage of using specially marked up comments as documentation is that modern code editors will parse them and make them available when a programmer imports a library module or header file. She can then just hover over a library provided function call and get a nice tool tip to remind her of what the relevant arguments should be etc.

This tool tip use of formatted comments is by itself quite useful and worth the price of admission. The library writer provides some brief standardized comments which are not too onerous to create. For example, in the class GF2::Matrix we have a identity(…​) method that begins as follows:

Here we are using comments marked up in one of the formats understood by one of the best know documentation generators Doxygen.

Doxygen is itself a sizable system and one with a large number of possible directives. However, if you just want to provide some useful tool tips you don’t need to understand anything about Doxygen at all (or even have it installed) and really only need to know a handful of fairly obvious directives—the three used above @brief, @param, and @return already cover a lot of ground and their purposes are pretty obvious!

This works because parsing trivial Doxygen markup is built in to most modern code environments either directly or through a plugin. For example, if you are editing code that calls that method in VSCode, hovering over the appropriate spot produces a tool tip like:

The tip is slightly wordy because GF2::Matrix is a template class (hence that <Block, Allocator> reference) but nonetheless, it does an adequate job of explaining what’s going on.

This particular method is so simple that it might be argued that any documentation at all is a bit of an overkill but we took the view it is better to be consistent and to provide this minimal level of documentation in-source for all public methods and functions in our library.

However, good documentation goes a lot further than just providing tool tips.

Introductory material may be needed, the rationale for a particular implementation given, gotchas enumerated, fallbacks enumerated if preconditions are not met, etc. Above all, short focused examples showing how a method might be used and what it might return are generally really useful for the end-user of a library. This is the material you expect to see in a professional library’s long form documentation site all nicely laid out and correctly linked together in a user friendly format.

Doxygen was built to handle all of that and there are projects which use it. However, putting all that material into the source code itself just seems clunky and pushes code commenting a bit too far. Apart from anything else, the number of comment lines balloons to the point where it is hard to scan the code itself. Writing a lot of material as comment blocks isn’t all that pleasant an experience either though there are plugins for editors that attempt to ease the pain.

Of course Doxygen can link in documents that are independent of the source code but at that point you start to lose the benefit of having one source for the code and its documentation.

If you browse through the headers of the standard library on your computer (or more likely get sucked into looking at one of those headers when your IDE finds an error in your code that leads you down to the dungeon) you quickly realize that these pearls of computer wisdom are nigh on unreadable!

The only sizable comments in the headers are often limited to some legal gobbledygook pertaining to licensing at the start of the file. Variables are tersely named and festooned with underscores that hurt the eye (those visually unappealing decorations are there to stop you shooting yourself in the foot by some inadvertent misuse of the preprocessor). Even if you can get past those festering carbuncles you will still likely be confounded by the sheer generality of the code which is expected to handle obscure corner cases and ancient computer architectures. That is all part of the genius & generality of the library but it certainly doesn’t make for a good read!

Of course if you are happily typing away in any reasonable IDE you will remain oblivious to the warts and instead get excellent tool tips for all standard library objects and calls. However, those do not come from comments embedded in the header files but instead are provided using some other mechanism. Your IDE will probably also provide links to really good long form documentation for the standard library complete with all the goodies we mentioned earlier.

For C++ developers, one long form documentation site really stands out, namely CPP Reference. To quote from their FAQ, the purpose of the site is as follows.

Quality is valued over quantity so the site sticks to just documenting the standard library, tempting though it might be to add things like some of the most popular Boost libraries.

To be fair, the site has a very particular look which might not be to everyone’s taste. For one thing, it somehow doesn’t look all that “modern” as there is a lot of text on every page all in a single undistinguished typeface ( DejaVuSans ) and to boot, some of that text is in quite small point sizes.

Nevertheless, the quality of the documentation is excellent and the writing style and format is consistent across the entire large project which is quite the task given that the whole thing is a community effort (it makes me suspect that a huge amount of the material is actually written or at least heavily edited by a relatively small number of contributors).

In many ways CPP Reference sets the gold standard for documentation of a C++ library. So emulating it would seem like a reasonable approach to documenting a new library. Achieving a comparable writing style is of course up to the author but unfortunately mirroring the page format turns out to be quite difficult.

CPP Reference is a wiki built using the same MediaWiki software that powers Wikipedia. While the basic markup used isn’t all that hard, everything in the site is wrapped in “quite complex templates”. The editors basically suggest that a contributor just concentrates on writing some content and then let the experts take over to put the text into the correct format.

Moreover, the MediaWiki software was designed to be used in a browser which is very clunky and basic compared to using any fully featured modern editor.

Lightweight Doxygen comments embedded in the library source code is an excellent way to provide short-form, tool-tip style documentation. On the other hand, long form documentation complete with cross links, references, examples etc., is best built and maintained outside the source code itself though most likely in files that are located in the same repository.

What markup format should those long form pages use?

Markdown is by far the best known lightweight markup language. These days it is used just about everywhere on the web. It is so simple to learn that you can get up to speed in just a couple of minutes, yet it provides enough features to enable you to write most shorter documents. It has chapters/sections, paragraphs, the usual text formatting such as italic, bold, monospaced etc., along with lists, tables, and so on. Code blocks can be made stand out from regular text which is handy for documentation purposes.

The format is ubiquitous so if you download a Markdown file to your tablet or computer and click on it the chances are excellent that it will open and get rendered in some very reasonable way (though not necessarily the exact same way that it looks to someone reading it on a different device or application).

Markdown is just simple text with some character combinations used to suggest formatting. If you can write text, you can write Markdown. Moreover, there are a huge number of tools available that enable you to view the formatted output more or less as you type it. Every editor will have a plugin that renders Markdown. There are also some excellent dedicated Markdown editors.

Of course, our GF2++ is a math library so it is handy to be able to have a formula or two in the documentation. That wasn’t part of the original Markdown but these days most flavors allow you to embed \(\TeX\) equations like:

However, even with all that, Markdown is far from ideal for writing software documentation. It shines when everything fits in a single file but it misses some key features once a project spills over to many files. Links between files can be created in Markdown but you can’t really even simply include one file or part of a file in another (for example have a code snippet in one file and then include the relevant section as text into another).

There is also no native support for some really useful things that are common in technical documentation. For example, callouts where you highlight lines of sample code with explanatory materials, and admonitions where you warn the user of some particular issue or other. Of course Markdown does allow you to embed HTML in the document so all things are possible but, once you go that route, definitely no longer easy.

As the documentation grows you will inevitably be led to exploring alternatives and the one we settled on is AsciiDoc.

AsciiDoc itself can be a pretty lightweight markup language. In fact, if you restrict yourself to the same feature set as Markdown, an AsciiDoc .adoc file will look very similar to the equivalent Markdown .md one. Here is a cheat-sheet that shows the equivalence between the two forms of markup.

However, AsciiDoc is extensible and much more powerful than Markdown.

For example, the AsciiDoc include directive allows you to embed some snippets of text in multiple locations without resorting to copy-paste.

Explanations can easily be added to particular code lines by using callouts such as:

AsciiDoc also has admonitions as a way to highlight warnings so the reader’s attention is drawn to particular gotchas that might trip them up. Those can also be used to highlight tips and informational notes etc. such as:

AsciiDoc is not nearly as well known as Markdown but nonetheless your editor probably has a plug-in to make writing it pleasant and to enable previewing the formatted output as you go.

These days, Asciidoctor is the principal tool used to process AsciiDoc files. It is a mature open source program that can convert AsciiDoc to HTML, PDF, EPUB, DocBook, manual pages etc. It is written in Ruby and works on all the usual platforms, Windows, macOS, and Linux.

There is also a JavaScript version so you can install a browser extension and then simply open an AsciiDoc file to see it nicely rendered with active links etc. That can be quite handy when you are authoring material as you can see previews in a browser even if your editor doesn’t have any specific AsciiDoc support.

Of course you can hardly expect all your library users to install a browser plugin just to read your documentation!

Instead you might use Asciidoctor to convert all the .adoc files into HTML.

Suppose the AsciiDoc content is in files and subdirectories sitting beneath a documentation/manual/pages/ directory. Then you can sit in the parent documentation/manual/ and use the the following Asciidoctor command to convert all the .adoc files in pages/ into their .html equivalents.

The .html files will in be found in an identical directory layout in the sibling documentation/manual/html/ directory.

A reader can then browse through those HTML files without any need for an AsciiDoc plug-in. However, all Asciidoctor gives you is a simple tree of files without any of the navigation niceties you would expect in a modern website so the job is not yet done!

Like its simpler Markdown cousin, AsciiDoc is a readable markup language that is easy to create in any text editor. As we explained, it is primarily aimed at creating technical content. The Asciidoctor program can transform that content into various output formats amongst which is HTML, the language of the web.

If you have run asciidoctor as shown here then you can open the files in documentation/manual/html/ directory and read the documentation in a browser from there. No plug-in will be required as you are reading HTML rendered nicely by your browser.

However, Asciidoctor is not a web-site builder. So while the styling will be good, the documentation tree is still just that — a simple tree of files without any navigation bar, menus, search box etc. so not a real website. Navigation is limited to the forward and back buttons in your browser.

Antora aims to fill the void. It is a static website generator that creates sites where the content is written in AsciiDoc and where the corresponding files are stored in a version control system like git. It is used to add elements like menus, navigation bars, search boxes, etc. — elements we expect to see when we visit a website.

Antora has been designed to facilitate very large scale technical documentation projects where the content might be spread across many different repos. In fact it can be used to consistently document a complete operating system with thousands of libraries and hundreds of applications. Not only that, its design means it can gracefully handle different versions of each of those through their lifecycle.

This generality is both a blessing and a curse!

While there is copious Antora documentation, it tends to focus on the great flexibility of the system without nearly enough reference to specific simple complete examples. The suggested use cases are quite heavy weight with lots of documentation in different repositories and possibly even a team of people with specialized technical writing roles and so on. Even the Antora Quick-start isn’t all that useful as it relies on existing content pulled from other repositories instead of building a small site from scratch in a single repository. This all makes Antora appear quite daunting.

Of course it is great that the system was designed to scale up to the very large but it is quite hard to get started with a much simpler scenario like documenting our little header-only library GF2++.

Well we do know there are couple of requirements up front:

That’s it! You have now got a local git repository that Antora will be happy to work with even though you haven’t checked in anything to it.

Let’s focus on a simple scenario where you are building an application FabulousApp whose code base sits in a single repository and which is documented with an installation manual and a user guide.

In broad strokes the FabulousApp repository might be laid out along the following lines:

We now want to use Antora to build a website that hosts that documentation in a decent looking, easily navigable website.

In common with most static website generators, Antora expects that the content for a documentation website is laid out in a particular structure of directories and files. Because Antora is designed to potentially handle very large documentation projects that structure might be a bit deeper than you might think strictly necessary.

The GF2++ project is stored in a git repository the following structure:

Before we talk in more detail about the documentation we should mention that the library really has just two substantial headers Vector.h and Matrix.h. That assert.h is a short helper file defining macros used do optional assertions such as bounds checking primarily for debug builds. The library ha some headers than this but for the purposes of this discussion we will stick to those.

There is a convenience header GF2.h that just includes all the others. So, assuming your include path allows it, you use the library by just adding one line include <GF2/GF2.h> to the top of your program.

As you can see there are three separate modules in the docs/ directory:

The /docs/reference-manual/pages/ directory closely mimics the layout of the library source code in the /include/GF2/ directory. That pages/ directory looks somethings like:

The documentation is in the form of AsciiDoc files stored under the pages/ subdirectory. This layout makes it possible to easily use the static website builder Antora.

The pages/overview.adoc file introduces the library as a whole (this corresponds to the usual index.html for a website). Then each class and macro in GF2++ has a corresponding documentation sub-directory in pages/.

For example, the fairly trivial <GF2/assert.h> header just defines the three related assert style macros and those are all documented in the one assert.adoc file.

In contrast, the GF2::Vector and GF2::Matrix classes have a large number of associated methods and quite a few documentation files. Rather than having one documentation file per method we instead group methods and functions together where it logically makes sense for them to share a single file.

For example, the Vector/access.adoc file documents both GF2::Vector::operator()(i) methods, both GF2::Vector::operator[](i) methods, as well as the GF2::Vector::test(i) method. These are grouped together because each provides access to the individual element at a given index i in a bit-vector. The signature for each of the five is explained, how bounds checking can be enabled for the index, and the access.adoc file includes a short sample code that exercises the methods and the output from that code is shown. The user should be able to copy-paste that snippet, compile, and reproduce the shown results.

The main entry point for documenting the GF2::Vector class is the file Vector/Vector.adoc and for the GF2::Matrix class it is Matrix/Matrix.adoc. Those two files lay out the overall rationale behind the classes as well as providing links to all the other documentation in a nice table format.

The whole effect is quite similar to what you get if say you look up std::vector at CPP Reference/vector.

Expanding a little on our earlier picture the GF2++ repo now looks something like:

The top level docs/ directory is organized in a natural manner though we have made an effort to keep it compatible with the way Antora would like things. In particular, we have split the documentation into modules and then under each module stored the actual AsciiDoc content under a pages/ subdirectory, images under an images/ subdirectory and so on.

It does seem very sensible to keep the Antora specific parameter files, build instructions, needed support tools etc. in their own sandbox. In fact, the Antora documentation suggests having a completely separate repo for all that stuff. For a small library like ours that seems a bit of an overkill so instead we settled for having a dedicated antora/ directory at the top of the repo which looks like:

Not everything is plain sailing with either AsciiDoc or Antora. On the plus side the combination is powerful enough to tackle large scale documentation projects. However, they aren’t the easiest set of tools to learn and can seem complicated for smaller scale problems.

Antora also extends AsciiDoc by introducing the family directory concept. This is a key Antora idea that is particularly important for large multi-repo projects. However, perhaps more thought should have been given to making it backward compatible with plain standard AsciiDoc — at least for simpler projects.

Writing basic AsciiDoc is really no different from writing Markdown but of course that isn’t really a testament to its usability.

You are only going to write AsciiDoc to access the extra features it unlocks. The essential cost benefit comparison is then whether those “extras” come at a low enough cost to justify the effort of learning a new syntax.

Certain features AsciiDoc provides out of the box are without doubt really useful for writing technical documentation and very easy to use! In particular, callouts and admonitions are enormously useful, much used, and take very little effort to learn.

Other features of AsciiDoc are occasionally useful but you will probably find yourself needing to go to the manual to refresh your memory on how to invoke them. This is because AsciiDoc syntax can be a bit wordy and frankly some aspects simply aren’t explained all that well in the core documentation so it is hard to develop an intuition how to do things.

One specific example is math mode in AsciiDoc. Technical documentation will often need to embed mathematical equations both inline and display style. The good news is that you can readily get AsciiDoc to support \(\TeX\) style equations but unfortunately in a somewhat wordy manner.

Using \(\TeX\) you write $A \cdot x$ to get the inline equation \(A \cdot x = b\).

In AsciiDoc (certainly if you want to use Antora) you instead need to write the longer stem:[A \cdot x = b]. It is even more wordy and space consuming to write the display version:

This does display correctly as

but the raw source you are editing is much more visually cluttered than the equivalent \(\TeX\) code:

One other thing that is less of a criticism but something to bear in mind is that at the end of the day, AsciiDoc is just a markup language for documents. As such it lies in complexity and capability between its simpler cousin Markdown and something like \(\LaTeX\) which is heavily used in many scientific disciplines.

You can write some very simple macros in AsciiDoc but those are mostly for doing substitutions. So you can assign an short attribute to say a long messy URL and then refer to that URL by using the attribute. This is definitely useful but a long way short of the type of things you can do (granted often painfully) in \(\TeX\).

And of course while AsciiDoc allows you to link documents together in a cohesive manner, it is not by itself by any means a website generator. You use the language to write the content for a website but bear in mind that you will need to go elsewhere to actually get a reasonable looking website.

A key improvement of AsciiDoc over Markdown is that it allows you to include one file (or even just a tagged portion of a file) in another one.

The target file in an AsciiDoc include directive is usually specified by simply giving its location relative to where you doing the actual include::. For example to include a source code file main.cpp that lives a couple of levels up in the source/ directory you might write:

AsciiDoc does support some variations on this theme but they are all equally straightforward.

The same sort of simple target resolution holds for inter-document cross references (an AsciiDoc xref) and for locating image files (the AsciiDoc image directive).

The downside is that this can be fragile especially if you want to reference files that exist in different repositories. AsciiDoc will let you do that but only by hard coding the URL of the target repo and the location of target file from there. If anything changes you end up with a lot of URLs to fix in your AsciiDoc files.

Antora is specifically targeted at large documentation projects where the components are spread across many repositories so this is a key problem it aims to fix.

It does so by applying a core notion. In Antora that all resources that go into the website (pages, images, snippets/partials, downloadable attachments etc.) are given a unique identifier—​an Antora ID.

During the build process Antora gathers all of the website resources into a virtual filesystem and assigns one of those ID’s to each item. Antora more or less insists that you use those ID’s when you want to include a resource either through the include, xref, or image directives.

In the greatest generality you do that by using all of the target resource’s Antora coordinates. These coordinates are exhaustive but also human friendly and include a version tag, a component name, a component module, an Antora family, and finally the name of the resource itself. These Antora coordinates are independent of the physical layout of the resource files and so eliminate the need for those somewhat hacky ../.. references in your documentation files.

That’s great and even better is the fact that most of the time you don’t need to specify almost any of those Antora coordinate elements. A directive like:

tells Antora to find SomeImageFile.png in the appropriate family directory which is always called images/ and always is a sibling to pages/ (in fact all the recognized family directories are siblings of pages/).

AsciiDoc is documented on the AsciiDoc documentation website, and Antora on the Antora documentation site.

AsciiDoc has been around a while so you can find other sources of tips by Googling. Though it’s not been added to in a while, I found the Awesome Asciidoctor site to be useful (the site’s content is also available as a book).

Antora is newer so there isn’t that much readable material available beyond the official documentation site.

There is an active forum you can follow on Zulip. However, the Antora Zulip site doesn’t appear to be indexed by the standard search engines which is annoying! If you type an Antora related search term into Google you will not see any results from the forum (though you may pick something up from an earlier one that existed prior to the migration to Zulip). A new user is expected to find Zulip through the Antora docs and then use the internal search tool there. This adds another point of friction when you are learning the system.