Documentation is one of the important, yet often overlooked, parts of a software engineering project. Good documentation enables users to quickly get started in using the project and answer questions as they arise. Bad documentation and no documentation look very similar; They waste the users time and force them to either read the code to guess at how something works or ask the authors of the project. The authors of the project pay costs for poor documentation as well. Projects that are harder to learn how to use have a harder time gaining traction. Users that are interested, but unable to learn what they need from reading the code, will ask the authors their questions. A project with many users will end up with the same questions being asked every time - a linear penalty for authors who want to focus on developing the project further! Given all this, why is it such a trope of the industry that software engineers write poor documentation?
Stacked bar chart showing how a project that becomes more popular can result in less time being able to work on it. Escaping this trap is hard, but documentation can help.
It takes great effort to write good documentation. Unlike other skills learnt in college: algorithms, data structures, hands on experience with the popular programming language of the day, technical writing is not always taught and rarely explored deeply. Understanding the importance of good technical writing is something that engineers don’t normally realize until they hit a Senior level. A lack of documentation is also something which fails silently. A syntax error in the code causes the compiler to complain. A program which doesn’t work results in a code change, and (ideally) a regression test. A program which doesn’t work repeatedly can be analyzed with modern tools for test coverage, static, and dynamic analysis. But what of documentation? Low quality documentation results in people reaching out to ask the author questions. Good documentation results in people not needing to ask the author as much. This is worse than a silent failure, it’s practically a perverse effect of doing a good job. Undifferentiated high frequency user interactions are not a good thing. If your users are asking questions instead of requesting features, finding bugs, and getting involved, the project is being held back by the lack of good documentation. To fix this, authors must learn to not only write code, but also documentation.
Writing
Before diving head first into technical writing an author must learn what good technical writing looks like. This is a skill which can be taught, learnt, and practiced. A good technical writer can explain a topic concisely. Good technical writing has intuitions about when to use pictures and when to use words. Words should be used with precision and ambiguity avoided. My favorite resource for learning technical writing is Google’s Technical Writer Course: https://developers.google.com/tech-writing. This course offers self paced studies that can help someone understand the basics and how to develop themselves further. From there an author needs to put in the hours. Nothing beats practicing documenting one section of a project, revising it, then documenting a new section.
When I write docs, I find it greatly helps my final output if I keep the following in mind:
Define the Audience
Every doc should start by defining the intended audience, as early as is reasonable. Defining the audience means stating who is intended to read it and what they are expected to know before reading it. Is the intended audience graphic designers who understand skeuomorphic design principles? Or is it software engineers familiar with distributed systems, git, and sqlite?
The purpose of this is to help both yourself and the reader know what to focus on. If you want to discuss, for example, a new type of database index, it will shorten the document if a reader is expected to know both what a database is and how an index is used in a database. It also removes uncertainties from the author about how much detail to go into outside of the intended work to capture. This in turn keeps the doc focused instead of meandering into unnecessary diversions.
Divide Content Into Sections
Technical docs should make finding necessary information easy and straightforward. Full text search is a lovely feature, but a good index can both work faster and let users understand how something works overall. Consider a user who wants to learn about how a build system works end to end. This is all summarized in a single doc. Is it easier if they:
- (a) Have each component of the build system listed in it’s own labeled section, with subsections broken out as needed for large components.
- (b) Have one long doc, with no clearly defined chapters or breaks, that lays out everything.
(b) is an extreme example, the technical equivalent of a massive run on sentence, but it is also clearly a bad way to lay out information. (a) is easier to understand, it keeps the information in each section tightly focused, and even reading the index can be instructive for a user. There is also an advantage grouping information into small, digestible, pieces. Understanding technical docs is not like reading a novel for pleasure, boring parts cannot be as easily glossed over. Keeping sections short helps users avoid fatigue, especially in longer documents.
Summarize the Section, Up Front
Don’t bury the lede! Put summaries at the start, where they can be quickly seen and read.
The US military calls it Bottom Line Up Front, or BLUF. People with MBAs call it the Executive Summary. No matter it’s name it is a brief summary of what is to follow below. This is a great tool that allows a reader to decide, at the start, if they want to read more below. Every section should start with a summary of what is to follow in it. Every document should start with a brief summary of the sections that will follow and how they will all tie together.
I didn’t mention tl;dr here, because it’s the opposite of what is desired. tl;dr is what journalists would call “burying the lede” when the important parts appear at the end of a document. Summaries should always be at the start.
Use Pictures
Pictures have the ability to convey a large amount of information, in a concise manner. When possible, pictures should be used to explain concepts. This doesn’t mean using abstract images to break up walls of text, as Harvard Business Review is frequently guilty of, but to show things that require large amounts of words to explain. One easy example is discussing what a skin condition looks like. You can say rash, such and such shade, size, shape, but it helps to put a picture of it at the top, and then describe in detail with the text below.
A spool of analog film, unwound. Beautiful, evocative in a non-offensive manner. Barely tied to the subject manner. This would be perfect for HBR. Photo by Denise Jans on Unsplash
Pictures do not need to be limited to photographs either. Flow charts, sequence diagrams, and Gantt charts are all valid examples of information dense ways to convey information. Information density is the key place to look for when deciding where pictures can help. If a thousand words can be replaced with one picture, add a picture. If a half page of lists can be replaced, that’s enough to add a picture. There is no hard and fast rule beyond conveying information in the easiest possible manner to understand.
Every picture, even one which seems obvious, should also be captioned. There are two reasons for this:
- Readability: Readers tend to flip through documents looking for information. Their eyes get drawn to pictures and they stop and look at it, ignoring the text around it. Going back to the example of a skin condition, readers may not want to read the paragraphs before and after to learn what condition it is, it is much easier to caption the specific condition being shown in the photo.
- Accessibility: Being able to see, without any sort of issues such as far sightedness, color blindness, or other ailments is a wonderful thing. Not everyone is able to do so and adding a caption can help make it more clear to others what is being shown. This is most true for people using screen readers, but it applies to many lesser situations as well.
Use Lists
Lists are another powerful technique to break down text into easily digestible chunks. Keep in mind, technical docs are not novels, they should strive to be concise, focused, and present information in small standalone chunks for ease of reading and looking up key material.
Places where lists are useful include:
- Enumerating a defined set of items
- Enumerating a defined set of use cases
- Listing out steps to follow in a process.
- Anywhere else that a number of elements are being written out. Consider the difference in parsing between: “There was a girl, she wore a red beret, carried a translucent umbrella, and had a green and black striped dress. On her feet were black patent leather shoes.” This is descriptive, but not a good way to lay out information for easy parsing in a technical doc. Instead consider: There is a girl. She wore:
- Red beret
- Translucent umbrella
- Green and black striped dress
- Black patent leather shoes While not as poetic this does let her clothing be easily discovered, looked up, even potentially cross referenced against a picture.
Precision and Professionalism
The final piece of advice on writing technical docs is to pay attention to the words used. As a technical writer it is expected that you write in a direct, precise manner. Writing should also be as international and professional as possible. The reader should not be assumed to be your friend nor from the same region as you.
Precision is the opposite of ambiguity and ambiguous technical docs just raise questions as you read them. As an example I will walk through a technical service that encrypts a message.
- “The service uses cryptography to protect the message in transit” - This is very ambiguous, there are many kinds of cryptography. Does the service sign a message, does it encrypt the message, does it hash the message?
- “The service uses encryption to protect the message in transit” - This is more precise, but there are many encryption algorithms out there. Which one is used?
- “The service uses AES to protect the message in transit” - This is better, but there are also many possible settings for AES. What is the key size and the mode used for AES?
- “The service uses AES-256-GCM to protect the message in transit” - This is precise. It describes concisely the type of cryptography used with all the required settings. A reader can understand immediately what is being done by reading this sentence.
The other enemy of technical writers is writing informally. A good technical doc should not have the familiarity of a carnival barker, which is itself a very American saying. The writer should write as though the reader is the defined audience (say, “an engineer familiar with distributed systems”) and not assume any understanding of:
- Sports - No sayings such as “Touchdown!”, “Doing this is a red card move”, “Double header”, etc.
- Regional Politics - Avoid anything involving descriptions of your country’s politics because they may not make sense to international readers or offend local ones: “Two semaphores may interact in a way that causes a deadlock. This is similar to when Republicans control the House and Democrats win the Senate.”
- Sayings in your country or native language: “This image processing library can detect itsy-bitsy tell-tale signs of funny money without a cashier needing to dilly dally” is not going to make sense to non-native speakers of English. Instead frame it as a more precise “This image processing library can quickly detect signs of counterfeit currency.”
- References to the reader: The reader is not a “gentle reader”, is not “going to see it is easy”, and should not be told “You will feel happy with yourself to know these things”. These are technical docs and one should just stick to the facts.
Do note that shibboleths of your profession are probably okay, since readers are defined to be members of the profession. This is an example of an okay sentence: “The phrase ‘My code base is a mess’ is so common among SWEs it could be a form of greeting. This tool helps to refactor code and prevent the user from needing to say that quite so often.”
Tooling
Once one understands the “what” of good technical writing, the next part is “how” to achieve that. Documentation is something which should be a part of every feature. If you wouldn’t ship without testing, you shouldn’t ship without writing documentation. But good docs also need to be kept up to date. Stale docs are poor docs and poor docs might as well be no docs. I’ve found the following to be very beneficial in keeping docs useful and up to date:
Keep Docs Close to Code
Many observations here echo what Daniel says in his haxx.se article. He has a very good point here that keeping docs close to the code is very helpful in keeping them up to date. Docs next to the code make it easy to keep them up to date. Authors can update docs as they make changes. Tooling can enforce this as well by requiring that either the doc files are modified or the author at least confirms that they reviewed them and no changes are needed.
Markdown is an excellent language to use for doc files stored next to code. It’s easy to write, it allows for some formatting, and many tools support working with it. It’s also very easy to lint and understand what you will get out of it.
WYSIWYG is nice if you can’t
What You See Is What You Get (WYSIWYG) docs, such as Google Docs, are a reasonable alternative to keeping docs in the code. With a richer editor you are making a trade off on one form of ease in updating (seeing it in the VCS as you work) for another (easier editing).
Graph showing different options for managing docs. Each one has it’s tradeoffs in ease of use and ability to express complex details with ease of effort.
Rich editing experiences for WYSIWYG editors are a very nice bell and whistle because it becomes easy to add formatted text, pictures, hyperlinks, and whatever else you desire, with even more ease and expressivity than markdown allows for.
Instant publication, as is the case with Google Docs and any other cloud based tool, is another plus. No need to have a pull request, a review, potentially fire a CI job. Just type it out, maybe hit publish, and you are done with an update.
Ease of communication (I swear I’m not trying to sell Google Docs specifically) is another benefit. Docs that people can comment on, help update, and revise can keep docs alive as well.
Always Make Design Docs
Design docs are an important part of any feature and project that is going to be more than an hour of work or involve more than one person. I choose “one hour” as an arbitrary value, with the intent of saying “anything with more than mild complexity should have a clear design.”
When working on anything even moderately complex there are multiple moving pieces that may take a long time to properly figure out. If each piece talks to every other piece through an API, that’s great! You just need to make sure that both the user and provider of the API do not feel that there is any ambiguity in that API. If that isn’t the case, if there are functions calling into each other, running programs, or other increasingly vague connections between pieces, those are all places where issues can arise. If multiple people, perhaps in different timezones, are working on this as well, there are more opportunities for failures in pieces to behave as expected. On top of it, as projects drag on, people may join and leave, and memories can become hazy. “Why did we decide on our SQL table having this schema?” isn’t a question that should ever be asked. It should be evident why that decision was made, even if that table is changed later on.
The best way to solve this is to have a design doc that specifies the problem, the solution, and how that solution is to be implemented. Everyone implementing the solution can then work off of it and stay in sync across time and space. Once complete, it forms the basis for any future documentation as well. In many cases, this may be the only documentation for something!
A picture is worth 1k words
Pictures are a fantastic way to convey information that would otherwise be more cumbersome to write out. Protocol exchanges, flowcharts for algorithmic behavior, statistics, can all be easily expressed through pictures. The problem is how to relate those to good technical documentation.
A mermaid diagram showing a set of actions for Christmas shopping. As of 2023-06-19 this is the default example on https://mermaid.live.
Oftentimes good technical documentation lacks pictures because they are a pain to edit. Pictures are kept separate from the documentation, and require cumbersome and tedious work such as hand mapping edges to nodes, arranging everything so it doesn’t collide, and then remembering to save it somewhere so it can be edited later. It’s all a huge pain and leads to pictures either not being done or being left out of date because it is hard to update them.
At the same time, it’s hard to just read text. Even RFC’s, a text only format, benefits from diagrams. To see an example, look no further than the TCP state diagram from RFC9293.
+---------+ ---------\ active OPEN
| CLOSED | \ -----------
+---------+<---------\ \ create TCB
| ^ \ \ snd SYN
passive OPEN | | CLOSE \ \
------------ | | ---------- \ \
create TCB | | delete TCB \ \
V | \ \
rcv RST (note 1) +---------+ CLOSE | \
-------------------->| LISTEN | ---------- | |
/ +---------+ delete TCB | |
/ rcv SYN | | SEND | |
/ ----------- | | ------- | V
+--------+ snd SYN,ACK / \ snd SYN +--------+
| |<----------------- ------------------>| |
| SYN | rcv SYN | SYN |
| RCVD |<-----------------------------------------------| SENT |
| | snd SYN,ACK | |
| |------------------ -------------------| |
+--------+ rcv ACK of SYN \ / rcv SYN,ACK +--------+
| -------------- | | -----------
| x | | snd ACK
| V V
| CLOSE +---------+
| ------- | ESTAB |
| snd FIN +---------+
| CLOSE | | rcv FIN
V ------- | | -------
+---------+ snd FIN / \ snd ACK +---------+
| FIN |<---------------- ------------------>| CLOSE |
| WAIT-1 |------------------ | WAIT |
+---------+ rcv FIN \ +---------+
| rcv ACK of FIN ------- | CLOSE |
| -------------- snd ACK | ------- |
V x V snd FIN V
+---------+ +---------+ +---------+
|FINWAIT-2| | CLOSING | | LAST-ACK|
+---------+ +---------+ +---------+
| rcv ACK of FIN | rcv ACK of FIN |
| rcv FIN -------------- | Timeout=2MSL -------------- |
| ------- x V ------------ x V
\ snd ACK +---------+delete TCB +---------+
-------------------->|TIME-WAIT|------------------->| CLOSED |
+---------+ +---------+
Figure 5 of RFC9293 showing states in a TCP connection.
Section 3.3.2 of the diagram lists out all the states, it lists how one transitions from one to the next, but the eye is still drawn to this picture more so than the text, because this picture is much easier to read and follow along with. Now imagine what to do if you realize that FIN-Wait-1 needs a transition to LAST-ACK. Try editing the ASCII to add that. Ugh!
Mermaid is a standard, based on markdown, which makes this much easier. Under Mermaid, drawings become written inputs, which are laid out automatically. Mermaid is supported in a number of places including GitHub, VSCode, and many markdown editors. By having the picture be the output of text it becomes easier to both comment on specific problematic elements, as well as make changes with ease.
Here is Figure 5, redone in Mermaid:
Figure 5 of RFC9293, redone as a mermaid graph.
The code which I used to generate this is below. I used the “elk” renderer on https://mermaid.live to allow for a better layout in the render.
%%{init: {"flowchart": {"defaultRenderer": "elk"}} }%%
flowchart TD
CLOSED
LISTEN
SYNRCVD["SYN RCVD"]
SYNSENT["SYN SENT"]
ESTAB
FINWAIT1["FINWAIT-1"]
FINWAIT2["FINWAIT-2"]
CLOSING
CLOSEWAIT["CLOSE WAIT"]
LASTACK["LAST-ACK"]
TIMEWAIT["TIME-WAIT"]
CLOSED -->|passive OPEN\ncreate TCB|LISTEN
LISTEN -->|CLOSE\ndelete TCB|CLOSED
LISTEN -->|rcv SYN\nsnd SYN,ACK|SYNRCVD
SYNRCVD -->|"rcv RST (note 1)"|LISTEN
LISTEN -->|SEND\nsnd SYN|SYNSENT
SYNSENT -->|CLOSE\ndelete TCB|CLOSED
CLOSED -->|active OPEN\ncreate TCB & send SYN|SYNSENT
SYNSENT -->|rcv SYN\nsnd SYN,ACK|SYNRCVD
SYNRCVD -->|rcv ACK of SYN\nx|ESTAB
SYNSENT -->|rcv SYN,ACK\nsnd ACK|ESTAB
ESTAB -->|CLOSE\nsnd FIN|FINWAIT1
SYNRCVD -->|CLOSE\nsnd FIN|FINWAIT1
ESTAB -->|rcv FIN\nsnd ACK|CLOSEWAIT
CLOSEWAIT -->|CLOSE\nsnd FIN|LASTACK
LASTACK -->|rcv ACK of FIN\nx|CLOSED
FINWAIT1 -->|rcv ACK of FIN\nx|FINWAIT2
FINWAIT1 -->|rcv FIN\nsnd ACK|CLOSING
CLOSING -->|rcv ACK of FIN\nx|TIMEWAIT
FINWAIT2 -->|rcv FIN\nsnd ACK|TIMEWAIT
TIMEWAIT -->|Timeout=2MSL\ndelete TCB| CLOSED
With the above code, try adding a transition from FINWAIT-1 to LAST-ACK. How long will it take you compared to editing an ASCII diagram?
Document your APIs, document your functions
APIs, the place where others use your work, must be clear to use. Their inputs should be obvious, as should any expected outputs and error conditions. This is an obvious place where docs excel, and where you often see docs. It is still worth mentioning docs because they are essential here.
What is even better is to bring documentation to internal functions as well. By documenting all but the most obvious internal functions, the same purpose is served as in a good design doc. Documented functions let others work on them, or call them, with ease since it is clear what they do. Even if you are the sole author, coming back to well documented functions after months away will make the process of refreshing one’s memory easier. For documented functions, these docs should always live in the code, ideally right at the start of the function.
Examples are 💯 gr8 ✨
There is a joke that a lot of programming is done by copying from Stack Overflow. In 2021, they did the math ™ and determined that it’s really really accurate. Some languages, such as Golang and Rust, embrace this and provide support for having examples which can even be run (and altered!) in a web browser. This is really powerful.
The best way to get someone to use something is to make that on-ramp of learning as short as possible. If you can provide a working example someone can run in a browser, that is great 🥇 Something that can be copy-and-pasted for running elsewhere, such as a terminal, also good 🥈 If these examples stop being accurate however… that is bad 😡
The best way to ensure an example stays working is via…🥁 automation! Examples which are co-located with the code can be validated automatically and flagged if they stop working. Golang and some other languages provide this as built in support. For an example see: https://pkg.go.dev/testing#hdr-Examples.
Docs must be easy to view
The harder a doc is to reference, the worse it is. This applies to even the best written docs, that have the most helpful of examples. This is a place where Markdown and cloud hosted docs show their value. They are easy to read, search, and understand. In contrast, languages such as LaTeX and Sphinx are less valuable. Effort needs to be made to even make use of the doc, typically via some compilation steps. Needing to compile a doc before it is usable decreases the ability for a user to get value from the doc.
Whenever possible, favor styles of documentation that are easy to view. The easier a doc is to view, the easier it is to use.