Why it’s worth it to invest in internal docs

Good internal documentation leads to more stable and innovative development, and a better experience for users and developers alike. Here’s a look at some best practices, and how engineering orgs can make documentation a part of their culture.

Part ofIssue 6 August 2018

Every company, regardless of its age or size, holds a wealth of experience and knowledge—and it’s never too early to start investing in documenting it all. Internal technical documentation, which covers an engineering organization’s internal infrastructure, architecture, development processes, and collective knowledge, contributes to smarter, more efficient, and more productive development.

Internal documentation pays dividends, no matter how small the company. When you’re a company of one, you can document the many new things that arise and free your mind to focus on the next thing. When your company is growing, good documentation means your most experienced developers aren’t forced to spend their time troubleshooting basic issues, and new developers are empowered to be self-sufficient. The earlier you start investing, the more manageable your documentation becomes.

GitHub’s 2017 Open Source Survey lists “incomplete or confusing documentation” as the number one problem encountered in open-source software development, observed by 93 percent of respondents. Yet 60 percent of contributors said they “rarely or never” contribute to documentation. GitHub concludes that “documentation is highly valued, but often overlooked,” summing up a major problem with internal documentation: Without the proper structure in place, there’s no accountability.

Other findings from GitHub’s survey reflect the broader impact of documentation:

You might not think that internal documentation could so greatly affect onboarding success, create a more inclusive environment, or help a significant number of engineers who may not be fluent in English (according to the survey, nearly a quarter of the open-source community reads and writes English less than “very well”). But GitHub’s findings show that documentation has an immense impact on the engineering community. Likewise, I’ve seen large internal developer satisfaction surveys list internal documentation as a top-three issue affecting their productivity. And I’ve seen dramatic improvements take place when companies invest in a documentation strategy.

Proprietary data is precious to technology companies, yet some of the most valuable data lives solely in the minds of engineers. These engineers change projects, move teams, get sick, go on vacation, leave the company. They take this knowledge with them—knowledge sometimes accrued over years of learning the intricacies of a company’s technology stack and development processes. Without documentation, these lessons are only passed on to those lucky coworkers who ask(ed) the right questions. Documenting this knowledge, on the other hand, makes it available to everyone, forever.

Investing in internal documentation is, in many ways, still a novel concept. With an external product, the lines are clear: You have a product, its use cases with customers are divided by personas, and you map the content accordingly. Internally, however, engineers play the dual role of author and user. And without public or customer pressure keeping things honest, it’s easy to overlook the needs of internal users.

When it comes to internal documentation, you’re not just focused on creating the content, but on reshaping the company culture around it. The good news is that both of these goals can be accomplished through a set of relatively straightforward initiatives.

Think your internal docs are bad? Find out how bad. Measuring the state of your internal documentation will help you prioritize your efforts and provide benchmarks to ensure that you’re making marked improvements. The most important documentation metric is customer satisfaction (CSat); in the case of internal technical documentation, those customers are generally your engineering teams. Ultimately, you want to know if your documentation is useful to them. You can find this out by offering users the ability to rate a page with a prompt like, “Was this page helpful? Yes/No,” and allowing them to submit open feedback with additional comments.

Open feedback comments are crucial for small sample sizes, which are often the reality when gathering feedback on internal documentation. Once you’ve reached a sample size of 50 ratings, you can begin to analyze the results. For any document with over 60 percent positive ratings, you can generally assume that all of the steps in the document work. Above 80 percent is stellar, and also suggests that the product itself has a great user experience. Below 50 percent positive ratings means that something is inaccurate or incomplete on the page that prevents users from successfully following its steps. The 50 to 60 percent range tends to indicate user confusion, which can often be reduced by improving the language or structure.

If you’ve taken action on the feedback from a poorly rated page and feel confident that the information properly reflects the functionality of the product, then the problem is likely with the user experience of the product itself. Everything aside from CSat, including page views, top search terms, and time on page, should help contextualize and prioritize the findings from CSat.

Internally, you may run into some resistance to grabbing off-the-shelf tools for CSat or analytics. Security teams are often uneasy about using tracking tools that involve placing third-party JavaScript on internal pages that contain sensitive information. And third-party feedback tools can be expensive and hard to justify when they’re only used on internal docs. One possible alternative is to use open-source tools like Matomo (formerly Piwik), which offer similar analytics capabilities to Google Analytics, but in a self-hosted solution. But if you keep your analytics simple with just a Yes/No rating option and an open feedback form, you can probably find an engineer to build a tool in-house.

There are three common scenarios that frustrate engineers looking for documentation:

To combat this, create a canonical index, which is a trusted subset of internal user documentation. This is the documentation engineers need to be successful in their day-to-day roles, such as guides, tutorials, runbooks, and FAQs. Even if your internal knowledge base consists of 20,000 documents across multiple sources like Confluence and Google Docs, the set of documents that will actually make up a canonical index is around 1 percent—a much less intimidating number.

The first sift through the entire corpus to find qualified documentation for your blessed canonical index is laborious, but once you get your 20,000 pages down to 200, you have a much more manageable set to maintain. This set of documentation is what you will continue to curate and set expectations for, while the rest of the documentation can live on in wiki form.

Look for logical demarcations in your infrastructure that you can latch onto in order to determine your documentation requirements. For example, if you have a microservice architecture, you can require that each mission-critical service has a design document, a README, and a runbook. This foundation makes it easier to write additional documentation.

When you walk into a hardware store, things are placed on the shelves and in the aisles in a logical manner. The wood is in the lumber section; the chandeliers are with the lighting fixtures. There are also sections that offer collections of different types of items used to perform a certain task, like plumbing or gardening. Gardening tools like trowels aren’t on the opposite side of the store from the planters; they’re next to the hoses and the soil. Although these are completely different objects, they form a logical whole; you’re likely to be looking for several items related to the same task. When you’re browsing for an item, you can deduce that what you’re looking for is nearby based on the other items around you. Imagine how much harder a simple home improvement project would be if every item in the store were kept in alphabetical order, or worse, placed in aisles at random.

The way your content is organized and presented to users is your hardware store, also known as information architecture. The user experience relies on the visual representation of the information architecture, which consists of the homepage, the landing pages, and how the documentation itself is organized. Information architecture shapes the browsing experience for users. It should create user trust by being consistent and predictable.

The canonical index is the parent-child hierarchy in a table in your database. The information architecture is the visual representation of this structure for users of the knowledge base, and it pulls from the canonical index as the source of truth. While every page has a single home in your canonical index, pages may be used in multiple places in your information architecture. For example, “Testing Standards” would probably live in the “Standards and Best Practices” category as well as the “Getting Started” category in your information architecture, but its true home would be “Standards and Best Practices” as its parent in the canonical index.

Even well-structured and properly formatted data in your canonical index requires proper information architecture and UX so users can navigate and traverse your documentation with ease. (If they can’t, you haven’t done your job.)

Investing in first-time user experience (FTUE) can be paramount to a project’s success. When it comes to documentation, the homepage is key to FTUE, as it sets the stage for the rest of your knowledge base. A homepage should be simple and approachable, surfacing the top three to six most important topics for your engineers, such as “Getting Started,” “Development Process,” “On Call,” and “Standards and Best Practices.” Each subsequent landing page should be like a mini-homepage for that topic.

You don’t need to spend a ton on design resources, but the same principles apply to your internal users as your external users: Focus on approachability and usability. Users tend to avoid unpleasant experiences, such as navigating something disorganized or unsightly.

The browsing experience is crucial. A common assumption is that an index of files and a good search engine will get users the answers they need. But when you’re dealing with code names, teams, orgs, people, and other company-specific language, you don’t know what you don’t know, so how can you search for it? Browsing is a core aspect of the web experience, so you must account for it.

If the content is properly organized with a solid set of landing pages, users can easily navigate around and know where to find documentation, browsing by product or use case. Every company has different information architecture needs, so it’s important to interview users, look at the analytics, put something in place, and incorporate feedback as you go.

A good authoring experience is the key to getting engineers to produce strong documentation. When selecting a tool for your engineers, make sure it’s not unnecessarily complex. Some engineers like writing documentation close to their code in the repo in Markdown, which unlocks the use of auto-generation tools. Some engineers like to write in prose and prefer the WYSIWYG (what you see is what you get) experience, which is ideal for long-form documents, such as design proposals. There’s no perfect authoring tool, so gather the use cases from your engineers, make tradeoffs, and choose a tool that works best for your needs. Does your documentation frequently undergo changes? You’ll want something simple and fast. Does your documentation involve long-form manuals that require plenty of content reuse? You may need something a little more powerful in a CMS.

Whatever you do, choose just one tool for your canonical source of content. Introducing multiple tools for every use case might seem like it will please everyone, but users end up suffering when content is created in different places, making it difficult to search and browse. Having a multitude of tools also strips away the uniform experience and consistency you need for a good UX.

Great content is useless if no one can find it. Well-organized content not only creates a better browsing experience, but also feeds structured relational content to your search engine and yields better search results. It’s much easier to retrieve a page via search when you have a good idea of what you want. Browsing is a good fallback option when you’re not sure what your query should be, or even where to begin on a certain topic. Search isn’t so bad to retrofit, so you may want to wait until your team surpasses 50 or more engineers before investing resources in a good search solution. You can probably get by with the stock search engine included in your knowledge base tool until then. It’s a challenging problem, but there are now several promising open-source or SaaS offerings, such as Elasticsearch and Algolia.

Look for ways to introduce automation into your documentation. If you provide steps in a runbook or code examples in a guide, why not introduce automated testing on a recurring basis to make sure they still work? A non-functioning code example is often more useless than no code example at all. You can use tools like Selenium for basic use cases that involve UI, or incorporate tests into your CI tool to make sure new commits don’t break corresponding code examples in the documentation.

Documentation on your development standards and best practices is the seed of your internal canonical documentation, so it requires a central investment. If your team is small, this should be relatively straightforward. If it’s large, a developer productivity team or technical writing team can assemble the common workflows and best practices that engineers should use as a starting point. This includes language style guides, code review practices, testing requirements, and product lifecycle checklists. This content is extremely important and should take the most effort up front. Changes to this content will likely be incremental, so upkeep shouldn’t be that hard. This initial investment will continue to reap returns as you grow.

As far as documentation best practices go, over time, your engineering team should know what their preferred forms of documentation are. Few things are more intimidating than staring at a blank page; creating simple templates, like documents with headers containing annotations of what to fill in, allows newer engineers to start contributing quickly.

Ownership is crucial to maintaining the documentation in place, creating accountability and clarity of expectations. Very few companies invest in dedicated internal documentation teams, and even when they do, they’re greatly outnumbered. Ownership must be decentralized down to the team and individual authors.

Start by defining what owning a document means. Does it mean reviewing and stamping the document every month to certify that it’s up to date? Does it mean keeping a page to a minimum of a 60 percent page-rating score over the previous 30 days? Does it mean actioning user feedback within 48 hours of receiving it? These are all valuable parameters to evaluate, depending on how mission-critical documentation is to your organization. It’s natural for engineering teams to own certain areas of documentation, but every page should also have an individual owner. This creates accountability and prevents feedback from falling through the cracks.

If your documentation tool doesn’t allow you to set a page owner, manually input it at the top or bottom of each page. As part of the offboarding process when an engineer leaves the team or the company, every page they own should be reassigned. All too often, important pieces of documentation fall out of date and never recover once their previous owner leaves. In order to establish proper ownership, it’s important to provide clarity on what it means to own a document and ensure that owners have the resources they need to maintain high quality, including templates, metrics, and feedback on the pages they own.

Two things are proven to drive human behavior: passion and rewards. Realistically, there may only be a few engineers in your organization who are true believers in documentation—those who will continue to evangelize its merits internally and fight to get documentation work on their sprints in the face of more development work. Documentation should be baked into the development process by ensuring that planning includes budgeting for documentation time and that engineers know how to meet documentation requirements.

Over time, the work that will actually get done will closely align with how engineers are selected and how they’re rewarded. This means the job description and ladder must call out documentation as part of the expectations of the software developer job. If documentation isn’t mentioned anywhere in the job ladder, why would an engineer prioritize it against other work when doing sprint planning or project selection? Including something like “Writes well-tested and properly documented code” in a ladder or role expectation allows for a broad interpretation and enforcement of documentation principles. Define what “properly documented” means at each phase of the development cycle, from the design phase to user documentation, by providing guides and templates.

Just throwing “documentation” into a job ladder won’t change things overnight. Managers need to take this requirement seriously come performance review time. Set the expectation in the interview phase that documentation is an integral part of the software development expectations at your company, provide the proper resources to teach engineers how to properly document, and reinforce good documentation behavior throughout the development lifecycle and at performance review time.

Instead of tirelessly admonishing developers at all levels to properly document their work, focus on mobilizing your senior engineers to enforce good documentation practices through design and code reviews. This can be especially helpful for junior engineers who might feel that their success as software developers hinges on writing as much code as possible. They’re more likely to listen to senior engineers sending back pull requests with actionable feedback within the context of a project than to an engineering manager talking at them to document better. Circulate a rule or two—for example, that code must be properly documented and kept up to date to be considered complete and functioning—to help shape the mindset of engineers.

“Code should be self-documenting in place of documentation.” “No one reads the documentation anyway.” “Everything is just going to change, so why bother documenting it?” These are common assumptions that can be used as a crutch to skip documentation work. Educate your developers and arm your managers and senior engineers with the proper justifications to explain the rationale behind your documentation strategy. Documentation, after all, is more than just instructions on a page: It creates a more inclusive organization and onboards engineers faster.

It can be hard to justify investing resources into something that customers can’t see—but the many benefits of internal documentation do greatly benefit customers, just in more indirect ways. Documentation allows engineers to be more productive more quickly, and that feeling of productivity leads to greater job satisfaction. The investment in internal documentation pays dividends many times over, as more stable and innovative development leads to a better experience for all.

Dave Nunez is the documentation manager at Stripe and a former technical writer. He lives in San Francisco with his wife and kids, and is always looking for book and restaurant recommendations.

linkedin.com/in/djnunez