More and more companies are handing their documentation to AI agents. The idea is tempting: a developer changes the code, and an agent updates the user manual automatically. But how good is the result? Until recently, nobody had measured it systematically.
That has now changed. The documentation platform Promptless has published DoGBench (Documentation Generation Benchmark), the first benchmark that tests whether AI agents can produce user-facing documentation that an experienced technical writer would accept in review. The short answer: not yet. (Source: Promptless)
How the benchmark works
DoGBench consists of 292 tasks drawn from real open source projects such as Helm, PostHog and Mautic. In 205 of them the documentation needs to change: new content, updates or the removal of outdated passages. In the other 87, the right answer is to leave the documentation alone.
Each agent receives the repository as it was before the change, plus a trigger such as a code change or a reported documentation gap. It has to decide for itself whether users are affected and whether the documentation needs updating at all. The results are scored against detailed criteria developed together with the projects’ maintainers: accuracy, completeness, reader guidance, placement, style and the project’s conventions. Anyone who makes a critical error cannot score more than 60 out of 100 for that task, however polished the rest is.
The results: polished, but not reliable
Seven combinations of AI model and agent software were tested. The best one reached a score of 47.3 out of 100. Even more revealing are the details:
- Knowing when to update is hard. Some agents documented internal changes that are irrelevant to users. Others missed necessary updates because there was no existing page on the topic. For tasks that needed no change, the agents got it right between 28.6% and 85.7% of the time.
- Readers get stuck. In an audit of 1,267 submissions, 45.5% had a gap that would stop readers from completing their task, such as a missing prerequisite, step or verification.
- Errors are hard to spot. 36.6% contained technical inaccuracies. These are often subtle: a misstated default value, or a conditional behaviour presented as universal. Reviewers who only skim will easily miss them.
- Few results are usable without errors. At best, 39% of the delivered texts were free of critical errors.
According to the researchers, the problems start before writing begins, in the investigation. The agents often stopped at the first plausible page, filled gaps with assumptions, or described an interface without checking how readers actually use it.
What this means for your technical documentation
The study confirms what we see every day at Lexsys: good documentation is not about well-written sentences. It comes from understanding who the readers are, what they want to achieve, and which information they need for it. That knowledge is often not in the code itself.
AI agents can be a useful aid, for example for first drafts or for finding the pages a change affects. But their output needs expert review. The authors of the study recommend checking every AI draft from the reader’s point of view:
- Can readers find the information, complete the steps and verify the result?
- Does every technical statement hold, and under which conditions?
- Do related pages contradict the new content, or are they now out of date?
This applies all the more to multilingual documentation. Every error in the source text multiplies with each translation.
Our conclusion
DoGBench gives documentation managers solid evidence for the conversation with management: AI can speed up documentation, but it cannot yet replace professional technical writers. The best results come from combining the two, with AI as a tool and experts who make sure the documentation really helps readers.
Would you like to use AI in your documentation process without losing quality? Find out how we can help with your technical documentation.
