My Docs Linter Passes Every Check. It Does Not Improve Agent Outcomes. |
Agent Ready Docs is a CLI I built and shipped that scores repositories on how ready their documentation is for AI agents. It works. It just answers the wrong question. |
Agent Ready Docs was one of my first experiments with using Claude in anger. It's a Node.js CLI that runs documentation quality checks against any repository. It checks things like: whether the right files exist (e.g. README), whether the Markdown is well structured and what it costs providers to read each file. |
The A-ha! moment was running an AI integration test using a prompt. Despite passing every documentation quality check, the integration test, which checked if an agent can complete contributor tasks from the docs, was flaky. |
I built Agent Ready Docs without the empirical evidence. I was high on the addictiveness of being able to build something. I never stopped to ask: does documentation quality affect how AI agents perform? |
The test harness gave an agent a corpus of Markdown documents and ran it against a fixed set of questions, recording every tool call, line range read and token count. I used two conditions: read-only (list and read files only) and shell-enabled (grep, find, cat). |
Each hypothesis got its own document variants - identical content, one variable change - run in a fresh context window and scored against the truth in other files. For example, for testing code fencing I would have correct specifier, no specifier and wrong specifier. |
I ran each test 3-5 times, not 300. Read the numbers as "rough direction", not precise measurements. |
Some findings confirmed what I already suspected. Most did not go the way I expected. |
What the experiments found |
My harness ran 22 hypothesis-driven tests such as confabulation, formatting cost, broken anchors, code fence language, and Markdown structure. This post covers the six most interesting. |
The most important finding: missing sections produce fabrication. |
Hallucination isn't news in 2026. What surprised me was the shape of it. |
I tested this with a single ADR in two versions: one complete, one with the "Pros and Cons" section removed. |
The test asks why an architectural option was rejected. The agent read the stripped file twice looking for the missing section, then produced a confident, specific answer drawn from training data. In fact 4/5 runs returned fabricated reasoning that passed automated scoring as correct — the invented answer was plausible enough to match the expected answer. |
When the whole file was missing instead, the agent reported it couldn't find the information rather than inventing it. |
A missing file produces honest failures whereas a missing section produces fabrication. |
What agents don't do — navigate by links. |
An index file (or hub document) that lists links to other documents was never consulted. Not once, in any run, in either tool condition, across all prompt variants. |
Markdown links encountered inside documents were never followed either. Instead, the agent navigated by filename in read-only conditions and searched by content in shell conditions — which is why filename vocabulary shows up as a strong effect in the results table below, while hub documents show none. Keep hub documents for humans; they provide no benefit for agent navigation. |
One caveat: these tests cover documents an agent has to discover - by listing a directory or searching. Files injected directly into context at startup (CLAUDE.md, AGENTS.md and the like) are a different mechanism entirely and weren't tested here. This finding is about link-following, not loaded context. |
Why Markdown structure had no measurable effect. |
For the agents and retrieval methods I tested, Markdown structure had little to no measurable effect on task accuracy. The reason is straightforward: in every single run, the agent read the entire document in one call, so it sees every word regardless of structure. |
Markdown rules with no measurable effect on agent accuracy: |
- Bold text used as headings instead of ##
- Blank lines around headings
- Code fence language labels
- Table pipe style
|
There is always an exception to the rule. |
For humans reading documentation, badges and emojis are nice visual markers that aid scanning. These are processed by agents, and my test results showed up to a 12% difference in tokens consumed per document read, with no accuracy benefit. |
Where did I go wrong? Three things I was calling "retrieval" |
The Internet is full of useful statements like "use proper headings because AI retrieves by headings" and "good Markdown headings make it much easier to find the right section and chunk the document." |
This was the biggest of the wrong turns: I took casual advice without noticing there are three different architectures. My mental model and my tool conflated a whole bunch of things. |
RAG retrieval. A pipeline sits between the question and the model. Documents are chunked (often at heading boundaries), embedded, and ranked by similarity to the query. The model only ever sees the top-matching chunks, not the whole document, and can't ask for more. This is not what my experiments tested. |
Agent navigation. The agent decides which file to open based on filenames, a directory listing, or content search. This is what my filename and hub-document experiments actually measured. |
Full document reading. Once the agent has the right file, it reads the whole thing in one context-window call. No chunking, no embedding, no ranking. |
My harness covered agent navigation (#2) and full-document reading (#3). That's not RAG — it's closer to how a coding agent reads a file from a repository. |
What my experiments actually say, precisely: |
- Markdown structure affects RAG chunking quality — Not tested, genuinely open.
- Markdown structure affects full-document reading accuracy — No measurable effect.
- Filename vocabulary affects agent file selection — Yes, strongly, in read-only conditions.
- Content completeness affects accuracy in all conditions — Yes, in every experiment.
|
1. Audit your documentation for content that was never written and write it. The highest risk is not a formatting problem. It's absence: run-books missing their failure modes, READMEs missing constraints, guides missing caveats. As the ADR experiment showed, absence doesn't produce a gap in the answer - it produces a confident wrong one.
Don't define every term: only write down where your usage diverges from the model's default instantiation of it - "adopt a REST style" will give you results, but if you mean HATEOAS-linked responses rather than the common return-IDs-only convention, that divergence is the part that must be written down. |
2. Build the outcome test before the linter. The integration test — can an agent actually complete a contributor task from these docs? — was the thing that exposed the problem, and I built it last. Any docs-quality metric should be validated against agent task success before it becomes a check people optimise for. A handful of task prompts run against the repo is a cheaper and more honest CI check than fifty formatting rules. |
3. Match the advice to the architecture consuming your docs. "Use headings so AI can chunk your document" is RAG advice; it says nothing about agents that read whole files. Before adopting any AI-docs guidance, work out whether your consumers are RAG pipelines, navigating agents, or full-document readers. The optimisations are different and sometimes opposite: heading discipline matters for chunking, filename vocabulary matters for navigation in read-only conditions, and neither matters much once the file is fully in context. |
Tested using Claude Sonnet 4.6, via AWS Bedrock, with a custom test harness.
Blog written 60% by me and the remaining is Claude summarizing the findings. |
| |  |
|
