Auditing Docs for AI-Readiness: A Knowledge Engineering Self-Study |
Diagnosing documentation before agents try to read it. |
Many corporate-authority documents were written for humans. AI coding assistants, automated reviewers, Specification Driven Development pipelines are trying to read them too and can't reliably due to how documents are stored and structured. |
Making the invisible visible is something that I love to do. AI is enabling me to learn new techniques such as knowledge engineering and easily build pipelines to extract interesting facts from various corpora. |
It began with our ADRs (Architecture Decision Records). |
Starting Point: 2,677 ADRs Across 174 Repos |
First, I tackled our 2677 ADRs across 174 repositories followed by our corporate-authority documents. |
My interest was sparked by building an ADR knowledge pipeline. It's built across 5 stages: inventory (scrape ADRs from Git), classify (filter real ADRs from templates/diagrams), vocabulary (Claude-extracted canonical terms with synonym mappings), themes (BERTopic clustering into semantic themes), and analysis (report generation). |
One finding was that 62% of ADRs rely on generic terms like 'data', 'service', 'api' and this is an issue for search and retrieval problems. When every ADR is tagged as 'data' or 'api' then those tags carry no signal. You can't filter by them meaningfully. As the corpus grows the signal to noise ratio gets worse, not better. So I learned a controlled vocabulary with specific terms is what makes a corpus searchable and governable at scale. |
This ADR exercise got me animated about learning knowledge engineering techniques. |
Next: Corporate Policy Documents |
Many organizations have existing policy documents covering security, incident management, release management and how and what to log. |
Knowledge is often spread across different repositories such as Confluence, SharePoint or other corporate sites. To exploit the benefits of SDD frameworks such as SpecKit and OpenSpec, decomposing existing corporate policies into agent consumable constraints is going to be something many organizations are tackling. |
To begin my analysis I needed an agent consumable knowledge base of our standards. I built an ingestion pipeline to capture the raw data, preserving the structure and stripping nothing. Once I had pages in markdown I was able to get cracking. |
Four Questions I Asked of The Corpus |
Here is the approach that I took to understanding our corpora. |
1. What do you have in the corpora? |
Claude helped me build a corpus inventory, document classification, and perform content model analysis. This is what let me map what exists, how it is organized, whether it is flat or hierarchical, and how mature or fresh the corpus is. |
2. Where does the knowledge come from? |
An authority chain shows who or what has the final say, in order. Understanding authority chains in AI is critical for agent reasoning. If one of your specs has a rule that appears in multiple documents from different authorities, and two rules contradict each other then who wins? |
Alongside an authority-chain map I created a traceability matrix. It is a simple table that shows one row per relationship, mapping source document → target document → relationship type → evidence. Some examples: |
- Password Policy → Encryption Standard → Implements → Standard mandates AES-256 per policy's data-at-rest requirement
|
- Data Retention Policy → Backup Standard → Depends on → Standard's retention periods are defined by the policy's minimum-retention rule
|
Before any corporate-authority document is decomposed for AI consumption, it needs to be de-duplicated and contradictions need to be resolved or explicitly arbitrated. This is an authoring problem. |
3. What can cause the knowledge to change? |
What change triggers make your knowledge out of date and forces it to be revised? This is not about who edits the file, it is the specific kind of event that would render the content incorrect or incomplete. With regards to corporate-authority documents this could mean a new policy or security rule. |
Change-trigger analysis matters because if two pieces of information get updated for different reasons or at different times, that's a sign they shouldn't be crammed into the same document. Understanding this nugget has been a big A-Ha! moment for me. In one case I've seen one of our documents has seven multiple independent triggers. |
4. How easily can consumers acquire the knowledge they need? |
Acquiring knowledge in any large organization can be challenging. Often teams have to navigate Confluence, Sharepoint, maybe an internal developer portal and Git. Many of these sites require internal authentication. |
The corpora revealed a multi-domain information architecture that spans nine distinct documentation repositories, each owned by a different organizational unit and each governed by a different access model. A reader trying to acquire knowledge must cross several of these repositories along with several permission boundaries before they acquire the knowledge. |
An AI agent reading a standard has no path to that content in a default setup. Agents will reach a dead-end if the operationalizing detail of a rule is in a gated source. |
The techniques for this is permission-boundary analysis and knowledge journey mapping. This connected repository ownership, access controls, and dead ends in the path an engineer or agent must take to actually acquire the knowledge. |
What I'd Tell Someone Starting This Tomorrow |
AI has been totes amazeballs at making me feel super human at the same time as making me feel slightly denser. Writing this blog is one way to ensure the lessons stick. |
The last few months have quietly turned into a pile of related side projects: an ADR classifier (taxonomy/classification), an agent-ready docs pipeline (retrieval-ready structuring), markdown pre-commit hooks (quality gates), and lexical drift/synonym mapping analysis across documents. |
Here is what I've learned:- |
- Treat knowledge objects, not documents, as the primary asset. The tribal knowledge of a particular concept such as compliance logging (or whatever) evolves independently from how it is published (a standard or policy document). An example of a knowledge object is "passwords must be rotated every 90 days". If you audit at the document level, you'd count three documents containing rotation rules and miss that they disagree. If you audit at the knowledge-object level, the 90-day rule is one thing with one authoritative source. If its mentioned in other documents then its either a correct restatements or drift to be fixed. (Can't wait to write about my lexicon drift experiments).
- Every knowledge object should have an identified authority that is one clearly named "source of truth" plus a clear precedence path when when multiple documents mention it. Without a named authority, teams and agents cannot tell which statement wins when two documents differ.
- Terminology affects output quality in ways that are hard to trace. The difference was not in the underlying requirement but in the vocabulary used to describe it. Different framings of the same requirement will produce different implementations, and not always in traceable ways. Consistency of terminology in how we author and inject corporate policies into SDD pipelines will matter more than we might expect.
|
None of this makes me a knowledge engineer. But it's made me a better reader of our own documentation and that turned out to be a super useful skill. This is diagnostic work, not a finished system. I've found the drift and the gaps, not fixed them yet.
|
| |  |
|
