|
For the past month or two, my colleague and I have spent
time running a bazillion evaluations on trying to figure
out how to roll out corporate rule sets across the
organization. We tested prompts, RAG, context, memory and
guardrails.
The ceiling is the knowledge gaps in definition,
ownership and access, not by model capability.
Challenges with All Approaches
Trying to pin point when things work or not feels
impossible given the separate variables. There is much to
contend with:
-
the consuming team experience - how
are teams going to work with these rule sets day by
day, how do we distribute them?
-
the authoring experience - how are
rulesets to be written, structured, and broken into
pieces before that can be cited by teams.
-
the spec frameworks - OpenSpec vs
SpecKit and the nuances that they present.
-
individual AI tools - the
combinations are endless Claude vs. OpenAI models,
teams local preferences and settings. The list goes
on.
How Old Skool Knowledge Management and Information
Architecture Techniques Have Helped
My active research focus in recent months has been getting
to grips with old skool knowledge management and IA
techniques. My sense is that these concepts map to AI
native systems:
-
knowledge object decomposition (breaking a document
into independently evolving answerable units) is the
upstream input to chunk boundary design (documents
with multiple knowledge objects produce noisy chunks
in a RAG layer)
-
controlled vocabularies (elects one canonical term per
concept) map to entity resolution making sure every
mention of the same thing, however it's worded,
gets tied back to one record instead of being tracked
as separate, disconnected things.
Old wine in new bottles it seems to me.
Understanding vocabulary fragmentation across your corpus
helps because when a human reads "service" in
one standard and "application" in another then
they can fill in that context. An agent sees two different
things. When you write the word 'key' do you
mean encryption key, API key, SSH key or something else?
The invisible problems in documentation are
invisible precisely because humans compensate for them and
we know agents don't compensate.
A few things have stood out for me corpus after corpus
analysis:
-
Tag rules, not documents - policy
documents are not just one thing - they are bundles of
rules, each with their own type, obligation level and
obligation owner. Metadata at the rule level rather
than the document level help an agent understand
what's guidance, whats mandatory and who to
escalate it to when its unclear.
-
Enforce terminology - vocabulary
drift has been super interesting - same concept, three
names, no canonical mapping. Agents can't tell if
"scope", "boundary" and
"containment" are being used
interchangeably. My sense is that many teams are
relying on accidental good behavior without very
through evaluation frameworks. Teams should check
cause not just appearance.
None of these are new. Taxonomy becomes ontology.
Controlled vocabularies becomes entity resolution and
metadata schemes become structured context.
Building the tooling to check my own claims.
While many teams are battling with the complexities of
learning AI workflows and tooling, I've been drawn to
the underlying knowledge. These are new disciplines to me
and not something that I've had deep exposure to
however I do wonder if every organization should now be
hiring librarians?
Sharing some skills that
you can run against a corpora of markdown: here is
https://github.com/JoyceMarieStack/markdown-ia-skills. Its a three-skill sequence (corpus discovery ->
content model discovery -> vocabulary governance).
The work is ongoing and the findings (and
design) are rough in places. That's the honest
version. I'd rather show the audit in progress than a
polished conclusion I haven't earned yet.
If you run these skills then please let me know. If
anything, I hope they are a learning resources. I'd
like to hear about it.
Unsubscribe
| Av. Joaquín Costa 16, Badajoz 06001, Spain
|
