I Broke Every Table Formatting Rule and Claude Didn't Care - Except for One |
This week's AI rabbit hole was tabular data in markdown files. |
The following (imperfect) experiment was supposed to give me evidence for an AI markdown guide so I could draft a skill. I started with tables as I learned from @dachary that tables tend to be more efficient than prose. |
My thinking was given a table the author is writing, a skill would provide format guidance and flags structural issues that would degrade agent comprehension. It's not a linter as such but an opinionated guide. |
If I deliberately break the structure of a documentation table, does an AI agent extract the wrong values? |
I set up a baseline table with 12 rows in a Markdown README file. The kind of table that exists in thousands of repos. I created six versions of the same table: |
- the healthy well structured baseline
- headers removed
- cells emptied
- headers abbreviated to two letter codes
- two tables merged into one
- prosed stuffed into cells
|
All experiments were run using Claude UI / Claude Sonnet 4.6. I ran each experiment in a new window three times so I had 18 result files. |
 example table data |
The deliberate ambiguity. |
The table was designed so values are meaningless without context. 3000 appears twice - once as a port and once as a timeout. US and EU appear in different rows — one is a locale, one is a deployment region. Without headers, the agent has to figure out which is which. |
I expected more failures. I expected that the removal of headers would most definitely be an issue. It wasn't. I expected the ambitious 3000 number to be an issue and that the merging of different tables would confuse it no end. |
I had Claude generate the prompt. Twelve questions, one extraction prompt, a couple of ground rules — use only what's in the document, no guessing, no filling in gaps from training data. |
Some questions were simple. What's the default port. Some needed filtering. Which variables are required. Some were designed to trip it up - two variables default to 3000, name both and explain the difference. |
Same prompt. Same questions. Six different versions of the table. Score the answers against a known answer key. |
"Did the same model design the test and take the test?" I hear you ask. The answer is yes, and that's a legitimate concern. |
Five of six variants scored 12/12. |
Only one defect moved the score: empty cells. When I removed the PORT default and the CMS_BASE_URL required flag, the agent reported incomplete answers. But it only said so because I told it to. The prompt explicitly instructed the model to report NOT FOUND rather than guess. |
A real agent working in your repository doesn't get that instruction. It gets "set up this project" and fills in the blanks from whatever it knows. |
An empty cell in a controlled test produces an honest gap. An empty cell in a real Copilot session produces a confident guess you won't notice is wrong. |
My take aways.
I set out to write an AI guide for markdown tables. I ended up with one rule.
|
Don't leave cells empty. Use a placeholder — N/A, none, — — whatever works. An empty cell is ambiguous to a human and invisible to an agent. A placeholder is a decision made visible. |
The other rules I had lined up - headers, full words rather than abbreviations, one concept per table, no prose in cells are all still good practice for humans.
But Claude compensated for every structural defect I threw at it. I'm not going to dress those up as evidence-based AI guidance. |
Fill in your cells. That's what the data tells me.
Further reading |
These studies informed the experiment. None tested documentation tables which is why I ran my own. The findings from these papers can't be extrapolated to Markdown. Still interesting insights. |
Sui et al. (WSDM 2024) — How LLMs process tables across six formats. Found structural cues like headers improve performance. My experiment didn't confirm this on Sonnet. arxiv.org/abs/2305.13062 |
Liu et al. (NAACL 2024) — Broke table structure while holding format constant. Found performance dropped. The paper that most predicted my no-headers variant would fail. It didn't. aclanthology.org/2024.naacl-long.26 |
MMTU (2025) — 25 table tasks, 30,000+ questions. Models handle lookups well but struggle with reasoning across columns. My experiment only tested lookups. cse.engin.umich.edu |
JTON / Zen Grid (2026) — Tested 10 LLMs. Format preference was model-specific. No format won universally. arxiv.org/abs/2604.05865 |
ViTaB-A (2026) — Models get right answers from tables but can't reliably point to which cell they read. arxiv.org/abs/2602.15769 |
| |  |
|
