We ran a controlled experiment in early 2026 to answer one question: if you serve AI crawlers Markdown instead of HTML, do they visit your pages more often?
We tested 381 pages across six websites and tracked bots like ChatGPT, ClaudeBot, and PerplexityBot using Camino5's Agent Analytics.
The short answer: no. Markdown pages showed a small directional bump, but the range of possible true effects was wide enough to include zero. The result is consistent with random noise. Good, structured HTML remains the right focus. Engineering a separate Markdown delivery system for AI bots is not worth the effort.
1. Why This Matters
AI-driven traffic is growing fast. Between early 2024 and mid-2025, AI-sourced web traffic grew 527% year-over-year. By late 2025, AI bots were making roughly 50 billion crawler requests per day on the Cloudflare network alone.
A theory spread through SEO communities: Markdown is cleaner than HTML, so AI bots should find it easier to parse, crawl more frequently, and cite more often. This experiment was designed to test whether that theory holds up.
2. How We Ran It
381 pages, six content verticals, 20 days (January 19 to February 8, 2026).
Half the pages served normal HTML. The other half served Markdown to AI bots via middleware. We tracked unique visits per page from AI-specific crawlers only, excluding traditional search engines like Googlebot.
Bots tracked: ChatGPT-User, OAI-SearchBot, GPTBot, ClaudeBot, Claude-Web, PerplexityBot, Meta-ExternalAgent, FacebookBot, DuckAssistBot.
3. What We Found
Markdown pages had slightly higher raw numbers, but the difference wasn't meaningful.
After adjusting for pre-experiment variance, the estimated lift was +13.9%. But the confidence interval ran from -11.2% to +46.8%. The true effect could be zero. Statistical tests confirmed: no significant difference.
ChatGPT dominated. It accounted for 73% of all AI bot traffic. Meta was second at 20%. ClaudeBot was 2%.
The bump only helped already-popular pages. Pages below the median traffic threshold saw no difference at all. The Markdown effect, such as it was, only appeared for pages already getting significant bot visits. Markdown does not appear to drive discovery. At most, it slightly amplifies what is already working.
4. What Industry Experts Say
The null result lines up with what the major platforms have said publicly. Google's John Mueller has stated that LLMs have been trained on normal web pages from the start and have no trouble with HTML. Microsoft's Fabrice Canel has questioned whether bot-specific formats add any meaningful signal. A parallel study on llms.txt found no positive correlation with AI crawler activity either.
5. Limitations Worth Knowing
The experiment could only reliably detect effects of roughly 41% or larger. A smaller but real gain of, say, 10% would not have shown up here. Also, this study measured crawling, not citations. A page can be visited more often without being included in AI answers more often. Those are different things.
6. What To Do Instead
Don't over-invest in Markdown delivery systems. Focus on the fundamentals:
- Write quality content that provides real value to LLM training and retrieval.
- Use clean HTML with proper headings, lists, and tables.
- Don't block AI crawlers with aggressive rate-limiting or CAPTCHAs.
- Track bot behavior with tools like Camino5's Agent Analytics to see what is being visited and by whom.