Agent-friendly websites: what's real, what's hype, and what I'm building

My feed has been full of “make your website agent friendly” for weeks. Checkers, scores, bundles of new protocol files.

I’m building Porchsite (porchsite.dev), a website that is built and maintained entirely by your AI assistant. There is no human editor, no dashboard and no login for building or updating it. The owner tells their assistant what they want, and the assistant does the rest on the owner’s own Cloudflare account. You can also self-host it.

That puts agents on both sides of the site: the assistant that builds it, and the agents that later read it. So I need to know if this agent-friendly stuff is real.

I had my team research it. Standards, proposals, what shows up in real server logs, what the AI and search companies say, and what people on X are saying. Here’s what I learned.

The traffic part is real

Read the labels: these are companies reporting on their own networks, and some sell the fix.

• Cloudflare (company-reported): more than half of the traffic on its network is now automated, and daily AI-agent requests are up more than 1,700% in a year.
• Vercel (company-reported): 58% of its traffic is bot-originated, up from 32% in January 2024.
• Mintlify (company-reported): 257M agent requests vs 131M human page loads in August on sites it hosts. Mintlify sells agent-ready docs.

Now the other side. A Cloudflare Radar scan of the top 200,000 domains in April 2026 found:

• 3.9% answered a request for Markdown correctly
• 4% declared Content Signals
• fewer than 15 had MCP server cards or API catalogs

So lots of agents, very few sites ready for them. That gap is what the buzz is about.

The finished standards are the boring ones

Most things being sold as “agent friendly” are not standards. The ones that are finished are stuff you’ve probably ignored for years:

• robots.txt (RFC 9309). Worth knowing: the RFC itself says the rules “are not a form of access authorization.” It’s a request, not a lock. OpenAI says its user-triggered ChatGPT-User fetches “may” not follow it.
• Sitemaps (de facto, referenced by RFC 9309).
• HTTP caching with ETag and 304 “not modified” (RFC 9110/9111). An agent re-checking an unchanged page gets a tiny reply instead of the whole page.
• Link headers (RFC 8288), well-known URLs (RFC 8615), and the API catalog (RFC 9727) if you have an API.
• schema.org structured data in JSON-LD (JSON-LD 1.1 is a W3C Recommendation).

None of that is exciting. All of it works.

What agents actually use: Markdown

Serving a Markdown version of each page is a convention, not a formal standard. But it has the best real-world evidence of anything I looked at.

• GoodBarber’s logs (one site, two weeks, self-reported): GPTBot followed the page’s “alternate” link to the Markdown copy 99 times in 100. Claude Code asked for Markdown on 99% of its requests. Assistants answering a human’s question mostly fetched plain HTML. Googlebot made zero .md requests.
• Cloudflare’s Feb 2026 test of 7 agents: only Claude Code, OpenCode and Cursor asked for Markdown by default.
• Size: GoodBarber measured Markdown 79% lighter. One builder’s page went from 158 KB to 17 KB. Cloudflare says “up to 80%” fewer tokens.

The simplest version needs no clever code: write a .md copy of each page at publish time and point to it with <link rel="alternate" type="text/markdown">.

llms.txt: cheap, mostly unread, sometimes useful

Against:

• In Ahrefs’ study of 137,000 domains (May 2026), 97% of llms.txt files got zero requests.
• Google says Search ignores it. No major AI provider says its crawler reads it.
• SE Ranking looked at about 300,000 domains and found no link between having an llms.txt and getting cited by AI tools.

For:

• In Mintlify’s benchmark (a vendor benchmark, 2,400 agent runs), giving agents a link to llms.txt cut their guesses at dead URLs by about 90%. The map mattered more than the Markdown.

My take: it’s one tiny file. Keep it short, link it to your Markdown pages, and never sell it as SEO.

One warning: a study of 62,751 real llms.txt files found 2,035 asking agents to take actions, and a test showed an agent can be steered by one. Generate yours from your own content only, with no instructions to agents.

Still being built, or just hype

Still being built. The permission and identity layer: Content Signals (a Cloudflare feature), IETF AI Preferences (a draft that says its text “DOES NOT REFLECT CONSENSUS”), Web Bot Auth for signed agents (a draft), and pay per crawl (a Cloudflare closed beta). Mostly Cloudflare-led. Worth watching.

Only matters if you expose actions or an API. WebMCP is a proposal in a Chrome origin trial, and WebKit formally opposes it. NLWeb needs a vector store and LLM calls per question. The MCP server card path has already changed once.

Hype to discount:

• A “73% of context wasted on HTML boilerplate” stat going around with no source. The published figures are Cloudflare’s “up to 80%” and GoodBarber’s 79% on its own pages.
• Checkers selling a dozen extra “agent protocol files.” Most have no known consumers.
• “Agent readiness” scores sold as traffic predictors. They’re checklists, not measured results.

What I’m doing in Porchsite

Because assistants build and maintain every Porchsite, the site and its agent-friendly layer come from the same place. Nothing depends on a person remembering to tweak a setting. You can see the project at porchsite.dev.

Two rules I set for the team.

1. Anything an agent can reach gets made agent friendly. That’s every public page. It’s also private, passkey-protected content, for an agent that holds the login. Your own agent should get the same light Markdown and structured data a public crawler gets. No separate “agent mode”.

Being agent friendly never widens access, though. Anonymous visitors to a private site still get a blank 404, and logged-in responses are never cached publicly.

2. Every feature works self-hosted, not just on Cloudflare. Cloudflare is a speed-up, not a dependency. Example: Cloudflare’s “Markdown for Agents” converts pages at the edge, but it drops the ETag, adds an “ai-train=yes” signal if your site sets none, and is listed for paid plans. Lots of owners are on free accounts. So Porchsite builds the Markdown copies itself at publish time, and the self-hosted server serves the same files with the same headers.

The build list, cheapest for the server first:

1. Light, pre-built HTML with real cache headers and ETags
2. robots.txt that treats search crawlers, user-triggered agents and AI-training crawlers differently, a sitemap with accurate dates, and real 404s
3. A Markdown copy of every page, linked from the page
4. JSON-LD generated from the owner’s own business data
5. A short generated llms.txt, owner content only
6. An owner switch for AI-training crawlers, and we always write the content signal ourselves
7. The same treatment for logged-in agents on private content

Not now: MCP servers, NLWeb, WebMCP, signed agents, pay per crawl, agent cards.

The caveats

• No ranking or traffic promise. The evidence supports efficiency: fewer bytes, fewer dead URLs, fewer re-crawls. Not more traffic or citations. I won’t sell it to owners that way.
• Read the labels. A lot of these numbers are company-reported, vendor benchmarks or one site’s logs, measured on different populations. They aren’t comparable to each other.
• The ground keeps moving. llms.txt v2 came out in August. Finished standards first, proposal stuff kept small and easy to drop.

The cheap checklist

Copy this.

• [ ] Put all your content in the HTML (not loaded later by JavaScript), with real headings, lists and tables
• [ ] Let ETag and 304 replies work (and set cache headers on any dynamic routes yourself)
• [ ] robots.txt with separate rules for search, user-triggered agents and training crawlers, plus a Sitemap: line
• [ ] Sitemap with accurate lastmod dates
• [ ] Real 404s for missing files, including /llms.txt
• [ ] schema.org JSON-LD that matches what’s visible on the page
• [ ] A .md copy of each page, linked with <link rel="alternate" type="text/markdown">
• [ ] If you hand out Markdown based on the Accept header, send Vary: Accept
• [ ] A short llms.txt linking to your Markdown pages, no instructions to agents, not called SEO
• [ ] On Cloudflare’s Markdown for Agents? Set your own Content-Signal so it doesn’t default training to yes
• [ ] Count requests to .md and /llms.txt by user agent before you build anything fancier

Make it light, make it cacheable, and be honest about what it does. That’s most of the agent-friendly web, and it was sitting in the HTTP specs the whole time.

Also published as an Article on X by @cooksbayouboy.