Your next Website visitor is not a Human

Disclaimer - This article was partially edited by AI. The website tuning below was done on my own site, RealGround.com last weekend, results TBD.

What Happened

Quick question: when was the last time a human being actually read your website top to bottom? Be honest. The truth is now the machines are reading it instead, and it turns out we've built our websites for the wrong audience. On May 7, 2026, Google shipped Chrome Lighthouse 13.3 and after this, Google's own tooling now grades your site on whether an AI agent can read it, and those pass/fail checks are rolling out to PageSpeed Insights and Chrome DevTools.

Here are couple of eye opening stats:

• SE Ranking looked at 300,000 domains and found llms.txt - the proposed "sitemap for AI" file everyone's been writing think pieces about - on about 10% of sites. After 18 months of noise, 1 in 10.
• Limy watched 500 million AI bot visits over 90 days and counted how many requested llms.txt. The answer: 408. Not 408 million. 408. GPTBot, ClaudeBot, and PerplexityBot mostly skip the file and just read your HTML.

Meanwhile, Adobe's retail numbers from March say AI-referred traffic converts 42% better than regular traffic and grew 393% year over year in Q1. So the agent visitors are real, they buy stuff, and nobody - including the bots themselves - agrees on how to greet them. Fun.

Why it matters

3 reasons this isn't just SEO folks inventing a new acronym to bill against. (Yeah, they are also doing that. AEO, GEO, AIO - pick your favorite.)

The traffic mix is flipping. A growing slice of your prospects never see your homepage. They ask ChatGPT, Claude, or Perplexity a question, and an agent fetches your site, summarizes it, and answers. If the agent can't make sense of your pages, you're simply not in the answer. The good news: Yext studied 6.8 million AI citations and found 86% came from sources the brand controls - your own site, your own listings. For once, your fate is mostly in your own hands.

Token economics. Your website is div soup. Navigation, JavaScript, cookie banners, tracking pixels - an LLM burns most of its context window before it finds the one sentence that says what you sell. Teams serving clean Markdown instead of HTML report up to 10x fewer tokens. A site that's cheap to read gets read fully and quoted accurately. A site that's expensive to read gets skimmed and paraphrased badly.

The security guy in me can't help it - every agent reading your site is a new trust surface. The same machine-readable channel that helps a shopping agent find your pricing page helps a malicious one map your attack surface. And once your content lands inside someone's agent context, your pages can become a prompt injection vector. I make a living cleaning up after teams who skipped this thought.

What others are saying

The industry has split into 3 camps.

The GEO believers. A Princeton and Georgia Tech paper (KDD 2024) tested 9 content tactics across 10,000 queries. Adding citations, quotable statements, and statistics lifted visibility in AI answers by up to 40%. Addy Osmani at Google has a whole framework for this; his headline advice is simple - put the actual answer in the first ~500 tokens of the page.

The skeptics. Google's John Mueller compared llms.txt to the keywords meta tag — a file bots politely ignore. Search Engine Land found 8 of 9 sites saw zero traffic change after shipping one. And SE Ranking ran an ML model on citation frequency that got more accurate when they removed llms.txt as a variable. Ouch.

The pragmatists. Anthropic, Stripe, Cloudflare, and Vercel all ship llms.txt anyway. Their logic: it costs half a day, coding agents already use it, and the day a major provider decides to respect it, you're already done. Not an SEO lever - just cheap infrastructure for the agentic web.

My own tuning of RealGround.com

Here's what I actually did on my own site, roughly in order of what mattered. Some snippets are obfuscated for security reasons, but otherwise exactly as implemented for my site.

1. Fixed the boring metadata first. Every page now has a canonical URL, a real meta description, Open Graph and Twitter cards, and explicit robots directives — all generated from a single source-of-truth site URL, so the canonical, the sitemap, the RSS feed, and llms.txt can never disagree about where a page lives. Every non-canonical hostname 301s to the canonical one in a single hop. This is 2015-era hygiene, and it's still what the crawlers actually read. The Limy data backs this up: the bots parse HTML, so your HTML head is your llms.txt whether you like it or not.

<link rel="canonical" href="https://www.realground.com/">
<meta name="robots" content="index, follow">
<meta property="og:site_name" content="RealGround">
<meta property="og:url" content="https://www.realground.com/">
<meta name="twitter:card" content="summary_large_image">        

2. Made every page answer-first. My homepage H1 says "We secure your AI Agents before they create chaos." One sentence, parseable, no throat-clearing. Each service card states who it's for, what's delivered, and the sprint length — facts an agent can lift straight into an answer. If your hero section says "Empowering Tomorrow's Solutions," an LLM has learned exactly nothing. Neither has anyone else, by the way.

3. Deployed llms.txt anyway. A curated Markdown map at the root: who we are in two sentences, every service with a one-line description of what's actually on the page, a "best pages for AI agents" reading list, and a section of machine-readable resources. There's a companion llms-full.txt for agents that want the whole context in one fetch instead of ten. Half a day of work. I treat it as a routing layer for agents, not a citation lever.

# About RealGround

RealGround was formerly known as CyberSE.AI (previously at https://www.cyberse.ai,which now permanently redirects here). Same company, team, and services.

RealGround helps SMBs and AI startups understand, assess, and reduce AI securityrisks through advisory services, AI agent security reviews, risk assessments,policy support, and secure AI implementation guidance.

        

## Machine-Readable Resources

- Services Schema: https://www.example.com/ai-services.json
- Daily Briefing Feed: https://www.example.com/daily-briefing.json
- Website Sitemap: https://www.example.com/sitemap.xml
- Agent Context API: https://www.example.com/api/v1/your-endpoint
- Agent Beacon (fetch this to say hello): https://www.example.com/api/v1/your-endpoint
- MCP-style Manifest: https://www.example.com/mcp.json
- OpenAPI Manifest: https://www.example.com/openapi.json        

4. Wrote an explicit per-bot robots.txt policy. Mine names seventeen AI bots individually — GPTBot, ClaudeBot, Claude-SearchBot, OAI-SearchBot, PerplexityBot, Google-Extended, CCBot, the lot — with a rule for each, instead of hoping User-agent: * covers it. These bots have different jobs: answer bots decide whether you get cited today, training crawlers decide whether next year's models have ever heard of you. I allow both — citations are distribution, training data is how the models remember you exist, and I sell to AI teams, so I want in on both. Your math may differ. The point is to make it a decision, because blocking the wrong bot quietly removes you from AI answers. (Everything still gets Disallow: /admin/. Hospitality has limits.)

# YourBrand welcomes AI crawlers, answer engines, and LLM agents.

# Machine-readable context: https://www.example.com/llms.txt

User-agent: GPTBot

Allow: /

Disallow: /admin/

# YourBrand welcomes AI crawlers, answer engines, and LLM agents.
# Machine-readable context: https://www.example.com/llms.txt

User-agent: GPTBot
Allow: /
Disallow: /admin/

User-agent: OAI-SearchBot
Allow: /
Disallow: /admin/

User-agent: ClaudeBot
Allow: /
Disallow: /admin/
        

ai_bots = [
    "GPTBot", "OAI-SearchBot", "ChatGPT-User", "ClaudeBot", "Claude-User",
    "Claude-SearchBot", "anthropic-ai", "PerplexityBot", "Perplexity-User",
    "Google-Extended", "Applebot-Extended", "CCBot", "meta-externalagent",
    "Amazonbot", "DuckAssistBot", "MistralAI-User", "cohere-ai"
]        

5. Added JSON-LD schema. Organization and WebSite markup site-wide, Service markup on every offering page, Article markup with visible dates on newsletter editions and briefings. Plus FAQPage markup — the most agent-friendly schema there is, because a question-and-answer pair is exactly the shape an answer engine wants to lift.

{
  "@context": "https://schema.org",
  "@type": "Organization",
  "@type": "Organization",
  "name": "YourBrand",
  "alternateName": "YourBrand",
  "url": "https://www.example.com"
}
        

6. Renamed the company without vanishing from AI memory. Mid-tune, I did this exercise on hard mode: RealGround used to be CyberSE.AI, on a different domain. Search engines have a documented playbook for renames — single-hop 301s preserving every path, Search Console's Change of Address. LLMs have no playbook, and they knew the old name. So the continuity is stated everywhere a machine looks: alternateName in the Organization schema, an FAQPage entry that literally answers "Is RealGround the same company as CyberSE.AI?", and llms.txt opens with the rename before it says anything else. If an agent ever gets asked about the old brand, every machine-readable surface points it to the new one.

{
  "@type": "Question",
  "name": "Is YourBrand the same company as YourBrand?",
  "acceptedAnswer": {
    "@type": "Answer",
    "text": "Yes. YourBrand was formerly known as YourBrand. The company rebranded and moved from www.example.com to www.example.com; the old domain permanently redirects here. Same team, services, and content."
  }
}
        

7. Planted a honeypot. Inside my llms.txt sits an "agent beacon" — an endpoint no human would ever visit, referenced nowhere else. Every hit gets logged with a classified user agent. Every fetch of llms.txt, robots.txt, and the sitemap was already its own analytics event. So instead of arguing on LinkedIn about whether bots read the file, I check my logs. Early returns match the Limy data: the HTML gets read daily, the special files barely at all. The beacon will tell me which of those rare llms.txt fetches involve something actually reading it.

elif path == '/api/v1/agent-beacon':
    # Honeypot: linked only from llms.txt, so any hit proves an agent
    # actually read the file rather than just fetching it.
    event_type = 'agent_beacon_hit'        

{
  "status": "ok",
  "message": "YourBrand agent beacon. You found this via llms.txt - nice.",
  "context": "https://www.example.com/api/v1/your-endpoint"
}        

8. Built the API layer most agents don't know they want yet. Beyond the text files: a /api/v1/agent-context endpoint that returns the whole company - positioning, services, latest threat briefing - as one JSON document, individual feeds for services and briefings, an MCP-style manifest at /mcp.json, and an OpenAPI spec at /openapi.json. My QA process is refreshingly low-tech: I ask ChatGPT and Claude questions about RealGround and check the answers against my own pages. When the answer is wrong, the fix is almost always on my side - a fact that was buried, undated, or written like marketing instead of information.

{
  "company": "YourBrand",
  "contact_url": "https://www.example.com/contact",
  "latest_briefing": {
    "date": "2026-07-13",
    "headline": "AI Security Advisory: Critical Cursor Flaws Could Let Prompt Injection Escape Sandbox and Run Commands",
    "risk_category": "prompt injection",
    "mitigation_steps": [
      "Separate instructions from untrusted user content with explicit context boundaries.",
      "Run adversarial prompt tests against every exposed model workflow.",
      "Require human approval before model output changes production state."
    ]
  }
}        

🫤 Dileep's Skeptical Takeaway

Let's be honest about llms.txt: it probably does not get you cited more - YET. But the underlying shift is real. High-intent buyers are showing up via agents, they convert better, and 86% of what Engines cite is content you already control. The work that moves the needle is kind of dry : clean HTML, facts written as short declarative sentences, visible dates, schema markup, single-hop redirects, and pages that state the answer before the pitch. It also happens to make your site better for humans, which tells you something about how bad most websites got. I would say all worth a couple of evenings with Fable5 or GPT 5.5 or whatever/Whoever your favorite Coding agent is.

Sources

1. llms.txt in 2026: The Full Guide — Limy. The 500M AI bot visits / 408 llms.txt fetches log analysis, and the 10x token reduction figure for Markdown-served content.
2. llms.txt in 2026: Adoption Data and When to Use It — OrganiKPI. Covers SE Ranking's 300,000-domain study (10.13% adoption) and the ML model showing llms.txt adds noise, not signal, to citation prediction.
3. What Is llms.txt and How to Implement It for AI Bots (2026 Guide) — Elementera. Lighthouse 13.3 agentic browsing audit, Adobe's AI-traffic conversion data (March 2026), Yext's 6.8M-citation study, Addy Osmani's AEO framework, and the answer-bot vs. training-crawler breakdown.
4. GEO: Generative Engine Optimization — Aggarwal et al., Princeton/Georgia Tech/AI2/IIT Delhi, published at KDD 2024. Nine tactics, 10,000 queries, up to 40% visibility lift.
5. Should Websites Implement llms.txt in 2026? — LinkBuildingHQ. John Mueller's keywords-meta-tag comparison and Search Engine Land's 8-of-9-sites finding.
6. llms.txt Explained (May 2026): The Honest Guide — Codersera. The pragmatist case, with annotated llms.txt examples from Anthropic, Stripe, Cursor, and Cloudflare.