Skip to content
rev·ductive

AI5 min read

llms.txt is bullshit. Do it anyway.

97% of llms.txt files never get requested and Google ignores them. I built one anyway, found AI was reading a third of my site, and rebuilt it so it isn't.

Kevin Stout

Founder, Revductive

Share on LinkedIn
Editorial illustration in bone, black and orange: a long ribbon of dense unreadable text feeds leftward into the mouth of a matte black reader, across a pale wedge of light. A hard orange line slashes down through the ribbon partway along, and past that cut the paper turns to shadow and crumples away unread into the dark.

The SEO world is selling llms.txt as the way to get your company into ChatGPT responses. Ahrefs pulled the server logs on 137,000 sites in May. About 28% of them publish one. 97% of those files got zero requests that month, from a bot or a person. Google says in its own documentation that Search doesn’t use them.

So as an SEO play or getting you into AI responses, it does nothing (yet, maybe it will someday). I built one anyway, plus a few other files around it, because I came up with something potentially as useful (we’ll see). It didn’t work right the first time, but I think I have it working well now.

The page I built

Last week I put a page on my website called “Point your AI at this page.” The idea was simple. People are running me past ChatGPT before they ever book a call, so instead of letting it crawl the site and guess, I turned the whole site into one text file and put a button on it that opens ChatGPT or Claude with the file already in the prompt. All 53 pages, word for word, not summaries (which is the traditional recommendation for llms.txt).

I posted about it on LinkedIn and said I had no idea if it would work. It turns out, it didn’t. Not the way I thought.

What actually happened

I pretended to be a prospect and asked ChatGPT some questions and it missed really clear answers I knew I had somewhere on the website.

I went and looked at what the AI was actually receiving. The file is about 93,000 tokens. A token is roughly four characters, so call it 373,000 characters of text. The web fetch tool inside ChatGPT and Claude (basically their browser) stopped reading somewhere around 25 to 30 thousand tokens in my tests. And it doesn’t tell you. For me, that meant AI got the first third of the file, the cut lands mid sentence on some random page, and it answers like it read everything (because AI loves to lie!).

The about page, my bio, a bunch of blog posts, some pillar pages, and more were all past that cut. No AI that read that link had ever seen them.

I tried the obvious things first.

  1. Raise the token limit in the model settings. Nope. The limit is in the fetch tool, not the model. (Learned something new there!)
  2. Ask it to fetch the file again and keep going. It gets the same first third. None of the fetch tools let you ask for a byte range, so “read from where you left off” isn’t a thing.

So the only way to get the rest of the site read is via separate URLs. Three files that each fit under the limit, instead of one file that doesn’t.

The fix

Here’s what’s on the site now. A build script generates all of it from the finished HTML every time the site deploys, so nothing is written by hand and gets out-of-date. No, I have no idea how the hell that works exactly, Claude Code and Astro are carrying me again (check out my blog post about that).

  1. One core file that fits in a single fetch. Home, pricing, about, how we operate, the RevOps and AI pages, both audits, the case studies, contact. About 19,000 tokens. Pricing and the bio are near the top on purpose, because the end of the file is what gets lost if anything does.
  2. A short “quick answers” block at the top of it. The questions people ask on discovery calls: what it costs, who does the work, how an engagement gets started, etc. Every sentence in it is copied from a page on the site, and the build fails if one of those sentences ever disappears from the site. I didn’t want a summary that drifts away from the pages it’s summarizing.
  3. llms.txt as the index. Every page, one line each, with links. The prompt on the page now says: read the core file first, then look at the index and go get two to four more pages if they’re relevant to what you’re working on. This is closer to the current thinking around the structure of the llms.txt doc.
  4. Every file says how big it is and what its last section is. Right at the top. This file is about 19,000 tokens, most fetch tools cut off around 25,000, the last section is headed X, and if you didn’t reach it you didn’t get the whole file. The prompt tells the AI to say so plainly if a fetch came back short. Before this there was no way for the model, or for you, to know.
  5. A markdown copy of every page. /pricing.md sits next to /pricing. So when the AI does go get one more page, it gets clean text instead of the HTML.

The full file is still there for downloading and uploading as an attachment. A small percentage of people will actually do that.

Astro generates all of this when the site builds, and no, I don’t really know how. Claude Code does. The other post explains that part. You do NOT need to know a bunch of technical stuff to build the same workflow. Point Claude to this post and that one.

Does any of this do anything for SEO

No. This is for the person who points their AI at the page or uploads the file. It’s not generating top-of-funnel for you.

Where I’d start if you want this on your site

  1. Check what some AI tools are grabbing when you point them at a page. It could be cutting off a lot of the page.
  2. Keep any file an AI is going to fetch under about 20,000 tokens. That’s roughly 80,000 characters. Put the thing you most need read at the top.
  3. Consider multiple llms.txt files with different purposes. This isn’t the going recommendation right now, but I bet on best practices around this stuff being a lot more flexible than the SEO rules of the past.

If you try it and hit a wall, tell me. I’m still learning this one as I go. The short version of this, plus the two tools I’d buy this month, is in issue 003 of the newsletter. And if you’re trying to work out which AI tool belongs where before you build any of it, that’s a different post.

The page itself is at /for-your-ai. Point something at it and see what comes back.

Want a look at your setup?

30 minutes, free. Leave with a recommended tool stack, no matter what.

Book a 30-min call →

Common questions

Does llms.txt help you rank in ChatGPT or Google AI answers?

No. Ahrefs pulled server logs on 137,000 sites in May 2026: about 28% publish an llms.txt, and 97% of those files received zero requests that month from a bot or a person. Google's own documentation says Search does not use them. Nothing is crawling the file, so it does nothing for SEO or AI search visibility today.

Why build an llms.txt file if nothing crawls it?

Because of the people who point their own AI at your site before they talk to you. They paste a URL into ChatGPT or Claude and ask what your company does and what it costs. The file is what that model reads instead of guessing from whichever three pages it decided to crawl. It is a better answer for a prospect already looking at you, not a way to acquire new ones.

How large can a file be before an AI fetch tool truncates it?

Keep any file an AI will fetch under about 20,000 tokens, which is roughly 80,000 characters. In my testing the web fetch tools inside ChatGPT and Claude stopped somewhere around 25,000 to 30,000 tokens. They do not tell you they stopped, they cut mid-sentence, and the model answers as though it read the whole thing.

Can you ask an AI to resume a fetch that got cut off?

No. None of the fetch tools accept a byte range, so "keep reading from where you stopped" is not available and a second fetch returns the same first chunk. Raising the model's token limit does not help either, because the limit lives in the fetch tool rather than the model. The only fix is splitting the content across separate URLs that each fit in one fetch.

Should you publish a markdown version of every page?

Yes, if you want an AI to read individual pages cleanly. A markdown copy at /pricing.md next to /pricing gives the model plain text instead of navigation, scripts and markup, so more of what it fetches is actual content. Generate them from the built pages at deploy time rather than by hand, or they drift out of date the first time you edit a page.

Related