Getting found by ChatGPT and AI assistants: what you really need and what you can skip
ChatGPT, Claude, Gemini and Perplexity read websites to answer people's questions. A few changes genuinely help them, and many only push up a score. Here is how we tell them apart.
Who reads your website today
For years a website had two kinds of readers: people and search engines. Today there is a third. AI assistants such as ChatGPT, Claude, Gemini and Perplexity read pages to answer someone's question, or to complete a task on their behalf.
It is not a single bot. Each company uses several, for different purposes, and robots.txt lets you treat them differently. For OpenAI and Anthropic the split is as follows (OpenAI, Anthropic):
| Purpose | OpenAI | Anthropic | If you block it |
|---|---|---|---|
| Model training | GPTBot | ClaudeBot | Future content is left out of training data |
| Search inside the assistant | OAI-SearchBot | Claude-SearchBot | The site may not appear among the sources of answers |
| Visit requested by a user | ChatGPT-User | Claude-User | The assistant cannot read the page when someone asks it to |
The practical consequence: saying no to training does not mean disappearing from answers. You can block GPTBot and let OAI-SearchBot through. OpenAI notes that robots.txt rules may not apply to visits requested by a user, because a person is asking.
How it is measured: Cloudflare's score
In April 2026 Cloudflare launched isitagentready.com, a free scanner that gives an agent-readiness score (Cloudflare). It checks four areas:
| Area | What it checks |
|---|---|
| Discoverability | robots.txt, sitemap, Link headers |
| Content | A Markdown version of the page, served when the agent asks for it |
| Bot access control | Rules for AI bots, Content Signals, Web Bot Auth |
| Capabilities | API catalog, OAuth, MCP Server Card, Agent Skills, WebMCP |
It also checks protocols for agentic commerce, but does not count them in the score. llms.txt is not among the default checks: it can be added by hand.
The baseline shows how empty the field still is. Across 200,000 of the most visited domains, Cloudflare found a robots.txt in 78% of cases, but almost always written for traditional search engines. Preferences on AI use were declared on 4% of sites, the Markdown version worked on 3.9%, and MCP Server Cards and API catalogs appeared on fewer than 15 sites in total (Cloudflare).
It is a good starting point, but it should be read for what it is. It measures whether files and signals exist, not whether agents use them or whether the content is any good. And the number changes a lot with the chosen profile: in a test published by Search Engine Journal, the same site on the same day scored 33 out of 100 on the default profile and 67 on the content-site profile (Search Engine Journal).
What we do, in this order
On the sites we manage we start with the changes that help everyone and cost little. The others come only if the site has something to offer an agent.
- 01
Clear rules for AI bots
In robots.txt we decide bot by bot, together with the client. Usually yes to AI search and to visits requested by users; on training the choice is the company's, and we write it down explicitly.
- 02
Content Signals
Three signals in robots.txt state how content may be used: search for indexing, ai-input for generated answers, ai-train for training. A typical line is Content-Signal: search=yes, ai-input=yes, ai-train=no. They are preferences, not a technical block; according to Cloudflare they count as a reservation of rights under EU Directive 2019/790 (Cloudflare).
- 03
A Markdown version of every page
Clean text, without menus, scripts or graphics, served when the agent asks for it. Sites on a Cloudflare Pro plan or above can turn on Markdown for Agents, which converts pages on the fly (Cloudflare); on WordPress sites our plugin does it.
- 04
An llms.txt index
A file that describes the site and points to the pages that matter, in a format a model reads in one go. It costs little and does not interfere with SEO.
- 05
The basics in order
An up-to-date sitemap, Link headers pointing to the sitemap and alternate versions, structured data consistent with the visible text of the page.
- 06
Measure before and after
One scan before the work and one after, with the same profile, so the comparison is fair.
Where to stop
For Google the answer is clear. To appear in AI Overviews and AI Mode you do not need AI files, special markup or special structured data: the same basics as traditional search apply (Google Search Central).
On llms.txt Google speaks with two voices. Search considers it unnecessary for its AI features, while Lighthouse, Chrome's auditing tool, checks it from version 13.3 among the requirements for agentic browsing (Search Engine Journal). Our reading: llms.txt is for the assistants and agents that read it, not for ranking on Google.
The items in the Capabilities area (MCP, API catalog, OAuth) only make sense if the site really lets you do something: book, order, check a price list through an API. On a brochure site, adding them to raise the score is work with no benefit, and more surface to maintain and protect.
Finally, none of this guarantees being cited. It is the technical precondition for an assistant to read the site properly. Whether it then cites you depends on the content: clear, up to date, with verifiable facts.
How we do it
On the WordPress sites we manage, the agent layer lives in our own plugin, CF - Agent Discovery, kept separate from the theme and the page builder. It handles AI bot rules, Content Signals, llms.txt and a Markdown version of the pages. It updates as standards change and can be removed cleanly.
The first complete project was a business funding portal, measured before and after with isitagentready.com. The same approach is applied to this site: cloudfactory.it publishes its own llms.txt and a Markdown version of every page, regenerated on every update.
Want to know how agents see your site?
Send us your site's address. We run the scan and tell you what is missing, what is worth doing and what is not needed.
How we wrote this article
At Cloudfactory we also use artificial intelligence in our editorial work: it helps us find and cross-check sources, compare data, set up the structure and polish the language. The system we use is configured by us, with instructions, examples and procedures built around the way we write. Our team checked every source, corrected the text and approved the published version. The assessments and choices are Cloudfactory's, and we are accountable for them.
Frequently asked questions
If I block GPTBot or ClaudeBot, do I disappear from ChatGPT and Claude?
No. Those bots are for training. Search inside the assistants uses OAI-SearchBot and Claude-SearchBot, which you can let through.
Does Google use llms.txt?
Not for search: Google says AI Overviews and AI Mode need no AI files. llms.txt is for the assistants and agents that read it.
Do we need to rebuild the site?
No. Rules, signals, the index and the Markdown version are added to the existing site, without touching theme or content.
Does the isitagentready score affect rankings?
No. It is a Cloudflare tool: it checks whether files and signals for agents are present, not the quality of the content or the position in results.
Do Content Signals stop my content from being used?
No. They express preferences that bots may or may not respect. To actually block a bot you need rules on the server or the firewall.
Sources
- Cloudflare, Introducing the Agent Readiness score
- Cloudflare, Content Signals Policy
- Cloudflare, Markdown for Agents
- OpenAI, Overview of OpenAI crawlers
- Anthropic, Anthropic crawlers
- Google Search Central, AI features and your website
- Search Engine Journal, Google's llms.txt guidance depends on which product you ask
- Search Engine Journal, All you need to know about Cloudflare's Agent Readiness score