AI search is when a search tool writes an answer instead of, or as well as, showing a list of links. Google’s AI Overviews and AI Mode, ChatGPT search, Perplexity and Claude all do it, and all of them draw on web pages. This guide explains how each one finds sources, what the platforms themselves say about being included, which popular tactics you can ignore, and what a small business should do first.
A lot of what’s written about AI search is guesswork sold as a new discipline. Where this guide makes a claim about how a platform works, it’s from that platform’s own documentation, linked in the sources. Where it’s my view, I say so.
[Table of contents]
What AI search is
AI search uses a large language model to read search results and write a direct answer, usually with links to the pages it drew on. Instead of choosing between ten blue links, the searcher gets a summary, and the businesses and pages named in it get the attention.
There are two broad kinds:
| Kind | Examples | Where the answer comes from |
|---|---|---|
| AI inside a search engine | Google AI Overviews, Google AI Mode | Google’s own search index and ranking systems |
| AI assistants that search the web | ChatGPT search, Perplexity, Claude | Their own crawlers and search partners, plus live fetches of pages when users ask |
Both kinds depend on web pages being accessible, understandable and trustworthy. That’s why so much of AI search visibility turns out to be ordinary SEO done well.
How search got here
Search has been moving from links towards answers for years. Featured snippets, knowledge panels and “People also ask” boxes all answered questions on the results page. AI search takes that further by writing the answer from several sources at once.
For most of Google’s history, a search returned a list of pages and the searcher did the reading. Over time, Google added more ways to answer directly: featured snippets pulled a paragraph from a page, knowledge panels summarised facts about businesses and people, and local results showed opening hours and reviews without a click.
The arrival of AI assistants that could search the web, and then Google’s own AI Overviews and AI Mode, changed the scale of it. Instead of lifting one paragraph, AI search reads many pages and writes a new answer, often naming products and businesses along the way. The question for a business shifted from “do we rank?” to “are we one of the sources, and are we named?”.
How Google’s AI Overviews and AI Mode work
AI Overviews and AI Mode are built on Google’s normal search systems. Google says they may use “query fan-out”, running several related searches across subtopics to build an answer, and that there are no extra requirements to appear beyond being indexed and eligible to show a snippet in Google Search.
Google’s documentation on AI features and your website sets out the essentials:
- No special requirements. Google says there are “no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary”, and no special schema.org structured data to add.
- Indexing and snippets. To be shown as a supporting link, a page must be indexed and eligible to be shown in Google Search with a snippet.
- Query fan-out. Both features may issue multiple related searches across subtopics and data sources to develop a response. That’s why a page can be cited for a question it doesn’t rank first for.
- Controls. The nosnippet, data-nosnippet, max-snippet and noindex controls limit what Google shows from a page in Search, including in AI features.
- Measurement. Traffic from AI features is included in Search Console’s Performance report, within the Web search type. Google’s documentation updates in June 2026 confirmed that AI Mode is counted in those totals.
Google’s AI optimisation guide goes further. It says the best practices for SEO continue to be relevant “because our generative AI features on Google Search are rooted in our core Search ranking and quality systems”, and that optimising for generative AI search is “still SEO”.
In other words, there isn’t a separate Google algorithm for AI answers to game. The pages Google trusts in normal results are the pages it tends to draw on for AI answers.
How ChatGPT, Perplexity and Claude find sources
AI assistants use their own crawlers, which are separate from the ones used to collect AI training data. OpenAI, Perplexity and Anthropic each document a search crawler, a training or general crawler, and a user-triggered fetcher, and each can be allowed or blocked in robots.txt independently.
| Company | Search crawler | Training crawler | User-triggered fetcher |
|---|---|---|---|
| OpenAI | OAI-SearchBot | GPTBot | ChatGPT-User |
| Perplexity | PerplexityBot | Not used for training, per Perplexity | Perplexity-User |
| Anthropic | Claude-SearchBot | ClaudeBot | Claude-User |
What the documentation says:
- OpenAI. Its crawler documentation says OAI-SearchBot is for search, GPTBot is used to crawl content that may be used in training its foundation models, and each setting is independent. A site can allow OAI-SearchBot to appear in ChatGPT search results while disallowing GPTBot. OpenAI notes it can take around 24 hours for its systems to adjust after a robots.txt change.
- Perplexity. Its crawler documentation says PerplexityBot is designed to surface and link websites in Perplexity’s search results and isn’t used to collect content for training AI foundation models. It also says Perplexity-User, which fetches pages when a user asks, generally ignores robots.txt because a user requested the fetch.
- Anthropic. Its help centre article says ClaudeBot collects web content that could contribute to model training, Claude-SearchBot improves search result quality, and Claude-User accesses websites when people ask Claude questions. Blocking Claude-SearchBot may reduce a site’s visibility in Claude’s search results.
- Google. Google-Extended is a robots.txt token for Google’s AI products outside Search. Google’s crawler documentation says it doesn’t affect a site’s inclusion in Google Search and isn’t used as a ranking signal there.
The practical upshot is that “should we block AI bots?” is really several separate decisions. Blocking a training crawler is a reasonable choice for many businesses. Blocking a search crawler means that assistant can’t read your pages when answering, which for most small businesses is the opposite of what they want.
What the evidence shows so far
AI search is already changing which pages get clicks. On sites I work on, informational pages have lost clicks to AI Overviews answering simple questions, while businesses with clear, specific pages are being named in AI answers. Being cited and being clicked are now two different measures.
Some examples from real sites:
Named first in an AI Overview. Search Google in the UK for “best garage door threshold seal” and the AI Overview names GaraDry’s threshold seal kit first, with GaraDry’s site ranking number one in the results below. The pages behind that clearly explain what each product does, who it suits and how it’s fitted, the same work that ranks them in normal results. The GaraDry case study covers the strategy.
Education content losing impressions. When Search Console showed 20 of Kwiat’s jewellery education pages losing impressions, the main causes were AI Overviews answering the questions and competitors improving their content. Those pages were rewritten with deeper content, answer passages written to be quoted and full FAQs. The Kwiat case study covers what was done.
Declining posts with mixed causes. On condoms.uk, 29 blog posts were losing clicks. AI Overviews answering simple questions was one cause, alongside outdated product details, thinner content than competitors and broken internal links. AI search was part of the story, not all of it, which is typical. The condoms.uk case study covers the refresh programme.
These examples point to a pattern worth knowing. When impressions hold steady or rise while clicks fall, AI answers are a likely cause, but not the only one. Why your impressions are up but clicks are down explains how to tell the difference.
What helps you appear in AI answers
The things that help most are the things that already help in search: indexable pages, genuinely useful and specific content, clear answers near the top of pages, a clearly identified business, and a reputation other sites reflect. Google specifically recommends non-commodity content that goes beyond what others have already said.
Be crawlable, indexable and snippet-eligible
If Google can’t index a page, or it’s blocked from showing a snippet, it can’t be a supporting link in AI Overviews or AI Mode. Check your key pages in Search Console, and check robots.txt isn’t blocking the search crawlers of the assistants you want to appear in.
Write non-commodity content
Google’s AI optimisation guide says creating content people find “unique, compelling, and useful” will likely influence your presence in generative AI search more than any of its other suggestions. It contrasts generic advice with content that brings first-hand perspective and specific outcomes.
For a small business, that’s an advantage. You have things a content farm doesn’t: real jobs, real customers, real prices, real photos and real mistakes you’ve learned from.
Answer the question clearly, early
Pages that state the answer in the first paragraph under a clear heading are easy for readers and AI systems to use. That doesn’t mean chopping content into fragments. Google’s guide says there’s no requirement to break content into tiny pieces. It means not burying the answer beneath an introduction.
Make the business unmistakable
AI answers name businesses, so it helps when yours is easy to identify: a consistent business name, a clear About page, named authors with real credentials, and consistent details across your site and profiles. Google’s guide also says Merchant Center feeds and Google Business Profiles can help products and services be visible in AI responses as well as other search results.
Earn a genuine reputation
Google’s guide notes that its AI features can show what’s being said about products and services across the web, in blogs, videos and forums, but warns that seeking inauthentic mentions isn’t as helpful as it might seem. Real reviews, press coverage and recommendations are the kind of reputation that shows up.
Keep product and service information accurate
When assistants recommend products or businesses, outdated prices, discontinued products and wrong opening hours work against you. Accurate structured data helps search engines read those details, even though Google says structured data isn’t required for its AI features.
The AI citation readiness service checks all of this for a site, and how to appear in Google AI Overviews goes into more practical detail.
What you can ignore
Several popular AI search tactics aren’t needed for Google, according to Google itself. Its AI optimisation guide says you can ignore llms.txt files and other special AI files, chunking content into tiny pieces, and chasing inauthentic mentions, and that structured data isn’t required for its generative AI features.
| Tactic | What Google says |
|---|---|
| llms.txt and other AI text files | Not needed to appear in Google Search, including its AI features, as Google Search doesn’t use them. Fine to keep for other services |
| Chunking content into tiny pieces | No requirement to break content up for AI to understand it |
| A page for every query variation | Creating content for every variation or fan-out query primarily to manipulate rankings or AI responses violates the scaled content abuse policy |
| Inauthentic mentions | Not as helpful as it might seem |
| Special schema for AI | Structured data isn’t required, and there’s no special schema.org markup to add |
Other AI assistants may use llms.txt files or other signals differently, and that may change. For Google, the guidance is clear. Does llms.txt matter? covers the wider debate.
One related change: FAQ rich results stopped showing in Google Search, and Google removed their documentation in 2026. Visible FAQs still help readers and give AI systems clear question-and-answer content. They just no longer change how a listing looks in Google.
How to measure AI search visibility
Measure AI search in three ways: Search Console for Google, where AI Overviews and AI Mode traffic is included in the Performance report; analytics for visits referred by AI assistants; and regular manual or tracked checks of whether your business is named for the questions that matter.
Search Console. AI Overviews and AI Mode are counted within the Web search type in the Performance report. They aren’t broken out separately, so watch for pages where impressions hold or rise while click-through rate falls, especially on informational searches.
Analytics. Visits from links in AI assistants often arrive as referrals from the assistant’s domain, such as chatgpt.com or perplexity.ai, although some arrive without referrer information and show as direct traffic. How to track traffic from ChatGPT in GA4 walks through setting up a report.
Visibility checks. Being named without a click doesn’t show in either tool. For that, you need a fixed set of questions your customers ask, checked regularly across the assistants that matter to you, recording whether your business is mentioned or linked. Results vary by location, personalisation and wording, so the trend over time is more reliable than any single check. That’s what AI visibility tracking does.
What’s likely to change next
My expectation is that AI answers will spread across more searches, that being named will matter more than ranking for simple questions, and that AI agents will start completing tasks such as bookings and purchases. That’s a forecast based on the direction the platforms have taken, not a certainty.
Everything below is my view, based on what the platforms have announced so far. Here’s what I’d expect:
- More searches answered directly. Simple informational questions will keep sending fewer clicks. Searches where people compare, buy, book or need detailed help should keep sending visits.
- Trust and brand carry more weight. When an assistant names two or three businesses, the ones it picks tend to be clearly identified and well regarded. Reputation becomes part of search visibility.
- Agents that act, not just answer. Google’s AI optimisation guide already includes initial guidance on AI agents that perform tasks such as booking a reservation or comparing product specifications. Clear product data, simple checkouts and accessible booking will matter more.
- Preferred sources and personalisation. Google’s preferred sources feature, which lets people choose sites they want to see more of, began rolling out to AI Overviews and AI Mode in May 2026. Loyal readers may start to shape visibility as well as rankings.
- Measurement stays imperfect. Until platforms report AI citations more clearly, businesses will rely on a mix of Search Console, analytics and visibility tracking.
What’s genuinely uncertain is how quickly AI answers reach commercial searches, and how the platforms balance answering questions with sending visitors to websites. I’ll update this guide as the documentation changes.
What I learned
The businesses I see named in AI answers aren’t doing anything exotic. They’re the ones with the clearest, most specific pages on exactly what customers ask, on sites search engines can crawl easily. GaraDry is named first for “best garage door threshold seal” because its pages explain the product better than anyone else’s, not because of an AI-specific trick.
What’s changed is how I measure success. Clicks alone no longer tell the whole story, because a business can be the source of an answer without getting the visit. I now look at impressions, clicks and whether the business is named, side by side, and I treat a page losing clicks to an AI Overview as a prompt to make it more useful rather than a lost cause.
If you’re starting out, this is what I’d do:
- Check robots.txt allows the search crawlers of the assistants you want to appear in, whatever you decide about training crawlers.
- Confirm your most important pages are indexed in Search Console.
- Write down the ten questions customers ask most, search them in Google and ChatGPT, and note who’s named.
- Make sure each of those questions has a clear, specific answer on your site, near the top of the relevant page.
- Run key pages through the free SEO checker, which includes AI readiness checks.
If you’d like to know where your business stands, a free assessment is a good starting point, and AI search visibility explains how the work is done for clients. The what is SEO guide covers the foundations everything here builds on.
FAQ
Do I need to optimise for AI search separately from SEO?
For Google, no. Google says its AI features are rooted in its core ranking and quality systems, and that optimising for generative AI search is still SEO. Other assistants use their own crawlers, so it’s worth checking they can access your site, but the content that helps is the same.
Should I block AI bots in robots.txt?
It depends which bots. OpenAI, Perplexity and Anthropic each separate their search crawlers from training crawlers or user-triggered fetchers. Blocking a training crawler stops your content being used to train models. Blocking a search crawler stops that assistant reading your pages when answering, which reduces your visibility there.
Do I need an llms.txt file?
Not for Google. Google’s AI optimisation guide says you don’t need llms.txt or other special AI files to appear in Google Search or its AI features, because Google Search doesn’t use them. It’s fine to keep one for other services that might, but it isn’t a substitute for useful content.
Does structured data help with AI Overviews?
Google says structured data isn’t required for its generative AI features and there’s no special schema.org markup to add. Structured data still helps search engines understand products, businesses and articles, and supports rich results. It’s useful, but it won’t earn an AI Overview citation on its own.
How do I know if AI Overviews are taking my clicks?
Look in Search Console’s Performance report for pages or searches where impressions and position hold steady but clicks and click-through rate fall, especially on informational questions. Then search those questions yourself. If an AI Overview answers them fully, it’s a likely cause, although other changes to the results page can have similar effects.
Sources
- AI features and your website, Google Search Central, accessed 15 September 2026. Supports: no additional requirements for AI Overviews and AI Mode, no special structured data, indexing and snippet eligibility, query fan-out, preview controls, AI features counted in the Performance report within the Web search type.
- AI optimization guide, Google Search Central, accessed 15 September 2026. Supports: AI features rooted in core ranking and quality systems, “still SEO”, non-commodity content, llms.txt and chunking not needed, inauthentic mentions, structured data not required, scaled content abuse and fan-out variations, Merchant Center and Business Profiles, AI agents.
- Latest Google Search documentation updates, Google Search Central, accessed 15 September 2026. Supports: FAQ rich result deprecation and documentation removal, AI Mode counted in Search Console totals, preferred sources in AI Overviews and AI Mode, llms.txt clarification.
- Overview of OpenAI crawlers, OpenAI, accessed 15 September 2026. Supports: OAI-SearchBot, GPTBot and ChatGPT-User roles, independent settings, around 24 hours to adjust.
- Perplexity crawlers, Perplexity, accessed 15 September 2026. Supports: PerplexityBot for search and not training, Perplexity-User generally ignoring robots.txt.
- Does Anthropic crawl data from the web, and how can site owners block the crawler?, Anthropic, accessed 15 September 2026. Supports: ClaudeBot, Claude-SearchBot and Claude-User roles.
- Google’s common crawlers, Google, accessed 15 September 2026. Supports: Google-Extended doesn’t affect inclusion in Google Search and isn’t a ranking signal.
- Live Google search for “best garage door threshold seal”, google.co.uk, 15 September 2026. Supports: GaraDry named first in the AI Overview and ranking first organically.
- Kwiat and condoms.uk, Google Search Console data and content audits, 2026. Supports: 20 Kwiat education pages losing impressions, 29 condoms.uk posts losing clicks and their causes.