Most businesses that want customers to find them should not block AI crawlers across the board. The bots that feed ChatGPT search, Claude and Google’s AI answers are separate from the bots that collect training data, and each company lets you block them one at a time in your robots.txt file. If you are a builder, a restaurant, a clinic or a property developer, keep the search bots open so AI tools can read and recommend you, and decide on the training bots as a separate question. On 30 October 2026 Google said it plans to support, along with other AI providers, one shared signal for opting out of AI training, which should make that second decision simpler from 2027. xCreative™ is a Melbourne creative agency in Chadstone that has built websites since 2016 and now runs SEO and AI search work for Australian businesses, and this is the advice we give owners who ask us.
Training bots and search bots do different jobs
The question usually arrives as “should I block ChatGPT?”, but there is no single ChatGPT bot. OpenAI’s crawler documentation lists GPTBot, which collects pages that may be used to train its models, and OAI-SearchBot, which finds pages for ChatGPT’s search answers. OpenAI says a site that blocks OAI-SearchBot will not appear in those search answers, while blocking GPTBot only signals that the content should not be used for training. It also notes that robots.txt changes for search can take about 24 hours to apply.
Anthropic splits its bots the same way. Anthropic’s help article for site owners names ClaudeBot for training, Claude-SearchBot for search and Claude-User for pages a person asks Claude to read. Anthropic says blocking either of the last two may reduce your visibility in answers to user searches.
Google works differently again. Its list of common crawlers describes Google-Extended as a separate token that controls whether crawled content is used for Gemini training and for grounding in Gemini Apps. Google says Google-Extended does not affect inclusion in Google Search and is not used as a ranking signal. So a rule against Google-Extended keeps your pages out of Gemini training without touching your normal Google listing.
What changed on 30 October 2026
Until now an owner who wanted to opt out of AI training had to write a separate rule for every company. Google’s Search Central post of 30 October 2026 says it plans to support a shared training control built on work from the IETF’s AI preferences group. If a site uses that shared token to opt out of training and does not mention Google-Extended, Google plans to treat it as if Google-Extended were blocked. Google aims to support it in 2027, and its existing controls keep working alongside it.
The token is defined in an Internet-Draft called Machine-Readable Training Preference Signal, written by an Anthropic engineer and published on the same day. The example in the draft is short:
- User-Agent: __usage_ai-training
- Disallow: /
Two details matter for owners. The draft says the token states a preference and is not an instruction about crawling, so it does not stop any bot from visiting. It also has no formal standing yet inside the IETF. Treat it as a useful extra line, not a replacement for the named rules you may already have.
When blocking makes sense for a business
Blocking training bots is a reasonable choice when your content is the product. A publisher, a course provider, a stock photographer or a firm that sells research reports has a real reason to keep that material out of training data. In those cases we block the training tokens (GPTBot, ClaudeBot, Google-Extended and the new shared token once it is supported) and leave the search bots alone, because those businesses still want to be found.
For most service businesses the trade is different. A Melbourne builder’s project pages, a restaurant’s menu or a developer’s floor plan summary are written to be read and repeated. Customers now ask AI tools which builder to call or where to book dinner, and the tool can only recommend you if it can read your pages. Blocking search bots to protect content you published for free usually costs more than it saves.
Google AI Overviews are a separate setting again. Since 31 August 2026 every site can use a Search Console control to leave AI Overviews and AI Mode. Google says opted out sites stop receiving traffic and impressions from those features, and that the choice is not a ranking signal for ordinary results. We have not recommended this to any client, because those answers are often where a local search starts.
Check you are not blocking AI by accident
The more common problem we find is a website that blocks AI tools without the owner knowing. Security plugins, hosting firewalls and content delivery networks can all ship settings that turn away AI bots, and some of them write their own lines into robots.txt. The page still loads for people and Google still indexes it, so nothing looks broken. The first sign is that AI tools never mention the business.
Three quick checks will tell you where you stand:
- Open yourdomain.com.au/robots.txt in a browser and look for lines naming GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot or Google-Extended. A Disallow under a search bot is the one to question.
- Ask ChatGPT or Claude to summarise a specific page on your site by pasting its address. If it says it cannot read the page, something is blocking it.
- Ask whoever manages your hosting, firewall or CDN whether an AI bot setting is switched on, and which bots it covers.
These checks are part of our website maintenance work, and they belong on any handover checklist when you change developers, because a firewall rule set by a previous provider can sit there for years.
What we do on our own website
Our own robots.txt at xcreative.com.au allows every crawler, and we publish an llms.txt file that lists our services and case studies for AI tools to read. We chose that because we want AI assistants to describe our work accurately when someone asks for a Melbourne branding or web design agency, and that only happens if they can read the source. The approach is set out on our AI search optimisation page, and the wider search work on our SEO and AI search page.
If an agency offers to make you visible in AI answers, ask what it changes in your robots.txt and firewall before anything else. Our guide to choosing an AI SEO agency and the questions to ask an SEO company cover the rest. If you would like us to check what your site allows today, get in touch.
Frequently asked questions
Will blocking GPTBot remove my business from ChatGPT?
No. OpenAI uses GPTBot for training and OAI-SearchBot for ChatGPT search, and each follows its own robots.txt rule. Blocking GPTBot signals that your content should not be used for training. Your site can still appear in ChatGPT search answers as long as OAI-SearchBot is allowed.
Does blocking Google-Extended hurt my Google rankings?
Google says Google-Extended does not affect inclusion in Google Search and is not a ranking signal. It controls Gemini training and grounding in Gemini Apps. AI Overviews and AI Mode have their own opt out in Search Console, which is a separate decision.
Should I add the new shared AI training token now?
You can add it if you already want to opt out of training, but keep your existing named rules. The token was published as an Internet-Draft on 30 October 2026, it expresses a preference rather than a crawling instruction, and Google says it aims to support it in 2027.
