Should Shopify merchants block AI crawlers in robots.txt, or let them read product pages?
Your product pages are public because customers need to find them. AI systems increasingly read those same pages when they answer product questions, show shopping results, or send a buyer to a store.
That can make `robots.txt` feel like a simple privacy switch: block AI crawlers, keep products out of AI answers. It is not that simple.
`robots.txt` has different effects for different crawlers. It does not control Shopify Catalog. It does not guarantee a crawler will stay away. And blocking one Google crawler may leave another Google surface untouched.
For most stores, the sensible default is to leave public product pages crawlable and make them useful when they are read. If you have a specific concern about AI training, use the crawler rule that matches that concern without accidentally blocking the discovery you still want.
Start with the decision you are actually making
There are at least four separate choices hiding inside the word “AI.”
- Whether AI systems can discover and summarize your public product pages.
- Whether a provider can use page content for future model training.
- Whether your products are available through Shopify Catalog and agentic storefronts.
- Whether products can appear on Google Search or Google Shopping surfaces.
These choices overlap, but they are not the same switch.
Say you sell a jacket with three fits, a detailed fabric blend, and care instructions. You may want an assistant to find the product page and tell a shopper which fit suits them. At the same time, you may not want the page used for future model training. That is a narrower decision than blocking every AI-related crawler.
Before changing anything, write down which of those outcomes you want. It will stop you from treating `robots.txt` as a catch-all control it cannot be.
The default answer: let public product pages be read
Shopify says its default `robots.txt` is optimized for most stores. It recommends leaving it alone unless you have a specific reason to change it. Shopify also warns that custom edits are unsupported and incorrect use can result in loss of all traffic.
That is strong reason not to add broad blocks just because AI crawlers sound unfamiliar.
Shopify also says product-detail pages should serve both customers and AI systems that crawl them. The useful information is the same information a real buyer needs:
- A clear price.
- Current availability.
- Features and detailed specifications.
- Comparison information where it helps someone choose.
- Structured data.
- Sizing, materials, and care instructions.
This is the practical part of the decision. A crawler cannot explain what makes one of your products right for a buyer if the page only has a product name, a photo, and a vague description.
You do not need an AI app to fix that. Open one product page and ask whether a person could answer these questions from it:
- What exactly is included?
- Who is it for?
- Which option should I choose?
- What is it made from?
- Is it in stock?
- What will it cost?
If the answer is unclear, improve the page first. That work helps the customer standing in front of the page too.
Google also recommends making product pages crawlable and placing `Product` structured data in the initial HTML. It warns that product markup generated by JavaScript can make shopping crawls harder.
Check your current robots.txt before changing it
This takes less than a minute.
Open this address in a browser:
`https://your-store-domain.com/robots.txt`
Replace the domain with your own store domain. You are looking at the instructions your store publishes for crawlers.
Do not assume a line mentioning one crawler applies to every AI product or search surface. Crawler names matter.
Also remember that Shopify describes `robots.txt` rules as directional and advisory. Not every crawler is guaranteed to follow them. Shopify handles network-layer bot management for Shopify-hosted stores, and says merchants do not need to take action there.
So `robots.txt` is a request to compliant crawlers. It is not a wall around public page content.
If your concern is ChatGPT, separate search from training
OpenAI uses different crawlers for different purposes.
`OAI-SearchBot` is the crawler to allow if you want pages included in ChatGPT summaries and snippets. `GPTBot` is the crawler to disallow on pages you want excluded from potential training.
That means a store can make a more precise choice than “ChatGPT yes” or “ChatGPT no.” You might allow `OAI-SearchBot` for product discovery while disallowing `GPTBot` for training.
There is an important limit. OpenAI says a page that is disallowed can still appear as a link and title if the URL is discovered elsewhere. If you do not want that, OpenAI recommends `noindex`.
`noindex` is a much bigger decision than a training preference. It tells search systems not to index the page. Do not add it to a product page casually.
For a normal public catalogue, blocking `GPTBot` while allowing `OAI-SearchBot` is the cleaner option if your concern is specifically potential training rather than product discovery.
Google has separate controls too
Google’s `Google-Extended` token controls whether content may be used for future Gemini training and certain grounding systems. Google says it does not affect inclusion in Google Search and is not a Google Search ranking signal.
Again, this lets you make a narrow choice. You can set a preference for Gemini-related training and grounding without treating it as a Google Search block.
Google Shopping is separate as well. `Storebot-Google` affects Google Shopping surfaces, including the Shopping tab. `Googlebot` controls Google Search and its Search features.
Blocking `Storebot-Google` does not necessarily block Google Search. Blocking `Googlebot` does not necessarily settle what happens on Google Shopping.
If shopping visibility matters to your store, do not change either rule without knowing which surface you are trying to affect.
Robots.txt does not control Shopify Catalog
Shopify Catalog is a separate path for product data.
Shopify says Catalog is enabled by default through the Allow Shopify to manage for me setting. When Catalog access is active, AI channels can use product titles, descriptions, images, prices, and availability for discovery. Some channels also support direct checkout.
Blocking an AI crawler in `robots.txt` affects open-web discoverability only. It does not stop product data being sent through Shopify Catalog to agentic storefronts you have activated.
The reverse is also true in an important way. Turning off Shopify Catalog access for ChatGPT does not guarantee that products disappear from ChatGPT. Shopify says products may still be displayed or referenced through external discovery methods, including web crawling and indexing. Shopify also says removing Catalog access can take up to 7 days.
So check both places when you are making a product-data decision:
- Your public web crawl rules in `robots.txt`.
- Your Shopify Catalog and agentic storefront settings.
One cannot stand in for the other.
We learned to prefer narrow controls
When we built our own `llms.txt` publisher, we shipped it without permission to write to a store theme. That was deliberate. Fewer permissions meant less for a merchant to trust.
It forced us to publish the file as a Shopify page with a redirect instead of writing a theme file. That turned out cleaner anyway.
The lesson applies here. Start with the smallest control that solves the actual problem.
Do not broadly block product pages because you want to limit training. Do not turn off Catalog access because you only meant to affect open-web crawling. Do not block Google Shopping because you meant to change Google Search.
Each broad move can remove a path you still wanted buyers to use.
A practical default for most stores
For most Shopify merchants, this is a reasonable starting position:
- Leave Shopify’s default `robots.txt` unchanged.
- Keep public product pages crawlable.
- Make product pages complete enough for a person or crawler to understand them.
- Check that `Product` structured data is in the initial HTML.
- Review Shopify Catalog settings separately from crawl rules.
- If you have a clear training concern, consider the specific crawler token involved rather than a broad AI block.
There are valid reasons to change crawler rules. The point is not that every store should make the same choice. The point is that you should know which system and outcome each change touches.
Your next step is simple: open your store’s `/robots.txt`, then open one best-selling product page. Check whether the crawler rules match your intent and whether the page gives a buyer enough detail to choose.