Should Shopify merchants treat robots.txt as a security setting?
You might open `/robots.txt`, see a list of blocked paths, and assume those parts of your store are protected.
That is an easy assumption to make. Shopify’s default file includes paths such as `/admin`, `/cart/`, `/checkout`, `/account`, and `/orders`.
But `robots.txt` is not a lock on those doors.
It is a public set of instructions for crawlers. Good crawlers may follow those instructions. Visitors can still request the file, read every path in it, and try those URLs themselves.
Treat `robots.txt` as a crawl and visibility setting. Treat passwords, authentication, and bot protection as security settings.
What robots.txt actually does
A `robots.txt` file lives at the root of your domain. On a Shopify store, you can see it by adding `/robots.txt` to your storefront URL.
For example:
`https://yourstore.com/robots.txt`
The file gives crawler rules in plain text. A rule might say:
```text User-agent: * Disallow: /cart/ ```
That means: “Crawler, please do not crawl URLs under `/cart/`.”
It does not mean: “Nobody can open the cart.”
The Robots Exclusion Protocol is clear on this point. It is not a substitute for real content security. Paths named in the file are publicly discoverable, so a file can even draw attention to URLs you hoped nobody would notice.
Google describes robots rules as a way to control which URLs crawlers access. They can reduce crawling of duplicate or unimportant pages and help manage crawler traffic.
That is useful work. It is just different work from access control.
Why Shopify blocks some pages by default
Shopify’s default rules keep crawlers away from areas that are not useful search results. That includes administrative, customer-specific, checkout, and duplicate pages.
A shopper’s account page should not become a search result. Neither should a cart URL or checkout step.
Shopify’s default file also allows crawling of public content, including product, collection, page, blog, and policy URLs. Shopify says its default setup is intended to suit SEO for most stores.
That split is sensible:
- Public store content can be discovered.
- Private or duplicate paths are discouraged from crawling.
- The actual protection of sensitive areas happens elsewhere.
For a quick check, open your live `/robots.txt` now. Look for two things:
- Whether your product and collection pages are broadly allowed.
- Whether a custom rule blocks a section you expect search engines to find.
Do not read a `Disallow` line as proof that the underlying page is private.
A blocked URL can still show up in search
There is another reason not to use robots rules as a hiding place.
Google says a URL disallowed by `robots.txt` can still appear in Google Search when other sites link to it. The crawler may not access the page content, but the URL itself can still be known.
Say you have a page you no longer want people to find. Adding its path to `robots.txt` is not the right first move if your real goal is to remove it from search.
Google points to different tools depending on the situation:
- Use password protection when visitors should not access the content.
- Remove the content when it should no longer exist.
- Use `noindex` when the page can stay live but should not appear in search results.
Those are different decisions. Start with the outcome you need, not the file that feels easiest to edit.
Use the setting that matches the problem
Here is a simple way to sort common situations.
| If you need to... | Use... | | --- | --- | | Reduce crawling of duplicate or low-value URLs | `robots.txt` | | Keep a storefront private before launch | Shopify password protection | | Keep an account, admin area, or customer data private | Authentication and Shopify’s built-in controls | | Deal with abusive traffic or bots | Shopify’s traffic protections | | Keep a public page out of search results | `noindex` or remove the page |
For a storefront that is not ready for visitors, Shopify offers Private mode, also called password protection. You can find it under Online Store → Preferences.
Use a password you do not use for your Shopify admin account. Shopify explicitly advises against reusing the admin password there.
For bot and traffic protection, Shopify says online-store requests pass through Cloudflare. Web Application Firewall and DDoS protection are active by default, and Shopify also uses hCaptcha. Shopify Plus stores have an additional checkout bot protection option.
You do not need to edit a crawler file to solve those problems.
Be careful when editing Shopify’s robots file
A custom `robots.txt` can be appropriate. You may need a narrow crawl rule for a specific part of the store, or you may have a clear SEO reason to change crawler instructions.
But on Shopify, adding a `robots.txt.liquid` template replaces Shopify’s generated output.
That detail matters. You are not adding one line to a fixed Shopify file. You are taking responsibility for the output that Shopify generated before.
Before publishing a custom template, save a copy of your current live file. Then compare the new output at `/robots.txt` after publishing.
Shopify supports Liquid objects for this work, including `robots`, `group`, `rule`, `user_agent`, `sitemap`, and `request`. In practice, the safest custom file usually starts from Shopify’s existing generated structure and changes only the rule you need.
Do not build a short replacement file from memory. It is easy to leave out a sitemap line or a default group you did not know was there.
We learned a version of this lesson while building our free store scanner. We chose to read only public URLs: `robots.txt`, the catalogue endpoint, product markup, and policy pages. No install. No login.
That choice keeps the boundary clear. A public file is useful for understanding what crawlers are being told. It is not proof that an unlisted URL is inaccessible, and it is not a reason for a scanner to treat private store areas as public.
Blocking a crawler is not the same as blocking product distribution
Shopify also warns that robots rules are directional and advisory. Not every crawler has to follow them.
This is especially worth remembering when you are deciding what to do about AI crawlers. A rule in `robots.txt` can affect open-web discoverability for crawlers that respect it. It does not create a technical barrier around your content.
It also does not stop Shopify Catalog from sending product data to activated agentic storefronts such as ChatGPT or Microsoft Copilot.
So ask a narrower question before changing an AI crawler rule: what channel are you trying to affect?
If you mean open-web crawling, a robots rule may be relevant. If you mean product data shared through Shopify Catalog, it is not the control for that job.
Keep the file in its lane
`robots.txt` is worth checking. It can stop useful pages from being crawled, waste crawler attention on the wrong URLs, or reflect an old decision nobody revisited.
It is not a security boundary.
Open your live `/robots.txt` this week. If you find a rule added to hide sensitive content, move that job to the right place: password protection, authentication, page removal, `noindex`, or Shopify’s built-in traffic controls.