Crawlers

An assistant cannot cite a page it is not allowed to fetch.

ChatGPT, Claude, Perplexity, and Google each send a crawler. Shopify stores control those crawlers in robots.txt, and each page can still say noindex.

robots.txt

A disallow for GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, or Google-Extended means that crawler should stay away. That is a choice. It is also the reason an assistant may talk about a competitor and skip you.

noindex and passwords

A noindex tag, a canonical that points somewhere else, or a storefront password means the public web does not have the page. A password is right before launch. It also means the check has nothing to read until you provide that password or remove it.

llms.txt

llms.txt is a plain-text map at the root of the shop. It is optional. It helps when it lists real URLs in short prose. An HTML error page at that path does not count.

The SEO check reads the page you paste, then robots.txt and llms.txt for that host.