Discovery is a chain, not a single switch
A page can be discovered through links from other pages, an XML sitemap, a search engine index, a product specific crawler, or another data source. Discovery only means that a system has encountered the URL. The system may still choose not to crawl, index, retrieve, summarize, or cite it.
Website owners control several important inputs: whether the URL is public, whether it returns useful HTML and a successful status, whether robots rules permit the named crawler, whether a canonical identifies the preferred page, and whether internal links make the page reachable. They do not control the final source selection inside an answer.
robots.txt controls crawling by user agent
A robots.txt file lives at the root of a host and contains groups for crawler user agents. An Allow or Disallow rule can grant or restrict crawling for matching paths. Rules are public and should never contain secrets. A sitemap location can also be declared in the file.
Robots exclusion is not an authentication system. A blocked URL may still be known through links or other sources, and malicious crawlers may ignore the rules. Sensitive content requires real access control. Test robots behavior against the exact host, protocol, path, and user agent because a rule on one subdomain does not automatically govern another.
OpenAI separates search discovery from training controls
OpenAI documents OAI-SearchBot as the crawler used to surface websites in ChatGPT search. It documents GPTBot separately for potential use in training generative AI models. A publisher can allow OAI-SearchBot while disallowing GPTBot when it wants search discovery without granting the separate training crawler access.
OpenAI also notes that robots changes can take time to be reflected. The exact user agent tokens and published IP information should be checked in the current official documentation before changing production controls. A website should not rely on a third party list copied months earlier.
Perplexity and Google publish their own crawler guidance
Perplexity documents PerplexityBot and Perplexity-User, with different roles in indexing and user initiated retrieval. Google documents Googlebot and the robots protocol it supports. Each provider can define user agents, behavior, and controls differently, so one broad statement about all AI crawlers is unreliable.
Create a small policy table for the business. Record the product, user agent, purpose described by the provider, current decision, source URL, and review date. This turns crawler access into a maintained business choice instead of a one time copy and paste operation.
| Question | What to record |
|---|---|
| Which crawler? | Exact official user agent token and provider documentation. |
| What purpose? | Search indexing, user initiated retrieval, model training, or another published purpose. |
| What decision? | Allow, disallow, or path specific rule with a named owner. |
| When reviewed? | Date checked and next review date. |
What an accessible crawler should find
When access is allowed, the page still needs useful output. Serve a descriptive title, canonical, main heading, visible body content, internal links, and appropriate metadata. Avoid placing the entire meaning behind a click, animation, canvas, or client side request that may fail. For JavaScript sites, use static generation or server rendering when practical.
Link related pages with descriptive anchor text. Include the URL in an XML sitemap when it is canonical and intended for indexing. Return a real 404 or 410 for removed content rather than a visually missing page with a successful status. Keep redirect chains short and point moved pages to the closest relevant destination.
Citation requires usefulness and selection
A crawler permission does not create a citation. The answer product may evaluate relevance, freshness, authority, corroboration, safety, and response quality. It may cite a page directly, rely on an underlying search index, use another source, or decide that the query does not need the page.
Improve citation readiness by answering a specific question, naming the subject clearly, providing original and checkable information, using dates, identifying the responsible author or organization, and connecting the page to related evidence. Avoid writing repetitive passages that restate a keyword without adding usable information.
Monitor access without overinterpreting it
Server logs can show requests from user agents, but a user agent string alone can be spoofed. Use provider published verification guidance when available. Search Console and analytics can show some discovery and traffic patterns, while manual citation observations can document what appeared for a controlled question set.
Separate these states in reporting: discovered, crawled, indexed, shown, cited, visited, and converted. They are not interchangeable. If a source does not provide enough evidence for one state, label it unknown instead of inferring completion from the previous step.
Crawler readiness checklist
- Confirm the canonical production host and protocol.
- Review robots.txt for wildcard and named crawler groups.
- Check current provider documentation before changing a user agent rule.
- Keep sensitive content behind authentication, not robots exclusion.
- Serve important content and links in reliable semantic HTML.
- Publish a current XML sitemap with absolute canonical URLs.
- Test status codes, canonicals, redirects, and removed pages.
- Record crawl, index, citation, visit, and conversion states separately.
Primary official sources
These sources support the platform, standards, and implementation guidance in this article. Product behavior and documentation can change, so confirm the current source before a consequential implementation.
- Publishers and developers FAQ (OpenAI)Official OAI-SearchBot and GPTBot guidance.
- Perplexity crawlers (Perplexity)Official PerplexityBot and Perplexity-User documentation.
- Robots.txt specifications (Google Search Central)Google supported robots protocol and syntax.
- Build and submit a sitemap (Google Search Central)Official discovery and sitemap guidance.