Last reviewed: June 6, 2026. This policy can change if crawler behavior, provider tokens, server load, or content-use risk changes.
Short version
- Public MyIPScan pages are intended to be crawlable by search and answer-retrieval systems.
- Internal, API, temporary, audit, backup, and partial-template paths are not intended for crawling.
- Crawler access is not a promise of indexing, ranking, traffic, AI answers, or AI citations.
- Robots.txt is a public crawler policy file, not an access-control or privacy-security layer.
Recommended access policy
MyIPScan currently allows major search and answer-retrieval crawlers to access public pages because the site is built for public privacy education, tool-led SEO, and citation-friendly explanations. This includes Googlebot, Bingbot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-SearchBot, Claude-User, and Applebot.
MyIPScan also currently allows selected broad AI and web-corpus crawlers such as GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, and Bytespider on public pages. This favors visibility and possible downstream discovery, but it increases content reuse and server-load risk. If logs show abusive behavior, excessive load, or misuse, MyIPScan should rate-limit or block the specific crawler rather than changing the whole public crawl policy first.
Blocked paths
The public robots policy keeps crawlers out of paths that are not content destinations, including API endpoints, partial templates, audit output, temporary directories, backups, and similar operational files. These paths are not intended for search results or AI summaries.
What llms.txt is for
The llms.txt file gives AI and answer systems a concise map of MyIPScan's public pages, methodology, safe citation wording, and privacy-claim limits. It is an emerging guidance file. It is not a universal standard, not a ranking factor guarantee, and not a substitute for normal HTML, sitemap, robots, schema, and editorial quality.
What this policy cannot guarantee
This policy cannot guarantee that Google, Bing, ChatGPT, Claude, Perplexity, Gemini, Copilot, or any other system will crawl, index, rank, summarize, or cite MyIPScan. It only states the intended public access policy and the paths that should stay out of crawler scope.