# hempdata.io — proprietary regulatory data moat # Policy: block training crawlers (don't let our curated dataset train competitors' models). # Allow user-fetch + AI-search bots so AI assistants can cite us live. # --- Training crawlers — BLOCK (protects the moat) --- User-agent: ClaudeBot Disallow: / User-agent: GPTBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: CCBot Disallow: / User-agent: Bytespider Disallow: / User-agent: Amazonbot Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Meta-ExternalAgent Disallow: / User-agent: anthropic-ai Disallow: / User-agent: cohere-ai Disallow: / User-agent: PerplexityBot-Internal Disallow: / User-agent: AI2Bot Disallow: / User-agent: Ai2Bot-Dolma Disallow: / User-agent: Diffbot Disallow: / User-agent: FriendlyCrawler Disallow: / User-agent: ImagesiftBot Disallow: / User-agent: Timpibot Disallow: / User-agent: Webzio-Extended Disallow: / User-agent: YouBot Disallow: / # --- User-fetch crawlers — ALLOW (AI assistant citations) --- User-agent: Claude-User Allow: / Disallow: /login Disallow: /signup Disallow: /forgot-password User-agent: ChatGPT-User Allow: / Disallow: /login Disallow: /signup Disallow: /forgot-password User-agent: PerplexityBot Allow: / Disallow: /login Disallow: /signup Disallow: /forgot-password User-agent: Meta-ExternalFetcher Allow: / Disallow: /login Disallow: /signup Disallow: /forgot-password # --- AI-search indexing — ALLOW (appear in AI search answers) --- User-agent: Claude-SearchBot Allow: / Disallow: /login Disallow: /signup Disallow: /forgot-password User-agent: OAI-SearchBot Allow: / Disallow: /login Disallow: /signup Disallow: /forgot-password # --- Traditional search + default --- User-agent: Googlebot Allow: / Disallow: /login Disallow: /signup Disallow: /forgot-password User-agent: Bingbot Allow: / Disallow: /login Disallow: /signup Disallow: /forgot-password User-agent: * Allow: / Disallow: /login Disallow: /signup Disallow: /forgot-password Sitemap: https://hempdata.io/sitemap.xml