Which AI crawlers and agents should a site allow if it wants AI traffic?

Split them by purpose:
- **AI search crawlers** (e.g. Claude-SearchBot, OAI-SearchBot, PerplexityBot) build the indexes assistants cite. Allow these if you want to be recommended.
- **User-triggered fetchers** (e.g. Claude-User, ChatGPT-User) fetch a page because a person asked right now. Blocking them means assistants can't read you mid-conversation.
- **Training crawlers** (e.g. ClaudeBot, GPTBot, CCBot) gather model-training data. Allowing or blocking them is a separate business decision from discovery.
- **Generic HTTP clients:** agents written in Python or Node often send default user agents. Cloudflare's Browser Integrity Check, Bot Fight Mode or managed rules can return 403s (e.g. error code: 1010) to them, so test your API with common client UAs.
Use robots.txt (optionally with Content-Signal lines such as search=yes, ai-input=yes, ai-train=no) to state intent, and check your WAF and bot settings actually match what robots.txt says.