[User-Agent] (AI Search Crawlers)

[jlarmstrongiv]

Paste the full User-Agent String here

Amazonbot
Applebot
PerplexityBot
YouBot
Amazonbot

Is this for Addition / Removal?

• Addition
• Removal
• Keep a watch on this one

Did the User-Agent request robots.txt first?

• Yes
• No

Post Log Excerpt to show User-Agent behavior (10-20 lines is enough)

Post log excerpt here (paste in between the ```     ``` markers)

Additional information

They should be watched, but they cite websites so may be okay for now

See also https://darkvisitors.com/agents

[mitchellkrogza]

Will scan some logs for these

[TheArcaneBrony]

AmazonBot: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot) Chrome/119.0.6045.214 Safari/537.36
OpenAI/GPTBot: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)
Anthropic/ClaudeBot: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)

[mitchellkrogza]

AmazonBot: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot) Chrome/119.0.6045.214 Safari/537.36 OpenAI/GPTBot: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot) Anthropic/ClaudeBot: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)

Thanks @TheArcaneBrony what kind of hit rate are you getting from these?