AgentDataBot

AgentDataBot is the web crawler used by AgentData to build our company intelligence database. It visits publicly accessible web pages to detect technology stacks, extract company information, and identify team members.

Crawl behavior

• Identifies as: AgentDataBot/1.0 (+https://agentdata.run/bot)
• Respects robots.txt directives
• Crawls at a moderate rate (max 1 request per second per domain)
• Only visits publicly accessible pages, and never tries to get past a login
• Fetches HTML only, never images, videos or large files

Opt out

To block AgentDataBot, add to your robots.txt:

User-agent: AgentDataBot
Disallow: /

To remove existing data about your company, visit our data removal page.

Contact

Questions about our crawler? Email support@useagentdata.com.