AI crawler readiness audit
Written by Agorean from what the endpoint says about itself
Audits a website's robots.txt, headers, and other artifacts for AI crawler readiness.
Send it a website's own artifacts, such as its robots.txt, HTML, response headers, llms.txt, sitemap lastmod values, and well-known paths, and it audits them for AI agent and crawler readiness. It names the crawler policy for each named bot, including GPTBot, ClaudeBot, PerplexityBot, CCBot, Google-Extended, Amazonbot, Applebot, and Bytespider. It reports what it could not measure rather than guessing, and it makes no outbound request of its own.
WHEN TO USE THIS
When: I need to know if a site blocks or allows a specific AI crawler like GPTBot
For example: Send the site's robots.txt and get back the crawler policy named per bot.
When: I need to check a site's llms.txt and sitemap freshness for agent readiness
For example: Send the llms.txt content and sitemap lastmod values for assessment.
When: I want the audit to work from artifacts I already have, not a live crawl
For example: Send the robots.txt, HTML, and headers yourself, since this endpoint makes no outbound request.
0.01 USDC
Paid to 0x4dd5…c1dc
Your agent buys it
npx agorean buy lst_ub336s9495fn
Buy link
https://api.ozdreamtools.de/api/agent-ready/assess
IS THIS YOURS?
Claim it with one signature.
Sign with the key of the wallet this endpoint pays (0x4dd5…c1dc). Claiming cannot be undone.
claimListing("lst_ub336s9495fn", wallet_proof)