Robots.txt crawl permission checker
Written by Agorean from what the endpoint says about itself
This checks whether a given URL path can be crawled by a specific bot, such as Googlebot, GPTBot, ClaudeBot or Bingbot, or any custom user-agent token. It fetches the site's robots.txt and reports allowed or disallowed, the matching rule, the rule group used, any crawl-delay, and any sitemaps the site declares. It is for crawlers and SEO tools that need to check permission before fetching a page.
WHEN TO USE THIS
When: A crawler is about to fetch a page and needs to know if it is allowed first
For example: Check a path for GPTBot before fetching it, and read the matching rule and crawl-delay back.
When: An SEO tool is auditing a site's crawl rules
For example: Check several paths against Googlebot and Bingbot and compare which rule group applies to each.
When: A developer wants to find a site's declared sitemaps without loading the whole robots.txt manually
For example: Run the check and read the declared sitemaps field in the result.
0.001 USDC
Paid to 0xfc24…ebb3
Your agent buys it
npx agorean buy lst_ufed32irsxd6
Buy link
https://intel.rallylive.ca/monitor/robots-allowed
IS THIS YOURS?
Claim it with one signature.
Sign with the key of the wallet this endpoint pays (0xfc24…ebb3). Claiming cannot be undone.
claimListing("lst_ufed32irsxd6", wallet_proof)