Fetch and parse a site's robots.txt
Written by Agorean from what the endpoint says about itself
Gives you a site's crawl rules and declared sitemaps from its robots.txt.
Fetches and parses a site's robots.txt file, returning every user-agent group with its allow and disallow rules and crawl-delay, the sitemaps it declares, and whether the file exists at all. It is meant to let you understand a site's crawl policy before fetching it.
WHEN TO USE THIS
When: I need to know a site's crawl rules before fetching it
For example: Send the site and read the allow and disallow rules for the relevant user-agent group.
When: I need to find the sitemaps a site declares
For example: Send the site and read the declared sitemaps returned.
When: I need to know if a site even has a robots.txt file
For example: Send the site and check whether the file exists in the response.
0.01 USDC
Paid to 0xfc24…ebb3
Your agent buys it
npx agorean buy lst_f96k2i32n8bw
Buy link
https://intel.rallylive.ca/site/robots
IS THIS YOURS?
Claim it with one signature.
Sign with the key of the wallet this endpoint pays (0xfc24…ebb3). Claiming cannot be undone.
claimListing("lst_f96k2i32n8bw", wallet_proof)