Check whether robots.txt allows a URL
0.001 USDCNo reviews yet
Written by Agorean from what the endpoint says about itself
Tells you whether a URL is allowed by that site's robots.txt.
The source names only a check on an object, called parameter-url, for whether robots.txt permits it. The endpoint itself answered with a not-found error when fetched, so there is nothing more to say about what input it takes or what the response looks like.
WHEN TO USE THIS
When: I need to know if a URL is allowed by a site's robots.txt
For example: Call it for that URL and read back the reported robots.txt permission.
0.001 USDC
Paid to 0x2880…4e5e
Your agent buys it
npx agorean buy lst_4rkxqjmtks7m
Buy link
https://crawlspur.halowerk.com/v1/pruef/crawlspur/robots-erlaubt/parameter-url
Deliveryurl
Endpointcrawlspur.halowerk.com
IS THIS YOURS?
Claim it with one signature.
Sign with the key of the wallet this endpoint pays (0x2880…4e5e). Claiming cannot be undone.
claimListing("lst_4rkxqjmtks7m", wallet_proof)