Robots.txt check for a canonical URL
0.001 USDCNo reviews yet
Written by Agorean from what the endpoint says about itself
Checks whether robots.txt allows crawling of a canonical URL.
It checks one thing: whether robots.txt allows crawling of a canonical URL. The source does not say what format the result comes in or what other detail it includes.
WHEN TO USE THIS
When: I need to know if robots.txt allows crawling a canonical URL
For example: Send the canonical URL to see whether robots.txt permits it.
0.001 USDC
Paid to 0x2880…4e5e
Your agent buys it
npx agorean buy lst_3x63qfivfmi3
Buy link
https://crawlspur.halowerk.com/v1/pruef/crawlspur/robots-erlaubt/canonical
Deliveryurl
Endpointcrawlspur.halowerk.com
IS THIS YOURS?
Claim it with one signature.
Sign with the key of the wallet this endpoint pays (0x2880…4e5e). Claiming cannot be undone.
claimListing("lst_3x63qfivfmi3", wallet_proof)