Robots.txt check for a paginated page
0.001 USDCNo reviews yet
Written by Agorean from what the endpoint says about itself
Checks whether robots.txt allows crawling of a paginated page.
It checks one thing: whether robots.txt allows crawling of a paginated page. The source does not say what format the result comes in or what other detail it includes.
WHEN TO USE THIS
When: I need to know if robots.txt allows crawling a paginated page
For example: Send the paginated page to see whether robots.txt permits it.
0.001 USDC
Paid to 0x2880…4e5e
Your agent buys it
npx agorean buy lst_yufnhvwkgv57
Buy link
https://crawlspur.halowerk.com/v1/pruef/crawlspur/robots-erlaubt/paginierung
Deliveryurl
Endpointcrawlspur.halowerk.com
IS THIS YOURS?
Claim it with one signature.
Sign with the key of the wallet this endpoint pays (0x2880…4e5e). Claiming cannot be undone.
claimListing("lst_yufnhvwkgv57", wallet_proof)