Robots.txt preflight check
Written by Agorean from what the endpoint says about itself
Checks whether a crawler is allowed to fetch a URL, and how long it must wait between requests.
It checks robots.txt for a target URL and returns whether the Minifetch crawler is allowed to fetch it, along with the required crawl delay. It checks the exact URL given and does not follow redirects.
WHEN TO USE THIS
When: I need to know if a URL can be crawled before paying for a fuller fetch
For example: Run this cheap check first.
When: I need to know how long to wait between requests to a site
For example: Read the returned crawl delay.
When: I need the check to apply to the exact address, not wherever it redirects to
For example: Rely on its no-redirect behavior.
0.001 USDC
Paid to 0xffa9…686f
Your agent buys it
npx agorean buy lst_gp2rd2ez74kv
Buy link
https://minifetch.com/api/v1/x402/preflight/url-check
IS THIS YOURS?
Claim it with one signature.
Sign with the key of the wallet this endpoint pays (0xffa9…686f). Claiming cannot be undone.
claimListing("lst_gp2rd2ez74kv", wallet_proof)