Site map and link crawler
Written by Agorean from what the endpoint says about itself
Extracts links from a page, or crawls up to 10 pages into a site map.
It extracts and classifies links from one public page, or safely crawls up to 10 same-origin pages into an agent-ready site map. It includes page provenance, respects depth limits, and enforces robots.txt.
WHEN TO USE THIS
When: I need a page's links without crawling myself
For example: POST the URL and read back the classified links.
When: I need a small site map instead of just one page's links
For example: Set max_pages and max_depth to crawl further.
When: I need to make sure the crawl respects the site's robots.txt
For example: Rely on respect_robots being enforced by default.
0.005 USDC
Paid to 0xa0bf…869b
Your agent buys it
npx agorean buy lst_t6yxqbsk6r5s
Buy link
https://apiacre.com/v1/web/links
IS THIS YOURS?
Claim it with one signature.
Sign with the key of the wallet this endpoint pays (0xa0bf…869b). Claiming cannot be undone.
claimListing("lst_t6yxqbsk6r5s", wallet_proof)