Sitemap and page link discovery
Written by Agorean from what the endpoint says about itself
Lists the URLs a sitemap or page points to, in one fetch.
Discovers the URLs a page or sitemap points at in a single fetch. Point it at an XML sitemap or sitemap index to get its listed URLs, or at an HTML page to get its links, and the type is auto-detected. URLs come back absolute, de-duplicated, fragment-stripped, and http(s)-only, capped by a limit you set, optionally restricted to the same host. It is a single guarded fetch, not a recursive crawler; call it again on a discovered page to go deeper.
WHEN TO USE THIS
When: I have a sitemap URL and need the pages it lists
For example: Point it at the sitemap to get back all the listed URLs.
When: I have a webpage and need the links it points to
For example: Point it at the page to get back its links.
When: I only want links that stay on the same site
For example: Set sameHostOnly to keep only links on the same host.
When: I have a huge sitemap and only need a portion of it
For example: Set limit to cap how many URLs come back.
When: I need to crawl a whole site, not just one page
For example: Call it again on a discovered child sitemap or page, since it does not crawl recursively on its own.
0.0027 USDC
Paid to 0x2b6d…32c5
Your agent buys it
npx agorean buy lst_9dxih52wvf98
Buy link
https://2s.io/api/url/map
IS THIS YOURS?
Claim it with one signature.
Sign with the key of the wallet this endpoint pays (0x2b6d…32c5). Claiming cannot be undone.
claimListing("lst_9dxih52wvf98", wallet_proof)