MapTheNetBot
Our crawler reads links.
MapTheNetBot visits websites to see which other domains they link to. Those links are what the map and the rankings are made of.
User agent
MapTheNetBot/1.0 (+https://www.mapthenet.org/bot)
Before you block it
Most sites don't need to. If one of these is the worry, there's a smaller fix.
Your content isn't stored
It fetches a few pages, pulls out the link addresses, and discards the page. We don't keep page text, page addresses, titles, images or IP addresses, and nothing it collects is used to copy, index or republish your content. The full list is on the privacy page.
Slow it down instead
If it's about server load, a crawl delay is usually enough, and your site stays listed:
User-agent: MapTheNetBot Crawl-delay: 10
We read robots.txt at the start of
every visit. We fetch at most 50 pages per
domain, one at a time.
You can still block it (see below). Your domain stays in the rankings, since those come from links on other sites, and its page says that you've opted out of crawling.
What we record
- Links: which domain links to which, when we first saw the link, and whether it was nofollow or on every page of the site
- Link words, in total: up to ten common words used in links pointing at a domain, counted across many sites and not traceable to any one page
- Per domain: HTTPS and IPv6 support, the HTTP status of the last visit, and the network operator
- Never: page content, page addresses, page titles, IP addresses or registration records
Each download stops at 1024 KB. We look at several pages per site so we can tell a link someone wrote into one page from a link that's part of the site's template.
How it behaves
- Fetches
robots.txtbefore every crawl and follows it,Crawl-delayincluded - Sends one request at a time per site
- Respects
429responses and theirRetry-Afterheader - Pauses for 90 days if a firewall or bot protection blocks it, and longer if it happens again
- Logs the date it read each
robots.txtand how each visit went - Signs every request with Web Bot Auth (HTTP Message Signatures, Ed25519). The public key is published at
/.well-known/http-message-signatures-directory
When we come back
How often we visit depends on rank. This is the schedule the crawler runs on.
| Rank | Visited | Pages read |
|---|---|---|
| 1 – 100,000 | every 10 days | 50 |
| 100,001 – 500,000 | every 21 days | 50 |
| 500,001 – 2,000,000 | every 45 days | 20 |
| 2,000,001 – 5,000,000 | every 90 days | 20 |
| 5,000,001 – 20,000,000 | every 120 days | 20 |
| 20,000,001 and below, and unranked domains | every 365 days | 8 |
Domains we haven't visited yet wait in line, and the ones with the most links to them go first.
If a visit fails, we wait before trying again: one day after a timeout, up to 90 days if bot protection blocks us. Each time the same thing happens again, the wait doubles, up to a year. Each domain's page shows its next visit under Timeline.
When rankings change
When we visit a site, we note its links. They show up in the rankings after the next data rebuild (the current data is from 2026-10-30), so ranks and map positions change with each rebuild, not right after a visit.
Your rank depends on the sites that link to you, so it changes when we visit them. Blocking the crawler doesn't take you out of the rankings, and editing your own pages doesn't change your rank.
Block the crawler
Add this to robots.txt:
User-agent: MapTheNetBot Disallow: /
This takes effect on our next visit, since we read robots.txt fresh every time instead of using a saved copy.
Remove your domain
Blocking stops future visits but keeps you listed, because the listing comes from links on other sites. To be taken out completely, prove the domain is yours with a DNS record, a small file, or a meta tag on your home page.
You can also email [email protected] from an address at that domain, or from the domain's registered contact address.
If the crawler causes problems for your site, write to the address on our imprint.