Understanding the structure of the internet through open research.
MapTheNet.org is a non-profit research initiative that systematically crawls the public web to record domain-level link relationships. By analysing which domains link to which other domains, we construct a graph that represents the connective tissue of the internet.
The resulting dataset is released openly so that researchers, journalists, educators, and policymakers can study how the web is organized, identify structural vulnerabilities, and track changes over time.
The internet is often described as a network, yet relatively few people have seen its actual structure. Commercial search engines index pages for relevance; we index domains for structure. Our data answers questions like:
How we build the map, from crawl to visualization.
Our distributed crawler visits publicly accessible domains, following outbound links while respecting robots.txt and enforcing per-domain rate limits. Only the source and target domain of each link are recorded.
Raw link observations are deduplicated and aggregated into a weighted directed graph. Each edge carries a weight representing the number of distinct pages on the source domain that link to the target domain.
Domains are enriched with metadata: TLD, country of registration (via public WHOIS), category (via heuristic classification), and basic HTTP response headers such as server software and status codes.
The graph is rendered using a force-directed layout algorithm. Nodes are positioned so that heavily connected domains cluster together, revealing the natural communities and hierarchies of the web.
MapTheNet is maintained by a small group of volunteers.
Responsible for overall direction, architecture, and data governance.
Builds and operates the distributed crawling infrastructure.
Processes crawl data, generates exports, and maintains the map UI.
We welcome contributors of all skill levels. Use the contact form to reach the maintainers and we will reply as soon as possible.
We are grateful for the support of organizations that share our mission.
Sponsorship slots are available. If your organization supports open data, internet research, or digital rights, please reach out via our contact form.