Crawling the Complete IPv4 Reverse DNS Space

In November 2023, we published a feasibility study that asked a simple question: is it practical to look up the reverse DNS record of every routable IPv4 address? The back-of-the-envelope answer was "probably": roughly 20 small servers querying at full speed for three weeks.

That estimate turned out to be pessimistic. The crawler is now in production: every month it sweeps all 3,702,258,432 routable IPv4 addresses, collects 1,039,377,899 PTR records, and finishes in two to four days. The complete result includes every hostname and the exact DNS status of every address that did not resolve, yet compresses into a single artifact of roughly 1 GB.

This article explains how the crawl works and, more interestingly, what a billion hostnames are actually good for: hosting detection, VPN and Tor corroboration, connection type inference, and internet-scale geolocation mining. Along the way we survey the current research frontier, from CAIDA's Hoiho to LLM-guided extraction systems like The Aleph, before discussing what is next: faster incremental re-crawls and the IPv6 reverse tree. The raw dataset itself is published as the ipapi.is Reverse DNS Database, and it feeds directly into the ipapi.is IP intelligence API.

100.34.2.29	pool-100-34-2-29.phlapa.fios.verizon.net
184.27.32.74	a184-27-32-74.deploy.static.akamaitechnologies.com
3.137.190.186	ec2-3-137-190-186.us-east-2.compute.amazonaws.com
88.198.1.1	static.88-198-1-1.clients.your-server.de

Why Reverse DNS Matters More in 2026

A PTR record maps an IP address back to a hostname via the special in-addr.arpa DNS zone. The operator of the address space controls the record, and many operators encode surprisingly rich information in it:

$ dig +short -x 100.34.2.29
pool-100-34-2-29.phlapa.fios.verizon.net.

This single hostname reveals the ISP (Verizon Fios), the access technology (a residential pool), and even the city (phlapa = Philadelphia, PA). Multiply that by a billion and you have one of the most information-dense public datasets on the internet.

Reverse DNS also stopped being optional for a large class of infrastructure. Since April 2024, Google and Yahoo require bulk email senders to have a valid PTR record with forward-confirmed reverse DNS (FCrDNS: the PTR hostname must resolve back to the same IP), and Microsoft enforced the same for Outlook in May 2025. What used to be a courtesy convention (RFC 1912 already recommended that every internet-reachable host should have a name) is now a hard deliverability requirement, which means PTR data keeps getting better maintained, and more useful to crawl.

The Crawl: 3.7 Billion DNS Queries, Politely

The open-source crawler is a small Go program deployed on a fleet of 20 cloud servers. The design goals were simple: complete the full sweep in days, be a polite DNS citizen, and record an honest status for every address, not just the ones that resolve.

Throughput, and Why the Next Pass Is Much Faster

In the 2023 feasibility study, a single server pushing everything through 8.8.8.8 managed about 112 lookups per second. The production fleet is far quicker. The full sweep of all 3,702,258,432 routable addresses completes in about three days. The exact figure lands in a two-to-four-day window depending on how many networks are slow to answer. That works out to an aggregate of roughly 14,000 PTR lookups per second across the fleet, about 700 per second per node, or on the order of 1.2 billion lookups per day. It is more than a hundred times the single-server baseline, and not one public resolver sits in the hot path.

The bottleneck is waiting on the slowest authoritative servers, not CPU or bandwidth. A PTR query and its answer are tiny. That is why the tuning favors a short timeout and few retries: at 800 concurrent in-flight lookups per node, a query stuck waiting is 1/800th of that node's capacity doing nothing, so the fleet is deliberately configured to give up quickly and move on.

Subsequent Passes Only Re-Query What Can Change

The first sweep has to visit all 3.7 billion addresses because it knows nothing in advance. Every sweep after that does not. Since the crawler records the DNS status of every address, the next pass can skip the roughly 1.94 billion addresses that returned an authoritative nxdomain, a definitive "no reverse record exists here" that changes only when an operator provisions new naming, which is rare.

That leaves only the addresses whose answer can realistically change or was never captured cleanly: the ~1.04 billion that have a PTR record (hostnames get renamed, ranges get re-purposed) and the ~561 million that timed out last time (a slower re-query with a longer timeout may finally catch them). Re-crawling that set (roughly 1.6 billion addresses instead of 3.7 billion, well under half) brings a full refresh down to somewhere around a day. Zone-aware probing pushes this further still: because reverse DNS is delegated per /24, a single SOA/NS query can confirm that an entire 256-address block is undelegated, replacing 256 individual lookups with one. The architecture for this incremental, self-scheduling updater is described in the crawler's DESIGN.md.

Negative Answers Are Data Too

Most reverse DNS datasets only store the hostnames that resolved. We deliberately store the DNS outcome of every queried address, because the failure modes are a dataset of their own. The July 2026 sweep breaks down as follows:

Two observations are worth making. First, the often-cited figure of 1.25 billion PTR records (from the 2021 paper "IP Geolocation through Reverse DNS") is higher than our 1.04 billion because a strict single-pass sweep with a 2-second timeout classifies slow authoritative servers as timeout rather than waiting them out; some fraction of those 561 million timeouts would resolve with patient retries. We trade a few percentage points of coverage for a sweep that completes in days and reports every failure honestly.

Second, the reverse DNS tree is in remarkably poor shape. Independent research confirms this: the study "Advancing in Reverse: A Comprehensive Characterization of IN-ADDR.ARPA Deployment" (ACM SIGCOMM CCR) found that only about 43% of the allocated IPv4 space is well-configured for reverse DNS. Our per-status data makes that misconfiguration measurable month over month, network by network.

Storing a Billion Hostnames in One Gigabyte

A naive dump of the crawl is a 40+ GB text file that is unpleasant to ship and slow to parse. The crawler instead writes a purpose-built binary format (.rdnsz) that exploits the structure of PTR data:

• Delta-encoded IPs: records are sorted, so each address is stored as a varint difference from the previous one, usually a single byte.
• IP templating: 65.7% of all PTR hostnames embed their own IP address (ec2-52-28-99-71…, pool-100-34-2-29…). The encoder detects the four common decorations (dashed, dotted, forward, reversed) and replaces the embedded IP with a 1-byte marker that is reconstructed on read.
• Block compression: records are grouped into blocks of 64K and each block is independently compressed with zstd, so the file can be streamed and processed in bounded memory without ever being fully decompressed.

The result is a complete sweep (all 3.7 billion query outcomes, including every hostname) that ships in about 0.99 GB. That is less than a byte per queried IPv4 address.

What a Billion PTR Names Actually Look Like

A histogram of full hostnames is useless: almost every one is unique, because operators generate them per IP. The signal lives in the shared suffix and the naming template:

hostname:  node-h2l.pool-1-0.dynamic.nt-isp.net
suffix:    dynamic.nt-isp.net                     (shared by the whole pool)
template:  node-h#l.pool-#-#.dynamic.nt-isp.net   (digit runs collapsed)

Aggregating by suffix and token over the full crawl paints a clear picture of what the IPv4 internet is made of. These are some of the most frequent tokens across all one billion hostnames:

The largest operators by PTR volume are a mix of US broadband, European telecoms and hyperscale clouds:

One counterintuitive insight: the 65.7% of hostnames that merely encode their own IP address are not useless. An auto-generated PTR is itself a strong signal that the address belongs to a mass-provisioned eyeball or cloud network rather than to individually managed infrastructure. We exploit this fact below.

From Hostnames to IP Intelligence

The processing pipeline classifies each hostname with two layers. The first layer is a curated set of provider suffixes: compute.amazonaws.com is AWS cloud, clients.your-server.de is Hetzner hosting, hsd1.*.comcast.net is Comcast residential, and so on. A known provider suffix is authoritative and overrides everything else. The second layer is a set of keyword rules for the long tail: vps, colo, dedicated, dhcp, pppoe, fiber, cable, lte, 5g, and dozens more.

The ordering matters more than it looks. Generic tokens like server, host or node appear constantly in residential ISP naming schemes. A hostname like static-server-3.dsl.someisp.net must not flip an ISP network into a datacenter verdict. Weak hosting keywords are therefore guarded: they only count when no ISP, mobile or dynamic signal is present on the same host. Getting this discipline wrong is how IP databases end up flagging entire broadband ranges as datacenters.

Classification covers 83% of all one billion PTR records. The result:

On top of the categories, the classifier extracts per-IP connection signals at a scale no manual process could reach: 199.3M dynamically assigned IPs, 83.4M static assignments, 149.2M datacenter IPs, 629.0M residential IPs. Per-IP results are aggregated to /24 networks, and a network only earns a verdict when enough of its sampled IPs agree. Single stray hostnames never label a whole subnet.

Keeping Score: How Much New Signal Did the Crawl Buy?

It is easy to generate impressive-sounding overlays; it is harder to know whether they add anything. So every processing run audits itself against the existing ipapi.is hosting database: of the 132.5 million hosting IPs detected from reverse DNS, 93.8% were already covered by our WHOIS-based hosting detection. This is reassuring cross-validation of two fully independent methods. Meanwhile, 8.2 million IPs and 47,721 complete /24 networks were genuinely new hosting coverage that WHOIS alone had missed. Those flow back into the is_datacenter field of the API.

VPN and Tor: Learning From Ground Truth, Not Grepping for "vpn"

The tempting shortcut is to flag every hostname containing vpn or tor. It is also how you build a false-positive machine: vpn.corp.example.com is an employee gateway, not an anonymity service, and cust-tor-98-49.isp.net is a customer in Toronto. We deliberately have no keyword rule for either.

Instead, rules are learned from ground truth. Our VPN enumeration pipeline continuously collects confirmed exit-node IPs from commercial VPN providers, and the Tor consensus publishes exit relays. The learner resolves the PTR records of those confirmed IPs, finds candidate rDNS patterns, and then asks the crucial question against the full crawl: of all IPs on the internet that carry this pattern, what fraction is confirmed VPN or Tor? That purity score is what separates a dedicated exit domain from shared infrastructure. A hosting provider whose network contains thousands of VPN exits alongside 159,000 unrelated servers scores a purity of 0.04 and is rejected; a dedicated exit domain scores near 1.0 and is promoted.

The honest summary of this technique: only about 12% of VPN exit IPs have a PTR record at all (Tor: about 45%), so reverse DNS can never be the primary VPN detection source. But as a precision layer it is excellent. The last run promoted 59 pattern rules with near-zero false-positive risk by construction and discovered 666 anonymizer IPs that the enumeration pipeline had not yet seen.

The Geography Hidden in Hostnames

The most fascinating use of reverse DNS is geolocation. Network operators name their equipment for their own troubleshooting convenience, and those names encode physical locations: IATA airport codes, city names, telephony CLLI codes, US state abbreviations, and countless in-house mnemonics.

This has been a research topic for over two decades. Manually maintained regex collections (GTrace, 1999) gave way to automated convention learning with DRoP (CAIDA, 2014), Microsoft's "IP Geolocation through Reverse DNS" (2021) applied machine learning to the problem, and CAIDA's Hoiho (CoNEXT 2021) learns per-operator regexes validated by round-trip-time measurements, and powers a public API. CAIDA's analysis of internet router names found no dominant convention: 54% of learned rules extract IATA codes, 38% place names, 11% CLLI codes and 1% UN/LOCODEs.

The newest entrant is The Aleph (Thiagarajan, Carisimo & Bustamante, ACM CoNEXT 2025), which uses large language models to generate the extraction regexes and hint dictionaries automatically. Applied to 1.16 billion PTR records, it mapped 19% of them to cities and found that 58% of network operators encode location hints in their naming schemes across more than 6,000 cities in over 200 countries. The direction is clear: hostname geolocation is moving from hand-curated dictionaries to learned, per-operator conventions at full internet scale.

Why This Is Harder Than It Looks

Geographic tokens are treacherous. ash is the IATA code for Nashua, New Hampshire, but operators use it for Ashburn, Virginia. tor is Toronto. sat is San Antonio's airport, but in a hostname it usually means satellite. And AT&T still labels hosts with conventions inherited from a merger twenty years ago (99-170-164-205.lightspeed.tukrga.sbcglobal.net: tukr + ga = Tucker, Georgia). Any naive dictionary lookup produces confident nonsense.

Our pipeline therefore never trusts a token globally. It learns conventions per provider suffix, and a candidate rule must survive a series of validation gates before promotion: the extracted location must be consistent across the suffix's IPs, must agree with the authoritative country of the address space, must dominate its ambiguous alternatives, and must be plausible against the observed geographic centroid of the network. In the July run, 20,808 rules were promoted, while 20,000 candidate rules were rejected by exactly these gates. The survivors:

A subtle detail: a state code like az is only believed under a suffix where the convention is proven. For comcast.net, at least six distinct US state codes must appear, each geographically consistent, before any of them is trusted. That is how the pipeline knows az means Arizona under comcast.net without assuming it means Arizona anywhere else.

At full scale, the promoted rules located 511 million IP addresses, or 49% of all PTR records. These aggregate into 473,840 city-level and 1.95 million country-level network verdicts. The entire mining pass over a billion records takes about 30 minutes on a single machine. The emitted networks look like this:

network,country,region,city,latitude,longitude,granularity,tier,weight,ip_count,rule
1.21.12.0/24,JP,Kyoto,Kyoto,35.02107,135.75385,city,city,0.55,254,vectant.ne.jp|kyoto

Corroborate, Never Override

How much should a hostname hint be trusted against our other geolocation sources? Our answer: structurally less. Every rDNS-derived row enters the geolocation merge with a low, tier-dependent weight between 0.25 and 0.60. This is always below our higher-confidence sources such as validated operator geofeeds. Reverse DNS corroborates and fills gaps; it never overrides a stronger source. The effect is deliberate:

• Where another geolocation source and the hostname agree, the independent corroboration raises the confidence score of the result.
• Where the only alternative is a country-level fallback, a "somewhere in Arizona" hint wins and beats a wild guess at the most populated city by hundreds of kilometers.
• Where the hostname disagrees with the winning source, the conflict is flagged: 1,292 rules currently point at networks where the hostname and our other sources disagree. Those disagreements are a review queue for stale data; reverse DNS does not need to be right about where, only to signal doubt.

What About IPv6? Limits and Feasibility

Everything above is IPv4. The obvious question is why we do not simply do the same for IPv6. The answer is a useful illustration of how different the two protocols are in practice.

IPv4 is a 32-bit space: 4.3 billion addresses, of which about 3.7 billion are routable. That is large, but it is enumerable: you can literally count through every one, which is exactly what our crawler does. IPv6 is a 128-bit space. A single /64 subnet (the smallest block typically assigned to one customer) contains 18.4 quintillion addresses, more than four billion times the entire IPv4 internet. Brute-forcing even one home network's subnet would take longer than the age of the universe. Linear enumeration is simply off the table, and with it every technique this article is built on.

There is, however, a clever way in. Because IPv6 reverse DNS also lives in a tree (ip6.arpa, one level per hex nibble, 32 levels deep), you can walk that tree instead of enumerating addresses. Under RFC 8020, a nameserver answering NXDOMAIN means "there is nothing here or anywhere below this point." So a crawler can start at a prefix, query one nibble at a time, and only descend into branches that return NOERROR, which prunes the impossible 2¹²⁸ space down to just the tiny inhabited slivers. This is the technique introduced by Peter van Dijk and demonstrated at scale in the PAM 2017 paper "Something From Nothing (There)", which harvested 5.8 million IPv6 addresses purely from DNS, and it is codified in RFC 7707 on IPv6 network reconnaissance.

So it is feasible, but with real limits that explain why we have not shipped it yet:

• It only finds what is in the reverse tree. Unlike IPv4, where we can ask about every address directly, tree-walking only discovers IPv6 addresses that have a PTR record. IPv6 reverse DNS is far less consistently populated than IPv4, so the result is a partial, biased sample. It is useful for infrastructure discovery, not a census.
• It depends on well-behaved nameservers. The pruning trick only works if servers correctly return NXDOMAIN per RFC 8020. Many do not, which forces the crawler to seed from BGP-announced prefixes and fall back to slower strategies.
• Dynamically generated zones are a trap. Some ISPs auto-generate a PTR for every address in a large block, so the tree looks infinitely populated. A walker has to detect these (probe a handful of random deep names; if they all "exist", the zone is synthetic) and skip them, or drown.
• The payoff is lower, for now. The IP intelligence value we extract (hosting detection, connection type and geolocation) is concentrated in the eyeball and cloud IPv4 space that still carries the bulk of client traffic. Per unit of engineering effort, IPv4 rDNS remains the higher-yield dataset in 2026.

There is also a shortcut that sidesteps tree-walking entirely: dual-stack bridging. A large fraction of our billion IPv4 hostnames forward-resolve to an AAAA record as well, which directly reveals the operator's IPv6 address for the same service. This provides a way to bootstrap IPv6 coverage from data we already have. IPv6 reverse crawling is firmly on the roadmap; it is a different crawler (a tree walk, not a linear sweep), and it will lean on both the ip6.arpa NXDOMAIN walk and dual-stack bridging rather than the brute force that makes the IPv4 sweep possible.

What We Have Not Built Yet: The rDNS Frontier

A complete, monthly, per-status PTR snapshot is a general-purpose instrument. These are the applications on our roadmap, including both designed features and open research:

1. Forward-confirmed crawler verification. Google and Bing officially document exactly one way to verify their crawlers: the IP's PTR must end in googlebot.com (or search.msn.com), and the hostname must forward-resolve back to the same IP. A full PTR sweep provides the reverse half for the entire address space for free. This is enough to both unmask fake Googlebots in stale bot lists and discover new crawler IPs before published ranges update.
2. A residential "innocence" veto. The single worst error an IP intelligence API can make is flagging a family broadband line as a proxy. With 629 million residential IPs identified via rDNS, high-confidence residential /24s can act as a veto layer that suppresses weak community proxy-list hits, while never overriding enumerated exit-node ground truth. False-positive reduction as a first-class feature.
3. Monthly diffs as an infrastructure change feed. Each release is a complete, self-contained snapshot, so diffing two months reveals the internet's plumbing in motion: new hosting ranges being deployed, ISPs renumbering, address space changing hands (increasingly interesting in the age of IPv4 leasing), and CGNAT rollouts. Sudden PTR churn in a network is also a promising abuse signal because fast-flux infrastructure behaves nothing like a sleepy broadband pool.
4. LLM-assisted convention mining. The Aleph demonstrated that LLMs can decode operator mnemonics that no dictionary contains. The experiment cost about $500 for 2,646 networks. The obvious synthesis: let an LLM propose per-operator extraction rules for the long tail, then subject every proposal to the same deterministic purity and consistency gates we already run. Hypothesis generation by language model, verification by measurement: hallucination-proof by construction.
5. A DNS hygiene observatory. We record servfail, timeout and empty-zone status for every address on the internet, every month. That is a longitudinal measurement of reverse-tree misconfiguration that research currently reconstructs with considerable effort. Since FCrDNS became a mail deliverability requirement, operators have a concrete reason to care about it.
6. Device and infrastructure exposure. Tokens like camera, cctv, nvr, router and voip already tag 35.8 million infrastructure hosts in our classification. A finer device-class taxonomy would map exposed IoT populations per network and feed risk scoring without sending a single probe packet.

The Dataset Is a Product

Everything described above is derived from one artifact, and that artifact is available to buy. The ipapi.is Reverse DNS Database ships the complete monthly sweep (all 3,702,258,432 query outcomes, all 1,039,377,899 hostnames, and every negative status) as a single ~1 GB download in the compact .rdnsz format, re-crawled from scratch and re-published every month. Unlike most reverse DNS datasets, it records the DNS outcome for every IPv4 address rather than only the ones that resolved, so you can diff two monthly releases to watch the internet's infrastructure change over time.

This is the same pipeline that improves ipapi.is, our IP intelligence API: the crawl feeds the is_datacenter, VPN, connection-type and geolocation fields you get back from a lookup at api.ipapi.is. In other words, one monthly crawl serves two purposes: it raises the quality of the ipapi.is API and is sold directly as the Reverse DNS Database for teams that want the raw PTR data for their own research, security and enrichment work.

Conclusion

In 2023 we asked whether crawling the reverse DNS of the entire IPv4 internet was feasible. In 2026 it is a monthly production job: three weeks of estimated effort became three days of actual runtime, 40 GB of raw text became one gigabyte of structured data, and a billion hostnames became measurable signals including 8.2 million newly discovered hosting IPs, purity-verified VPN and Tor rules, and half a billion addresses carrying a validated geographic hint.

The deeper lesson is about discipline rather than scale. Reverse DNS is operator-controlled, stale in places, and full of traps: tor means Toronto, ash means Ashburn, and server usually means a DSL line. Used as a sole oracle it produces confident nonsense. When used as an independent witness to corroborate, veto and fill gaps under strict validation gates, it is one of the highest-leverage datasets in IP intelligence. That is how we use it, and the numbers in every API response are better for it.

References

1. Dan, O., Parulkar, V., Davison, B. D.: IP Geolocation through Reverse DNS, ACM Transactions on Internet Technology, 2021.
2. Huffaker, B., Fomenkov, M., claffy, k.: DRoP: DNS-based Router Positioning, ACM SIGCOMM CCR, 2014.
3. Luckie, M., Huffaker, B., Marder, A., Bischof, Z., Fletcher, M., claffy, k.: Learning to Extract Geographic Information from Internet Router Hostnames, ACM CoNEXT, 2021. See also the Hoiho API.
4. Thiagarajan, K., Carisimo, E., Bustamante, F. E.: The Aleph: Decoding Geographic Information from DNS PTR Records Using Large Language Models, Proceedings of the ACM on Networking (CoNEXT), 2025.
5. Arouna, A., Livadariu, I., van Rijswijk-Deij, R., Jonker, M.: Advancing in Reverse: A Comprehensive Characterization of IN-ADDR.ARPA Deployment, ACM SIGCOMM CCR, 2023.
6. Fiebig, T., Borgolte, K., Hao, S., Kruegel, C., Vigna, G.: Something From Nothing (There): Collecting Global IPv6 Datasets From DNS, PAM, 2017.
7. RFC 8020, NXDOMAIN: There Really Is Nothing Underneath; RFC 7707: Network Reconnaissance in IPv6 Networks.
8. Google Search Central: Verifying Googlebot and Other Google Crawlers; Google Workspace: Email Sender Guidelines.