Update: Second IPv4 Reverse DNS Crawl

In July we published the first complete sweep of the IPv4 reverse DNS space: 3,702,258,432 routable addresses queried, 1,039,377,899 PTR records collected, and an honest DNS status recorded for every address that did not resolve. That article ended with a promise: the first sweep has to visit everything, but every pass after that does not, because the recorded statuses tell us exactly which addresses are worth asking again.

The second crawl is now complete, and it kept that promise. It re-queried only 1,747,686,124 addresses — 47% of the space — and it answers two questions the baseline could not: how many of the 561 million timeouts were real, and how much a billion PTR records actually change in one month. The result is 120,489,757 newly resolved hostnames, a measured monthly churn rate on the valid set, and an updated Reverse DNS Database that now contains 1,157,507,027 PTR records — +11.4% more than the July release.

The Update Pass: Re-Query Only What Can Change

The baseline sweep recorded one of eight DNS statuses for every routable IPv4 address. Those statuses split the address space into two very different halves. An authoritative nxdomain is a stable "no": the operator's nameserver positively asserted that no PTR record exists, and that answer only changes when someone provisions new naming — rare enough that re-asking 1.94 billion addresses every month is wasted work. The other half is where the information lives:

Skipped entirely: 1.94 billion nxdomain addresses and 18.7 million clean noerror_empty answers — definitive negatives from working infrastructure. Mechanically, each of the 20 crawler nodes receives the ~57 MB .rdnsz shard it produced in July and streams its own target list out of it: the compact observation format doubles as the crawl plan for the next pass, which is exactly why we store the status of every queried address rather than only the hits.

Patience Pays: Re-Tuning the Timeout Budget

The July sweep prioritized completing a full pass in days, so it gave each address at most 2 seconds and one retry. That was the right call for a first census — and it honestly labeled 15.2% of the space as timeout, an artifact of impatience as much as a statement about the address. For the update pass the economics flip: the target set is half the size, and recovering previously timed-out addresses is one of its main purposes. So the tuning changed:

The retries are cheaper than they look. Each node runs its own recursive unbound resolver, and a retry hits that resolver's already-warmed cache: attempt two does not start recursion from scratch, it continues roughly where attempt one gave up. Fifteen seconds of patience aligns with the point where unbound itself stops trying; beyond that, longer timeouts mostly park worker slots on dead address space. The fleet, the sharding (ip % 20) and the per-node concurrency stayed identical to July — the pass simply trades some speed for answers on the exact set of addresses where extra patience can pay off.

Recovered: What the 561 Million Timeouts Really Were

The most interesting question of this pass: when you re-ask 561 million previously-unanswered addresses with 15 seconds of patience instead of 4, what comes back? The answer, from the compare step that joins both passes record by record:

The headline: 118,776,810 addresses — 21.2% of everything that timed out in July — had a PTR record all along. The baseline's 4 seconds of patience were simply not enough for the nameservers responsible for them. Another 24.3% answered with an authoritative nxdomain once given time, so 46.0% of the re-queried timeout space now has a definitive answer instead of silence. Only 26.5% timed out again — the genuinely dead, firewalled or blackholed part of the reverse tree.

The largest single destination is an odd one at first glance: 27.6% moved from timeout to servfail. That movement is an artifact of patience, and it is worth understanding. With a 4-second budget, the client gives up while the local unbound is still walking a broken delegation — the observation is recorded as timeout. With 15 seconds, unbound has time to reach its own conclusion and return SERVFAIL: the delegation is actually broken. It is the same dead infrastructure in both passes — the update pass just labels it more precisely.

The July servfail set tells the mirror-image story: it is sticky. 88.6% of it failed the same way again — broken DNSSEC chains and lame delegations do not fix themselves in a month — and only 1.17% (1,712,947 addresses) came back with a hostname. A ~1% hit rate is a strong argument for re-visiting this bucket far less often than the rest, and that conclusion goes straight into the re-crawl scheduler (What Is Next).

Summed across all three re-crawled buckets, 120,489,757 addresses that had no hostname in the July dataset have one now, against 2,360,629 authoritative removals — a net gain of 118,129,128. The recovered names concentrate exactly where slow authoritative infrastructure lives: .net (37.9M), .com (16.8M), .jp (14.3M) and .it (10.4M) lead the TLD histogram of the gains — large carrier and telco zones whose nameservers answer reliably, just not quickly. 57.2% of the recovered hostnames also forward-confirm (FCrDNS): these are maintained, consistent DNS entries, not leftovers.

Churn: What a Billion PTR Records Do in One Month

The second purpose of the update pass is measurement. Reverse DNS churn rates are the key input for any intelligent re-crawl scheduler: how often must the valid set be refreshed before the data goes stale? Re-crawling all 1,039,377,899 July PTR records 27 days later gives the answer directly:

The reverse DNS space is slow. 98.09% of addresses that had a hostname in July still have one, and for 97.86% it is the exact same hostname. The number that matters for freshness is the second row: 0.23% of the valid set changed its name in 27 days — 0.0084% per day. Authoritative deletions are equally rare at 0.23%. At those rates a monthly update keeps well over 99% of the published hostnames current — and the average certainly hides structure: dynamic residential pools rename orders of magnitude faster than datacenter naming, which is precisely what per-zone volatility tracking (see below) will exploit.

Forward-confirmation moves on its own schedule, too. Of the 1.02B addresses whose PTR survived and where both passes performed the forward check, 60.0% were FCrDNS-confirmed in both passes, 0.38% lost the confirmation and 0.85% gained it — the forward zone drifts slowly and mostly independently of the reverse zone. For consumers of the fcrdns_match bit this means the flag is stable enough to build policy on.

Folding the Update In: Merge Semantics and the Grace Policy

An update pass produces a second observation log, not a new database. To publish an updated dataset, the re-crawl is folded into the baseline with a deterministic merge: for every address that was re-crawled, the newer observation wins; every address outside the target set keeps its baseline record untouched. One rule deserves explanation, because it is the difference between a maintained dataset and a naive snapshot:

Everything else is bookkeeping: duplicate observations from crawler restarts are deduplicated (first observation wins, matching the comparison semantics), and the merged shards inherit the same 20-way layout — so the merged output is itself a valid baseline for pass three. The merge of all 3.7 billion records runs in about half an hour on a single machine.

The Updated Database

After the merge, the Reverse DNS Database covers the same 3,702,258,432 routable IPv4 addresses with the following status distribution — shown next to the July baseline for comparison:

Three movements dominate the table. The PTR set grew by +118,129,128 to 1,157,507,027 hostnames — 31.26% of the routable space, up from 28.07% in July. The timeout bucket collapsed from 561.4M to 155.0M (4.19% of the space): what was the biggest pocket of uncertainty in the dataset is now a residual. And nxdomain grew by +146.2M — a sizeable share of the old timeout space turned out to be provably empty rather than merely silent. The growth of the servfail row is the re-labeling effect explained above: patience attributes dead delegations more precisely, it does not mean reverse DNS got more broken.

The updated database is live: the product page now also documents a crawl history with the summary statistics of every pass, so you can track how the dataset evolves release over release. Existing customers download the new package with their usual command; the format, the shard layout and the bundled rdnsz_query.js reader tool are unchanged.

What Is Next

This pass validated the core of the incremental architecture described in the crawler's DESIGN.md: status-targeted re-crawls plus a deterministic merge keep the dataset fresh at half the query cost of a full sweep. The measured numbers now feed the next steps:

1. Adaptive TTLs from measured churn. 0.23% of the valid set changed in a month — but that average hides structure. Dynamic residential pools churn orders of magnitude faster than datacenter naming. Per-zone volatility feedback (re-crawl what changed recently, back off what did not) is the difference between spending the query budget where data moves and spreading it uniformly.
2. Zone-aware probing. Reverse DNS is delegated per /24. A single SOA/NS probe can confirm that an entire 256-address block is undelegated, replacing 256 individual queries with one — the biggest remaining efficiency win, especially for the still-dead timeout space.
3. A master state database. With repeated passes, per-IP history becomes valuable: first seen, last seen, last changed. That state array is what turns monthly snapshots into a change feed — "which PTR records changed this month" as a queryable product.
4. The IPv6 reverse tree. Unchanged from the first article, and unchanged in priority: an ip6.arpa NXDOMAIN tree walk plus dual-stack bridging, not a linear sweep.

Conclusion

The second pass did what the incremental architecture promised. For 47% of the query cost of a full sweep it recovered 120,489,757 hostnames the first pass had missed, measured the real-world churn rate of a billion-record PTR dataset (0.0084% per day), and converted most of the timeout uncertainty into definitive answers. Just as importantly, it validated the operational loop — status-targeted re-crawl, deterministic merge with a grace policy, automated export — that turns a one-off measurement stunt into a maintained dataset. The next passes get cheaper and smarter from here.

The complete updated dataset — every hostname plus the exact DNS status of all 3.7 billion routable IPv4 addresses — is available as the ipapi.is Reverse DNS Database, and its signals flow directly into the ipapi.is IP intelligence API.