Understanding the deployment of public recursive resolvers
Monday, May 6th, 2024 by Matthew LuckieThis is the third in a series of essays (two earlier blog posts [1, 2]) about CAIDA’s new effort to reduce the barrier to performing a variety of Internet measurements.
We were recently asked about running Trufflehunter, which infers the usage properties of rare domain names on the Internet by cache snooping public recursive resolvers, on Ark. The basic idea of Trufflehunter is to provide a lower bound of the use of a domain name by sampling caches of large recursive resolvers. One component of Trufflehunter is to identify the anycast instances that would answer a given vantage point’s DNS queries. Trufflehunter uses a series of TXT queries to obtain the anycast instance of a given public recursive resolver, and considers four large public recursive resolvers: Cloudflare (1.1.1.1), Google (8.8.8.8), Quad9 (9.9.9.9), and OpenDNS (208.67.220.220). The original paper describes the queries in section 4.1, which we summarize below, highlighting the interesting parts of each response with blue color.
Cloudflare returns an airport code representing the anycast deployment location used by the VP:
$ host -c ch -t txt id.server 1.1.1.1
id.server descriptive text "AKL"
Google returns an IP address representing the anycast deployment used by the VP, which can be mapped to an anycast deployment location with a second query:
$ host -t txt o-o.myaddr.l.google.com 8.8.8.8
o-o.myaddr.l.google.com descriptive text "172.253.218.133"
o-o.myaddr.l.google.com descriptive text "edns0-client-subnet <redacted>/24"
$ host -t txt locations.publicdns.goog
locations.publicdns.goog descriptive text "34.64.0.0/24 icn " ... "172.253.218.128/26 syd " "172.253.218.192/26 cbf " ...
Quad9 returns the hostname representing the resolver that provides the answer:
$ host -c ch -t txt id.server 9.9.9.9
id.server descriptive text "res100.akl.rrdns.pch.net"
and finally, OpenDNS returns a bunch of information in a debugging query, which includes the server that handles the query:
$ host -t txt debug.opendns.com 208.67.220.220
debug.opendns.com descriptive text "server r2004.syd"
debug.opendns.com descriptive text "flags 20040020 0 70 400180000000000000000007950800000000000000"
debug.opendns.com descriptive text "originid 0"
debug.opendns.com descriptive text "orgflags 2000000"
debug.opendns.com descriptive text "actype 0"
debug.opendns.com descriptive text "source <redacted>:42845"
Each of these queries requires a slightly different approach to extract the location of the anycast instance. Our suggestion is for experimenters to use our newly created python library, which provides programmatic access to measurement capabilities of Ark VPs (described in two earlier blog posts [1, 2]). The code for querying the recursive resolvers from all VPs is straight forward, and is shown below. First, on lines 15-27, we get the Google mapping, so that we can translate the address returned to an anycast location. This query requires TCP to complete, as the entry is larger than can fit in a UDP payload (the response is 9332 bytes at the time of writing this blog). This query is synchronous (we wait for the answer before continuing, because the google queries that follow depend on the mapping) and is issued using a randomly selected Ark VP. Then, on lines 29-38, we issue the four queries to each of the anycasted recursive resolvers from each VP. These queries are asynchronous; we receive the answers as each Ark VP obtains a response. On lines 40-82, we process the responses, storing the results in a multi-dimensional python dictionary, which associates each Ark VPs with their anycast recursive resolver location, as well as the RTT between asking the query and obtaining the response. Finally, on lines 84-96, we dump the results out in a nicely formatted table that allows us to spot interesting patterns.
01 import argparse
02 import datetime
03 import ipaddress
04 import random
05 import re
06 from scamper import ScamperCtrl
07
08 def _main():
09 parser = argparse.ArgumentParser(description='get public recursive locs')
10 parser.add_argument('sockets')
11 args = parser.parse_args()
12
13 ctrl = ScamperCtrl(remote_dir=args.sockets)
14
15 # pick an ark VP at random to issue the query that gets the mapping
16 # of google recursive IP to location
17 goog_nets = {}
18 obj = ctrl.do_dns('locations.publicdns.goog',
19 inst=random.choice(ctrl.instances()),
20 qtype='txt', tcp=True, sync=True)
21 if obj is None or len(obj.ans_txts()) == 0:
22 print("could not get google mapping")
23 return
24 for rr in obj.ans_txts():
25 for txt in rr.txt:
26 net, loc = txt.split()
27 goog_nets[ipaddress.ip_network(net)] = loc
28
29 # issue the magic queries to get the instance that answers the query
30 for inst in ctrl.instances():
31 ctrl.do_dns('o-o.myaddr.l.google.com', server='8.8.8.8', qtype='txt',
32 attempts=2, wait_timeout=2, inst=inst)
33 ctrl.do_dns('id.server', server='1.1.1.1', qclass='ch', qtype='txt',
34 attempts=2, wait_timeout=2, inst=inst)
35 ctrl.do_dns('id.server', server='9.9.9.9', qclass='ch', qtype='txt',
36 attempts=2, wait_timeout=2, inst=inst)
37 ctrl.do_dns('debug.opendns.com', server='208.67.220.220', qtype='txt',
38 attempts=2, wait_timeout=2, inst=inst)
39
40 # collect the data
41 data = {}
42 for obj in ctrl.responses(timeout=datetime.timedelta(seconds=10)):
43 vp = obj.inst.name
44 if vp not in data:
45 data[vp] = {}
46 dst = str(obj.dst)
47 if dst in data[vp]:
48 continue
49 data[vp][dst] = {}
50 data[vp][dst]['rtt'] = obj.rtt
51
52 for rr in obj.ans_txts():
53 for txt in rr.txt:
54 if dst == '8.8.8.8':
55 # google reports an IPv4 address that represents
56 # the site that answers the query. we then map
57 # that address to a location using the mapping
58 # returned by the locations TCP query.
59 try:
60 addr = ipaddress.ip_address(txt)
61 except ValueError:
62 continue
63 for net, loc in goog_nets.items():
64 if addr in net:
65 data[vp][dst]['loc'] = loc
66 break
67 elif dst == '1.1.1.1':
68 # Cloudflare replies with a single TXT record
69 # containing an airport code
70 data[vp][dst]['loc'] = txt
71 elif dst == '9.9.9.9':
72 # Quad9 reports a hostname with an embedded
73 # airport code.
74 match = re.search("\\.(.+?)\\.rrdns\\.pch\\.net", txt)
75 if match:
76 data[vp][dst]['loc'] = match.group(1)
77 elif dst == '208.67.220.220':
78 # opendns reports multiple TXT records; we want the one
79 # that looks like "server r2005.syd"
80 match = re.search("^server .+\\.(.+?)$", txt)
81 if match:
82 data[vp][dst]['loc'] = match.group(1)
83
84 # format the output
85 print("{:15} {:>13} {:>13} {:>13} {:>13}".format(
86 "# vp", "google", "couldflare", "quad9", "opendns"))
87 for vp, recs in sorted(data.items()):
88 line = f"{vp:15}"
89 for rec in ('8.8.8.8', '1.1.1.1', '9.9.9.9', '208.67.220.220'):
90 cell = ""
91 if 'loc' in recs[rec]:
92 rtt = recs[rec]['rtt']
93 loc = recs[rec]['loc']
94 cell = f"{loc} {rtt.total_seconds()*1000:5.1f}"
95 line += f" {cell:>13}"
96 print(line)
97
98 if __name__ == "__main__":
99 _main()
The code runs quickly — no longer than 10 seconds (if one of the VPs is slow to report back), illustrating the capabilities of the Ark platform.
The output of running this program is shown below. We have highlighted cells where the RTT was at least 50ms larger than the minimum RTT to any of the large recursive resolvers for the given VP. These cells identify low-hanging fruit for operators, who could examine BGP routing policies with the goal of selecting better alternative paths.
# vp google couldflare quad9 opendns
abz-uk.ark lhr 24.4 LHR 16.5 lhr 16.9 lon 16.4
abz2-uk.ark lhr 23.4 MAN 8.7 man 8.4 man1 8.5
acc-gh.ark jnb 246.7 JNB 167.3 acc 0.5 cpt1 236.6
adl-au.ark mel 28.1 ADL 5.5 syd 22.6 mel1 14.0
aep-ar.ark scl 29.4 EZE 8.0 qaep2 4.6 sao1 34.0
aep2-ar.ark scl 38.2 EZE 6.6 qaep2 5.7 sao1 42.5
akl-nz.ark syd 26.9 AKL 2.6 akl2 4.0 syd 26.1
akl2-nz.ark syd 29.2 AKL 4.9 akl2 4.0 syd 26.9
ams-gc.ark grq 5.0 FRA 6.3 ams 0.8 ams 1.1
ams3-nl.ark grq 6.0 AMS 2.1 ams 1.3 ams 1.9
ams5-nl.ark grq 5.1 AMS 1.2 ams 1.9 ams 1.7
ams7-nl.ark grq 6.0 AMS 2.4 ams 1.8 ams 2.3
ams8-nl.ark grq 14.8 AMS 11.5 fra 17.7 ams 11.8
arn-se.ark lpp 9.7 ARN 2.0 arn 0.8 cph1 10.5
asu-py.ark scl 55.4 EZE 22.6 asu 0.8 sao1 57.0
atl2-us.ark atl 9.8 DFW 23.6 qiad3 20.5 atl1 4.6
atl3-us.ark atl 22.7 ATL 13.1 atl 17.0 atl1 19.7
aus-us.ark dfw 80.7 DFW 56.6 dfw 46.6 dfw 17.6
avv-au.ark mel 142.3 MEL 9.4 syd 22.6 mel1 8.5
bcn-es.ark mad 38.0 BCN 13.3 bcn 0.5 mad1 9.8
bdl-us.ark iad 13.3 EWR 5.0 lga 4.9 ash 10.6
bed-us.ark iad 30.6 BOS 15.2 bos 13.5 bos1 19.8
beg-rs.ark mil 62.5 BEG 1.2 beg 0.7 otp1 12.9
bfi-us.ark dls 11.7 SEA 3.8 xsjc1 96.9 sea 10.5
bjl-gm.ark bru 87.5 CDG 71.1 bjl 6.7 cdg1 73.4
bna-us.ark atl 11.4 BNA 9.0 atl 9.5 atl1 9.3
bna2-us.ark cbf 25.5 ORD 19.7 ord 12.8 chi 12.8
bos6-us.ark iad 17.6 BOS 5.6 qiad3 15.5 bos1 5.7
bre-de.ark grq 21.4 TXL 23.1 ber 17.2 ams 23.7
btr-us.ark dfw 20.3 ORD 29.5 dfw 13.2 dfw 11.4
bwi2-us.ark iad 12.7 IAD 4.4 qiad3 3.6 rst1 3.1
cdg-fr.ark mil 24.9 MRS 1.5 mrs 1.0 mrs1 11.6
cdg3-fr.ark bru 8.5 CDG 3.9 ams 13.7 cdg1 5.5
cgs-us.ark iad 4.5 EWR 7.9 iad 2.2 ash 2.1
cjj-kr.ark hkg 63.4 qhnd2 70.7 hkg 47.3
cld4-us.ark lax 22.7 LAX 15.7 bur 104.1 lax 14.3
cld5-us.ark lax 52.6 LAX 20.7 bur 103.1 lax 21.9
cld6-us.ark lax 14.5 LAX 5.2 qlax1 7.0 lax 6.0
cos-us.ark cbf 22.9 DEN 7.3 ord 36.0 den1 12.6
dar-tz.ark jnb 224.6 NBO 14.0 dar 1.0 jnb 51.1
dar2-tz.ark jnb 223.0 DAR 0.9 dar 0.9 jnb 48.0
dmk-th.ark sin 32.2 SIN 28.0 bkk 2.4 nrt2 95.0
dtw2-us.ark cbf 18.1 DTW 4.6 dtw 4.2 chi 10.7
dub-ie.ark lhr 18.9 DUB 1.2 dub 0.7 dub1 0.7
dub2-ie.ark lhr 18.7 DUB 1.8 dub 1.1 dub1 1.2
dub3-ie.ark lhr 35.5 ZRH 45.6 dub 16.9 dub1 11.6
ens-nl.ark grq 9.3 AMS 5.2 ams 4.0 ams 4.9
eug-us.ark dls 15.7 SJC 18.3 sea 6.5 sea 6.5
fra-gc.ark fra 9.1 FRA 1.1 fra 1.1 fra 1.1
gig-br.ark gru 125.2 GIG 2.2 qrio1 1.4 rio1 1.1
gva-ch.ark ZRH 5.3 gva 1.0 fra 10.8
gye-ec.ark chs 128.7 MIA 66.8 quio2 10.7 mia 77.1
ham-de.ark grq 19.7 HAM 8.5 ber 11.0 fra 17.7
her2-gr.ark mil 64.9 ath 7.5 mil1 32.6
hkg4-cn.ark tpe 13.6 HKG 1.0 qhkg3 4.9 hkg 9.5
hkg5-cn.ark hkg 16.3 HKG 2.6 qhkg3 3.9 hkg 3.6
hlz2-nz.ark syd 40.9 AKL 16.9 akl 13.8 syd 38.2
hnd-jp.ark nrt 6.5 NRT 3.4 qhnd2 80.3 nrt 3.6
hnl-us.ark dls 82.3 HNL 1.1 sea 75.2 sea 75.3
iev-ua.ark waw 45.5 FRA 29.8 qwaw2 14.3 wrw1 14.9
igx2-us.ark iad 16.1 EWR 16.9 qiad3 11.5 atl1 11.8
ind-us.ark atl 25.1 IND 1.1 ord 8.5 atl1 11.0
ixc-in.ark del 78.8 DEL 0.8 qsin1 0.7 mum2 0.7
jfk-us.ark iad 9.6 EWR 1.5 lga 0.6 nyc 1.0
kgl-rw.ark jnb 231.6 NBO 15.0 kgl 3.3 jnb 73.6
ktm-np.ark del 98.6 DEL 30.4 ktm 7.6 mum1 94.4
las-us.ark lax 15.5 LAX 8.4 pao 15.9 lax 8.2
lax3-us.ark lax 10.5 LAX 3.0 bur 2.0 lax 2.0
lcy2-uk.ark lhr 11.5 MAN 7.4 lhr 11.2 lon 2.2
lej-de.ark fra 21.2 FRA 9.7 fra 12.5 fra 9.6
lex-us.ark iad 17.9 IAD 15.7 iad 17.3 atl1 25.9
lgw-uk.ark lhr 31.3 LHR 24.2 lhr 24.6 lon 23.6
lhe2-pk.ark dia 195.0 KHI 35.0 qsin4 113.1 sin 130.6
lis-pt.ark mad 33.0 LIS 4.5 lis 3.9 mad1 18.0
lke2-us.ark dls 21.7 SEA 21.4 sea 15.1 sea 11.7
lun-zm.ark jnb 189.9 JNB 32.2 jnb 23.3 jnb 23.3
lwc-us.ark tul 18.0 MCI 1.4 xsjc1 106.3 dfw 10.4
lwc2-us.ark dfw 24.4 DFW 15.7 qlax1 46.9 dfw 16.5
mdw-us.ark cbf 25.7 ORD 7.0 ord 3.4 chi 4.4
med2-co.ark mrn 96.4 MIA 50.7 qbog1 22.7 mia 48.9
mhg-de.ark fra 30.7 FRA 19.1 fra 21.4 fra 18.4
mia-gc.ark mrn 19.7 MIA 1.1 mia 1.0 mia 0.6
mnl-ph.ark hkg 87.9 MNL 1.8 qsin1 50.8 hkg 75.8
mnz-us.ark iad 13.4 IAD 5.8 qiad3 5.2 rst1 4.0
mru-mu.ark jnb 206.1 JNB 43.1 mru 1.0 jnb 43.3
msy-us.ark dfw 34.4 DFW 21.9 qlax1 55.2 dfw 23.0
mty-mx.ark tul 44.1 MFE 4.5 qiad3 39.6 dfw 16.7
muc-de.ark fra 30.3 FRA 9.1 fra 8.8 fra 8.9
muc3-de.ark zrh 25.9 MUC 6.1 fra 11.0 fra 12.2
nap2-it.ark mil 37.8 MXP 19.8 fra 38.0 mil1 15.2
nbo-ke.ark jnb 220.5 NBO 2.4 nbo 2.2 jnb 60.7
nic-cy.ark mil 61.3 LCA 1.0 mrs 35.2 mrs1 35.4
nrn-nl.ark grq 8.6 AMS 5.5 ams 2.9 ams 3.1
nrt-jp.ark nrt 4.8 NRT 1.2 hkg 101.9 nrt2 1.0
nrt3-jp.ark nrt 6.5 NRT 3.5 pao 101.7 nrt2 3.5
oak5-us.ark lax 19.3 SJC 5.9 sjc0 4.4 pao 4.4
okc-us.ark dfw 9.6 MCI 9.6 dal 8.2 dfw 7.3
ord-us.ark cbf 13.0 ORD 2.9 iad 19.0 chi 1.6
ory4-fr.ark bru 6.5 CDG 2.0 cdg 1.4 cdg1 1.7
ory6-fr.ark bru 6.4 LHR 8.8 lhr 8.4 lon 8.6
ory7-fr.ark bru 7.7 CDG 4.0 cdg 5.0 cdg1 4.0
ory8-fr.ark bru 6.3 CDG 2.0 lhr 25.7 lon 9.4
osl-no.ark lpp 46.5 OSL 0.9 osl 25.8 sto1 9.0
pbh2-bt.ark bom 105.2 MAA 92.0 pbh 1.8 sin 91.7
per-au.ark syd 48.9 PER 2.0 per 1.2 syd 45.3
per2-au.ark syd 140.0 PER 2.5 per 1.5 syd 44.9
phl-us.ark iad 15.9 iad 14.0 rst1 15.9
pna-es.ark mad 33.9 BCN 13.5 bcn 13.3 mad1 19.3
prg-cz.ark fra 16.9 PRG 1.1 qbts1 5.9 prg1 0.6
prg2-cz.ark fra 29.8 PRG 1.7 qfra3 14.0 prg1 1.5
pry-za.ark jnb 167.3 JNB 1.2 jnb 0.6 cpt1 18.4
puw-ru.ark lpp 18.2 DME 1.5 beg 67.1 fra 34.3
pvu-us.ark lax 25.2 SLC 3.8 slc 2.7 sea 18.4
rdu-us.ark iad 9.5 IAD 8.1 iad 7.7 ash 8.6
rdu2-us.ark iad 11.3 IAD 8.1 iad 8.5 ash 8.6
rdu3-us.ark iad 27.5 IAD 18.6 iad 18.4 atl1 23.8
rkv-is.ark lhr 49.0 KEF 2.0 kef 0.8 dub1 23.0
san-us.ark lax 10.4 LAX 4.7 bur 87.2 lax 3.1
san2-us.ark lax 39.0 LAX 6.5 qlax1 7.5 lax 5.6
san4-us.ark lax 26.6 LAX 20.2 bur 106.8 lax 17.3
sao-br.ark gru 117.3 GRU 2.5 qgru1 1.3 rio1 9.6
scq-es.ark mad 31.7 MAD 11.0 mad 10.8 mad1 11.3
sea3-us.ark dls 10.1 SEA 3.9 pao 24.6 sea 3.0
sin-gc.ark sin 4.0 SIN 1.5 qsin1 93.5 sin 1.1
sin-sg.ark sin 3.2 SIN 2.3 qsin1 12.2 sin 0.7
sjc2-us.ark lax 15.2 SJC 1.4 sjc0 0.5 pao 2.1
sjj-ba.ark waw 71.1 BUD 57.5 vie 213.3 mil1 89.4
sjo-cr.ark chs 63.0 SJO 2.2 mia 51.8 mia 78.6
snn-ie.ark lhr 20.4 DUB 4.4 dub 3.9 dub1 3.9
sql-us.ark lax 19.6 SJC 2.2 pao 1.2 pao 0.9
stx-vi.ark chs 38.3 MIA 26.4 mia 25.5 mia 25.1
svo2-ru.ark lpp 22.4 DME 5.8 fra 41.1 fra 38.2
swu-kr.ark hkg 77.6 ICN 6.5 qsin1 76.9 nrt2 38.5
syd3-au.ark syd 4.8 SYD 0.9 syd 1.0 syd 0.5
tij-mx.ark lax 15.8 LAX 6.6 qlax1 6.6 lax 5.6
tlv-il.ark mil 80.2 MRS 43.6 tlv 2.0 mil1 50.5
tlv3-il.ark mil 64.2 TLV 2.1 tlv 2.1 tlv1 3.0
tnr-mg.ark JNB 215.7 jnb 199.1
tpe-tw.ark tpe 10.3 TPE 5.2 tpe 3.1 sin 74.4
vdp-dk.ark lpp 21.9 CPH 5.6 arn 13.4 cph1 5.4
vie-at.ark fra 22.5 FRA 13.8 vie 2.1 prg1 6.5
waw-pl.ark waw 20.1 HAM 31.3 qwaw2 1.3 wrw1 0.9
wbu-us.ark cbf 13.4 DEN 2.7 den 1.8 den1 1.5
wlg2-nz.ark syd 35.3 AKL 17.0 akl2 16.5 syd 32.4
ygk-ca.ark yyz 29.5 YYZ 16.7 iad 35.5 yyz 16.1
yyc-ca.ark dls 26.5 YYC 4.6 sea 21.8 yvr 16.7
zrh-ch.ark zrh 14.5 ZRH 1.1 zrh2 0.6 mil1 8.1
zrh2-ch.ark zrh 16.2 ZRH 1.1 zrh2 1.0 fra 6.8
zrh4-ch.ark zrh 15.8 ZRH 1.5 zrh2 0.8 mil1 9.0