In the first part of our blog series, we introduced our brand-new python module for scamper, the packet-prober underpinning much of Ark’s ongoing measurements. One aspect that we highlighted was the ability for potential users of Ark to develop their code locally, before running it on the Ark platform. When I develop measurement applications, I use a couple of local Raspberry Pis and my own workstation to get the code correct, and then copy the code to the CAIDA system to run the experiment using available Ark vantage points. The goal of this blog article is to describe different ways to locally develop your measurement experiment code.

Example #1: Starting small with one scamper process.

The easiest way to begin is with one scamper process running on the same system where you develop your python code. With scamper installed (we recommend that you use a package listed on the scamper website), start a scamper process, and make it available for measurement commands on a Unix domain socket. For example, you might run scamper as follows:

$ scamper -U /tmp/scamper -p 100

This will create a Unix domain socket to drive scamper at /tmp/scamper, and tell scamper that it can probe at up to 100 packets/second. You can adjust these parameters to what is appropriate locally.

You can then develop and debug your measurement code in Python. To use this scamper process, your Python code might begin as follows:

01 from scamper import ScamperCtrl
02
03 # use the scamper process available at /tmp/scamper
04 ctrl = ScamperCtrl(unix="/tmp/scamper")
05
06 # do a simple ping to 8.8.8.8 and print the outcome
07 o = ctrl.do_ping("8.8.8.8")
08 if o.min_rtt is not None:
09   print(f"{o.min_rtt.total_seconds()*1000):.1f} ms")
10 else:
11   print("no reply")

Example #2: Coordinating measurements among VPs.

Once you are comfortable using the python module with a single local scamper instance, you might want to test your code with multiple scamper instances, each representing a distinct vantage point. The scamper software includes the sc_remoted interface to support that. sc_remoted has features for authenticating endpoints with TLS, but you might choose to initially operate endpoints without the complexity of TLS.

sc_remoted listens on a port for inbound scamper connections, and makes Unix domain sockets — one for each VP — available in a nominated directory. The best idea is to create an empty directory for these sockets. You might run sc_remoted as follows:

$ mkdir -p /path/to/remote-sockets
$ sc_remoted -U /path/to/remote-sockets -P 50265


The first command creates the directory, because sc_remoted will not create that directory for you. The second command starts sc_remoted listening on port 50265 for incoming scamper connections, and will place Unix domain sockets in /path/to/remote-sockets as they arrive.  Note, we use /path/to as a placeholder to the actual path in your local file system that is appropriate for your environment; you might put these sockets in a directory somewhere in your home directory, for example.

Then, on the systems that you want to act as vantage points, the following command:

$ scamper -p 100 -R 192.0.2.28:50265 -M foo.bar

will (1) start a scamper process, (2) tell it that it can probe at up to 100 packets-per-second, (3) connect it to the specified IP address and port to receive measurement commands from, and (4) tell it to identify itself as “foo.bar” to the remote controller. If you go into /path/to/remote-sockets, you might see the following:

$ cd /path/to/remote-sockets
$ ls -l
total 0
srwx------ 1 mjl mjl 0 Jan 22 16:57 foo.bar-192.0.2.120:12369

This socket represents the scamper process you just started. The filename begins with foo.bar, the parameter that you gave to scamper to identify itself. After the dash is the IP address and port number that the remote controller observed the remote system coming from. You can connect as many additional scamper instances as you like, and you will see them listed in the directory individually. You should name each differently with something meaningful to you (foo.bar, bar.baz, etc) so that you can identify them on the system on which you’re writing your python code.

The python code we wrote in Example #1 above might be modified as follows:

01 import sys
02 from scamper import ScamperCtrl
03
04 if len(sys.argv) != 2:
05   print("specify path to unix domain socket")
06   sys.exit(-1)
07
08 # use the remote scamper process available at the specified location
09 ctrl = ScamperCtrl(remote=sys.argv[1])
10
11 # do a simple ping to 8.8.8.8 and print the outcome
12 o = ctrl.do_ping("8.8.8.8")
13 if o.min_rtt is not None:
14   print(f"{o.min_rtt.total_seconds()*1000):.1f} ms")
15 else:
16   print("no reply")

And run as:

$ python ping.py /path/to/remote-sockets/foo.bar-192.0.2.120\:12369

If you have multiple remote-sockets in the directory, you can add them individually, or use all sockets in the directory. For example:

01 import sys
02 from datetime import timedelta
03 from scamper import ScamperCtrl
04
05 if len(sys.argv) != 3:
06   print("usage: single-radius.py $dir $ip")
07   sys.exit(-1)
08
09 ctrl = ScamperCtrl(remote_dir=sys.argv[1])
10 for i in ctrl.instances():
11   ctrl.do_ping(sys.argv[2], inst=i)
12
13 min_rtt = None
14 min_vp = None
15 for o in ctrl.responses(timeout=timedelta(seconds=10)):
16   if o.min_rtt is not None and (min_rtt is None or min_rtt > o.min_rtt):
17     min_rtt = o.min_rtt
18     min_vp = o.inst
19
20 if min_rtt is not None:
21   print(f"{min_vp.name} {(min_rtt.total_seconds()*1000):.1f} ms")
22 else:
23   print(f"no responses for {sys.argv[2]}")

and run this command:

$ python single-radius.py /path/to/remote-sockets 8.8.8.8

We encourage you to reach out via email if you have questions about using the module. In the first instance, you can email ark-info at caida.org.

This is the first in a series of essays about CAIDA’s new effort to reduce the barrier to performing a variety of Internet measurements.

Network operators and researchers often require the ability to conduct active measurements of networks from a specific location in order to understand some property of the network. However, obtaining access to a vantage point at a given location is challenging, as there can be significant trust barriers that prevent access: a platform operator has to provide vantage point hosts with guarantees about the activity that their vantage points will exhibit, and a platform operator has to trust that users will stay within those guarantees. Current access control to active measurement infrastructure has two extremes: access that allows for trusted users to run arbitrary code, and API access that allows arbitrary users to schedule a (limited) set of measurements, and obtain their results. Prior research thrusts in active measurement design have focused on interfaces that allow a user to request a host to send arbitrary packets, leaving the implementation of the measurement to the user. However, this design pattern limits the guarantees that a platform operator can provide a vantage point host regarding what their vantage point will actually do. A domain-specific language for conducting active measurements can alleviate these concerns because it (1) allows a platform operator to specify the measurements that a user can run, and communicate to the host what their vantage point will do, (2) provides users reference implementations of measurement applications that act as building blocks to more complex measurements.

Over the past six months, in consultation with members of the active measurement community, CAIDA has been working towards a next-generation measurement infrastructure, built on the existing Archipelago (Ark) platform. One aspect of this platform is the notion of a researcher development environment that allows for complex, distributed, and reactive measurements built on a well-defined set of measurement primitives. In an effort to make the Ark platform easier to use for measurement researchers, while also providing important access control, CAIDA has developed the scamper python module that provides an interface to the measurement primitives available on each of the Ark nodes. Today, we are releasing the source code of that module, so that researchers can develop and test complex measurements locally, before running vetted experiments on Ark.

The architecture of scamper. Measurement tasks are supplied from one or more input sources, including from an input file, from the command line, or from a control socket.

The module provides user-friendly interfaces to existing scamper functionality. We illustrate the module with some examples.

Example #1: Implementation of RTT-based geolocation measurement

The following is an implementation of the well-known single-radius measurement, which conducts delay measurements to an IP address from a distributed set of vantage points, and reports the shortest of all the RTTs obtained with the name of the monitor, which on Ark, is derived from its location (e.g., lax-us, hlz2-nz, ams-nl, and so on). Researchers and operators might use this technique to understand approximately where a system is located, using the RTT constraint of the vantage point that reports the shortest delay.

01 import sys
02 from datetime import timedelta
03 from scamper import ScamperCtrl
04
05 if len(sys.argv) != 3:
06   print("usage: single-radius.py $dir $ip")
07   sys.exit(-1)
08
09 ctrl = ScamperCtrl(remote_dir=sys.argv[1])
10 for i in ctrl.instances():
11   ctrl.do_ping(sys.argv[2], inst=i)
12
13 min_rtt = None
14 min_vp = None
15 for o in ctrl.responses(timeout=timedelta(seconds=10)):
16   if o.min_rtt is not None and (min_rtt is None or min_rtt > o.min_rtt):
17     min_rtt = o.min_rtt
18     min_vp = o.inst
19
20 if min_rtt is not None:
21   print(f"{min_vp.name} {(min_rtt.total_seconds()*1000):.1f} ms")
22 else:
23   print(f"no responses for {sys.argv[2]}")

This implementation takes two parameters (lines 5-7) — a directory that contains a set of unix domain sockets, each of which represents an interface to a single vantage point, and an IP address to probe. On line 9, we open an interface (represented by a ScamperCtrl object) that contains all of the vantage points in that directory. We then send a ping measurement to each of the vantage point instances (lines 10-11). These ping measurements operate in parallel — the ping measurements on each of the nodes operate asynchronously. We then collect the results of the measurements (lines 13-18), noting the minimum observed RTT, and the vantage point where it came from. We pass a 10-second timeout on line 15, so that a vantage point that experiences an outage after we send the measurements does not hold up the whole experiment. Finally, on lines 20-23, we print the result of the measurement.

Example #2: RTTs to authoritative name servers of a specific domain

The next example shows a more complex measurement. Let’s say we want to know the RTTs to the authoritative name servers for a zone from a single vantage point.

01 import sys
02 from datetime import timedelta
03 from scamper import ScamperCtrl
04
05 if len(sys.argv) != 3:
06   print("usage: authns-delay.py $vp $zone")
07   sys.exit(-1)
08
09 ctrl = ScamperCtrl(remote=sys.argv[1])
10
11 # get the list of NS for the zone
12 o = ctrl.do_dns(sys.argv[2], qtype='NS', wait_timeout=1, sync=True)
13
14 # issue queries for the IP addresses of the authoritative servers
15 ns = {}
16 for rr in o.ans():
17   if rr.ns is not None and rr.ns not in ns:
18     ns[rr.ns] = 1
19     ctrl.do_dns(rr.ns, qtype='A', wait_timeout=1)
20     ctrl.do_dns(rr.ns, qtype='AAAA', wait_timeout=1)
21
22 # collect the unique addresses out of the address lookups
23 addr = {}
24 for o in ctrl.responses(timeout=timedelta(seconds=3)):
25   for a in o.ans_addrs():
26     addr[a] = o.qname
27
28 # collect RTTs for the unique IP addresses
29 for a in addr:
30   ctrl.do_ping(a)
31 for o in ctrl.responses(timeout=timedelta(seconds=10)):
32   print(f"{addr[o.dst]} {o.dst} " +
33         (f"{(o.min_rtt.total_seconds() * 1000):.1f}"
34          if o.min_rtt is not None else "???"))

This implementation takes two parameters (lines 5-7) — a single vantage point, and a zone name to study. As before, we open an interface to that VP on line 9, and then issue a DNS query for the authoritative name servers for the zone (line 12).

There are a couple of interesting things to note about line 12. First, we do not pass a handle representing the VP instance to the do_dns method, as the ScamperCtrl interface only has a single instance associated with it — it is smart enough to use that single instance for the measurement. Second, we pass sync=True to make the measurement synchronous — the method does not return until it has the results of that single measurement. This is shorter (in lines of code) and more readable than issuing an asynchronous measurement and then collecting the single result. Then, we issue asynchronous queries for the IPv4 and IPv6 addresses for the name servers returned (lines 14-20) and send ping measurements for each of the addresses (lines 29-30). Finally, we print the names of the nameservers, their IP addresses, and the minimum RTT observed to each.

The scamper python module supports most of the measurement primitives currently available in scamper: ping, traceroute, DNS query, alias resolution, HTTP, and simple UDP probes. We’ve used the module internally to: (1) reproduce the data collection methodology of a recent router fingerprinting method; (2) study the deployment of Netflix fast.com speed test endpoints, (3) implement MIDAR; (4) study anycast open resolvers, and (5) monitor serial number changes of zones amongst a set of nameservers authoritative for a zone. One important feature of our module is that it provides a python interface, which means that you can use our measurement module alongside existing python modules that parse JSON, etc. The documentation for the module is publicly available and we will write additional blog entries in the coming days that show more of its features, in a digestible form.

In the short term, measurement researchers can request access to the infrastructure to run vetted experiments by emailing ark-info at caida.org. Note: the access does not provide a login on any of the Ark nodes. Rather, it provides access to a system that can access the measurement primitives of the vantage points, as illustrated above. Again, we are releasing the python module to allow researchers to develop and debug experiments locally, before running them on Ark. We hope that this approach provides a convenient development lifecycle for researchers, as the CAIDA system you will have access to when running the experiments will not necessarily have the local development environment that you are accustomed to.

The module itself is written in cython, providing a wrapper around two C libraries in scamper that do much of the heavy lifting. You build and install the module using the instructions on the scamper website, install the module using the Ubuntu PPA (preferred if you are using Ubuntu), or install the module using one of the packages available on other operating systems as these become available.