Following Up On ‘A Day in the Life of the Internet’ Challenge
June 20th, 2007 by kc[okay, that took about four times as long as i’d hoped, but we’re done with a preliminary cataloging of the data collected for our “Day in the Life of the Internet” experiment for 2007. -k]
As a refresher, this is a follow-up to our last year’s announcement that we would try out this experiment recommended by a National Academy of Sciences workshop, specifically, to capture ‘a day in the life of the Internet’ (DITL) to support the needs of network research. We believe the research community now has more measurement data (indexed!) than ever before about a single day of the Internet, and while the data situation is still pretty bleak, a little data is better than even less. In terms of measurements executed, we did significantly better than our practice DITL run in 2006, so there is cause for optimism about the future of this kind of experiment. As the summary makes clear, this year’s progress was mostly due to contributions from outside the U.S., in particular from Korea and Japan, countries which have generally more successfully navigated data sharing issues for their research communities than the U.S. has. We are sorry to say we did not index a single trace from a commercial provider link this year, although we were pleased to get participation from 5 of the 13 root nameserver anycast cluster operators, up from 3 last year.
Access to the data collected during the DITL experiments varies based upon the policies of each collecting organization and the intended use of the data. We have indexed the datasets from participating DITL data providers in CAIDA’s DatCat. We recognize that data ownership policy issues are one of the biggest problems in the Internet research community, and we encourage feedback by email on any problems researchers have obtaining specific data sets, with as much detail as possible to help us improve the utility of this experiment in the future.
We should note that in the ten years that CAIDA has tried to tackle the measurement problem, the biggest obstacles have consistently been:
- economic (cost of instrumentation and data management)
- ownership (legal access to data), and
- trust (privacy and security obstacles to measurement).
It has become clear that if we do not find ways to tackle the economic and policy problems of Internet measurement, the incentive for technology investment in Internet research will weaken, as will our understanding of the Internet.