In late April 2019, social media was reportedly blocked and access to the Internet was shutdown in Benin during its 2019 parliamentary elections.

In this report, the Open Observatory of Network Interference (OONI) and the Center for Applied Internet Data Analysis (CAIDA) teams share OONI, IODA, and RIPE Atlas network measurement data that corroborate and provide insight into these recent censorship events in Benin.
Read the rest of this entry »

The CAIDA annual report summarizes CAIDA’s activities for 2018, in the areas of research, infrastructure, data collection and analysis. Our research projects span Internet topology, routing, security, economics, future Internet architectures, and policy. Our infrastructure, software development, and data sharing activities support measurement-based internet research, both at CAIDA and around the world, with focus on the health and integrity of the global Internet ecosystem. The executive summary is excerpted below:
Read the rest of this entry »

(Forgot to post this earlier, this is old news by now but fwiw..)
I presented at the 10th FTC Hearing on Competition and Consumer Protection in the 21st century this March, held in Washington D.C., giving a talk about Technological Developments in Broadband Networking which aims to address this question: Which (recent and expected) technological developments, or lack thereof, are important for understanding the competitiveness of the industry or impacts on the public interest?

A webcast of the presentation (my talk begins at 10m30s) is available. I also participated in a discussion panel, also webcast.

On December 12-13, 2018, CAIDA and the Massachusetts Institute of Technology (MIT) hosted the (invitation-only) 9th interdisciplinary Workshop on Internet Economics (WIE) at the University of California San Diego in La Jolla, CA.

The goal of this workshop series is to provide a forum for researchers, commercial Internet facilities and service providers, technologists, economists, theorists, policy makers, and other stakeholders to empirically inform emerging Internet regulatory and policy debates.

Presenters were asked to write talk abstracts on their presented topics, addressing four questions:

1. What is the policy goal or fear you’re addressing?
2. What data is needed to measure progress toward/away from this goal fear?
3. What methods do you propose (or are) being used to gather such data?
4. Who/how should such methods be executed, and the data shared, or not shared?

With a specific focus on measurement challenges, the topics we discussed included: analyzing the evolution of the Internet in a layered-platform context to gain new insights; measurement and analysis of economic impacts of new technologies using old tools; security and trustworthiness, reach (universal service) and reachability, sustainability of investment into public Internet infrastructure, as well as infrastructure to measure the public Internet.

Some of the takeaways from the workshop included:
Read the rest of this entry »

Presented at our 10th AIMS Workshop earlier this year, the MANIC project resulted in a prototype system to monitor interdomain links and their congestion state to support inference of persistent interdomain congestion. We announce the release of web and API-based methods to access the data. MANIC provides both a graphical user interface for conducting queries and visualizing results and programmatic access to the measurements via a queryable API. We used this MANIC infrastructure and data in our recent publication of “Inferring Persistent Interdomain Congestion”, which won the best paper award at ACM SIGCOMM 2018.

                                                       MANIC dashboard screenshot examples.

Excerpted from the paper:

“(4) We are publicly releasing our analysis scripts, and the underlying datasets via an interactive visualization interface and query API to encourage reproducibility of our results. Our data management system, based on the InfluxDB time-series database and Grafana visualization front-end, allows interactive data exploration, near real-time views of interdomain links, and longitudinal views. While this paper focuses on data from U.S. broadband access providers, we are publicly releasing measurements from VPs outside the U.S. as well.”

For access to the MANIC dashboard, or questions about the publicly accessible API, please contact manic-info@caida.org. (It is a beta prototype, in progress!)

Congratulations to Amogh Dhamdhere, David Clark, Alexander Gamero-Garrido, Matthew Luckie, Ricky K.P. Mok, Gautam Akiwate, Kabir Gogia, Vaibhav Bajpai, Alex Snoeren, and kc claffy, for being awarded Best Paper at SIGCOMM 2018!

The abstract from the paper, “Inferring Persistent Interdomain Congestion“:

There is significant interest in the technical and policy communities regarding the extent,scope, and consumer harm of persistent interdomain congestion. We provide empirical grounding for discussions of interdomain congestion by developing a system and method to measure congestion on thousands of interdomain links without direct access to them. We implement a system based on the Time Series Latency Probes (TSLP) technique that identifies links with evidence of recurring congestion suggestive of an under-provisioned link. We deploy our system at 86 vantage points worldwide and show that congestion inferred using our lightweight TSLP method correlates with other metrics of interconnection performance impairment. We use our method to study interdomain links of eight large U.S. broadband access providers from March 2016 to December 2017, and validate our inferences against ground-truth traffic statistics from two of the providers. For the period of time over which we gathered measurements, we did not find evidence of widespread endemic congestion on interdomain links between access ISPs and directly connected transit and content providers, although some such links exhibited recurring congestion patterns. We describe limitations, open challenges, and a path toward the use of this method for large-scale third-party monitoring of the Internet interconnection ecosystem.

Read the full paper on the CAIDA website.

Detecting Internet outages world-wide and in real-time is no small feat. It requires distributed measurement infrastructure, tools and processing power to analyze the resulting data, plenty of storage to save it, and a powerful user interface to visualize the data. IODA (short for Internet Outage Detection and Analysis) is CAIDA’s solution to this problem.

In an attempt to make IODA more useful, we just launched @caida_ioda, a Twitter account to bring attention to select Internet outages. We inaugurated this account by revealing an outage that took place in Morocco, on July 19, from 11:30 pm to 3:50 am local time. The visualization below illustrates this outage. The blue time series represents our active probing data. This data comes from a cluster of twenty software instances, located at SDSC in San Diego, that repeatedly ping active hosts in the IPv4 address space. Each data point of the time series captures the normalized number of /24 network blocks in Morocco that responded to these pings. The data is normalized with respect to the maximum value observed in the inspected time interval. Starting at 10:20 pm UTC, this fraction dropped significantly (from ~19,700 /24 network blocks to as low as ~13,400) and slowly started to recover after a few hours. The green time series exhibits a drop at the same time—it represents the normalized number of /24 network blocks that are reachable according to BGP, and geolocated to Morocco. The gaps in the BGP time series are due to missing data points caused by temporary issues with our infrastructure. You can use our interactive dashboard to investigate this outage yourself.

Internet outages do not always affect entire countries; their scope is frequently limited to regions or autonomous systems (ASes). IODA can detect such sub-national outages and, coming back to our example, did so for Morocco. The map below suggests that not all of the country’s regions were affected equally. Note, however, that IP address geolocation (that is, the mapping from IP address to geographical location) is far from perfect, so take this information with a grain of salt.

IODA determines an anomaly score for each outage that it detects. Our help page provides more details on how we determine this score but in essence it’s a number that captures the severity of the outage. A look at IODA’s AS-level breakdown confirms that Maroc Telecom was affected the most—the ISP’s overall anomaly score is more than twice that of Itissalat Al-Maghrib, the ISP that ranked second.

So, what happened? IODA reveals where Internet outages happen but it cannot tell us why. Understanding an outage’s root cause still requires a human in the loop; mostly to read news reports and social media postings that mention the outage. In our example, a search of the Arabic-speaking part of the Internet for “morocco internet” led us to Maroc Telecom’s Facebook page, which cited a power outage as the cause:

The time span quoted by Maroc Telecom roughly confirms what IODA saw but our data suggests that the outage began earlier—our active probers first saw a decline in connectivity at 11:30 pm—about half an hour before the alleged start of the outage.

We are supporting public access to IODA’s dashboard for exploration of this and other outages; please use it and send feedback to ioda-info AT caida DOT org.

For the last ten years (with some gaps due to network upgrades), CAIDA has captured monthly traffic samples on Internet backbone links in several large U.S[ cities (San Jose, Chicago, and since March this year, New York City).
We publish statistics for these traces at http://www.caida.org/data/passive/trace_stats/, which illustrates the growth in IPv6 traffic, relative to IPv4. Over the 10-year period covered by our traffic captures, the increase follows a steady exponential trend (linear on a log-lin graph), increasing 10-fold every 3 years. Currently the IPv6 fraction hovers around 1%. Were this trend to continue, the ratios would be roughly 50% each around October 2022 (for packets) September 2023 (for bytes). The byte fraction increases more slowly, reflecting a slightly smaller average IPv6 packet size compared to IPv4.

We are not making any predictions, and note that CGN deployment is also increasing rapidly. We are just reporting the best available data we have.

We finally published our new Program Plan for 2018-2023. (Previous program plans are at http://www.caida.org/home/about/progplan.) Executive summary below:

For the last 20 years UC San Diego’s Center for Applied Internet Data Analysis (CAIDA) has been developing data-focused services, products, tools and resources to advance the study of the Internet, which has permeated disciplines ranging from theoretical computer science to political science, from physics to tech law, and from network architecture to public policy. As the Internet and our dependence on it have grown, the structure and dynamics of the network, and how it relates to the political economy in which it is embedded, is gathering increasing attention by researchers, operators and policy makers, all of whom bring questions that they lack the capability to answer themselves. CAIDA has spent years cultivating relationships across disciplines (networking, security, economics, law, policy) with those interested in CAIDA data, but the impact thus far has been limited to a handful of researchers. The current mode of collaboration simply does not scale to the exploding interest in scientific study of the Internet.

On a more operational dimension, large-scale Internet cyber-attacks and incidents — route hijacking, network outages, fishing campaigns, botnet activities, large-scale bug exploitation, etc. — represent a major threat to public safety and to both public and private strategic and financial assets. Mitigation and recovery, as well as prevention of further attacks of similar nature, are often impeded by the fact that such events can remain unnoticed or are hard to understand and characterize. Because of their macroscopic nature, identifying such events and understanding their scope and dynamics requires: (a) combining data of different type and origin; and (b) teamwork of experts with varied background and skills; (c) agile tools for rapid, cooperative, interactive analysis.

These two infrastructure research challenges will require high performance research infrastructure, and CAIDA will embark on a new stage in our infrastructure development endeavors to support these challenges, re-using and sharing software and data components wherever possible. We will integrate existing as well as develop new measurement and analysis components and capabilities into interactive online platforms, accessible via web interfaces as well as APIs. These novel developments will enable researchers from various disciplines including non-networking experts to access and productively use Internet data, thus advancing more complex and visionary scientific studies of the Internet ecosystem. We hope these efforts will enable us and others to widen access to and utility of the best possible Internet measurement data available to research, operational, and policy communities worldwide.

On the research side, we will continue our Internet cartography efforts, improving our IPv4 and IPv6 topology mapping capabilities, and our ability to measure and analyze interdomain congestion. We will also continue development our of Internet Topology Data Kit (ITDK) data sets, but shift our focus to simplified versions of the data and visual interfaces that are easier for researchers to use. We will undertake a new project that studies topological weaknesses from a nation-state security and stability perspective. We will explore implications of these analysis for network resiliency, economics, and policy. Among our new collaborations is an interdisciplinary project to model and design an ecosystem for market-mediated software defined communications infrastructure at the wireless edge. And in the intersection between research and infrastructure, we will start a new research project that explores an ambitious new way of designing measurement infrastructure platforms to facilitate broader deployment and sharing of nodes across scientific experimenters.

As always, we will lead and participate in tool development to support measurement, analysis, indexing, and dissemination of data from operational global Internet infrastructure. Our outreach activities will include peer-reviewed papers, workshops, blogging, presentations, educational videos, and technical reports.

Note that not all of the activities described in this program plan are fully funded yet; we are seeking additional support to enable us to accomplish our ambitious agenda.

Complete program plan for 2018-2023 at: http://www.caida.org/home/about/progplan/progplan2018/.

The CAIDA annual report summarizes CAIDA’s activities for 2017, in the areas of research, infrastructure, data collection and analysis. Our research projects span Internet topology, routing, security, economics, future Internet architectures, and policy. Our infrastructure, software development, and data sharing activities support measurement-based internet research, both at CAIDA and around the world, with focus on the health and integrity of the global Internet ecosystem. The executive summary is excerpted below:
Read the rest of this entry »