The CAIDA annual report summarizes CAIDA’s activities for 2019, in the areas of research, infrastructure, data collection and analysis. Our research projects span Internet mapping, performance measurement, security, economics, and policy. Our infrastructure, software development, and data sharing activities support measurement-based internet research, both at CAIDA and around the world, with focus on the health and integrity of the global Internet ecosystem. The executive summary is excerpted below:
(more…)

v2.1

(GraphQL/RESTFUL)

Responding to feedback from our user community, CAIDA has released version 2.1 of the AS Rank API. This update helps to reduce some of the complexity of the full-featured GraphQL interface through a simplified RESTful API.

AS Rank API version 2.1 adds support for historical queries as well as support for AS Customer Cones, defined as the set of ASes an AS can reach using customer links. You can learn more about AS relationships, customer cones, and how CAIDA sources the data at https://asrank.caida.org/about.

You can find the documentation for AS Rank API version 2.1 here https://api.asrank.caida.org/v2/restful/docs.

You can find documentation detailing how to make use of historical data and customer cones here https://api.asrank.caida.org/v2/docs.

CAIDA Team

v2(GraphQL)

The new AS Rank API^v2 is ready for use. This new version reflects a move from a RESTful (v1) API to a GraphQL (v2) API. This will allow clients to create queries that specify which values they require and contain multiple resources. GraphQL, as a strongly-typed language, allows clients to know what data is available, in what format, and verify responses.

The User Interface (UI) can be found at http://asrank.caida.org. The Application Programming Interface (API^v2) serves at https://api.asrank.caida.org/v2/graphql and GraphiQL interface can be found at https://api.asrank.caida.org/docs.

We will be operating AS Rank API^v1 (http://as-rank.caida.org/api/v1) until March 1st, 2020, but it will no longer be updated. Current users should migrate to the v2 API before this date. Contact asrank-info@caida.org for migration assistance.

For those unfamiliar with GraphQL, it is a bit of a paradigm shift from the use of a RESTful API, in that GraphQL requires the client to specify precisely which values it needs. In the following example, the client wants to know an ASN’s transit degree. With a normal RESTful API, the client must retrieve the full record and extract the information it wants. A GraphQL API client must specify that it wants the ASN’s transit degree.

# request ASN 3356's degree
query={
   asn(asn:"3356") {
      asnDegree {
         transit
      }
   }
}
        

data={
   "asn": {
      "asnDegree": {
         "transit": 5255
    }
}

# request ASN 3356's record
/asns/3356?populate=1
                

GraphQL supports mixed record queries. The same query can include different record types, and can specify bindings (“joins”) between those resources. This approach reduces the number of API queries needed to retrieve related resources.

# request ASN 3356's asnName and 
# organization LPL-141-ARIN's rank.

query={
   asn(asn:"3356") {
      asnName
      organization {
        orgId
      }
   }
   organization(orgId:"LPL-141-ARIN") {
      rank
   }
}
        

# request ASN 3356's asnName and 
# it's organization's rank.

query={
   asn(asn:"3356") {
      asnName
      organization {
         rank
      }
   }
}
        

data={
    "asn": {
      "asnName": "LEVEL3"
      "organization": {
         "orgId": "LPL-141-ARIN" 
      }
    },
    "organization": {
      "rank": 1,
    }
}
        

data={
    "asn": {
      "asnName": "LEVEL3",
      "organization": {
        "rank": 1
      }
    }
  }
}
        

# request ASN 3356's record
/asns/3356?populate=1
                 

data={
  "name": "LEVEL3",
  "org": {
    "id": "LPL-141-ARIN",
    "name": "Level 3 Parent, LLC"
  },
  "clique": "true",
  "source": "ARIN",
  "cone": {
     ...                

data={
    "name": "Level 3 Parent, LLC",
    "rank": "1",
    "degree": {
      "asn": {
        "transit": 6999,
        "global": 7024
      },
      "org": {
        ....
                    

In late April 2019, social media was reportedly blocked and access to the Internet was shutdown in Benin during its 2019 parliamentary elections.

In this report, the Open Observatory of Network Interference (OONI) and the Center for Applied Internet Data Analysis (CAIDA) teams share OONI, IODA, and RIPE Atlas network measurement data that corroborate and provide insight into these recent censorship events in Benin.
(more…)

The CAIDA annual report summarizes CAIDA’s activities for 2018, in the areas of research, infrastructure, data collection and analysis. Our research projects span Internet topology, routing, security, economics, future Internet architectures, and policy. Our infrastructure, software development, and data sharing activities support measurement-based internet research, both at CAIDA and around the world, with focus on the health and integrity of the global Internet ecosystem. The executive summary is excerpted below:
(more…)

(Forgot to post this earlier, this is old news by now but fwiw..)
I presented at the 10th FTC Hearing on Competition and Consumer Protection in the 21st century this March, held in Washington D.C., giving a talk about Technological Developments in Broadband Networking which aims to address this question: Which (recent and expected) technological developments, or lack thereof, are important for understanding the competitiveness of the industry or impacts on the public interest?

A webcast of the presentation (my talk begins at 10m30s) is available. I also participated in a discussion panel, also webcast.

On December 12-13, 2018, CAIDA and the Massachusetts Institute of Technology (MIT) hosted the (invitation-only) 9th interdisciplinary Workshop on Internet Economics (WIE) at the University of California San Diego in La Jolla, CA.

The goal of this workshop series is to provide a forum for researchers, commercial Internet facilities and service providers, technologists, economists, theorists, policy makers, and other stakeholders to empirically inform emerging Internet regulatory and policy debates.

Presenters were asked to write talk abstracts on their presented topics, addressing four questions:

1. What is the policy goal or fear you’re addressing?
2. What data is needed to measure progress toward/away from this goal fear?
3. What methods do you propose (or are) being used to gather such data?
4. Who/how should such methods be executed, and the data shared, or not shared?

With a specific focus on measurement challenges, the topics we discussed included: analyzing the evolution of the Internet in a layered-platform context to gain new insights; measurement and analysis of economic impacts of new technologies using old tools; security and trustworthiness, reach (universal service) and reachability, sustainability of investment into public Internet infrastructure, as well as infrastructure to measure the public Internet.

Some of the takeaways from the workshop included:
(more…)

Presented at our 10th AIMS Workshop earlier this year, the MANIC project resulted in a prototype system to monitor interdomain links and their congestion state to support inference of persistent interdomain congestion. We announce the release of web and API-based methods to access the data. MANIC provides both a graphical user interface for conducting queries and visualizing results and programmatic access to the measurements via a queryable API. We used this MANIC infrastructure and data in our recent publication of “Inferring Persistent Interdomain Congestion”, which won the best paper award at ACM SIGCOMM 2018.

                                                       MANIC dashboard screenshot examples.

Excerpted from the paper:

“(4) We are publicly releasing our analysis scripts, and the underlying datasets via an interactive visualization interface and query API to encourage reproducibility of our results. Our data management system, based on the InfluxDB time-series database and Grafana visualization front-end, allows interactive data exploration, near real-time views of interdomain links, and longitudinal views. While this paper focuses on data from U.S. broadband access providers, we are publicly releasing measurements from VPs outside the U.S. as well.”

For access to the MANIC dashboard, or questions about the publicly accessible API, please contact manic-info@caida.org. (It is a beta prototype, in progress!)

Congratulations to Amogh Dhamdhere, David Clark, Alexander Gamero-Garrido, Matthew Luckie, Ricky K.P. Mok, Gautam Akiwate, Kabir Gogia, Vaibhav Bajpai, Alex Snoeren, and kc claffy, for being awarded Best Paper at SIGCOMM 2018!

The abstract from the paper, “Inferring Persistent Interdomain Congestion“:

There is significant interest in the technical and policy communities regarding the extent,scope, and consumer harm of persistent interdomain congestion. We provide empirical grounding for discussions of interdomain congestion by developing a system and method to measure congestion on thousands of interdomain links without direct access to them. We implement a system based on the Time Series Latency Probes (TSLP) technique that identifies links with evidence of recurring congestion suggestive of an under-provisioned link. We deploy our system at 86 vantage points worldwide and show that congestion inferred using our lightweight TSLP method correlates with other metrics of interconnection performance impairment. We use our method to study interdomain links of eight large U.S. broadband access providers from March 2016 to December 2017, and validate our inferences against ground-truth traffic statistics from two of the providers. For the period of time over which we gathered measurements, we did not find evidence of widespread endemic congestion on interdomain links between access ISPs and directly connected transit and content providers, although some such links exhibited recurring congestion patterns. We describe limitations, open challenges, and a path toward the use of this method for large-scale third-party monitoring of the Internet interconnection ecosystem.

Read the full paper on the CAIDA website.

For the last ten years (with some gaps due to network upgrades), CAIDA has captured monthly traffic samples on Internet backbone links in several large U.S[ cities (San Jose, Chicago, and since March this year, New York City).
We publish statistics for these traces at http://www.caida.org/data/passive/trace_stats/, which illustrates the growth in IPv6 traffic, relative to IPv4. Over the 10-year period covered by our traffic captures, the increase follows a steady exponential trend (linear on a log-lin graph), increasing 10-fold every 3 years. Currently the IPv6 fraction hovers around 1%. Were this trend to continue, the ratios would be roughly 50% each around October 2022 (for packets) September 2023 (for bytes). The byte fraction increases more slowly, reflecting a slightly smaller average IPv6 packet size compared to IPv4.

We are not making any predictions, and note that CGN deployment is also increasing rapidly. We are just reporting the best available data we have.