A year ago January 2020, k claffy, CAIDA Director and UCSD Adjunct Professor of Computer Science and Engineering responded with collaborator David Clark, Senior Scientist at MIT’s Computer Science and Artificial Intelligence Laboratory to the Request for Information (RFI) put out by the National Science and Technology Council’s (NSTC) Joint Committee on the Research Environment (JCORE). The response establishes the critical importance of the internet to the infrastructure of society and the need for governments, and specifically the U.S. government, to send a strong signal to the private sector through high-level policy making that the only path to understanding the characteristics of the internet comes via data sharing and that responsible sharing of data for documented scientific research will not generate corporate liability.

Another focus and benefit to the policy for which we call comes with the development and delivery of academic training of professionals to work with large data sets focused on communications and networking.

You can see the complete response and more related material posted in CAIDA resource catalog.

The recent Cyberspace Solarium Commission report (1) set out a strategic plan to improve the security of cyberspace. Among its many recommendations is that the government establish a Bureau of Cyber Statistics, to provide the government with the information that it needs for informed planning and action. A recent report from the Aspen Institute echoed this call. (2) Legal academics and lobbyists have already started to consider its structure. (3) The Internet measurement community needs to join this conversation.

The Solarium report proposed some specific characteristics: they recommend a bureau located in the Department of Commerce, and funded and authorized to gather necessary data. The report also says that “the center should be funded and equipped to host academics as well as private sector and independent security researchers as a part of extended exchanges”. We appreciate that the report acknowledges the value of academic researchers and that this objective requires careful thought to achieve. The report specifically mentions “purchasing private or proprietary data repositories”. Will “extended exchanges” act as the only pattern of access, where an academic would work under a Non-Disclosure Agreement (NDA), unable to publish results that relied on proprietary data? Would this allow graduate students to participate, i.e., how would they publish a thesis? The proposal does not indicate deep understanding of how academic research works. As an illustrative example, CAIDA/UCSD and MIT were hired by AT&T as “independent measurement experts” to propose and oversee methods for AT&T to satisfy FCC reporting requirements imposed as a merger condition. (4) AT&T covered all the data we received by an NDA, and we were not able to publish any details about what we learned. This sort of work does not qualify as academic research. It is consulting.

In our view, the bureau must be organized in such a way that academics are able and incentivized to utilize the resources of the bureau for research on questions that motivate the creation of the bureau in the first place. But this requires that when the U.S. government establishes the bureau, it makes apparent the value of academic participation and the modes of operation that will allow it.

These reports focus on cybersecurity, and indeed, security is the most prominent national challenge of the Internet. But the government needs to understand many other issues related to the character of the Internet ecosystem, many of which are inextricably related to security. We cannot secure what we do not understand, and we cannot understand what we do not measure. Measurement of the Internet raises epistemological challenges that span many disciplines, from network engineering and computer science to economics, sociology, ethics, law, and public policy. The following guiding principles can help accommodate these challenges, and the sometimes conflicting incentives across academic, government, commercial, and civic stakeholders.

1. Incentivize academic participation. A national infrastructure must be organized in such a way that academics are able and incentivized to utilize its resources. This requires designing and implementing modes of operation that will incentivize independent researcher participation.
2. Demonstrate innovation and value through real projects that address national-scale problems with data-intensive science and engineering research. To justify substantial U.S. government investment in cyberinfrastructure, the research community must demonstrate its value as an independent voice with important results that help to inform the future of the Internet. This demonstration will not be effective if it is hypothetical. Real projects are tricky, because the data does not necessarily exist yet, and if it does, may be proprietary. So researchers must overcome the chicken-and-egg problem of how to demonstrate the value of an independent research community before the Bureau exists.
3. Start with public data and shared community infrastructure. The starting point must be to work with public data, and translate research results into forms that are meaningful to a constituency broader than the research community. But this path reveals more specific barriers: Who would fund such research? What are the incentives of the academic research community to undertake it? Yet if we do not recognize and overcome this challenge, the independent research community may essentially be written out of the story, as more and more data is proprietary and hidden away.
4. Make specific and concrete calls for data of national importance. In our view, the community needs a focal point for discussion about collection and use of data, presenting an opportunity and responsibility to transform abstract calls for access to data into more specific and concrete articulations.
5. Prioritize framework for research access to proprietary data. Sharing of proprietary data must address the reasons that data is considered proprietary. Understanding these reasons is required to design approaches to allow reasonable access for research purposes.
6. Integrate focus on and metrics to evaluate workforce training efforts. The other risk of continuing on the current path, rather than confronting the data access problem, is the lost opportunity to train students to interpret complex operational data about Internet infrastructure, which is crucial to developing a globally competitive U.S. cybersecurity workforce capable of securing Internet infrastructure.

Other parts of the globe have moved to regularize cybersecurity data, and they have explicitly recognized the importance of engaging and sustaining the academic research establishment in developing cybersecurity tools to secure network infrastructure (5). If the U.S. does not take coherent steps to support its research community, there is a risk that it is sidelined in shaping the future of the Internet. The European Union’s proposed regulation for Digital Services (6) also discussed the importance of ensuring access to proprietary data by the academic research community:

Investigations by researchers on the evolution and severity of online systemic risks are particularly important for bridging information asymmetries and establishing a resilient system of risk mitigation, informing online platforms, Digital Services Coordinators, other competent authorities, the Commission and the public. This Regulation therefore provides a framework for compelling access to data from very large online platforms to vetted researchers.

They clarify what they mean by “vetted researchers”:

In order to be vetted, researchers shall be affiliated with academic institutions, be independent from commercial interests, have proven records of expertise in the fields related to the risks investigated or related research methodologies, and shall commit and be in a capacity to preserve the specific data security and confidentiality requirements corresponding to each request.

This regulation emphasizes a structure that allows the academic community to work with proprietary data, sending an important signal that they intend to make their academic research establishment a recognized part of shaping the future of the Internet in the EU. The U.S. needs to take a similar proactive stance.

References

1. Cyberspace Solarium Commission report
2. The Aspen Institute: A National Cybersecurity Agenda for Digital Infrastructure
3. Lawfare: Considerations for the Structure of the Bureau of Cyber Statistics
4. CAIDA: First Amended Report of AT&T Independent Measurement Expert: Reporting requirements and measurement methods
   CAIDA: Report of AT&T Independent Measurement Expert Background and supporting arguments for measurement and reporting requirements
5. DIRECTIVE OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL on measures for a high common level of cybersecurity across the Union, repealing Directive (EU) 2016/1148
6. Regulation of the European Parliament and of the Council on a Single Market For Digital Services

Figure 1: This picture shows a line of floating buoys that designate the path of the long-awaited SACS (South-Atlantic Cable System). This submarine cable now connects Angola to Brazil (Source: G Massala, https://www.menosfios.com/en/finally-cable-submarine-sacs-arrived-to-brazil/, Feb 2018.)

The network layer of the Internet routes packets regardless of the underlying communication media (Wifi, cellular telephony, satellites, or optical fiber). The underlying physical infrastructure of the Internet includes a mesh of submarine cables, generally shared by network operators who purchase capacity from the cable owners [2,11]. As of late 2020, over 400 submarine cables interconnect continents worldwide and constitute the oceanic backbone of the Internet. Although they carry more than 99% of international traffic, little academic research has occurred to isolate end-to-end performance changes induced by their launch.

In mid-September 2018, Angola Cables (AC, AS37468) activated the SACS cable, the first trans-Atlantic cable traversing the Southern hemisphere [1][A1]. SACS connects Angola in Africa to Brazil in South America. Most assume that the deployment of undersea cables between continents improves Internet performance between the two continents. In our paper, “Unintended consequences: Effects of submarine cable deployment on Internet routing”, we shed empirical light on this hypothesis, by investigating the operational impact of SACS on Internet routing. We presented our results at the Passive and Active Measurement Conference (PAM) 2020, where the work received the best paper award [11,7,8]. We summarize the contributions of our study, including our methodology, data collection and key findings.

[A1]  Note that in the same year, Camtel (CM, AS15964), the incumbent operator of Cameroon, and China Unicom (CH, AS9800) deployed the 5,900km South Atlantic Inter Link (SAIL), which links Fortaleza to Kribi (Cameroon) [17], but this cable was not yet lit as of March 2020.

(more…)

One of CAIDA’s primary missions has been to improve our understanding of the Internet infrastructure, through data-driven science. To this end, CAIDA has collected and maintains one of the largest collections of Internet-related data sets in the world, and developed tools and services to curate and process that data. Along with this success has come the challenge of helping new students and researchers to find and use that rich archive of resources.

As part of our NSF-funded DIBBS project, CAIDA has developed a rich context resource catalog, served at catalog.caida.org. The goal of the catalog is to help both newcomers and experienced users with data discovery, and reducing the time between finding the data and extracting knowledge and insights from it.

In addition to linking datasets to related papers and presentations, the catalog will also link to code snippets, user-provided notes, and recipes for performing commons analytical tasks with the data.

Please explore and provide feedback!

On October 10, 2020 the Spoofer system logged its 1,000,000th measurement session. Finishing its 7th year under CAIDA stewardship, the project recently published its final report documenting the improvements to the software and hardware infrastructure made possible by support from the two-year DHS award “ASPIRE – Augment Spoofer Project to Improve Remediation Efforts” co-led by Matthew Luckie of the University of Waikato’s faculty of Computing and Mathematical Sciences. The report describes (1) updates to the open source client-server source address validation (SAV) testing system (developed under DHS S&T contract D15PC00188) to expand visibility of networks behind Network Address Translation devices (NATs); (2) expanded notifications and reporting through our operator-focused private reporting engine and public regionally-focused notifications to operational mailing lists; (3) several publications documenting analysis of the effectiveness of different approaches to stimulating remediation activities [1, 2, 3]. These tasks achieved testing and evaluation of work developed under the previous contract, and analysis of options for technology transition to a broader cross-section of security research, operations, risk management, and public policy stakeholders. The resulting technologies and data improved the U.S. government’s ability to identify, monitor, and mitigate the infrastructure vulnerability that serves as the primary vector of massive DDoS attacks on the Internet.

Excerpted from the ASPIRE final report:

Of the 587 remediation events we inferred between May 2016 and August 2019, 25.2% occurred in the U.S., and 23.5% occurred in Brazil. Figure 8 shows that nearly 90% of the remediation events in Brazil occurred after we began sending monthly emails to the Brazilian operator email list (GTER). We calculate the remediation rate by dividing the number of ASes for which we inferred a remediation event by the total number of ASes that sent a spoofed packet during the same interval. For the year prior to commencing the GTER emails to Brazilian network operators, 14 of 67 ASes (21%) remediated; in the year after, 52 of 168 ASes (31%) remediated. This improvement is supported by NIC.br’s “Program for a Safer Internet” [37], which offers training courses and lectures to support network operators to deploy security best practices in Brazil. The rate of remediation in the U.S. is lower; prior to sending the NANOG emails to U.S. network operators, 21 of 132 (16%) of ASes remediated; in the year after, 35 of 147 (24%) of ASes remediated. While the rate of remediation is lower in the U.S. than Brazil, the relative improvement in both is equivalent –≈50%. Note that remediation in Brazil has slowed since the outbreak of Covid-19 in Brazil.

Figure 8: Remediation in the U.S. and Brazil.

We hope you will take the time to read the full final report, download the client software and test your network to help us better understand the state of IP spoofing.

References:

1. M. Luckie, R. Beverly, R. Koga, K. Keys, J. Kroll, and k. claffy, “Network Hygiene, Incentives, and Regulation: Deployment of Source Address Validation in the Internet”, in ACM Computer and Communications Security (CCS), Nov 2019.

2. L. Müller, M. Luckie, B. Huffaker, k. claffy, and M. Barcellos, “Challenges in Inferring Spoofed Traffic at IXPs”, in ACM SIGCOMM Conference on emerging Networking EXperiments and Technologies (CoNEXT), Dec 2019.

3. L. Müller, M. Luckie, B. Huffaker, k. claffy, and M. Barcellos, “Spoofed traffic inference at IXPs: Challenges, methods and analysis”, Computer Networks, vol. 182, Aug 2020.

This visualization shows how the growing demand for those addresses transformed the governance model from a handful of scientists and engineers managing these addresses to the multi-stakeholder governance model we have today. IPv4 (the fourth version of the Internet Protocol) is the governing standard of today’s Internet. Similar to any other network, unique identifiers play an integral role in Internet routing. We group IP address blocks based on the organization that regulates its allocation as recorded in IANA’s IPv4 address space file and the RFC.

Please view the visualization at: https://www.caida.org/publications/visualizations/ipv4-history/

Screenshots of the visualization

This was created with the support of the National Science Foundation (NSF). For any questions or comments on this project, please contact info@caida.org.

The CAIDA annual report summarizes CAIDA’s activities for 2019, in the areas of research, infrastructure, data collection and analysis. Our research projects span Internet mapping, performance measurement, security, economics, and policy. Our infrastructure, software development, and data sharing activities support measurement-based internet research, both at CAIDA and around the world, with focus on the health and integrity of the global Internet ecosystem. The executive summary is excerpted below:
(more…)

v2.1

(GraphQL/RESTFUL)

Responding to feedback from our user community, CAIDA has released version 2.1 of the AS Rank API. This update helps to reduce some of the complexity of the full-featured GraphQL interface through a simplified RESTful API.

AS Rank API version 2.1 adds support for historical queries as well as support for AS Customer Cones, defined as the set of ASes an AS can reach using customer links. You can learn more about AS relationships, customer cones, and how CAIDA sources the data at https://asrank.caida.org/about.

You can find the documentation for AS Rank API version 2.1 here https://api.asrank.caida.org/v2/restful/docs.

You can find documentation detailing how to make use of historical data and customer cones here https://api.asrank.caida.org/v2/docs.

CAIDA Team

v2(GraphQL)

The new AS Rank API^v2 is ready for use. This new version reflects a move from a RESTful (v1) API to a GraphQL (v2) API. This will allow clients to create queries that specify which values they require and contain multiple resources. GraphQL, as a strongly-typed language, allows clients to know what data is available, in what format, and verify responses.

The User Interface (UI) can be found at http://asrank.caida.org. The Application Programming Interface (API^v2) serves at https://api.asrank.caida.org/v2/graphql and GraphiQL interface can be found at https://api.asrank.caida.org/docs.

We will be operating AS Rank API^v1 (http://as-rank.caida.org/api/v1) until March 1st, 2020, but it will no longer be updated. Current users should migrate to the v2 API before this date. Contact asrank-info@caida.org for migration assistance.

For those unfamiliar with GraphQL, it is a bit of a paradigm shift from the use of a RESTful API, in that GraphQL requires the client to specify precisely which values it needs. In the following example, the client wants to know an ASN’s transit degree. With a normal RESTful API, the client must retrieve the full record and extract the information it wants. A GraphQL API client must specify that it wants the ASN’s transit degree.

# request ASN 3356's degree
query={
   asn(asn:"3356") {
      asnDegree {
         transit
      }
   }
}
        

data={
   "asn": {
      "asnDegree": {
         "transit": 5255
    }
}

# request ASN 3356's record
/asns/3356?populate=1
                

GraphQL supports mixed record queries. The same query can include different record types, and can specify bindings (“joins”) between those resources. This approach reduces the number of API queries needed to retrieve related resources.

# request ASN 3356's asnName and 
# organization LPL-141-ARIN's rank.

query={
   asn(asn:"3356") {
      asnName
      organization {
        orgId
      }
   }
   organization(orgId:"LPL-141-ARIN") {
      rank
   }
}
        

# request ASN 3356's asnName and 
# it's organization's rank.

query={
   asn(asn:"3356") {
      asnName
      organization {
         rank
      }
   }
}
        

data={
    "asn": {
      "asnName": "LEVEL3"
      "organization": {
         "orgId": "LPL-141-ARIN" 
      }
    },
    "organization": {
      "rank": 1,
    }
}
        

data={
    "asn": {
      "asnName": "LEVEL3",
      "organization": {
        "rank": 1
      }
    }
  }
}
        

# request ASN 3356's record
/asns/3356?populate=1
                 

data={
  "name": "LEVEL3",
  "org": {
    "id": "LPL-141-ARIN",
    "name": "Level 3 Parent, LLC"
  },
  "clique": "true",
  "source": "ARIN",
  "cone": {
     ...                

data={
    "name": "Level 3 Parent, LLC",
    "rank": "1",
    "degree": {
      "asn": {
        "transit": 6999,
        "global": 7024
      },
      "org": {
        ....
                    

In late April 2019, social media was reportedly blocked and access to the Internet was shutdown in Benin during its 2019 parliamentary elections.

In this report, the Open Observatory of Network Interference (OONI) and the Center for Applied Internet Data Analysis (CAIDA) teams share OONI, IODA, and RIPE Atlas network measurement data that corroborate and provide insight into these recent censorship events in Benin.
(more…)