Unintended consequences of submarine cable deployment on Internet routing
December 15th, 2020 by Roderick Fanou, Ricky Mok, Bradley Huffaker and kc
Figure 1: This picture shows a line of floating buoys that designate the path of the long-awaited SACS (South-Atlantic Cable System). This submarine cable now connects Angola to Brazil (Source: G Massala, https://www.menosfios.com/en/finally-cable-submarine-sacs-arrived-to-brazil/, Feb 2018.)
The network layer of the Internet routes packets regardless of the underlying communication media (Wifi, cellular telephony, satellites, or optical fiber). The underlying physical infrastructure of the Internet includes a mesh of submarine cables, generally shared by network operators who purchase capacity from the cable owners [2,11]. As of late 2020, over 400 submarine cables interconnect continents worldwide and constitute the oceanic backbone of the Internet. Although they carry more than 99% of international traffic, little academic research has occurred to isolate end-to-end performance changes induced by their launch.
In mid-September 2018, Angola Cables (AC, AS37468) activated the SACS cable, the first trans-Atlantic cable traversing the Southern hemisphere [1][A1]. SACS connects Angola in Africa to Brazil in South America. Most assume that the deployment of undersea cables between continents improves Internet performance between the two continents. In our paper, “Unintended consequences: Effects of submarine cable deployment on Internet routing”, we shed empirical light on this hypothesis, by investigating the operational impact of SACS on Internet routing. We presented our results at the Passive and Active Measurement Conference (PAM) 2020, where the work received the best paper award [11,7,8]. We summarize the contributions of our study, including our methodology, data collection and key findings.
[A1] Note that in the same year, Camtel (CM, AS15964), the incumbent operator of Cameroon, and China Unicom (CH, AS9800) deployed the 5,900km South Atlantic Inter Link (SAIL), which links Fortaleza to Kribi (Cameroon) [17], but this cable was not yet lit as of March 2020.
Figure 2: This image shows the Angola Cables Network, which includes the SACS cable (Source: Angola Cable, https://www.angolacables.co.ao/en/network-map/, 2019).
Methodology:
Our methodology quantifies the end-to-end communication performance induced by a new submarine cable deployment on Internet paths. Our approach relies on existing subsea maps/databases [17] and public measurement infrastructures [3, 13]. Our method has four steps:
- Collect candidate IP paths that could have crossed the cable
- Identify router IP interfaces on both sides of the cable based on those candidate IP paths
- Search for corresponding paths (between same pairs of endpoints) in historical traceroute datasets
- Annotate collected paths with necessary information for analysis such as hostnames, ASes, IP geolocations, and round-trip time (RTTs) differences between consecutive hops
Consider a newly deployed subsea cable C of length l. To accurately identify IPs on both sides of C, we need samples of IP paths crossing the cable in both directions. Our first step involves executing (post-cable launch) traceroutes between monitors located within two networks, denoted AS1 and AS2, that are topologically close to the respective sides of C. The output of these measurements is a set of candidate IP paths containing IP addresses of routers traversed by packets from AS1 to AS2 or vice-versa via C as well as the round-trip times (RTTs) from the respective source IP addresses to each of them. We selected the networks hosting vantage points – VPs – (i.e. AS1, AS2) as well as the functional VPs within those networks using existing measurement platforms [3,13] and publicly available sea cable databases/maps [17].
Our second step involves determining the IP interfaces on respective sides of C within the collected candidate IP paths. Using the speed of light constraint and the known length of C, we deduced the minimum RTT to cross the considered cable by computing RTTmin using equation (a): RTTmin=(2×l)/((2⁄3)×c) where c is the speed of light in a vacuum [W1]. Further, we singled out from the candidate IP paths any pair of IP addresses IPA-IPB for which the difference between the respective round trips to the source IP address of the considered traceroute is greater or equal than the threshold RTTmin. We looked for cases where the countries of those IP interfaces, according to geolocation databases [12] (such as Netacuity [10] and Maxmind [9]), match the countries linked by the new subsea cable. Then, we inferred IPA and IPB from the potential IPs on each side of C. We looked for the router aliases of those IPs.
Third, we fetched from existing measurement platforms RIPE Atlas [R1] and CAIDA Ark [C1], historical traceroutes containing any pairs IPA-IPB or IPB-IPA, which we grouped into a set T’<s,d>. We then identified the source IP and the destination network (<s,d>) for which we collected available historical traceroutes T<s,d>. We annotated these IP paths with hostnames, Autonomous Systems (ASes), country, and RTT difference between consecutive hops.
Finally, we used three metrics to evaluate end-to-end performance and AS paths pre and post-cable launch:
- The RTTs to the common IP hops closest to the traceroute destinations quantify the time packets traveled from a source interface to a common point close to a given destination network measured pre and post cable launch.
- The AS-centrality of transit ASes represents the percentage of paths for which an AS plays the role of transit.
- The length of AS paths crossing the studied cable operator’s network pre and post-event.
Data collection:
We used CAIDA’s Ark (100+ monitors) [3] and RIPE Atlas (11,000+ monitors) [13] measurement platforms as the sources of historical traceroutes. We extracted router aliases information from CAIDA’s Macroscopic Internet Topology Dataset Kit (ITDK), IXP information from CAIDA’s Internet eXchange Point Dataset [3], and translated IPs to hostnames with qr and zdns. We used IP geolocation databases Netacuity [10] and Maxmind [9] combined with hostname geolocation to map IPs to countries.
Findings/Results:
Comparing RTTs before and After SACS
We started our analysis by comparing RTTs before and after SACS deployment. For the same source VP and destination prefix, we built the set of common IP hops in the traces before and after SACS, and selected the IP closest to the destination IPc as a point of comparison. Using the RTTs from VPs toward IP hop IPc from the traces pre- and post-SACS, we plotted the boxplots of Figure 3, clustering RTTs by continent and measurement platform.
Figure 3: Boxplots of minimum RTTs from Ark and Atlas VPs to the common IP hops closest to the destination IPs. The red line of every boxplot represents the median of these minimum RTTs; we marked the 75th and 25th percentile as well as the Interquartile range IQR.
Overall, our results show that SACS had little impact on latencies from all vantage points (VPs) to all destination prefixes. Interestingly, paths from South America experienced a median decrease of 38%, which was quite significant compared to paths from Oceania – Australia (8% decrease), and to paths from Africa (3%). At the country-level, we found predictable performance improvements (RTT decrease) for paths going from Africa to Brazil, or from South America to Angola. However, we found an asymmetrical RTT reduction: the decrease of the median RTT from Africa to Brazil (73ms) was a third of that from South America to Angola (226ms). We also noted some unpredicted and unreported performance degradations e.g., we saw packets suboptimally routed through SACS for paths going from North America to Brazil or Africa/Europe to Angola, leading to latency increases.
Comparing Transit Structure
We computed per AS the percentage of observed <s,d> pairs for which the AS path with the minimum observed RTT contained the considered AS.
 Figure 4: Partial AS paths from South America to Angola (Observed RTT improvements) |
BEFORE | AFTER | ||
|
AS-centrality (AS37468) |
46% | 100% | ||
| Median RTT of IP paths via AS37468 | 346ms | 159ms | ||
| Min RTT of IP paths via AS37468 | 353ms | 108ms | ||
Figure 4 depicts the impact of SACS deployment on transit ASes serving observed paths from South America to Angola (RTT improvement). The ovals inside Angola Cables are part of traceroutes post-SACS that we manually geolocated using hints in hostnames. Before SACS, 46% of observed <s,d> pairs crossed Europe and then Angola via AC’s network. SACS provided a more direct path between these two continents and improved performance. As a consequence, the median RTT between VPs and IPs on IP paths via AS37468 decreased from 346ms to 159ms
 Figure 5: Partial AS paths from Europe to Angola (Observed RTT degradation) |
BEFORE | AFTER | ||
| AS-centrality (AS37468) | 74% | 100% | ||
| Median RTT of IP paths via AS37468 | 156ms | 259ms | ||
| Min RTT of IP paths via AS37468 | 128ms | 180ms | ||
We then considered partial paths between Europe and Angola, cases in which we observed an RTT degradation from the collected traceroutes. Note that AC was the major transit provider for traffic from Europe to Angola throughout the entire period of the study. However, the use of SACS within AC significantly lengthened the physical path and thus the latency on the forward path. As a consequence, the median RTT on IP paths crossing AS37468 increased from 156ms to 259ms.
We provide more detail on our in-depth inspection of the paths pre and post-SACS in the paper [7].
Contributions and key findings:
The key contributions of this study can be listed as follows:
- We introduced a reproducible method to investigate the impact of a cable deployment on macroscopic Internet topology and performance.
- We applied our methodology to the case of SACS, the first trans-Atlantic cable from South America to Africa. We discovered that the RTT decrease for IP paths going from Africa to Brazil was roughly a third of that noticed on paths from South America to Angola. Further, we discovered surprising performance degradations to/from some regions and analyzed the root-causes of these unintended consequences.
- We suggested ways to avoid suboptimal routing post-activation of cables in the future.
- Finally, we published our code and data to facilitate reproducibility [8].
References:
[1] Angola Cables: SACS. https://sacs.angolacables.co.ao, Jan 2020
[2] Bischof, Z.S., Fontugne, R., Bustamante, F.E.: Submarine Cables and Internet Resiliency. https://www.iij.ad.jp/en/dev/iir/pdf/iir_vol41_ focus2_EN.pdf (2018)
[3] CAIDA: Archipelago (Ark) Measurement Infrastructure. https://www.caida.org/projects/ark/, Jan 2020
[4] CAIDA website. http://www.caida.org, Jan 2020
[5] CAIDA, Presentation, video, and PDF of the paper: https://www.caida.org/publications/papers/2020/unintended_consequences/, March 2020
[6] Digital Element: Netacuity. http://www.digital-element.net/ip_ intelligence/ip_intelligence.html, Jan 2020.
[7] Fanou, R., Huffaker, B., Mok, R. and Claffy, K.C. Unintended Consequences: Effects of Submarine Cable Deployment on Internet Routing. In International Conference on Passive and Active Network Measurement (pp. 211-227). Springer, March 2020.
[8] Fanou, R., Huffaker, B., Mok, R. and Claffy, K.C., Github repository of the paper “Unintended consequences: Effects of submarine cable deployment on Internet routing”, https://github.com/CAIDA/submarine-cable-impact-analysis-public, March 2020.
[9] MaxMind: GeoIP. https://dev.maxmind.com/geoip/geoip2/geoip2-city-country-csv-databases/, Aug 2019.
[10] Digital Element: Netacuity. http://www.digital-element.net/ip_ intelligence/ip_intelligence.html, Jan 2020.
[11] Phil, E.: A Map of all the Underwater Cables that Connect the Internet. https://www.vox.com/2015/3/13/8204655/submarine-cables-internet, Nov 2015.
[12] Poese, I., Uhlig, S., Kaafar, M.A., Donnet, B., Gueye, B.: IP Geolocation Databases: Unreliable? ACM SIGCOMM Computer Communication Review 4(2), 53–56 (2011)
[13] RIPE NCC: Global RIPE Atlas Network Coverage. https://atlas.ripe.net/results/maps/network-coverage/, Jan 2020
[14] Submarine Cable Networks. https://www.submarinenetworks.com/en/, Jan 2020
[15] Subsea World News. https://subseaworldnews.com, Jan 2020
[16] Telegeography: Submarine Cable Frequently Asked Questions, https://www2.telegeography.com/submarine-cable-faqs-frequently-asked-questions, Dec 2020
[17] Telegeography: Telegeography Submarine Cable Map. https://www.submarinecablemap.com/, Jan 2020
[18] Wikipedia: Speed of light, https://en.wikipedia.org/wiki/Speed_of_light, 2020