Above: Undersea cable connections charted by the Submarine Cable Map service. The purple line between Curacao and Trinidad was the point of failure.

Originally published in Sunday Newsday on December 13, 2020

On the evening of December 11, Kurleigh Prescod and two communications professionals sat for an exclusive virtual interview with technology journalist Mark Lyndersay to explain what happened to Flow’s Internet connection on the morning of December 07.

On December 01, the Internet alarms were ringing in Flow’s offices. Monitoring systems in Miami and Latin America were registering an interruption in service between Curacao and Guadeloupe.

That cable break affected all islands north of Trinidad and south of Guadeloupe, including Barbados, Dominica, St. Lucia, St Vincent and Grenada, but resilience in Flow’s networks made it possible to continue delivering service.

Land based teams were quickly mobilised to investigate, and it soon became clear that the break was beneath the water, between two landings of the CWC cable EC Link, a 987 kilometre run of fibre optic cable activated in 2007.
Tests on land suggested that the subsea break was closer to Guadeloupe, and divers were assigned to the repair.

According to Kurleigh Prescod, Country Manager of FlowTT, the company has dive teams on standby in the region, but they weren’t in Guadeloupe. Then, covid19 restrictions slowed the arrival of the teams to the site.

Flippers were in the water on December 06, and the damage was located during that first day. Repair splicing on the break in the line was completed two days later, drifting well into the evening hours.

But that wouldn’t be the company’s only problem.

In Curacao, a nationwide power outage shut the country down on the morning of December 07, while the EC Link cable route was still under repair.

The tiny island, just 24 per cent larger than Tobago at 444 sq km, declared a covid19-related state of emergency on December 11 and has been reeling from a series of national power shortfalls since September.

According to the Curacao Chronicle, power company Aqualectra is down to delivering 110 megawatts to a country with a demand for 120 megawatts.

That loss of connectivity crushed Flow’s capacity.

Because Flow parent company Liberty Latin America is a bulk bandwidth provider in the Caribbean region, the disruption affected other ISPs who experienced varying levels of network congestion.

“Our [uninterruptible power] batteries did keep the system going and the generators did kick in,” Prescod explained,“ the facilities on the island remained powered, but there was a disruption on another aspect of the infrastructure that led to the outage.”
“The vulnerability was in-country and was resolved between 10 and 10:30 that morning.”

The FlowTT manager declined to offer details on the nature of the Curacao failure, but noted that another nationwide power outage happened on December 10, providing an inadvertent, but successful, test of the infrastucture repair.

“We were able to keep the majority of services live after the first break,” Prescod said.
“While we did not have the capacity to carry full network traffic, we were able to do diagnostics.”

Prescod believes that more redundancy in the Internet connections to Trinidad and Tobago would be better for the country, noting that while there are four connections, only three are capable of supporting the throughput expected in 2020.

The oldest active connection, Americas-II, is now 20 years old and approaching end-of-life.
The cable itself is likely to be capable of more throughput, but the supporting equipment, both on land and under the Caribbean sea would have to be upgraded.

That’s likely to be a case of the candle costing more than the funeral for a connection that went live in August 2000.
Americas-I, the first cabled link to Trinidad in 1994 has been retired, and is now only a connection between Miami and the British Virgin Islands.

The other connections are the Suriname-Guyana Submarine Cable System (SG-SCS, a 1,249 km link opened for service in July 2010 and Digicel’s 3,000 km Southern Caribbean Fiber, active since September 2006

“It is not a trivial investment,” Prescod said.
“There are opportunities for growth that have been tabled, but a subsea cable system is not inexpensive, it costs tens of millions.”

The last subsea cable connection cost Flow US$35 million.

“You have to be able to build a return on a significant investment of that nature.”
“A collaborative effort among all the players [working] with the government might be able to build a business case for the development of a new connection.”

Prescod has some ideas about where that should happen, noting that all the connections to Trinidad are in the north of the country, an unfortunate congregation of vulnerability.
He’s also concerned about the fact that there’s only one data cable connecting Trinidad to Tobago.

“Ideally we should have a cable connection that connects on the south coast of the island to ameliorate risk and a subsea cable that linked Trinidad with Tobago and then with Barbados would make sense in a geographic line.”

If it seems absurd that a country just off the South American coast makes all its connections looking north, that’s a consequence of our consumption patterns. All our data comes from the north, so connecting to an Internet backbone in South America wouldn’t be as helpful as it might seem.

Kurleigh Prescod is also a key figure in the TT Internet Exchange Point project (TTIX), a network of local servers founded by a collaboration of local ISP’s to keep data traffic in-country as much as possible.

We had all these super-fast connections, the networks were up and connected, but because the app is based off-island, users could not make a connection.”

But that system didn’t work as well as it should have during the outage.

Several local banking systems failed, because they required offshore connections and there were problems with the Linx system, which is entirely local in focus.

Linx has four members and four licensees and three of the four members were unable to complete online transactions because they required connections to international circuits. Financial institutions using Linx which did not require offshore connections were able to function on the network, which continued operating locally.

“There were things that continued to work,” Prescod said.
“If your DNS was hosted here, it would resolve correctly. We had all these super-fast connections, the networks were up and connected, but because the app is based off-island, users could not make a connection.”

“We have reached out to the app providers to have local instances that can continue to operate in the absence of an overseas connection; an example would be WhatsApp.”

“If a [WhatsApp] server was hosted locally, it would transfer local messages. A lot of the app providers [including Facebook and Google] have content and servers located locally.”

The TTIX commissioned its own domain name root servers in 2015. DNS software and hardware manage domain names and directs URLs to unique Internet addresses.

The local servers host copies of a D-root server provisioned by University of Maryland, and an E-root server, provided by NASA.
While the primary intent of the local root servers is to speed up requests, the system also makes it possible to keep Internet traffic completely local when external connections are spotty or nonexistent.

Making the system more locally focused demands more investment in local infrastructure and a willingness to create and deliver local content and applications locally to provide a fall back position should external Internet connections fail.

But that isn’t as easy or as cheap as buying the commodity cloud services readily available in the first world.

Updated December 15, 2020 with additional information on the status of Linx connectivity during the outage.