Northwest Arkansas Democrat-Gazette
Nashville AT&T blast outages raise telecom network worries
The Christmas Day bombing in downtown Nashville, Tenn., led to phone and data service outages and disruptions over hundreds of miles in the southern U.S., raising new concerns about the vulnerability of U.S. communications.
The blast damaged a key AT&T network facility, a hub that provides local wireless, internet and video service, and connects to regional networks. Backup generators failed, which knocked out service for hours after the blast. A fire broke out and forced an evacuation. The building flooded, with more than 3 feet of water later pumped out of the basement.
The immediate repercussions were widespread. AT&T customers lost service — phones, internet or video — across large parts of Tennessee, Kentucky and Alabama. There were 911 centers in the region that couldn’t field calls. Others didn’t receive crucial data associated with callers, such as their locations. The Nashville Police Department’s phones and internet service failed. Stores had to shift to cash-only.
At some hospitals, electronic medical records, internet service or phones stopped working. The Nashville airport had to halt flights for about three hours on Christmas Day. Rival carrier T-Mobile also had service problems as far away as Atlanta, 250 miles away, because the company uses AT&T equipment for moving customer data from towers to the T-Mobile network.
“People didn’t even realize their dependencies until it failed,” said Doug Schmidt, a Vanderbilt University computer science professor. “I don’t think anyone recognized the crucial role that particular building played” in the region’s telecom infrastructure, he said.
The explosion, which took place in the heart of Nashville’s historic downtown, killed the bomber, injured several people and damaged dozens of buildings. Federal officials are investigating the motive and haven’t said whether the AT&T building was specifically targeted.
AT&T said 96% of its wireless network was restored Sunday. As of Monday evening, AT&T said “nearly all services” were back up. On Wednesday, it was “activating the last of the remaining wireline equipment.”
AT&T said it directed temporary cell towers to help in affected areas and rerouted traffic to other facilities as it worked to restore power to the Nashville building. But not all traffic can be rerouted, spokesman Jim Greer said, and there was physical equipment that had to be fixed in a building that was part of an active crime scene, which complicated AT&T workers’ access.
“We are all too dependent on phone, cellphone, TV and internet to have outages for any reason,” U.S. Rep. Jim Cooper, the Democrat who represents Nashville, said in an emailed statement Wednesday. He said the U.S. “needs to harden our telecom facilities so we have greater redundancy and reliability,” and called for congressional hearings on reducing telecom vulnerabilities.
The impact on emergency services may have raised the most serious flags. At one point, roughly 100 911 centers had service problems in Tennessee alone, said Brian Fontes, head of the National Emergency Number Association.
A 911 call center should still be operational even if there is damage to a phone company’s hub, said David Turetsky, a lecturer at the University at Albany and a former public safety official at the Federal Communications Commission. If multiple call centers were out of service for several days, “that is of concern,” he said.
Cooper and experts like Fontes gave AT&T credit for their work on reinstating services. “To be able to get some services up and running within 24 to 48 hours of a catastrophic blast in this case is pretty amazing,” Fontes said.
Local authorities turned to social media on Christmas Day, posting on Facebook and Twitter that 911 was down and reassuring residents by offering other numbers to call. A Facebook page for Morgan County 911 in northern Alabama said Saturday that Alabama 911 centers were up and running but advised AT&T customers with issues to try calling via internet, and to go to local police or fire stations for help if they couldn’t get through.
The Nashville Police Department uses the FirstNet system built by AT&T, which the carrier says can provide “fast, highly reliable interoperable communications” in emergencies and that is meant to prioritize first responders when networks are stressed. But problems emerged around midday Friday, said spokesperson Kristin Mumford. The department had to turn to a backup provider, CenturyLink, for its landlines and internet service at headquarters and precincts, and obtained loaner cellphones and mobile hot spots from Verizon.
The transition to backups was “actually rather seamless,” Mumford said, although the public couldn’t make calls to police precincts. She said the AT&T service started coming back Sunday and as of Wednesday morning, overall service with cellphones, internet and landlines was “about 90% up.”