Nashville bombing exposed telecom network
The vulnerability of the telecommunications system in Nashville and beyond became clear Christmas Day when AT&T’S central office in downtown became a crime scene.
Mayor John Cooper called the bombing on Second Avenue an attack on infrastructure. The effects of that attack are sure to ripple through the region for weeks, as the telecom giant scrambles to restore services while also maintaining the integrity of an active investigation site teeming with federal agents.
State and local officials and experts say the fact that a multi-state region could be brought to its knees by a single bombing is a “wake-up call,” exposing vulnerabilitiesmany didn’t know existed and predicting it would lead to intense conversations about the future.
The bombing and the damage to the AT&T office was a “single-point of failure,” said Douglas Schmidt, the Cornelius Vanderbilt professor of computer science at Vanderbilt University.
“That’s the Achilles Heel. The weak link,” he said. “When one thing goes wrong and everything comes crashing down.”
Now, the Tennessee Emergency Communications Board has called a special meeting for next week to address the “impact to 911 operations as a result of the bombing in downtown Nashville,” according to a public notice of the meeting set for Monday.
State lawmakers have begun to weigh in.
“I think Tennesseans deserve some answers,” said state Sen. Paul Bailey, RSparta, of the widespread disruptions. “There is just a lot of questions out there.”
While still piecing together a motive, investigators Monday said suspected bomber Anthony Quinn Warner sought “more destruction than death.”
He parked an RV outside the nondescript window-less red-brick building on the historic district, which houses a facility that includes connection points for regional internet and wireless communications.
Flames broke out in the building and three feet of water pooled in the basement. Temporary battery power kept services intact in the hours following the explosion, but fire and flooding damaged backup power generators to power those batteries
The disruption brought communications in the region, from Georgia to Kentucky, to a halt, affecting 911 call centers, hospitals, the Nashville airport, government offices and individual mobile users. Issues with credit card devices hamstrung businesses big and small.
AT&T reported Monday morning that the majority of services in Nashville had been restored through a combination of fixes, including generator repairs and a temporary network set up at Nissan Stadium.
“In this day in age having a critical facility in a major metropolitan area next to a street without any other protections than a thick wall is crazy,” Schmidt said.
“The silver lining here is nobody was killed,” he said. “But this is a wake up call that if people treat it right, will help with future situations and be better prepared.”
Better backup plans needed
When the situation settles down, Bailey, who most recently served as chairman of the Senate commerce committee, hopes the Tennessee legislature can hear from AT&T representatives about what type of plan they’re implementing to prevent this type of outcome in the future if another similar disaster occurs.
“They need to have better redundancies in place,” Bailey said, referring to AT&T’S backup systems to prevent widespread outages. “It’s just very concerning that we have 911 centers go down. Lots of emergency services losing communications. That’s really concerning to me.”
Metro Council member Freddie O’connell, who represents the downtown area, said the city must also follow up on how to create more redundancy in critical communication systems in the aftermath of the bombing.
“How does a city as a whole function if we go through something like this again or a natural disaster?” he said. “We learned our systems are not redundant enough when one major provider goes offline.”
Police officers on the scene Friday were issued burner phones, according to Metro police spokesperson Don Aaron. Nashville’s police department uses FirstNet network, a priority network for first responders to use on existing AT&T cell towers for voice and data.
Metro’s 911 line remained operational but officials were without access to administrative phone lines through Friday evening, according to Stephen Martini, director of Nashville Department of Emergency Communications.
In the absence of non-emergency phone lines, residents were encouraged to request services through hubnashville online — which officials monitored for a three-day period.
Martini said communications to emergency personnel via radio was never impacted over the weekend.
He declined to share details on how the department remained operational, citing sensitive public safety information, but said a redundancy plan, dubbed the P.A.C.E method (Primary, Alternate, Contingent, Emergency), was in place.
Metro’s director of information and technology services, Keith Durbin, said Verizon phones had to be driven to some staff on Christmas Day.
“This was one of the worst case scenarios that happened,” Durbin said. “AT&T runs the world. To have them completely taken out … was even broader impact than we thought.”
Luckily, he said, none of the city’s “internal network backbone” was affected, with issues primarily coming from smaller Metro facilities. Some were continuing to experience issues Monday, including the Davidson County Clerk’s office.
The city was able to switch from AT&T to a secondary internet carrier Friday. But the city doesn’t have a backup for phone services. It’s something officials have considered in recent years.
Those talks, Durbin said, will be revived after the bombing. He said he’s confident it’ll now get wide support.
Bailey said he heard from 911 center directors in his district reporting outages nearly 100 miles away from Nashville. Residents in the area received reverse emergency calls to inform them not to dial 911, but instead to use another phone number to get in touch with dispatchers.
And then there were the retail stores, the pharmacies, the businesses and hospitals that were impacted, he said.
Bailey is a self-described “less regulation guy.” He isn’t jumping the gun to call for legislative action.
He credits AT&T for working quickly to restore service. But he said it’s concerning that one incident could wipe out so much of the region’s communications capability.
“This affected our entire Southeast region,” Bailey said. “There were multiple states that had issues because of this.”
But as for whether the General Assembly wields much power to compel action from AT&T, “the short answer is no, we don’t,” Bailey said.
The state’s Public Utility Commission, a five-member board consisting of political appointees, also has a very limited ability to regulate for-profit communications companies. Much of that would be a federal issue, Bailey noted.
State and hospitals face outages
The city wasn’t alone in experiencing communication outages.
The Tennessee General Assembly, which has offices adjoining the Capitol downtown, also experienced outages over the weekend. The email system was down for a portion of the day Saturday, and staff were told to work from home
Monday after the building experienced phone outages until Sunday evening.
State government office buildings remained closed Monday due to safety hazards that the outages continued to pose, said Lola Potter, spokesperson for the Department of Finance and Administration.
Fire and safety alarm systems in state buildings in Nashville still weren’t fully functioning.
Other state services over the weekend were impacted, though state employees found workarounds and alerted authorities in some situations, such as with reporting child abuse.
“Obviously we were concerned about it, but we took precautions by reaching out to law enforcement,” said Jennifer Donnals, chief of staff for the Department of Children’s Services.
A state web form and app that field child abuse reports remained in service, Donnals said, and staff in different regions also alerted major children’s hospitals around the state to the hotline being down.
The timing of the disruptions occurring on a weekend — and a holiday weekend at that — meant the agency would likely have been receiving fewer complaints than on weekdays anyway.
Meanwhile, hospitals in the region have also had to work around outages since the weekend, mostly stemming from their landline phone systems going down and being unable to receiving incoming calls.
Vanderbilt University Medical Center reported were back in service by late Monday afternoon, while Tristar Centennial Medical Center was experiencing some intermittent outages.
Both hospitals had to set up new, temporary phone numbers for people to call, according to spokespeople for both hospital systems
“We began to experience issues with landline phone services soon after the explosion,” said Jessica Pasley, a VUMC spokesperson. “In response, teams developed workarounds using alternative wireless services that facilitated patient care communications.”