‘Holy moly!’: Inside Texas’ fight against a ransomware hack

DALLAS (AP) — It was the start of a steamy Friday two Augusts ago when Jason Whisler settled in for a working breakfast at the Coffee Ranch restaurant in the Texas Panhandle city of Borger. The most pressing agenda item for city officials that morning: planning for a country music concert and anniversary event.

Then Whisler’s phone rang. Borger’s computer system had been hacked.

Workers were frozen out of files. Printers spewed out demands for money. Over the next several days, residents couldn’t pay water bills, the government couldn’t process payroll, police officers couldn’t retrieve certain records. Across Texas, similar scenes played out in nearly two dozen communities hit by a cyberattack officials ultimately tied to a Russia-based criminal syndicate.

In 2019, ransomware had yet to emerge as one of the top national security concerns confronting the United States, an issue that would become the focus of a presidential summit between Washington and Moscow this year. But the attacks in Texas were a harbinger of the nowexploding threat and offer a vivid case study in what happens behind the scenes when small-town America comes under attack.

Texas communities struggled for days with disruptions to core government services as workers in small cities and towns endured a cascade of frustrations brought on by the sophisticated cyberattack, according to thousands of pages of documents reviewed by The Associated Press and interviews with people involved in the response. The AP also learned new details about the attack’s scope and victims, including an Air Force base where access to a law enforcement database was interrupted, and a city forced to operate its watersupply system manually.

In recent months, a ransomware attack led to gasoline shortages. Another, tied to the same hacking gang that attacked the Texas communities, threatened meat supplies. But the Texas attacks — which, unlike these prominent cases, were resolved without a ransom payment — make clear that ransomware need not hit vital infrastructure or major corporations to interrupt daily life.

“It was just a scary feeling,” Whisler, Borger’s emergency management coordinator, recounted in an interview.

In the early morning of Aug. 16, as most Texans were still asleep, hackers half a world away were burrowing into networks. They encrypted files and left ransom notes.

That afternoon, with the attack’s impact becoming apparent, the city manager of Vernon emailed colleagues about a “ransom type” virus affecting the police department. The city near the Oklahoma state line could get back online by paying the $2.5 million the hackers were demanding, he wrote, but that was “obviously” not the plan.

“Holy moly !!!!! ” replied city commissioner Pam Gosline, now the mayor.

The culprits were affiliated with REvil, the Russia-linked syndicate that last spring extorted $11 million from meat-processor JBS and more recently was behind a Fourth of July weekend attack that crippled businesses around the globe. In the Texas case, however, communities were ultimately able to recover most of their data and rebuild their systems without anyone paying ransom.

The hackers gained their foothold through an attack on a Texas firm that provides technology services to local governments, branching through screen-sharing software and remote administration to seize control of the networks of some of the company’s clients.

An early hint of trouble came with a 2 a.m. phone call to the firm’s president, Richard Myers. His company, TSM Consulting Services Inc., provides data communications service for Texas communities, linking police agencies to a statewide law enforcement database.

One of his client’s servers was unresponsive, he was told. Upon inspection, Myers noticed that someone who wasn’t supposed to be in the computer system was trying to install something remotely. He rebooted the server. Things initially seemed fixed until the department called back: One of its laptops had a ransom note on it.

It soon became clear the problem wasn’t isolated to a single client.

“I don’t think you can begin to express the terror that goes through your mind when something like that starts to unfold,” he said.

Within hours, state officials were hunkered inside an underground operations center normally used for calamities like hurricanes and floods. Gov. Greg Abbott declared it a cyber disaster. Texas National Guard cyber specialists were activated.

“If you needed to build something — you needed an inspection, something like that — out of luck for a week,” said Andy Bennett, the state’s then-deputy chief information security officer. “Records look-ups? Couldn’t go look up records. Basically, if there’s a municipal function that you would go down to a city hall for, or that you would rely on the police department for, it wasn’t available.” _____

In Borger, a city of fewer than 13,000, early indications were worrisome as the city raced to shut down its computers...

Vital records, like birth and death certificates, were offline. Signs posted on a drive-up window outside City Hall said the city couldn’t process water bill payments but that cutoffs would be delayed.

Because the city had paid for remote offsite backup, Borger could reformat servers, reinstall the operating system and retrieve data. The police department, however, retained its data locally and officers were unable to access previous incident reports, Spradling said.

Jeremy Sereno was working his civilian job at Dell when he was enlisted by the state to help. A lieutenant colonel and senior cybersecurity officer with the Texas Military Department, Sereno helped deploy Texas National Guard troops to hacked cities, where specialists worked to assess the damage, restore data from backed-up files and retake control of locked systems.

One of the first areas of concern was a small North Texas city. The attack locked the “human-machine interface” workers used to control the water supply, forcing them to operate the system manually, Sereno said. Water purity was not endangered.

“That’s what’s considered critical infrastructure, when you talk about water,” he said.

AP is not identifying the city at the urging of state officials, who said doing so could draw new attacks on its water system.

In Graham, the ransomware attacked a police server housing body-camera videos, causing hundreds to be lost. Instead of using mobile data terminals to run checks on people they encountered, officers had to rely on requests to dispatchers at a local sheriff’s office unaffected by the attack, said Chief Brent Bullock.

The impact wasn’t limited to local governments. Sheppard Air Force Base confirmed to AP that its access to a statewide law enforcement database used for background checks was temporarily disrupted.

One complication: TSM’s client list was encrypted, officials said. State officials didn’t immediately know which communities had been victimized.

They had to call around, said Nancy Rainosek, Texas’ chief information security officer. “There was one place that we contacted and they said, ‘no, no, we’re not hit,’” she said. Days later, “they said, ‘yes, we were.’”

Fortunately for Borger, most city services were restored within days. The city has since invested in additional cybersecurity protections.

“When you complain about having to change your passwords, you complain a lot more when it’s never happened to you and you don’t have anything to relate it to,” Spradling said. “You tend to complain a little less after you’ve had to answer the phone and tell 300 people they couldn’t pay their water bill.”

Even now, Spradling said, officials will go to pull an old report or address record — only to find it isn’t there.