Ransomware attack puts KQED in low-tech mode
Staff copes as public broadcaster struggles to recover
The journalists at San Francisco’s public TV and radio station, KQED, have been stuck in a time warp.
All Internet-connected devices, tools and machinery have been cut off in an attempt to isolate and contain a ransomware attack that infected the station’s computers June 15. More than a month later, many remain offline.
Though the stations’ broadcasts have been largely uninterrupted — minus a half-day loss of the online stream on the first day of the attack — KQED journalists said every day has brought new challenges and revealed the immeasurable ways the station, like many businesses today, has become dependent on Internet-connected devices.
“It’s like we’ve been bombed back to 20 years ago, technologywise,” said Queena Kim, a senior editor at KQED. “You rely on technology for so many things, so when it doesn’t work, everything takes three to five times longer just to do the same job.”
KQED’s experience offers a glimpse into the lasting impact of a ransomware attack, the devastating online assaults that have become more frequent, destructive and wide-reaching in recent months. Ransomware is a specific form of malware that encrypts files, rendering them unreadable, with a digital key that a hacker promises to deliver if paid.
It also underscores an uncom-
fortable truth: If KQED, an organization that had up-todate security systems and an awareness cultivated by routinely producing news stories about cyberattacks, can fall victim to such an attack, most other companies can, too.
“It was astonishing,” Holly Kernan, KQED’s executive editor, said of the attack. “It definitely showed us what kind of changes we need to make going forward. For example, we are going to have separate networks in different parts of the organization so that we’re all working in a more secure environment.”
In the hours immediately following the malware infection, KQED’s email server stopped working. All network connected devices were taken offline. The radio station’s online broadcast went silent for more than 12 hours overnight. Radio journalists lost hours of work. Everyone with computers running Microsoft Windows was told not to touch them.
The wireless Internet in the building didn’t work for several days. Email didn’t return for two weeks.
“We’ve basically been putting everything together with duct tape for a month,” said Marisa Lagos, a former San Francisco Chronicle reporter who covers state politics for KQED. “From an outside point of view, we really made it work. But what our listeners don’t know is that people have been doing really crazy things to make sure no one notices that anything is wrong.”
Lagos said the morning after the hack, she and several other journalists reported to work before 5 a.m. to do the California Report because the show they had recorded had vanished.
KQED’s television newscast recorded segments from UC Hastings for two weeks in a row because of persistent problems stemming from the hack, Kernan said.
Even now, more than a month later, simple tasks once accomplished at the push of a button continue to require manual effort and creative workarounds.
To make sure everyone sees a copy of the script for an upcoming broadcast, reporters have to plug one of the stillworking computers into an old ink-jet printer, print out copies of the script and drop one off in a box at the center of the newsroom, where everyone can find it.
The timing of segments, once done automatically through the newsroom’s content management system, is now done the old fashioned way — with a stopwatch.
Even getting in and out of KQED’s buildings has become an ordeal. A new reporter who started just before the hack could not report to work in KQED’s San Jose bureau because she couldn’t get into the building, Kim said. The company’s network-connected card readers had been deactivated.
“It’s sort of interesting to see all the stupid little things we’ve relied on technology for,” Kim said. “And you don’t notice how dependent you are until it all breaks down.”
No one is sure how the ransomware got into KQED’s system.
The company had just updated its antivirus systems the morning of the attack, chief technology officer Dan Mansergh said.
It had up-to-date firewalls, email-scanning software and multiple malware detection programs. But the malware that infected their computers was a “new piece of software” that was not among the viruses for which KQED’s security vendor had been scanning, he said.
The attack encrypted files on “a small percentage” of Microsoft computers, though it appeared that the virus had detected “many more” computers and servers and was preparing to encrypt their files, too, before KQED’s technical staff was able to isolate the bug.
Ransomware, like the kind that infected KQED’s systems, can be transmitted to other computers and servers if they are all connected to the same network. Once the malware is in a system, it works to encrypt any number of files and then asks the victim to pay a ransom to restore them.
The attackers who hit KQED asked for 1.7 bitcoin per file. That’s roughly $3,637 apiece. With hundreds of thousands or millions of files possibly stored on a single PC, the asked-for ransom would have been far larger than KQED’s annual revenue of $71.6 million, of which $39.7 million comes from audience contributions and membership fees, according to the station’s annual financial disclosure.
KQED does not break out figures on its information technology spending. Since the recovery effort is ongoing, KQED’s Mansergh could not estimate the cost.
The attack, KQED employees said, did not appear to be targeted. In fact, it didn’t seem that the hackers knew what kind of organization they had hit.
KQED reported the hack to the FBI. The company declined to pay the ransom, in line with law enforcement’s usual advice, and has since been rebuilding the systems it lost and fortifying its network security to ensure that a virus brought in through one part of the organization cannot spread to another in the future.
“In an abundance of caution, we are wiping and restoring all Windows computers,” Mansergh wrote in an email to The Chronicle this month. “We will also be implementing other security measures to reduce the risk or impact of a future attack.”
Ransomware viruses are usually spread through email attachments, infected links or files that make their way into a computer via a USB drive.
Mansergh said the virus appeared to be a newer version of an attack that had been circulating in 2016.
It was not related to the two global ransomware attacks that locked down computers in more than 150 countries: WannaCry, which affected more than 230,000 computers including those in hospitals and public infrastructure agencies across Europe, or Petya, which spread through large firms, including FedEx Corp.’s TNT unit, food companies and legal groups. Both seemed to capitalize on Windows software that had not been updated.
The problem inherent in securing a company like KQED is that because it’s a news outlet that relies on public donations, there is a lot of information available about the company, the journalists who work there and what they cover, said Jake Williams, founder of cybersecurity firm Rendition InfoSec.
Nearly half of all ransomware attacks are caused by email or phishing scams that use publicly available information to pose as a trustworthy source, according to research from cybersecurity firm Datto.
Despite the challenges, several KQED workers said, they have also found a silver lining: The ransomware attack forced them to find workarounds and get creative, journalists said, and appreciate how fragile the systems they rely on really are.
“It’s sort of interesting to see all the stupid little things we’ve relied on technology for. And you don’t notice how dependent you are until it all breaks down.” Queena Kim, KQED senior editor