U.S. spies lag rivals in seizing on open source data hiding in plain sight

Rep. Jim Himes, a Connecticut Democrat and longtime Intelligence Committee member, said he believed there needed to be “some cultural change inside places like the CIA where people are doing what they’re doing for the excitement of stealing critical secrets as opposed to reviewing social media pages.”

WASHINGTON — As alarms began to go off globally about a novel coronavirus spreading in China, officials in Washington turned to the intelligence agencies for insights about the threat the virus posed to America.

But the most useful early warnings came not from spies or intercepts, according to a recent congressional review of classified reports from December 2019 and January 2020. Officials were instead relying on public reporting, diplomatic cables and analysis from medical experts — some examples of so-called open source intelligence, or OSINT.

Predicting the next pandemic or the next government to fall will require better use of open source material, the review found.

“There is little indication that the Intelligence Community’s exquisite collection capabilities were generating information that was valuable to policymakers,” wrote the authors of the review, conducted by Democrats on the House Intelligence Committee.

That echoes what many current and former intelligence officials are increasingly warning: The $90 billion U.S. spy apparatus is falling behind because it has not embraced collecting open-source intelligence as adversaries including China ramp up their efforts.

This doesn’t diminish the importance of traditional intelligence. Spy agencies have unique powers to penetrate global communications and cultivate agents. They scored a high-profile success when the Biden administration publicized ultimately correct intelligence findings that Russian President Vladimir Putin intended to invade Ukraine.

But officials and experts worry that the U.S. hasn’t invested enough people or money in analyzing publicly available data or taking advantage of advanced technologies that can yield critical insights. Commercial satellite imagery, social media and other online data have given private companies and independent analysts new powers to reveal official secrets. And China is known to have stolen or acquired control over huge amounts of data on Americans, with growing concerns in Washington about Beijing’s influence over widely used apps like TikTok.

“Open source is really a bellwether for whether the intelligence community can protect the country,” said Kristin Wood, a former senior official at the CIA who is now chief executive at the Grist Mill Exchange, a commercial data platform. “We collectively as a nation aren’t preparing a defense for the ammunition that our adversaries are stockpiling.”

Intelligence agencies face several obstacles to using open source intelligence. Some are technological. Officers working on classified networks are often not able to easily access the unclassified internet or open data sources, for example. There are also concerns about civil liberties and protecting First Amendment rights.

But some experts also question whether agencies are held back by a reflexive belief that top-secret information is more valuable.

Rep. Jim Himes, a Connecticut Democrat and longtime Intelligence Committee member, said he believed there needed to be “some cultural change inside places like the CIA where people are doing what they’re doing for the excitement of stealing critical secrets as opposed to reviewing social media pages.”

In one 2017 test held by the National Geospatial-Intelligence Agency, a human team competed against a computer programmed with algorithms to identify Chinese surface-toair missile sites using commercial imagery.

Both the humans and the computer identified 90% of the sites, Stanford University professor Amy Zegart wrote in the book “Spies, Lies, and Algorithms,” but the computer needed just 42 minutes — and it took the human team 80 times longer.

Reports created using commercial satellites, online posts and other open sources — like the daily analyses on Russian and Ukrainian military tactics published by the Institute for the Study of War — are widely read by lawmakers and intelligence officials.

“There is a lot of opensource capability that the U.S. intelligence community can pretty much rely on to be there,” said Frederick Kagan, a senior fellow at the American Enterprise Institute who oversees the creation of those reports. “What it needs to do is figure out how to leverage that ecosystem instead of trying to buy it.”

Most of the 18 U.S. spy agencies have open-source programs, from the CIA’s Open Source Enterprise to a 10-person program in the Department of Homeland Security’s intelligence arm. But top officials acknowledge there isn’t consistency across those programs in how they analyze open-source information or how they use and share it.

“We’re not paying enough attention to each other and so we’re not learning the lessons that different parts of the (intelligence community) are learning, and we’re not scaling solutions,” said Avril Haines, the U.S. director of national intelligence, at an industry event last year sponsored by the Potomac Officers Club. “And we’re not taking advantage of some of the outside expertise and information and work that could be taken advantage of.”

The Open Source Enterprise headquartered at the CIA is the successor to the Foreign Broadcast Information Service, where for generations employees monitored broadcasts to translate them for analysts.