Facebook “just trying to keep the lights on” during pandemic
SAN FRANCIS CO » As the coronavirus spread around the world and people everywhere were ordered to stay home, phone calls over Facebook’s apps more than doubled. In many countries, messaging on Instagram and Facebook soared by more than 50%, while group calls in Italy jumped by more than 1,000%. And hungry for information, people clicked repeatedly on virus news stories shown by the social network.
Inside Facebook, that meant the pressure was on.
“We’re just trying to keep the lights on over here,” said Mark Zuckerberg, Facebook’s chief executive.
As airlines, hotels, restaurants and other companies struggle to stay afloat during the pandemic, Facebook is also laboring to cope with the fallout. But unlike those other businesses, the Silicon Valley giant is being strained by the coronavirus in a different way: Its usage is going through the roof.
Skyrocketing traffic and a crush of new users are now stressing Facebook’s systems just as its 45,000 employees are dealing with working remotely for the first time. The company is trying to keep its users’ data secure while employees who sift through posts to moderate content do so from home. At the same time, Facebook has added to its workload by promising to do more to limit virus misinformation.
It is a pressure test moment for
Facebook, which has for years grappled with a backlash over privacy and toxic content, but now has a chance to change that narrative and be seen as an essential communications and information tool during the outbreak.
“The usage growth from COVID-19 is unprecedented across the industry, and we are experiencing new records in usage almost every day,” Alex Schultz and Jay Parikh, two Facebook vice presidents working on infrastructure, said in a blog post Tuesday. “Maintaining stability throughout these spikes in usage is more challenging than usual now that most of our employees are working from home.”
What has saved Facebook’s network from crashing altogether,
Zuckerberg said, was that the virus and the quarantines have had the largest impact in just a few areas where Facebook operates. Facebook is banned in China, where the virus first appeared, for instance.
Those areas that have the highest concentration of people using Facebook’s services during peak hours from home are also spread out by time zone, Zuckerberg said, which staggers the swell of traffic.
“It really is a big technical challenge,” he said. “We’re basically trying to ready everything we can.” He said Facebook had mobilized its engineers to make sure the company has enough computing capacity and adequate support to handle the surges.
The strain has been compounded by Facebook’s workforce adapting to working from home, which had been discouraged in the past.
To communicate, Facebook employees were told to use BlueJeans, which provides technology for videoconferencing calls, they said. But they quickly found that calls were frozen, or the video quality so bad that it was hard to make out who was speaking. Many employees instead turned to Apple’s Facetime feature, Google video hangouts or Zoom conference calls.
Issues quickly piled up. Two days into working from home, some Facebook managers sent the engineering teams a message: Limit idle chat on work message boards.
Facebook employees had been posting on those boards at a record rate, according to one employee. While some workers were sharing tips and best practices of how to set up a home office, others were sharing links to buy heirloom seeds for at-home farming, and instructions on how to sew their own face masks, one employee said.
Other snafus surfaced. Earlier this month, a bug within Facebook’s system began marking thousands of posts by major news outlets such as Politico and The Sydney Morning Herald as spam. It took Facebook a day to correct the mistake, as engineers struggled to communicate remotely with one another over how the bug had been introduced and what it would take to fix it.
While they scrambled, rumors spread among Facebook’s users over the source of the bug, with many accusing the company of censoring people’s speech. Internally, Facebook managers said that while the bug was routine, the amount of time it took to fix it was not.
Working from home has also made moderating Facebook’s posts more difficult. This month, Facebook put its army of global contractors from outside agencies on paid leave. Those contractors, who number more than 15,000, are responsible for sorting through the posts, images and videos that flow through Facebook’s services on a daily basis to weed out sensitive, explicit or hateful material.
As the outbreak spread, contractors were ordered not to come into the office, where they worked on protected networks behind virtual firewalls to maintain user privacy. Many of those contractors do not have the same technology setup at home.
Facebook is still trying to figure out how to let the contractors continue working. For now, it is relying on full-time employees, who do not have the training or the time, to moderate the posts themselves.