Google hosts ‘Operation Rosehub’ to patch thousands of open source projects

To narrow down the impact of the infamous ‘Apache Commons Collections Deserialisation Vulnerability’ that had affected several Java-based programs, Google has silently kickstarted its ‘Operation Rosehub’. A 50-member team of Google employees drove the new initiative for the first time in 2016 to help patch over 2,600 open source projects.

Internally called ‘Mad Gadget’ by Google’s engineers, the vulnerability was spotted in early 2015. It grabbed the attention of companies like Oracle, Cisco, Red Hat, Jenkins, VMware, IBM, Intel, Adobe and HP a few months after its discovery by security researchers from Foxglove Security. The IT companies immediately issued security alerts to let enterprises patch their proprietary offerings. However, a Google employee took the step to fix the issue in affected open source projects.

“Operation Rosehub was organised from the bottom-up on companywide mailing lists. Employees volunteered and patches were sent out in a matter of weeks,” Google’s software engineer Justine Tunney wrote in a blog post.

The researchers had reported that the vulnerability was a part of the seven ‘gadget’ classes within the Apache Commons Collections library versions 3.0, 3.1, 3.2, 3.2.1 and 4.0. These were the classes for handling Java object deserialisation that was used alongside the serialisation function to convert data from one format to another.

As the library was vital for many software operations, it was deployed by various commercial and open source projects. The Google employee used the pull request feature on GitHub to inform developers to patch their community solutions at the initial stage. But to reach the masses, a new development was needed.

Google has the Rosie tool that helps developers implement large-scale changes to codebases owned by its engineering teams. But GitHub, where a large number of Mad Gadget-affected projects were hosted, does not offer any such help. This is the reason the search giant formed a special task force and started working on the patches. “Patches were sent to many projects, avoiding threats to public security for years to come,” said the Google engineer.

Though some of the patches were just one-line changes, Google’s engineers took months to fix the vulnerable projects on GitHub. They spent a part of their daily routine at Google to jointly work on the patches.

For the remaining projects that are yet to be patched, the Google team is using an open source data set on BigQuery. This helps the engineers to identify the vulnerability in listed solutions.

“Going forward, we believe the best thing to do is to build awareness. We want to draw attention to the fact that the tools now exist for fixing software on a massive scale, and that these work best when that software is open,” Tunney concluded.

The Mad Gadget flaw has led to some serious attacks in the past. It enabled a hacker to gain access to the San Francisco Municipal Railway system last November. And it caused a PayPal vulnerability that was discovered in December 2015.