Facebook open sources its data compression and storage technologies

Facebook has open sourced a collection of compression and storage tools. The social networking giant wants to empower enterprises to scale their existing platforms to reach a larger user base while minimising their storage needs.

The company has released its Zstandard compression algorithm that offers a lossless compression technology, replacing the existing libraries powered by the outdated Deflate algorithm. Facebook claims that its new offering is much faster and more efficient than standard compression models. The Zstandard compression algorithm is supposedly five times faster than Deflate and files are 10 per cent smaller. The core algorithm inside the popular compression formats (Zip, gzip and zlib) is the Deflate compression model. Deflate as well as other algorithms have offered either good compression quality or fast compression speed, but never both. However, Facebook is set to face the real challenge by introducing the new standard.

Apart from Zstandard, Facebook has open sourced MyRocks — the next generation MySQL storage engine. According to the company, MyRocks can save 50 per cent of storage space when compared to compressed InnoDB. Likewise, MyRocks-powered RocksDB has several advantages over the default InnoDB storage. RocksDB uses a minimal number of overall I/O per read or update. The database is comparatively lightweight, and puts a negligible strain on the write endurance of Flash storage. You can use MyRocks only via a fork of MySQL 5.6.

The best part is that both Zstandard and MyRocks have been tested at scale within Facebook for over six months. Facebook has noted impressive results with both the technologies. Moreover, the Mark Zuckerberg-led team believes that these technologies will be adopted more quickly as an open source solution than as a proprietary package of a single company.

Late last month, Facebook released the code of its three artificial intelligence (AI) tools that are an upgrade on the current state of machine vision technology.

The open source tools, namely DeepMask, SharpMask and MultiPathNet, are designed to enable image segmentation and label object masks in photographs stored on a system.