Ten Emerging Open Source Technologies

This article covers emerging open source technologies—the many trends that drive success today and will do so in the future. These range from how the entire tech industry is getting reshaped by digital transformation and the makeover of all tech enterprises from ‘Digital Immigrants’ to ‘Digital Natives’, to the rise of innovation accelerators and the emergence of next generation open source tech platforms.

With open source technology having been around since quite a long time now, a thriving global community has grown around it; so code is shared among developers and everyone can test, re-build and learn from each other. As industry has begun adopting open source technologies in almost all verticals, many new technologies have used open source as their very foundation.

Here are some of the areas in which groundbreaking open source technologies are set to revolutionise the world as we know it.

1. Open source in machine learning

Machine learning (ML) is the study of algorithms that use large data sets to learn, generalise and predict. The most exciting aspect of ML is that with more data, the algorithm improves its predicting power. ML has acted as a strong base for self-driving cars, speech recognition, home automation products and much more. Machine learning is closely related to computational statistics and also focuses on making predictions via computers. It is regarded as an effective method for deploying complex models and algorithms that lend themselves to prediction in the commercial space -- this is known as predictive analysis. Machine learning tasks are broadly categorised into supervised learning (semisupervised learning, active learning and reinforcement learning) and unsupervised learning.

The most recent ML engines that have been open sourced by various IT giants are Google Cloud Machine Learning Engine, TensorFlow by Google, Amazon’s ML engine for AWS, Unity ML Agents, Apache PredictionIO, Microsoft Distributed Machine Learning Toolkit, etc.

Google Cloud Machine Learning Engine: This is regarded as a managed service enabling users to easily build operational ML models to work on any type or size of data. It makes TensorFlow Model perform large scale training on managed clusters, and is also equipped to manage the trained models for large scale online and batch predictions. It is integrated with Google Cloud,

allowing users to access data on Google Storage.

Unity Machine Learning Agents: The Unity ML Agents SDK allows developers and researchers to transform games and simulations developed via Unity’s editor into environments, where intelligent agents can be trained effectively using deep reinforcement learning.

Apache PredictionIO: This is an open source ML stack integrated with Apache Spark, MLLib, HBase, Spray and Elastic Search to create predictive engines for all sorts of ML tasks.

Amazon Machine Learning: The Amazon ML recommendation engine is used to power Amazon Echo/ Dot, powered by Alexa, Drone-Prime Air, Amazon Go and AWS based cloud services.

Microsoft Distributed Machine Learning Toolkit: This toolkit by Microsoft provides a framework for training models of ML on Big Data. The toolkit contains both algorithm and system innovations to make tasks on Big Data highly scalable, efficient and flexible.

2. The R programming language

R is a free and open source ML language supporting statistical computing and the graphics language on a wide range of operating systems. It provides diverse statistical functionalities like linear and non-linear modelling, classical statistical tests, time-series analysis, classification, clustering and advanced graphical techniques. It is a highly dynamic, scalable and extensible language that provides advanced features for data manipulation, calculation, graphics display, array calculation, data analysis tools, etc. It is a programming language that also contains conditions, loops and other capabilities.

R contains more than 11,000 packages of various kinds, which are available through the Comprehensive R Archive Network (CRAN), Bioconductor, Omegahat, GitHub and other repositories.

3. Emerging trends in blockchains and Bitcoin

Blockchain technology is rapidly undergoing intense development due to great interest from academia and the industry sector. A blockchain is regarded as a shared, open source transactional database for tracking transactions of digital currency like Bitcoin.

According to Don and Alex Tapscott, “The blockchain is an incorruptible digital ledger of economic transactions that can be programmed to record not just financial transactions but virtually everything of value.”

A blockchain is a continuously growing list of records called blocks which are linked and secured using cryptography, with every block having a hash pointer as a link to the previous block, a time stamp and transaction data.

Blockchain platforms

ERIS: This allows everyone to create their own secure, low-cost application (that can run anywhere) using a blockchain and smart contract technology.

HydraChain: This is an extension to the Ethereum platform, with added support for creating permissioned distributed ledgers.

MultiChain: This enables organisations to rapidly design, deploy and operate distributed ledgers.

OpenChain: This is open source distributed ledger technology suited for organisations working on issues and managing digital assets in a highly secure and flexible manner.

Ethereum Project: This is a decentralised platform running smart contracts: Applications that run as such are programmed without any limitations of downtime, censorship, fraud or third party involvement. Hyperledger: This is an open source collaborative effort to advance cross-industry blockchain technology and ensure transparency, interoperability and support to bring blockchain technologies for commercial adoption.

4. Open source and the Internet of Things (IoT)

The Internet of Things (IoT) is highly fragmented and changing continuously. Open source is playing a crucial role in creating IoT platforms as well as ready-made prototypes in terms of development boards for R&D and automation.

IoT standards, together with Artificial Intelligence (AI), are controlling and interpreting a wide range of activities in a smart manner. Without the use of open source technologies, there is no Web. IoT requires the same level of ubiquitous common access in its core functions and to the Web for shared accessibility. IoT deployments in the near future are expected to connect and integrate globally with billions of devices, assets, sensors and end points.

IoT platforms are regarded as the middleware layer between IoT devices or end points and services that consume data outputs. These platforms offer sophisticated end point management to control the devices.

Open source IoT platforms

Kaa IoT: This is an efficient, open source and cloud based IoT platform which enables data management for connected objects and back-end infrastructure by providing server and endpoint SDK components. SiteWhere: This provides ingestion, storage, processing and integration of device data. It runs on core servers provided by Apache Tomcat, and contains MongoDB and HBase implementations.

ThingSpeak: This allows users to collect and store sensor data in the cloud and on popular IoT applications development platforms. It works well with Arduino, ESP8266, BeagleBone, Raspberry Pi, MATLAB, etc. DeviceHive: This provides Docker and Kubernetes deployment options. It has the power to connect to any device or hacker board via Rest APIs,

WebSockets, MQTT, etc.

Thinger.io: This is an open source platform for IoT and provides scalable cloud infrastructure for connecting devices. It supports all types of boards like Arduino, ESP8266, Raspberry Pi and Intel Edison.

Open source IoT hardware - development boards: Various hardware development boards like Arduino, Freeduino, Raspberry Pi, BBC MicroBit, Orange Pi, Pine A64 as well as operating systems like Raspbian, Kodi, etc, are based on open source.

5. Open source and Big Data analytics

IDC says that worldwide revenues for Big Data and business analytics will be up from US$ 130 billion in

2016 to more than US$ 203 billion in 2020 with an annual growth rate of 11.7 per cent. Nowadays, most organisations understand the value of capturing all the data streaming inside the business and hence employ open source Big Data analytics to gain crucial advantage from it.

Open source software and Big Data go hand in hand these days since today’s applications can handle diverse data in an effective manner, as it grows exponentially in variety, volume, velocity and veracity.

The popular Big Data analytics tools and platforms are: Apache Hadoop: Low cost distributed computing for Big Data

Grid Gain: Faster analysis of real-time data

Cassandra: Manages huge databases

Terrastore: Popular for scalability and elasticity KNIME: Best tool for performance management and data integration

Rapidminer: For faster processing of data and to simplify predictive analysis

Solr: Scalable and reliable tool for Big Data file transfer and aggregation

Terracotta: Enables enterprise applications to store and manage Big Data in server memory

AVRO: Data serialisation system based on JSONdefined schemas

Oozie: Coordinates scheduling of Hadoop jobs Zookeeper: Centralised service for maintaining configuration information, naming, distributed synchronisation and group services

6. Progressive Web Apps (PWAs)

Progressive Web Apps (PWAs) bring a mobile-app like experience to end users without any app installation requirements. Designed by Google, these apps were promoted in the Google I/O 2017 conference.

PWAs take advantage of the much larger Web ecosystem, plugins, community and the relative ease of deploying and maintaining a website when compared to a native application in the respective app stores. A PWA takes advantage of a mobile app’s characteristics, resulting in improved user retention and performance, without the complications involved in maintaining a mobile application.

Progressive websites are rapidly growing in popularity — as a way to build apps with JavaScript, CSS and HTML, and they have a level of performance and usability that’s nearly identical to native apps.

PWAs are able to work with most browsers and devices, fit in all screens with responsive designs, enable offline connectivity, and offer an app-like experience with features like push notifications and Web app manifest.

Tools for building Progressive Web Apps

React: Managed and supported by Facebook and is the foundation for React Native. It can easily port apps built with React to native apps.

Polymer Template: Supported by Google to use the PRPL pattern to optimise delivery of the app to the device. Webpack: Essential for complex and front-end driven progressive websites and comprises innumerable JavaScript applications.

Lighthouse: Powerful Google PWA performance monitoring tool to test load times and performance in page loads, as well as security in network connections, design and the user interface.

7. Open source and cyber security

When creating a security policy for any organisation, or when building a security operations or research centre, the prime requirement is to have the right people, processes and effective tools.

The open source market is filled with lots of security tools and even Linux distributions like Kali Linux, the Parrot Security Tool Set, Network Security Tool Kit, Cyborg Hawk and many more. All these are used by security and penetration testing professionals for real-time security operations and to find vulnerabilities and backdoors in existing networks.

Open source intelligence (OSINT) is a methodology for using open source tools to collect information from publicly available sources, carry out analysis on the data and take the right course of action. To prevent security attacks on a network, it is of utmost importance to understand the information being collected by the organisation and the software being used to collect it.

As per the latest open source security analysis reports (Blackduck Software-www.blackducksoftware.com), 67 per cent of the applications reviewed contain OS security vulnerabilities, with 40 per cent of open source vulnerabilities in each application being rated as ‘severe’.

In the US, it is expected that the 2018 budget for the Pentagon and the US’ Department of Defense (DoD) will include the launch of a new pilot programme, for which 20 per cent of the custom code developed will use open source software because as per Linus’s Law: “Open source software is more secure as compared to anything else.” GitHub hosts more than 70 million open source projects, of which 600,000 components are downloaded over 14 billion times.

The following are the top operating systems used by hackers and penetration testers. These are under continuous development to make them impenetrable against real-time threats:

Kali Linux

Parrot Security OS

BackBox Linux

Samurai Web Testing Framework

Deft Linux

CAINE

Network Security Toolkit (NST)

BlackArch Linux

ArchStrike Linux

Cyborg Hawk

Fedora Security Spin

BugTraq

Node Zero

Weakerthan

Dracos Linux

8. Open Source and virtual reality, augmented reality and mixed reality

Recent years have seen an increased interest in the implementation and usage of virtual reality, augmented reality and even mixed reality -- particularly in the areas of healthcare applications, military, fashion, sports, construction, media, telecommunications, films and entertainment, engineering and education. However, higher adoption is limited due to the high costs of developing such software, the lack of technical skills and the difficulties in the implementation environment. To overcome these hurdles, open source has come to the rescue.

Open source has deep roots in this domain, and these will grow deeper and more advanced in the coming few years -- in terms of live implementations, SDKs and software. With the implementation of open source, a new reality called XR or Extended Reality has evolved, which is pretty advanced and has many live implementations.

Virtual reality, augmented reality and mixed reality sales are expected to touch US$ 2.8 billion by 2018. Various companies like Facebook, Google, Microsoft, Magic Leap, HTC, Samsung, WorldViz, Unity, Snap and FirstHand Technologies have already started adopting open source based technologies for bringing out hardware and software cum SDKs supporting VR, AR and MR.

The Open Source Virtual Reality (OSVR) Consortium and Xilinx are also working on coming out with various VR hardware and software for end users.

The following are various toolkits, platforms and SDKs for VR, AT and MR:

OSVR - Open Source Virtual Reality for gaming

ARToolkit – An open source AR SDK

Apertus VR - An open source VR and AR engine OpenSpace3D - An open source platform for 3D environments Holokit - An open source MR kit

Mixare - An open source AR engine

High Fidelity - An open source shared VR app OpenMask - Open source middleware for VR CalVR - An open source VR software framework

10. Open source and cloud computing

Cloud computing is one of the most significant technologies to have emerged in the last couple of years and perhaps, even the next couple of years, resulting in billions of dollars in investments. The cloud computing industry is expected to touch US$ 241 billion by 2020. Common open source cloud applications are CloudStack, OpenNebula or OpenERP servers.

Innovations in cloud computing have led to remarkable trends like the increasing adoption of the public cloud; the adoption of container technologies like Docker, Kubernetes, Apache Mesos, LXD, etc; and the adoption of DevOps in application delivery and open source tooling by companies.

Cloud computing and open source share common goals like minimising costs by not paying licence fees. It is generally accepted that without open source software, cloud computing would not have been able to grow as rapidly. Today most cloud computing vendors use open source software to develop their systems. Examples include Red Hat for cloud operating systems and infrastructure, Eucalyptus and OpenStack for Infrastructure-as-a-Service implementation, Cloudera for the open source Hadoop software framework, OpenNebula for the open source VM, the Xen Supervisor for server virtualisation management, Cloud Foundry and OpenShift for an open Platform-as-a-Service, etc.

The following are some useful tools for cloud computing. OpenStack: This facilitates data centres to offer the combined resources of computing, storage, networking and a GUI via a dashboard for effective management. CloudStack: This deploys and manages large networks of virtual machines.

Eucalyptus: This facilitates easy migration of apps and data; it creates private and hybrid cloud environments. openQRM: Enables the building of private, public and the hybrid IaaS.

OpenShift: Enables easy management of cloud based app development.

Other open source cloud computing simulators include CloudSim, CloudAnalyst, iCanCloud, GreenCloud, CloudSched, etc.