Ten Emerg­ing Open Source Tech­nolo­gies

OpenSource For You - - Contents - By: Dr Anand Nayyar The au­thor is a pro­fes­sor at the Duy Tan Univer­sity in Viet­nam. He loves to work on open source tech­nolo­gies, em­bed­ded sys­tems, sen­sor net­works, etc. He can be reached at anand_ nayyar@ya­hoo.co.in. YouTube chan­nel: Gyaan with Anand www

This ar­ti­cle cov­ers emerg­ing open source tech­nolo­gies—the many trends that drive suc­cess to­day and will do so in the fu­ture. Th­ese range from how the en­tire tech in­dus­try is get­ting re­shaped by dig­i­tal trans­for­ma­tion and the makeover of all tech en­ter­prises from ‘Dig­i­tal Im­mi­grants’ to ‘Dig­i­tal Na­tives’, to the rise of in­no­va­tion ac­cel­er­a­tors and the emer­gence of next gen­er­a­tion open source tech plat­forms.

With open source tech­nol­ogy hav­ing been around since quite a long time now, a thriv­ing global com­mu­nity has grown around it; so code is shared among de­vel­op­ers and ev­ery­one can test, re-build and learn from each other. As in­dus­try has be­gun adopt­ing open source tech­nolo­gies in al­most all ver­ti­cals, many new tech­nolo­gies have used open source as their very foun­da­tion.

Here are some of the ar­eas in which ground­break­ing open source tech­nolo­gies are set to rev­o­lu­tionise the world as we know it.

1. Open source in ma­chine learn­ing

Ma­chine learn­ing (ML) is the study of al­go­rithms that use large data sets to learn, gen­er­alise and pre­dict. The most ex­cit­ing as­pect of ML is that with more data, the al­go­rithm im­proves its pre­dict­ing power. ML has acted as a strong base for self-driv­ing cars, speech recog­ni­tion, home au­to­ma­tion prod­ucts and much more. Ma­chine learn­ing is closely re­lated to com­pu­ta­tional statis­tics and also fo­cuses on mak­ing pre­dic­tions via com­put­ers. It is re­garded as an ef­fec­tive method for de­ploy­ing com­plex mod­els and al­go­rithms that lend them­selves to pre­dic­tion in the com­mer­cial space -- this is known as pre­dic­tive anal­y­sis. Ma­chine learn­ing tasks are broadly cat­e­gorised into su­per­vised learn­ing (semisu­per­vised learn­ing, ac­tive learn­ing and re­in­force­ment learn­ing) and un­su­per­vised learn­ing.

The most re­cent ML en­gines that have been open sourced by var­i­ous IT gi­ants are Google Cloud Ma­chine Learn­ing En­gine, Ten­sorFlow by Google, Ama­zon’s ML en­gine for AWS, Unity ML Agents, Apache Pre­dic­tionIO, Mi­crosoft Dis­trib­uted Ma­chine Learn­ing Tool­kit, etc.

Google Cloud Ma­chine Learn­ing En­gine: This is re­garded as a man­aged ser­vice en­abling users to eas­ily build op­er­a­tional ML mod­els to work on any type or size of data. It makes Ten­sorFlow Model per­form large scale train­ing on man­aged clus­ters, and is also equipped to man­age the trained mod­els for large scale on­line and batch pre­dic­tions. It is in­te­grated with Google Cloud,

al­low­ing users to ac­cess data on Google Stor­age.

Unity Ma­chine Learn­ing Agents: The Unity ML Agents SDK al­lows de­vel­op­ers and re­searchers to trans­form games and sim­u­la­tions de­vel­oped via Unity’s ed­i­tor into en­vi­ron­ments, where in­tel­li­gent agents can be trained ef­fec­tively us­ing deep re­in­force­ment learn­ing.

Apache Pre­dic­tionIO: This is an open source ML stack in­te­grated with Apache Spark, MLLib, HBase, Spray and Elas­tic Search to cre­ate pre­dic­tive en­gines for all sorts of ML tasks.

Ama­zon Ma­chine Learn­ing: The Ama­zon ML rec­om­men­da­tion en­gine is used to power Ama­zon Echo/ Dot, pow­ered by Alexa, Drone-Prime Air, Ama­zon Go and AWS based cloud ser­vices.

Mi­crosoft Dis­trib­uted Ma­chine Learn­ing Tool­kit: This tool­kit by Mi­crosoft pro­vides a frame­work for train­ing mod­els of ML on Big Data. The tool­kit con­tains both al­go­rithm and sys­tem in­no­va­tions to make tasks on Big Data highly scal­able, ef­fi­cient and flex­i­ble.

2. The R pro­gram­ming lan­guage

R is a free and open source ML lan­guage sup­port­ing sta­tis­ti­cal com­put­ing and the graph­ics lan­guage on a wide range of op­er­at­ing sys­tems. It pro­vides di­verse sta­tis­ti­cal func­tion­al­i­ties like lin­ear and non-lin­ear mod­el­ling, clas­si­cal sta­tis­ti­cal tests, time-se­ries anal­y­sis, clas­si­fi­ca­tion, clus­ter­ing and ad­vanced graph­i­cal tech­niques. It is a highly dy­namic, scal­able and ex­ten­si­ble lan­guage that pro­vides ad­vanced fea­tures for data ma­nip­u­la­tion, cal­cu­la­tion, graph­ics dis­play, ar­ray cal­cu­la­tion, data anal­y­sis tools, etc. It is a pro­gram­ming lan­guage that also con­tains con­di­tions, loops and other ca­pa­bil­i­ties.

R con­tains more than 11,000 pack­ages of var­i­ous kinds, which are avail­able through the Com­pre­hen­sive R Ar­chive Net­work (CRAN), Bio­con­duc­tor, Omega­hat, GitHub and other repos­i­to­ries.

3. Emerg­ing trends in blockchains and Bit­coin

Blockchain tech­nol­ogy is rapidly un­der­go­ing in­tense de­vel­op­ment due to great in­ter­est from academia and the in­dus­try sec­tor. A blockchain is re­garded as a shared, open source trans­ac­tional data­base for track­ing trans­ac­tions of dig­i­tal cur­rency like Bit­coin.

Ac­cord­ing to Don and Alex Tap­scott, “The blockchain is an in­cor­rupt­ible dig­i­tal ledger of eco­nomic trans­ac­tions that can be pro­grammed to record not just fi­nan­cial trans­ac­tions but vir­tu­ally every­thing of value.”

A blockchain is a con­tin­u­ously grow­ing list of records called blocks which are linked and se­cured us­ing cryp­tog­ra­phy, with ev­ery block hav­ing a hash poin­ter as a link to the pre­vi­ous block, a time stamp and trans­ac­tion data.

Blockchain plat­forms

ERIS: This al­lows ev­ery­one to cre­ate their own se­cure, low-cost ap­pli­ca­tion (that can run any­where) us­ing a blockchain and smart con­tract tech­nol­ogy.

Hy­draChain: This is an ex­ten­sion to the Ethereum plat­form, with added sup­port for cre­at­ing per­mis­sioned dis­trib­uted ledgers.

Mul­tiChain: This en­ables or­gan­i­sa­tions to rapidly de­sign, de­ploy and op­er­ate dis­trib­uted ledgers.

OpenChain: This is open source dis­trib­uted ledger tech­nol­ogy suited for or­gan­i­sa­tions work­ing on is­sues and man­ag­ing dig­i­tal as­sets in a highly se­cure and flex­i­ble man­ner.

Ethereum Project: This is a de­cen­tralised plat­form run­ning smart con­tracts: Ap­pli­ca­tions that run as such are pro­grammed with­out any lim­i­ta­tions of down­time, cen­sor­ship, fraud or third party in­volve­ment. Hyper­ledger: This is an open source col­lab­o­ra­tive ef­fort to ad­vance cross-in­dus­try blockchain tech­nol­ogy and en­sure trans­parency, in­ter­op­er­abil­ity and sup­port to bring blockchain tech­nolo­gies for com­mer­cial adop­tion.

4. Open source and the In­ter­net of Things (IoT)

The In­ter­net of Things (IoT) is highly frag­mented and chang­ing con­tin­u­ously. Open source is play­ing a cru­cial role in cre­at­ing IoT plat­forms as well as ready-made pro­to­types in terms of de­vel­op­ment boards for R&D and au­to­ma­tion.

IoT stan­dards, to­gether with Ar­ti­fi­cial In­tel­li­gence (AI), are con­trol­ling and in­ter­pret­ing a wide range of ac­tiv­i­ties in a smart man­ner. With­out the use of open source tech­nolo­gies, there is no Web. IoT re­quires the same level of ubiq­ui­tous com­mon ac­cess in its core func­tions and to the Web for shared ac­ces­si­bil­ity. IoT de­ploy­ments in the near fu­ture are ex­pected to con­nect and in­te­grate glob­ally with bil­lions of de­vices, as­sets, sen­sors and end points.

IoT plat­forms are re­garded as the mid­dle­ware layer be­tween IoT de­vices or end points and ser­vices that con­sume data out­puts. Th­ese plat­forms of­fer so­phis­ti­cated end point man­age­ment to con­trol the de­vices.

Open source IoT plat­forms

Kaa IoT: This is an ef­fi­cient, open source and cloud based IoT plat­form which en­ables data man­age­ment for con­nected ob­jects and back-end in­fra­struc­ture by pro­vid­ing server and end­point SDK com­po­nents. SiteWhere: This pro­vides in­ges­tion, stor­age, pro­cess­ing and in­te­gra­tion of de­vice data. It runs on core servers pro­vided by Apache Tom­cat, and con­tains Mon­goDB and HBase im­ple­men­ta­tions.

ThingS­peak: This al­lows users to col­lect and store sen­sor data in the cloud and on pop­u­lar IoT ap­pli­ca­tions de­vel­op­ment plat­forms. It works well with Ar­duino, ESP8266, Bea­gleBone, Rasp­berry Pi, MATLAB, etc. De­viceHive: This pro­vides Docker and Ku­ber­netes de­ploy­ment op­tions. It has the power to con­nect to any de­vice or hacker board via Rest APIs,

We­bSock­ets, MQTT, etc.

Thinger.io: This is an open source plat­form for IoT and pro­vides scal­able cloud in­fra­struc­ture for con­nect­ing de­vices. It sup­ports all types of boards like Ar­duino, ESP8266, Rasp­berry Pi and In­tel Edi­son.

Open source IoT hard­ware - de­vel­op­ment boards: Var­i­ous hard­ware de­vel­op­ment boards like Ar­duino, Free­duino, Rasp­berry Pi, BBC Mi­croBit, Or­ange Pi, Pine A64 as well as op­er­at­ing sys­tems like Rasp­bian, Kodi, etc, are based on open source.

5. Open source and Big Data an­a­lyt­ics

IDC says that world­wide rev­enues for Big Data and busi­ness an­a­lyt­ics will be up from US$ 130 bil­lion in

2016 to more than US$ 203 bil­lion in 2020 with an an­nual growth rate of 11.7 per cent. Nowa­days, most or­gan­i­sa­tions un­der­stand the value of cap­tur­ing all the data stream­ing in­side the busi­ness and hence em­ploy open source Big Data an­a­lyt­ics to gain cru­cial ad­van­tage from it.

Open source soft­ware and Big Data go hand in hand th­ese days since to­day’s ap­pli­ca­tions can han­dle di­verse data in an ef­fec­tive man­ner, as it grows ex­po­nen­tially in va­ri­ety, vol­ume, ve­loc­ity and ve­rac­ity.

The pop­u­lar Big Data an­a­lyt­ics tools and plat­forms are: Apache Hadoop: Low cost dis­trib­uted com­put­ing for Big Data

Grid Gain: Faster anal­y­sis of real-time data

Cas­san­dra: Man­ages huge data­bases

Ter­ra­s­tore: Pop­u­lar for scal­a­bil­ity and elas­tic­ity KNIME: Best tool for per­for­mance man­age­ment and data in­te­gra­tion

Rapid­miner: For faster pro­cess­ing of data and to sim­plify pre­dic­tive anal­y­sis

Solr: Scal­able and re­li­able tool for Big Data file trans­fer and ag­gre­ga­tion

Ter­ra­cotta: En­ables en­ter­prise ap­pli­ca­tions to store and man­age Big Data in server mem­ory

AVRO: Data se­ri­al­i­sa­tion sys­tem based on JSONde­fined schemas

Oozie: Co­or­di­nates sched­ul­ing of Hadoop jobs Zookeeper: Cen­tralised ser­vice for main­tain­ing con­fig­u­ra­tion in­for­ma­tion, nam­ing, dis­trib­uted syn­chro­ni­sa­tion and group ser­vices

6. Pro­gres­sive Web Apps (PWAs)

Pro­gres­sive Web Apps (PWAs) bring a mo­bile-app like ex­pe­ri­ence to end users with­out any app in­stal­la­tion re­quire­ments. De­signed by Google, th­ese apps were pro­moted in the Google I/O 2017 con­fer­ence.

PWAs take ad­van­tage of the much larger Web ecosys­tem, plug­ins, com­mu­nity and the rel­a­tive ease of de­ploy­ing and main­tain­ing a web­site when com­pared to a na­tive ap­pli­ca­tion in the re­spec­tive app stores. A PWA takes ad­van­tage of a mo­bile app’s char­ac­ter­is­tics, re­sult­ing in im­proved user re­ten­tion and per­for­mance, with­out the com­pli­ca­tions in­volved in main­tain­ing a mo­bile ap­pli­ca­tion.

Pro­gres­sive web­sites are rapidly grow­ing in pop­u­lar­ity — as a way to build apps with JavaScript, CSS and HTML, and they have a level of per­for­mance and us­abil­ity that’s nearly iden­ti­cal to na­tive apps.

PWAs are able to work with most browsers and de­vices, fit in all screens with re­spon­sive de­signs, en­able off­line con­nec­tiv­ity, and of­fer an app-like ex­pe­ri­ence with fea­tures like push no­ti­fi­ca­tions and Web app man­i­fest.

Tools for build­ing Pro­gres­sive Web Apps

Re­act: Man­aged and sup­ported by Face­book and is the foun­da­tion for Re­act Na­tive. It can eas­ily port apps built with Re­act to na­tive apps.

Poly­mer Tem­plate: Sup­ported by Google to use the PRPL pat­tern to op­ti­mise de­liv­ery of the app to the de­vice. Web­pack: Essen­tial for com­plex and front-end driven pro­gres­sive web­sites and com­prises in­nu­mer­able JavaScript ap­pli­ca­tions.

Light­house: Pow­er­ful Google PWA per­for­mance mon­i­tor­ing tool to test load times and per­for­mance in page loads, as well as se­cu­rity in net­work con­nec­tions, de­sign and the user in­ter­face.

7. Open source and cy­ber se­cu­rity

When cre­at­ing a se­cu­rity pol­icy for any or­gan­i­sa­tion, or when build­ing a se­cu­rity op­er­a­tions or re­search cen­tre, the prime re­quire­ment is to have the right peo­ple, pro­cesses and ef­fec­tive tools.

The open source mar­ket is filled with lots of se­cu­rity tools and even Linux dis­tri­bu­tions like Kali Linux, the Par­rot Se­cu­rity Tool Set, Net­work Se­cu­rity Tool Kit, Cy­borg Hawk and many more. All th­ese are used by se­cu­rity and pen­e­tra­tion test­ing pro­fes­sion­als for real-time se­cu­rity op­er­a­tions and to find vul­ner­a­bil­i­ties and back­doors in ex­ist­ing net­works.

Open source in­tel­li­gence (OSINT) is a method­ol­ogy for us­ing open source tools to col­lect in­for­ma­tion from pub­licly avail­able sources, carry out anal­y­sis on the data and take the right course of ac­tion. To pre­vent se­cu­rity at­tacks on a net­work, it is of ut­most im­por­tance to un­der­stand the in­for­ma­tion be­ing col­lected by the or­gan­i­sa­tion and the soft­ware be­ing used to col­lect it.

As per the lat­est open source se­cu­rity anal­y­sis re­ports (Black­duck Soft­ware-www.black­duck­soft­ware.com), 67 per cent of the ap­pli­ca­tions re­viewed con­tain OS se­cu­rity vul­ner­a­bil­i­ties, with 40 per cent of open source vul­ner­a­bil­i­ties in each ap­pli­ca­tion be­ing rated as ‘se­vere’.

In the US, it is ex­pected that the 2018 bud­get for the Pen­tagon and the US’ Depart­ment of De­fense (DoD) will in­clude the launch of a new pi­lot pro­gramme, for which 20 per cent of the cus­tom code de­vel­oped will use open source soft­ware be­cause as per Li­nus’s Law: “Open source soft­ware is more se­cure as com­pared to any­thing else.” GitHub hosts more than 70 mil­lion open source projects, of which 600,000 com­po­nents are down­loaded over 14 bil­lion times.

The fol­low­ing are the top op­er­at­ing sys­tems used by hack­ers and pen­e­tra­tion testers. Th­ese are un­der con­tin­u­ous de­vel­op­ment to make them im­pen­e­tra­ble against real-time threats:

Kali Linux

Par­rot Se­cu­rity OS

Back­Box Linux

Sa­mu­rai Web Test­ing Frame­work

Deft Linux


Net­work Se­cu­rity Tool­kit (NST)

Black­Arch Linux

ArchStrike Linux

Cy­borg Hawk

Fe­dora Se­cu­rity Spin


Node Zero


Dra­cos Linux

8. Open Source and vir­tual re­al­ity, aug­mented re­al­ity and mixed re­al­ity

Re­cent years have seen an in­creased in­ter­est in the im­ple­men­ta­tion and us­age of vir­tual re­al­ity, aug­mented re­al­ity and even mixed re­al­ity -- par­tic­u­larly in the ar­eas of health­care ap­pli­ca­tions, mil­i­tary, fash­ion, sports, con­struc­tion, me­dia, telecom­mu­ni­ca­tions, films and en­ter­tain­ment, en­gi­neer­ing and ed­u­ca­tion. How­ever, higher adop­tion is lim­ited due to the high costs of de­vel­op­ing such soft­ware, the lack of tech­ni­cal skills and the dif­fi­cul­ties in the im­ple­men­ta­tion en­vi­ron­ment. To over­come th­ese hur­dles, open source has come to the res­cue.

Open source has deep roots in this do­main, and th­ese will grow deeper and more ad­vanced in the com­ing few years -- in terms of live im­ple­men­ta­tions, SDKs and soft­ware. With the im­ple­men­ta­tion of open source, a new re­al­ity called XR or Ex­tended Re­al­ity has evolved, which is pretty ad­vanced and has many live im­ple­men­ta­tions.

Vir­tual re­al­ity, aug­mented re­al­ity and mixed re­al­ity sales are ex­pected to touch US$ 2.8 bil­lion by 2018. Var­i­ous com­pa­nies like Face­book, Google, Mi­crosoft, Magic Leap, HTC, Sam­sung, WorldViz, Unity, Snap and First­Hand Tech­nolo­gies have al­ready started adopt­ing open source based tech­nolo­gies for bring­ing out hard­ware and soft­ware cum SDKs sup­port­ing VR, AR and MR.

The Open Source Vir­tual Re­al­ity (OSVR) Con­sor­tium and Xil­inx are also work­ing on com­ing out with var­i­ous VR hard­ware and soft­ware for end users.

The fol­low­ing are var­i­ous tool­kits, plat­forms and SDKs for VR, AT and MR:

OSVR - Open Source Vir­tual Re­al­ity for gam­ing

ARToolkit – An open source AR SDK

Aper­tus VR - An open source VR and AR en­gine OpenS­pace3D - An open source plat­form for 3D en­vi­ron­ments Holokit - An open source MR kit

Mixare - An open source AR en­gine

High Fidelity - An open source shared VR app OpenMask - Open source mid­dle­ware for VR CalVR - An open source VR soft­ware frame­work

10. Open source and cloud com­put­ing

Cloud com­put­ing is one of the most sig­nif­i­cant tech­nolo­gies to have emerged in the last cou­ple of years and per­haps, even the next cou­ple of years, re­sult­ing in bil­lions of dol­lars in in­vest­ments. The cloud com­put­ing in­dus­try is ex­pected to touch US$ 241 bil­lion by 2020. Com­mon open source cloud ap­pli­ca­tions are CloudS­tack, OpenNe­bula or OpenERP servers.

In­no­va­tions in cloud com­put­ing have led to re­mark­able trends like the in­creas­ing adop­tion of the pub­lic cloud; the adop­tion of con­tainer tech­nolo­gies like Docker, Ku­ber­netes, Apache Me­sos, LXD, etc; and the adop­tion of DevOps in ap­pli­ca­tion de­liv­ery and open source tool­ing by com­pa­nies.

Cloud com­put­ing and open source share com­mon goals like min­imis­ing costs by not pay­ing li­cence fees. It is gen­er­ally ac­cepted that with­out open source soft­ware, cloud com­put­ing would not have been able to grow as rapidly. To­day most cloud com­put­ing ven­dors use open source soft­ware to de­velop their sys­tems. Ex­am­ples in­clude Red Hat for cloud op­er­at­ing sys­tems and in­fra­struc­ture, Eu­ca­lyp­tus and OpenS­tack for In­fra­struc­ture-as-a-Ser­vice im­ple­men­ta­tion, Cloud­era for the open source Hadoop soft­ware frame­work, OpenNe­bula for the open source VM, the Xen Su­per­vi­sor for server vir­tu­al­i­sa­tion man­age­ment, Cloud Foundry and OpenShift for an open Plat­form-as-a-Ser­vice, etc.

The fol­low­ing are some use­ful tools for cloud com­put­ing. OpenS­tack: This fa­cil­i­tates data cen­tres to of­fer the com­bined re­sources of com­put­ing, stor­age, net­work­ing and a GUI via a dash­board for ef­fec­tive man­age­ment. CloudS­tack: This de­ploys and man­ages large net­works of vir­tual machines.

Eu­ca­lyp­tus: This fa­cil­i­tates easy mi­gra­tion of apps and data; it cre­ates pri­vate and hy­brid cloud en­vi­ron­ments. openQRM: En­ables the build­ing of pri­vate, pub­lic and the hy­brid IaaS.

OpenShift: En­ables easy man­age­ment of cloud based app de­vel­op­ment.

Other open source cloud com­put­ing sim­u­la­tors in­clude CloudSim, CloudA­n­a­lyst, iCanCloud, GreenCloud, CloudSched, etc.

Newspapers in English

Newspapers from India

© PressReader. All rights reserved.