The Smart Cube Uses Open Source to De­liver Cus­tom Re­search and An­a­lyt­ics Ser­vices

The Smart Cube of­fers a range of cus­tom re­search and an­a­lyt­ics ser­vices to its clients, and re­lies greatly on open source to do so. The UK-head­quar­tered com­pany has a global pres­ence with ma­jor bases in In­dia, Ro­ma­nia, and the US, and em­ploys more than 65

OpenSource For You - - Fossbytes -

“The whole an­a­lyt­ics ecosys­tem is shift­ing to­wards open source,” says Nitin Ag­gar­wal, vice pres­i­dent of data an­a­lyt­ics, The Smart Cube. Ag­gar­wal is lead­ing a team of over

150 de­vel­op­ers in In­dia, of which 100 are work­ing specif­i­cally on open source de­ploy­ments. The com­pany has cus­tomers rang­ing from big cor­po­ra­tions to fi­nan­cial ser­vices in­sti­tu­tions and man­age­ment con­sult­ing firms. And it pri­mar­ily lever­ages open source when of­fer­ing its ser­vices. Ag­gar­wal tells Open Source For You that open source has helped an­a­lyt­ics de­vel­op­ments to be more ag­ile in a col­lab­o­ra­tive en­vi­ron­ment. “We work as a true ex­ten­sion of our clients’ teams, and open source al­lows us to im­ple­ment quite a high de­gree of col­lab­o­ra­tion. Open source so­lu­tions also make it easy to op­er­a­tionalise an­a­lyt­ics, to meet the daily re­quire­ments of our clients,” Ag­gar­wal states.

Apart from help­ing in­crease col­lab­o­ra­tion and de­liver op­er­a­tionalised results, open source re­duces the over­all cost of an­a­lyt­ics for The Smart Cube, and pro­vides higher re­turns on in­vest­ments for its clients. The com­pany does have some pro­pri­etary so­lu­tions, but it uses an op­ti­mal mix of open and closed source soft­ware to cater to a wide va­ri­ety of in­dus­tries, busi­ness prob­lems and tech­nolo­gies.

“Our clients of­ten have an ex­ist­ing stack that they want us to use. But cer­tain prob­lems cre­ate large-scale com­plex an­a­lyt­i­cal work­loads that can only be man­aged us­ing open source tech­nolo­gies. Sim­i­larly, a num­ber of prob­lems are best solved us­ing al­go­rithms that are bet­ter re­searched and de­vel­oped in open source, while many de­scrip­tive or pre­dic­tive prob­lems are eas­ily solved us­ing pro­pri­etary so­lu­tions like Tableau, QlikView or SAS,” says Ag­gar­wal.

The Smart Cube team also mon­i­tors mar­ket trends and seeks cus­tomer in­puts at var­i­ous lev­els to eval­u­ate new tech­nolo­gies and tools, ad­just­ing the mix of open and closed source soft­ware as per re­quire­ments.

The chal­lenges with an­a­lyt­ics

Per­form­ing data anal­y­sis in­volves over­com­ing some hur­dles. In ad­di­tion to the in­trin­sic art of prob­lem solv­ing that an­a­lyt­ics pro­fes­sion­als need to have, there are some tech­ni­cal chal­lenges that ser­vice providers need to re­solve to ex­am­ine data. Ag­gar­wal says that stan­dar­d­is­ing data from struc­tured and un­struc­tured in­for­ma­tion has be­come chal­leng­ing. Like­wise, ob­tain­ing a sub­stan­tial amount of good train­ing sets is also hard, and de­ter­min­ing the right tech­nol­ogy stack to bal­ance cost and per­for­mance is equally dif­fi­cult.

Com­mu­nity so­lu­tions to help ex­tract data

Ag­gar­wal di­vulges var­i­ous com­mu­ni­ty­backed so­lu­tions that jointly power the data ex­trac­tion process and help to re­solve the tech­ni­cal chal­lenges in­volved in the data anal­y­sis process. To serve hun­dreds of clients in a short span of time, The Smart Cube has built a cus­tom frame­work. This frame­work of­fers data col­lec­tion and man­age­ment so­lu­tions that use open source. There is Apache Nutch and Kylo to en­able data lake man­age­ment, and Apache Beam to de­sign the whole data col­lec­tion process.

The Smart Cube lever­ages open source of­fer­ings, in­clud­ing Apache Spark and Hadoop, to an­a­lyse the bulk of ex­tracted struc­tured and un­struc­tured data. “We deal with data at the ter­abyte scale, and anal­y­sis of such mas­sive data sets is be­yond the ca­pa­bil­ity of a sin­gle com­mod­ity hard­ware. Tra­di­tional RDBMS (re­la­tional data­base man­age­ment sys­tems) also can­not man­age many types of un­struc­tured data like images and videos. Thus, we lever­age Apache Spark and Hadoop,” Ag­gar­wal says.

Pre­dic­tive an­a­lyt­ics us­ing open source sup­port

The Smart Cube is one of the lead­ing ser­vice providers in the nascent field of pre­dic­tive an­a­lyt­ics. This type of an­a­lyt­ics has be­come vi­tal for com­pa­nies op­er­at­ing in a tough com­pet­i­tive en­vi­ron­ment. Mak­ing pre­dic­tions isn’t easy. But open source helps on that front as well.

“A wide va­ri­ety of pre­dic­tive an­a­lyt­ics prob­lems can be solved us­ing open source. We take sup­port from open source so­lu­tions to work on ar­eas like churn pre­dic­tion, pre­dic­tive main­te­nance, rec­om­men­da­tion sys­tems and video an­a­lyt­ics,” says Ag­gar­wal. The com­pany uses Scikit-learn­ing with Python, Keras and Google’s Ten­sorFlow, to en­able pre­dic­tive anal­y­sis and deep learn­ing so­lu­tions for ma­jor pre­dic­tion is­sues.

Ad­di­tion­ally, in Septem­ber 2017,

The Smart Cube launched ‘Con­cept Lab’ that al­lows the firm to ex­per­i­ment at a faster pace, and de­velop and test so­lu­tion frame­works for client prob­lems. “This ap­proach, en­abled by opt­ing for open source, has gained us a lot of trac­tion with our cor­po­rate clients, be­cause we are able to pro­vide the flex­i­bil­ity and agility that they can­not achieve in­ter­nally,” Ag­gar­wal af­firms.

The bright fu­ture of data an­a­lyt­ics

Open source is projected to help data an­a­lyt­ics com­pa­nies in the fu­ture, too. “We ex­pect open source to dom­i­nate the fu­ture of the an­a­lyt­ics in­dus­try,” says Ag­gar­wal.

The Smart Cube is fore­see­ing good growth with open source de­ploy­ments. Ag­gar­wal states that open source will con­tinue to be­come more main­stream for data an­a­lyt­ics com­pa­nies and will grad­u­ally re­place pro­pri­etary so­lu­tions. “Most of the new R&D in an­a­lyt­ics will con­tinue to be on open source frame­works. The mar­ket for open source so­lu­tions will also con­sol­i­date over time as there is a huge base of small play­ers at present, which some­times con­fuses cus­tomers,” Ag­gar­wal states.

Ac­cord­ing to NASSCOM, In­dia will be­come one of the top three mar­kets in the data an­a­lyt­ics space, in the next three years. The IT trade body also pre­dicts that the Big Data an­a­lyt­ics sec­tor in the coun­try will wit­ness eight-fold growth by 2025, from the cur­rent US$ 2 bil­lion to a whop­ping US$ 16 bil­lion.

Com­pa­nies like The Smart Cube are an im­por­tant part of In­dia’s growth jour­ney in the an­a­lyt­ics mar­ket, and will in­flu­ence more busi­nesses to opt for open source in the fu­ture.

Nitin Ag­gar­wal, vice pres­i­dent of data an­a­lyt­ics, The Smart Cube

Newspapers in English

Newspapers from India

© PressReader. All rights reserved.