Ro­botic pup­peteer

Meet Jamoora, the stu­dent-built mu­sic-in­ter­pret­ing robot that can be con­trolled us­ing mu­sic and voice com­mands

The Hindu - - EDGE -

Not all things tra­di­tional are bor­ing. Some, in fact, are down­right in­ter­est­ing, es­pe­cially when it in­volves the ageold tra­di­tion of sto­ry­telling, with pup­pets and mu­sic. Now, mix tra­di­tion with to­day’s uber­cool tech­no­log­i­cal ad­vance­ments, and what you have is best of both worlds.

Meet Jamoora, a mu­sic-in­ter­pret­ing robot de­signed by Var­nit Jain and Anant Sharma, stu­dents from the In­draprastha In­sti­tute of In­for­ma­tion Tech­nol­ogy, Delhi (IIIT-Delhi), and fac­ulty, Aman Par­nami. The robot is named af­ter jamoora or

ja­mura, a per­former who plays a spe­cific type of side­kick role — one who demon­strates un­ques­tion­ing obe­di­ence to the mas­ter's di­rec­tives — in the tra­di­tional folk the­atre.

The bot can be con­trolled us­ing mu­sic and voice com­mands, in­stead of the usual man­ual com­mands. “Our premise was to use ro­bots that can be con­trolled with­out us­ing hands. In the con­text of In­dian pup­petry, which is unique to our cul­ture, there is usu­ally an in­di­vid­ual con­trol­ling the pup­pet, and two or three oth­ers play­ing the mu­sic. But, what if there is just one per­son? How can he or she man­age all as­pects of sto­ry­telling, pup­pets, and mu­sic? With the help of this pup­pet, we play live mu­sic and nar­rate a story, and the robot emotes ac­cord­ingly,” says Var­nit.

Pro­fes­sor Aman Par­nami elab­o­rates on how the idea came to be. “At Weave Lab, our mis­sion is to ‘weave in­ter­ac­tiv­ity into ev­ery­day ob­jects and ex­pe­ri­ences’, and the call for par­tic­i­pa­tion in the Stu­dent In­no­va­tion con­test to be held dur­ing the 31st ACM User In­ter­face Soft­ware and Tech­nol­ogy con­fer­ence, aligned with our mis­sion. With the ex­cite­ment to par­tic­i­pate in this con­fer­ence, I be­gan look­ing for stu­dents, and couldn’t have cho­sen bet­ter peo­ple than Anant, a vo­cal­ist and mu­si­cian, and Var­nit, a pro­gram­ming ex­pert.


“We are us­ing voice com­mands for ex­plicit con­trol of the ac­tions of the robot, and mu­sic is used to ex­press the way the ac­tion is to be ex­e­cuted,” ex­plains Anant. He elab­o­rates on how there is a di­rect anal­ogy with dance, where the act of dance can be trig­gered by some­one in­struct­ing peo­ple to dance. How­ever, the move­ments and the in­ten­sity are driven by the mu­sic be­ing played. “We use this con­cept by in­cor­po­rat­ing In­dian clas­si­cal mu­sic for mu­sic de­tec­tion,” he adds. “The robot can be pro­grammed to per­form many func­tions. Raga de­tec­tion can also be pro­grammed, and can be im­proved to in­clude west­ern scales as well. This can be used as a tool kit for kids to cre­ate and per­form their own sto­ries. We of­ten think of cre­ative sto­ries as chil­dren, but do not have a good means to ex­press them.”


Anant ex­plains how the work­ing of the robot will help in recog­nis­ing the po­ten­tial in­clu­sion of mul­ti­ple func­tion­al­i­ties in it. “Jamoora com­mu­ni­cates through our lap­top, where the pro­cess­ing hap­pens. The lap­top re­ceives two sig­nals, an au­dio sig­nal from a key­board/har­mo­nium and a speech sig­nal through the mi­cro­phone of the lap­top. The madari gives com­mands to Jamoora us­ing keywords wo­ven into a script. The way Jamoora ex­e­cutes these ac­tions is con­trolled by the Hin­dus­tani clas­si­cal mu­sic played with his com­mand. We used the speech recog­ni­tion li­brary in Python, which uses Google Speech API to de­tect the speech. Jamoora looks for the keywords in the de­tected sen­tence and ex­e­cutes the ac­tions ac­cord­ingly.”

Con­se­quently, he adds, the au­dio sig­nals re­ceived from the key­board is pro­cessed in Su­perCol­lider and sent to Python, where their al­go­rithms de­tect the Hin­dus­tani clas­si­cal raga. based on the notes, and iden­ti­fies the cor­re­spond­ing emop­tions. For in­stance: Happy - Desh Gloomy - Jaun­puriya An­gry - Shree

Se­ri­ous - Kaafi “Jamoora, now has in­for­ma­tion about the ac­tion and the way the ac­tion needs to be ex­e­cuted. The pa­ram­e­ters of the move­ment are now changed, ac­cord­ing to the ac­tion to be ex­e­cuted and the emo­tion de­tected,” says Anant.


Jamoora, in its cur­rent form, is an ex­cel­lent ex­am­ple of the fu­sion of tech­nol­ogy with more hu­man as­pects such as mu­sic and sto­ry­telling, be­lieves, pro­fes­sor Par­nami. “That is why we have been ex­plor­ing other ap­pli­ca­tions in the do­mains of en­ter­tain­ment and ed­u­ca­tion tech­nol­ogy. For en­ter­tain­ment, we are look­ing at it as a toy or a con­troller for smart home ap­pli­ca­tions, whereas for ed­u­ca­tion, we are view­ing it as a tool to teach chil­dren about mu­sic, ro­bot­ics, and sto­ry­telling. We are also open to other uses of our con­cept. Since mu­sic is an in­put to the sys­tem, a ther­a­peu­tic use of it will be more of a side-ef­fect rather than the di­rect pur­pose. How­ever, if pup­petry has an over­all pos­i­tive ef­fect on pa­tients, the bot will be suit­able.”

Mad­hu­vanti S. Krishnan The robot can be pro­grammed to per­form many func­tions. Raga de­tec­tion can also be pro­grammed, and can be im­proved to in­clude west­ern scales as well

Newspapers in English

Newspapers from India

© PressReader. All rights reserved.