Meet Jamoora, the student-built music-interpreting robot that can be controlled using music and voice commands
Not all things traditional are boring. Some, in fact, are downright interesting, especially when it involves the ageold tradition of storytelling, with puppets and music. Now, mix tradition with today’s ubercool technological advancements, and what you have is best of both worlds.
Meet Jamoora, a music-interpreting robot designed by Varnit Jain and Anant Sharma, students from the Indraprastha Institute of Information Technology, Delhi (IIIT-Delhi), and faculty, Aman Parnami. The robot is named after jamoora or
jamura, a performer who plays a specific type of sidekick role — one who demonstrates unquestioning obedience to the master's directives — in the traditional folk theatre.
The bot can be controlled using music and voice commands, instead of the usual manual commands. “Our premise was to use robots that can be controlled without using hands. In the context of Indian puppetry, which is unique to our culture, there is usually an individual controlling the puppet, and two or three others playing the music. But, what if there is just one person? How can he or she manage all aspects of storytelling, puppets, and music? With the help of this puppet, we play live music and narrate a story, and the robot emotes accordingly,” says Varnit.
Professor Aman Parnami elaborates on how the idea came to be. “At Weave Lab, our mission is to ‘weave interactivity into everyday objects and experiences’, and the call for participation in the Student Innovation contest to be held during the 31st ACM User Interface Software and Technology conference, aligned with our mission. With the excitement to participate in this conference, I began looking for students, and couldn’t have chosen better people than Anant, a vocalist and musician, and Varnit, a programming expert.
“We are using voice commands for explicit control of the actions of the robot, and music is used to express the way the action is to be executed,” explains Anant. He elaborates on how there is a direct analogy with dance, where the act of dance can be triggered by someone instructing people to dance. However, the movements and the intensity are driven by the music being played. “We use this concept by incorporating Indian classical music for music detection,” he adds. “The robot can be programmed to perform many functions. Raga detection can also be programmed, and can be improved to include western scales as well. This can be used as a tool kit for kids to create and perform their own stories. We often think of creative stories as children, but do not have a good means to express them.”
Anant explains how the working of the robot will help in recognising the potential inclusion of multiple functionalities in it. “Jamoora communicates through our laptop, where the processing happens. The laptop receives two signals, an audio signal from a keyboard/harmonium and a speech signal through the microphone of the laptop. The madari gives commands to Jamoora using keywords woven into a script. The way Jamoora executes these actions is controlled by the Hindustani classical music played with his command. We used the speech recognition library in Python, which uses Google Speech API to detect the speech. Jamoora looks for the keywords in the detected sentence and executes the actions accordingly.”
Consequently, he adds, the audio signals received from the keyboard is processed in SuperCollider and sent to Python, where their algorithms detect the Hindustani classical raga. based on the notes, and identifies the corresponding emoptions. For instance: Happy - Desh Gloomy - Jaunpuriya Angry - Shree
Serious - Kaafi “Jamoora, now has information about the action and the way the action needs to be executed. The parameters of the movement are now changed, according to the action to be executed and the emotion detected,” says Anant.
Jamoora, in its current form, is an excellent example of the fusion of technology with more human aspects such as music and storytelling, believes, professor Parnami. “That is why we have been exploring other applications in the domains of entertainment and education technology. For entertainment, we are looking at it as a toy or a controller for smart home applications, whereas for education, we are viewing it as a tool to teach children about music, robotics, and storytelling. We are also open to other uses of our concept. Since music is an input to the system, a therapeutic use of it will be more of a side-effect rather than the direct purpose. However, if puppetry has an overall positive effect on patients, the bot will be suitable.”
Madhuvanti S. Krishnan The robot can be programmed to perform many functions. Raga detection can also be programmed, and can be improved to include western scales as well