Education app uses photos to help learners form proper sentences
PITTSBURGH, Pennsylvania: Students in the Carnegie Mellon University chapter of Design for America (DFA) have developed a free mobile app that uses visual cues to help children and adults — particularly those with language difficulties — form meaningful sentences that are grammatically correct.
The education app, called Sentence Mosaics, uses photos and colour-coded parts of speech to prompt users as they construct sentences.
The app is useful for elementary students and for a wide range of children and adults who are on the autism spectrum, or who have developmental disabilities, hearing disorders, stroke-related language deficits, dementia and other problems with language, said Alyse Baker, a certified speechlanguage pathologist who has honed the system for decades.
“Alyse has used coloured blocks, coloured cards and foam tiles attached with Velcro to a foam board as aids, but that’s not practical for use outside the classroom,” said Mehar Sawhney, a DFA@CMU member and a senior majoring in mathematics and physics. “We worked with her to translate her system into an iPad app that can be taken anywhere.”
Baker developed Sentence Mosaics based on long-standing research and practice within the field of speech-language pathology. She has long recognised that transforming the system into an app would make sense, but found the prospect daunting.
“When I first looked into developing an app, I was told it would cost US$20,000 to US$30,000 to write the code,” Baker said. “I did not have that much money in my hip pocket, so I knocked on doors for six years until a stroke of luck took me to David Kosbie.”
Kosbie, an associate teaching professor of computer science at CMU, connected her two years ago with DFA@CMU, an interdisciplinary group of students who design and engineer solutions for social good.
A group of DFA members, including Sawhney, computer science majors Elizabeth Board and Sashank Gogula, and cognitive science major Elizabeth La, worked with Baker to translate physical aids into digital ones. Jennifer Chou, who graduated last spring with a computer science degree, also was a major contributor to the project.
Photos are key to the system. Users write sentences that describe actions depicted in photos, preferably photos that include the student or show a scene from their everyday life. Board said the app can import photos from an iPad or iPhone.