The smart new app that serves as eyes for the blind

NEW YORK: As computers get better at navigating the world around them, they are also helping humans better navigate that world as well.

Thanks to advances in artificial intelligence and robotics, scientists from IBM Research and Carnegie Mellon University are working on new types of real-world accessibility solutions for the blind.

The goal is as audacious as it is inspiring: coming up with a technological platform that can help the visually impaired cope as well as everyone else.

The first pilot in the programme is a smartphone app for Apple products and Android called NavCog, which helps blind people navigate their surroundings by whispering into their ears through earbuds or by creating subtle vibrations on their smartphones.

The app operates similarly to the turn- by- turn directions offered by car GPS systems. The app analyses signals from Bluetooth beacons located along walkways and from smartphone sensors to help enable users to move without human assistance, whether inside buildings or outdoors.

The magic happens when algorithms are able to help the blind identify in near real-time where they are, which direction they are facing and additional surrounding environmental information.

The computer-vision navigation- application tool turns smartphone images of the surrounding environment into a 3-D space model that can be used to issue turn- by- turn navigation guidance.

The NavCog project has particular meaning for one of the lead researchers on the project, IBM fellow and visiting Carnegie Mellon faculty member Chieko Asakawa, who is visually impaired herself. It will soon be possible for her to walk across the Carnegie Mellon campus with the help of the NavCog app – and look just like any other person crossing the campus.

That’s just the beginning, says Kris Kitani of the Robotics Institute of Carnegie Mellon. A major goal is to extend the coverage beyond just the Carnegie Mellon campus that has been retrofitted with beacons. To encourage this, the scientists working on the project have made the entire NavCog platform open source by making it available to developers via the IBM BlueMix cloud. That makes it possible for other developers to build other enhancements for the system and speed the rollout to other physical destinations.

The other primary goal, Kitani said, is to make the system workable even in environments that do not include Bluetooth beacons. To make that possible, the university hopes to build on advances in computer vision as well as new work being conducted in the field of cogni- tive assistance, which is a research field dedicated to helping the blind regain information by augmenting missing or weakened abilities.

By using cameras for computer-aided vision, for example, it might be possible to develop a more accurate system that doesn’t require the presence of Bluetooth beacons. And this computer- aided vision, when combined with other localisation technologies, potentially could make it possible to recognise everyday landmarks like stairs or a barrier on the road that might not be picked up with today’s sensors.

There are plans to add other extras to the system that go beyond mere navigation. For example, a facial- recognition component would tell you in real- time if you are passing someone you know.

Moreover, sensors capable of recognising emotions on these faces – work that’s part of other Carnegie Mellon research into autism – could make it possible to recognise when those people passing you are smiling or frowning. Researchers also are exploring the use of computer vision to characterise the activities of people in the vicinity and ultrasonic technology to help identify locations more accurately.

If all goes according to plan, it’s possible to envision a virtuous feedback loop for machine intelligence and human intelligence. – Washington Post