Can computers identify emotions?
Q >> I mentioned to a friend that Apple now has a patent for “emotion detection” with its cameras and they said that was something that had been in digital cameras for years. Has it? Can a computer really detect human emotions?
A >> Let’s start with the Apple Patent, No. 11727724. According to The US Patent and Trademark Office (USPTO.GOV) filing, it is for “Emotion Detection” and is based on a research paper titled “Automatic detection of facial expressions and emotions for educational environments” from 2015.
In other words, yes, there’s clearly research work that’s been going on for many years on how to teach computer vision systems to identify specific facial expressions and extrapolate probable emotions.
At its most basic, this can be described with two words: “Say cheese!” You’ve heard it a thousand times, it’s the prompt to smile because someone’s about to take your photograph. Now imagine a smart camera that would automatically pick the best image out of a burst, or even analyze facial expressions to find the one where everyone’s smiling (and perhaps has their eyes open) simultaneously.
Smile Shot: Are you smiling yet?
In fact, that’s exactly what Olympus introduced way back in 2008 with its digital cameras, a feature called “Smile Shot.” On its cameras of the era, the feature was described as being able to “track your subject’s face to detect a smile and automatically fire off three consecutive shots so you never miss that perfect smile.” The feature was heralded as “great for parents hoping to capture hard-to-predict baby smiles.”
Olympus purged itself of the digital camera business back in 2020, but it’s interesting to note that even with the current dramatic leaps forward in digital photo processing and AI, no other company is offering such a feature.
Or are they? Google in 2019 announced that its Photobooth app could detect five facial expressions: smiles, tongue-out, kissy face, puffy cheeks, and surprise.
Imagine the frustration, however, when you want to take a picture of your child just to have the camera refuse to capture the image because they aren’t smiling. Forgetting you’ve enabled that feature, you’d immediately assume your camera (uh, smartphone) was broken. Not so good.
And what if you want to capture a picture of someone with a different emotion? “I’m sorry, Dave, I can’t take that picture because they’re sad.”
Meanwhile, the Apple patent
The real challenge is whether expression recognition is accurate across skin tones, facial physiognomy, and cultures. Then again, Apple’s interest might be more about smarter photo album searches. In that case, it might just be a great feature to be able to search for “pictures where baby Ashley is smiling.” If it works.
What do you think? Are you ready to have your smartphone camera — and computer — analyze and ostensibly respond to your facial expressions? Soon enough your smartphone might just be deciding if you really said “cheese” in your latest selfie!