Hey, Alexa, what can you hear? And what will you do with it?
Amazon ran a commercial on this year’s Super Bowl that pretended its digital assistant Alexa had temporarily lost her voice. It featured celebrities like Rebel Wilson, Cardi B and even the company’s chief executive, Jeff Bezos.
While the ad riffed on what Alexa can say to users, the more intriguing question may be what she and other digital assistants can hear — especially as more people bring smart speakers into their homes.
Amazon and Google, the leading sellers of such devices, say the assistants record and process audio only after users trigger them by pushing a button or uttering a phrase like “Hey, Alexa” or “O.K., Google.” But each company has filed patent applications, many of them still under consideration, that outline an array of possibilities for how devices like these could monitor more of what users say and do. That information could then be used to identify a person’s desires or interests, which could be mined for ads and product recommendations.
In one set of patent applications, Amazon describes how a “voice sniffer algorithm” could be used on an array of devices, like tablets and e-book readers, to analyse audio almost in real time when it hears words like “love,” bought” or “dislike.” A diagram included with the application illustrated how a phone call between two friends could result in one receiving an offer for the San Diego Zoo and the other seeing an ad for a Wine of the Month Club membership.
Some patent applications from Google, which also owns the smart home product maker Nest Labs, describe how audio and visual signals could be used in the context of elaborate smart home setups.
One application details how audio monitoring could help detect that a child is engaging in “mischief” at home by first using speech patterns and pitch to identify a child’s presence, one filing said. A device could then try to sense movement while listening for whispers or silence, and even program a smart speaker to “provide a verbal warning.”
A separate application regarding personalising content for people while respecting their privacy noted that voices could be used to determine a speaker’s mood using the “volume of the user’s voice, detected breathing rate, crying and so forth,” and medical condition “based on detected coughing, sneezing and so forth.”
The same application outlines how a device could “recognise a T-shirt on a floor of the user’s closet” bearing Will Smith’s face and combine that with a browser history that shows searches for Mr. Smith “to provide a movie recommendation that displays, ‘You seem to like Will Smith. His new movie is playing in a theater near you.’”
In a statement, Amazon said the company took “privacy seriously” and did “not use customers’ voice recordings for targeted advertising.” Amazon said that it filed “a number of forward-looking patent applications that explore the full possibilities of new technology,” and that they “take multiple years to receive and do not necessarily reflect current developments to products and services.”
Google said it did not “use raw audio to extrapolate moods, medical conditions or demographic information.” The company added, “All devices that come with the Google Assistant, including Google Home, are designed with user privacy in mind.”
Tech companies apply for a dizzying number of patents every year, many of which are never used and are years from even being possible.
Still, Jamie Court, the president of Consumer Watchdog, a nonprofit advocacy group in Santa Monica, California, which published a study of some of the patent applications in December, said, “When you read parts of the applications, it’s really clear that this is spyware and a surveillance system meant to serve you up to advertisers.”
The companies, Court added, are “basically going to be finding out what our home life is like in qualitative ways.”
Google called Consumer Watchdog’s claims “unfounded,” and said, “Prospective product announcements should not necessarily be inferred from our patent applications.”
A recent Gallup poll found that 22 per cent of Americans used devices like Google Home or Amazon Echo. The growing adoption of smart speakers means that gadgets, some of which contain up to eight microphones and a camera, are being placed in kitchens and bedrooms and used to answer questions, control appliances and make phone calls. Apple recently introduced its own version, called the HomePod.
But many consumers are also becoming increasingly nervous that tech companies are eavesdropping on them in order to serve them targeted ads, no matter how often the companies deny it. The recent revelations that a British political data firm, Cambridge Analytica, improperly harvested the information of 50 million Facebook users has only added to the public’s wariness over the collection and use of personal information.
Facebook, in fact, had planned to unveil its new internet-connected home products at a developer conference in May, according to Bloomberg News, which reported that the company had scuttled that idea partly in response to the recent fallout. Both Amazon and Google have emphasised that devices with Alexa and Google Assistant store voice recordings from users only after they are intentionally triggered. Amazon’s Echo and its newer smart speakers with screens use lights to show when they are streaming audio to the cloud, and consumers can view and delete their recordings on the Alexa smartphone app or on Amazon’s website (though they are warned online that “may degrade” their experience).