Who’s the smartest? Google, Alexa or Siri?
We put digital assistants to the test with an array of 150 questions
OK Google, we get it. You are smarter than the other assistants.
This has been the subtext of recent Google I/O developer conferences. The latest edition kicked off Tuesday in Mountain View, Calif., where the Internet giant showed off how its Assistant, available on phones and its smart speaker Google Home line, was getting more adept at mimicking human conversation.
Google has a long ways to catch up when it comes to the share of smart home devices. Amazon has Alexa in some 12,000 connected devices, compared to 5,000 for Google and just 194 for Apple’s Siri, says Voicebot.ai, a website that focuses on voice computing.
But in smarts, Google topped not just our informal survey but many recent ones as well, including surveys from online marketing firm Stone Temple and investment company Loup Ventures.
We spent the weekend asking the same 150 questions to the Google Assistant on Google Home, Amazon’s Alexa via the Echo speaker and Apple’s Siri on the iPhone. Google answered correctly 80% of the time, compared to 78% for Amazon and 55% correct for Siri.
A quick caveat for our methodology — if Google and Amazon gave us a complete, audio answer to the question, that counted as a successful response. When an assistant said it wasn’t set up to respond, or didn’t know, that counted as a fail.
And when Siri responded with a “Here’s what I found on the Web,” and a link to look it up ourselves, that also counted as a non-answer. (We tried some of those questions again with Apple’s HomePod, which is a $350 speaker and rival to the Echo and Google Home, but it didn’t fare much better. While it did answer one query, the rest returned with a “I can’t get the answer on the HomePod” response.)
Our questions came from a variety of courses: We cribbed from the 800 questions posed in recent surveys by Loup, the suggested queries on Amazon, Google and Apple’s websites to ask their assistants, and topics offered by social media.
Google, via the Home speaker, told us how to get to the nearest Mexican restaurant, what time the Avengers movie was playing at the cineplex, who won the Best Picture Oscar of 1989, the date and flight number of my next scheduled airline flight and the definition of a first cousin once removed.
What it couldn’t tell us was also quite interesting. Some notable Google Assistant failures. It couldn’t:
❚ Read our latest G-mail email aloud. Which Siri could do, but not Alexa.
❚ Google the movie Back to the Future.
❚ Tell me, “Who was Jesus Christ.”
❚ Answer if aliens really exist and why cats have whiskers. (Alexa has responses for both, and Siri was happy to tell me about Jesus.)
Alexa was surprisingly strong when it came to hard-core science factoids that Google excels in, such as naming the melting point of gold, how far away the moon is from the Earth and citing the weight of the sun.
It can translate “good morning” in German (all three can do this) and tell you how to say “thank you” in Japanese. (As can Google; Siri can’t.)
Both Google and Amazon can play your morning news briefings — Siri doesn’t do this. Alexa can read recipes, and being that it’s Amazon, also order paper towels, batteries, dog food or even shop for an iPad. Siri couldn’t do any of this with audio directions, instead answering “Here’s what I found on the Web,” and links.
Google Home could do all the shopping as well, by sending you to Google Express, its answer to Amazon, but it puts a $100 limit on purchases. I could shop for the iPad but not complete the purchase because of the limit; I could ship other products.
With Siri, as always, it’s a matter of managed expectations. It generally could answer most of the questions posed by Apple on the Siri section of its website, such as using it to call and text friends, set timers and appointments, relay information (“When is the L.A. Galaxy’s next home game?”) and tell what the weather would be like today, tomorrow and on the weekend.
But we’ve got some caveats. Apple suggests we ask Siri to play “the top song from 1985” on its website. Yet when I ask, it says, “Sorry, I don’t know what topped the charts on that date.”
Google Home has come a long way since we first reviewed it in 2016, and it couldn’t answer so many questions. We’d like to see simple improvements
“I’d like to see (Google) humanize it more, have it do more things you can do as a family,” like letting users know what’s on Netflix tonight. Bret Kinsella Voicebot publisher
such as learning how to “Google” information more effectively (the Back to the Future query was an easy one), read my latest email and texts aloud, improve on sports stats (“What’s Alabama’s record this season?”) and learn how to answer the Jesus question.
Bret Kinsella, publisher of Voicebot, thinks Google has done a great job showcasing the utility of the product — now it should be more fun.
“I’d like to see them humanize it more, have it do more things you can do as a family,” like letting users know what’s on Netflix tonight or adding more games.
But seriously, it’s Apple that has its work cut out for it, not Google. And we’d like to offer a simple suggestion. Stop having Siri direct us “to the Web” and instead announce actual answers, just like Google Home and Alexa.
Siri’s super-low 55% response rate on our survey is due to the fact it keeps offering non-hands-free, “Here’s what I found on the Web” links, when its rivals offer true audio replies.
When asked to tell me where to get a car repaired locally, instead of reading me choices, it sends links to the Firestone shop. Ask where to buy golf clubs, and you get a link to a nearby golf course. Inquire about recipes for a Tom Collins drink or how to make banana bread, and Siri directs you to look it up online.
Apple has the data. If Amazon and Google could do it, there’s no good reason why Siri couldn’t join this party. Make that one shift, and the successful query gap could be closed significantly overnight.