Every entrepreneur, side hustler and multitasking professional out there would love to have a virtual AI assistant to take on some of the dull daily tasks that come with existing in the modern era. As with any emerging technology, however, it can be difficult to separate the hype from the facts. There are four major players competing for attention: Amazon (Alexa), Apple (Siri), Google (Google Assistant) and Microsoft (Cortana).
If you're ready to jump into the early adopter lane (which not everyone is ready for) and get your very own personal assistant, but you want to know which is best, this faceoff is for you. I spent hours testing all four assistants by asking questions and giving commands that many business users would use. During the testing process, I noted the success of the AI's response to me, as well as other factors a prospective customer might care about, such as ease of setup, general ability to recognize my voice and contextual understanding.
Accessibility refers to how many devices the virtual assistant lives on. The best way to measure these AI assistants against each other in this category is to consider them individually. They are listed from most accessible to least accessible.
Google Assistant is available on all Android and iOS devices as well as on Chromebooks. Google has its own line of Google Home speakers that are built for use with Google Assistant, but it's also available on many third-party smart home speakers. Google Assistant is by far the most ubiquitous of all the AI assistants.Credit: Harman Kardon
Microsoft's AI, Cortana, comes standard on Windows machines, but it's also available for download on Android and iOS. Cortana will also be available on the $199.95 Harman Kardon Invoke with Cortana speaker, which is slated for release in fall 2017. Whether Cortana will be available on other third-party speakers or not remains to be seen. [Read related article: 5 Ways Businesses Can Use Cortana's AI to Get Organized]
Amazon's smart assistant is accessible through the Amazon Echo line of speakers as well as through Amazon's line of Fire tablets and Fire TV. There are also some third-party smart speakers and other devices that support Alexa, including the Huawei Mate 9 and Guess Women's Connect Smartwatch. There is also an Alexa app, but the app is intended primarily as a supplement to another Alexa device and not as a stand-alone AI assistant. [Read related article: Best Business Uses for Amazon's Alexa]
You can access Siri on nearly any Apple device, including its line of laptops, desktops, phones and tablets. Siri is also currently accessible via the iPod Touch, and it will be available on the HomePod when it's released in December 2017. Thus far, Siri is not available on any third-party speakers. [Read related article: How to Use Siri for Business]
Ease of setup
Ease of setup refers to how long it takes to get the assistant up and running, as well as how complicated that process is. All the AI assistants I evaluated have options to make the personal assistant experience more customized, including app integration, custom settings and adaptive responses.
To maintain an even testing ground, I did not consider additional setup steps that fall under the category of customization; I only considered how long it would take a new user to turn on the device and start asking questions. Google Assistant, Siri and Alexa all required virtually no setup time at all (you just sign in to a network and start) and were completely intuitive. None of these three took more than about 45 seconds for initial access, and that includes turning the device on or, in Alexa's case, plugging it in.
Cortana is a different story. If you want to access the voice features of Cortana, which is what this test was all about, you must access it on your device and then log in – oh, and you can't use a work or school email address for the login. Once you enter an acceptable email account, wait for a verification code and enter the code, you think you're in business, but you're not.
Ultimately, I successfully logged in by downloading the Cortana app and signing in on my phone. But when you do that, Cortana won't allow you to sign in with your phone number (even though it claims that's an option) because it's "not a Windows account." You can't sign in with your name either, just with your non-work and non-school email address.
After I signed in with my email address, Cortana cheerfully addressed me by that, my entire email address (dot-com included), for the remainder of our conversations, but at least it started working. Hopefully the setup process will be easier for the Harman Kardon speakers that will feature Cortana when they finally debut.
For voice recognition, all I wanted to know was how often the virtual assistant could recognize the words I was saying. I didn't consider context or the value of the response I was given, just basic recognition. I tested voice recognition at various distances from the devices as well as with varying levels of background noise.
Google and Siri understood me well when the room I was in was quiet and I was close to the devices. There were a couple funny misunderstandings, like when I asked Siri, "What's the date four weeks from now?" and it gave me the date for one week later, because it confused "four" with "for" and "weeks" with "week" and thought I was saying, "What's the date for a week from now?" All in all, though, the voice recognition was impressive under ideal conditions.
However, once I used Google or Siri to do anything sound-related, like read a news article or play music, the assistants could no longer hear me speaking at a normal volume. I shouted (literally), and eventually Google heard me, but Siri didn't hear me, even when I yelled. In fact, I had to manually turn off the sound on the iPhone to get Siri to stop playing the news, which kind of defeats the hands-free purpose.
Alexa was spotty with voice recognition. Like Google and Siri, it could not understand me at all when there was even soft music playing, and I had to manually shut down the speaker to get it to stop playing music. Unlike the other two assistants, Alexa also had issues understanding basic questions, even when the room was silent except for my voice commands.
I repeatedly asked for help getting plane tickets, and each time Alexa thought I was asking for movie tickets and directed me to Fandango. When I asked for future dates, such as, "What's the date for a week from now?" it simply replied, "Sorry, I don't know that," while other assistants had no trouble understanding identically phrased questions.
Cortana performed the worst by far in basic voice recognition. Microsoft's assistant had issues understanding me even with zero noise interference. Here are just a few examples of basic inquiries Cortana could not understand, even when stated slowly and clearly with no background noise:
- Cancel this task.
- I want to set a reminder.
- Does Amazon sell printer paper?
- I need help finding a restaurant.
- Do I have any reminders coming up?
It should be noted that these are all inquiries the other three assistants could understand, and some of the inquiries I made were understood by Cortana (like those about the weather), so it wasn't a microphone issue.
Success of queries and ability to understand context
The value of an AI assistant lies in its ability to understand natural language and context and deliver a useful response. To test this skill, I devised questions with context-reliant follow-ups. Here are a few questions I asked each of the assistants:
- How much is $5 in euros? What about in yen?
- Where is there an Applebee's near me? Can you make a reservation?
- How long will it take me to get to LaGuardia Airport in the car? How about by subway?
I also asked each of the assistants a laundry list of common questions about scheduling, setting reminders, online shopping, booking travel accommodations and getting directions.
Unfortunately, Cortana effectively removed itself from the running because it couldn't even hear or understand me on a basic level. When Cortana did understand me, it did not respond in a way that was intuitive or helpful. For example, when I asked for directions to the nearest airport, it searched for nearly a full minute and then returned with a list of results from Bing, which mostly linked to general airline ticket websites, such as Expedia.
When I asked Cortana to translate from USD to yen, which all the other three assistants did with ease, it did not reply aloud but rather brought up a page from Bing with a list of general responses about currency exchange from Answers.com. Anytime I asked follow-up questions, especially those dependent on context, Cortana either didn't respond at all, told me it didn't know or directed me to allow permissions from a third-party app.
While you can integrate third-party apps with Cortana (as you can with all the assistants), my guess is that the process for doing so is long-winded at best. In general, Cortana seems like a way to Bing things aloud and then receive answers on your laptop, rather than a full-fledged virtual assistant for business.
Alexa worked well when answering basic questions, especially those that pertained to purchasing items on Amazon and setting reminders. However, when I worked my way down the list and got to more complicated questions, which required context or close attention to detail, Alexa faltered. I attempted multiple times to get Alexa to help me purchase airline tickets, but the assistant consistently referred me to Fandango to buy movie tickets. Follow-up questions were rarely answered at all, and when they were, it was often with a polite but unhelpful "Sorry, I don't know that."
In general, Alexa could do part of the tasks I asked of it. For example, it could tell me where the closest Applebee's was, but it couldn't make a reservation. It could tell me how much $5 was in euros, but when I followed up with "What about yen?" it didn't know. When I asked how long it would take for me to get to LaGuardia Airport, it said, "As I don't know your speed, I can't tell you how long that will take."
Alexa seems to rely on very specific terminology for commands. For example, when I asked Alexa to read me the news, it did, but I couldn't get it to stop. I tried every possible direct command I could think of to get Alexa to stop reading the news, including "stop" and "turn volume off," but the only phrase that worked (which I had to look up online) was "stop flash briefing," which isn't exactly intuitive. It was easy to set up reminders with Alexa, but when the reminder time would come, Alexa would just blare an alarm, with no indication of what the alarm was for, and the alarm had to be turned off manually.
Alexa is an OK tool for ordering on Amazon, and it may be good for smart house integration, but it's not the best AI for natural language use, and anyone adopting the Alexa should peruse user guides before setting it up and understand its limitations.
Google Assistant vs. Siri
It's worthwhile to compare Google Assistant and Siri's ability to understand context and intuitive speech, because these two assistants are neck and neck when it comes to ease of use and intelligence.
When I asked for directions to One World Trade Center, both Google Assistant and Siri responded with clear driving instructions. When I added a follow-up question about public transportation directions instead, Google Assistant responded verbally with directions to and from the subway as well as an automatic link to step-by-step instruction on Google Maps. Unfortunately, Siri simply found me alternate driving directions. No matter how I asked, I could not get Siri to give me public transit directions.
Siri bested Google Assistant when it came to tasks like finding specific restaurants and making reservations. In fact, Siri was the only assistant I tested that could not only find a nearby restaurant (which Google Assistant did too) but also place a reservation for me. When I asked Google Assistant to find a specific restaurant, it did, but when I asked it to book a reservation for me, it inexplicably took me to Disneyworld.com. I tried several times to book a reservation through Google Assistant, and it just wasn't happening. Incidentally, it also failed to convince me to book a Disney vacation.
The ability to make hands-free restaurant reservations with Siri could be a real boon to business users who want to use their AI assistants for business and personal use. However, Siri falls short if you are not specific in your requests. For example, when I simply said, "I need help finding a restaurant," Siri responded with a list of Google results that were literally instructions on how to find a restaurant, while Google Assistant read between the lines and showed me restaurants nearby.
Google Assistant was also better than Siri when it came to travel information, but it still wasn't perfect on follow-through. When I asked Siri for flights from New York to Paris, it just Googled my question and showed me the responses. When I asked Google Assistant for flights from New York to Paris, it asked me for dates and started pulling up available reservations; however, when I tried to go all the way and book the flights, it simply said, "No problem!" and then did nothing. One cool thing Google Assistant offered me, without prompting, was the ability to email me if prices changed on the flights I was interested in, which is a neat feature.
When I asked each assistant to read me the news, Google Assistant immediately started playing a recent NPR podcast, while Siri Googled several news sources for me. I noticed throughout many different tasks that Siri rarely responded to me aloud, while Google Assistant nearly always did. While this is likely a matter of playing with a few settings, it's an interesting difference to note.
None of the AI assistants are perfect; this is young technology, and it has a long way to go. There was a handful of questions that none of the virtual assistants on my list could answer. For example, when I asked for directions to the closest airport, even the two best assistants on my list, Google Assistant and Siri, failed hilariously: Google Assistant directed me to a travel agency (those still exist?), while Siri directed me to a seaplane base (so close!).
Judging purely on out-of-the-box functionality, I would choose either Siri or Google Assistant, and I would make the final choice based on hardware preferences. None of the assistants are good enough to go out of your way to adopt. Choose between Siri and Google Assistant based on convenience and what hardware you already have.