Latest Research
Annual Digital Assistant IQ Test

Annual Digital Assistant IQ Test

We recently asked the leading digital assistants, Google Assistant, Siri, and Alexa, 800 questions each. Google Assistant was able to correctly answer 93% of them vs. Siri at 83% and Alexa at 80%. Each platform saw improvements across the board vs. one year ago. In July of 2018, Google Assistant correctly answered 86% vs. Siri at 79% and Alexa at 61%.

As part of our ongoing effort to better understand the practical use cases of AI and the emergence of voice as a computing input, we regularly test the most common digital assistants and smart speakers. This time, we focused on smartphone-based digital assistants. See past comparisons of smart speakers here and digital assistants here.

We separate smartphone-based digital assistants from smart speakers because, while the underlying technology is the same, use cases vary. For instance, the environment in which they are used may call for different language, and the output may change based on the form factor; e.g., screen or no screen. We account for this by adjusting the question set to reflect generally shorter queries and the presence of a screen which allows the assistant to present some information that is not verbalized.

We have eliminated Cortana from our test due to Microsoft’s recent shift in Cortana’s strategic positioning. Read more about what has changed with Cortana here.

Methodology

We asked each digital assistant the same 800 questions, and they are graded on two metrics: 1. Did it understand what was being asked? 2. Did it deliver a correct response? The question set, designed to comprehensively test a digital assistant’s ability and utility, is broken into 5 categories:

  • Local – Where is the nearest coffee shop?
  • Commerce – Order me more paper towels.
  • Navigation – How do I get to Uptown on the bus?
  • Information – Who do the Twins play tonight?
  • Command – Remind me to call Jerome at 2 pm today.

Note that we slightly modify our question set before each round of testing in order to reflect the changing abilities of AI assistants. This is part of an ongoing process to ensure that our test is comprehensive.

Testing was conducted using Siri on iOS 12.4, Google Assistant on Pixel XL running Android 9 Pie, and Alexa via the iOS app. Smart home devices tested include Wemo Mini plug, TP-Link Kasa plug, Phillips Hue Lights, and Wemo Dimmer Switch.

Results and Analysis

Google Assistant was once again the best performer, correctly answering 93% and correctly understanding all 800 questions. Siri was next, answering 83% correctly and only misunderstanding two questions. Alexa correctly answered 80% and only misunderstood one.

Google Assistant was the top performer in four of the five categories but fell short of Siri in the Command category again. Siri continues to prove more useful with phone-related functions like calling, texting, emailing, calendar, and music. Both Siri and Google Assistant, which are baked into the OS of the phone, far outperformed Alexa in the Command section. Alexa lives on a third-party app, which, despite being able to send voice messages and call other Alexa devices, cannot send text messages, emails, or initiate a phone call.

The largest disparity was Google’s outperformance in the Commerce category, correctly answering 92%, vs Siri at 68% and Alexa at 71%. Conventional wisdom suggests Alexa would be best-suited for commerce questions. However, Google Assistant correctly answers more questions about product and service information and where to buy certain items, and Google Express is just as capable as Amazon in terms of actually purchasing items or restocking common goods you’ve bought before.

We believe, based on surveying consumers and our experience using digital assistants, that the number of consumers making purchases through voice commands is insignificant. We think commerce-related queries are more geared toward product and service research and local business discovery than actually purchasing something, and our question set reflects that.

Overall, the rate of improvement of these systems continues to surprise us. We perform this test two times per year and see improvement across each assistant in each category every time. Many of the same trends continue; Google outperforms in information-related questions, Siri handles commands best, and the ranking of utility based on the number of questions answered has remained the same (Google Assistant, Siri, Alexa), but there have been dramatic improvements on each platform and in each category in the few short years that we have been tracking the progress of digital assistants.

Improvement Over Time

As measured by correct answers, over a 13-month period, Google Assistant improved by 7 percentage points, Siri by 5 points, and Alexa by 18 points.

Alexa made significant improvement across all five categories, most noticeably in Local and Commerce. While Alexa’s 18 point jump still leaves it behind Siri and Google Assistant, it represents the largest jump in correct answers year over year that we have recorded. The chart below shows the improvement of each assistant over time broken down by category.

With scores on our test quickly approaching 100%, it may seem like digital assistants will soon be able to answer any question you ask them, but we caution this is not the case. Today, they are able to understand, within reason, everything you say to them, and the primary use cases are well built out, but they are not generally intelligent.

Further improvements will come from extending the feature sets of these assistants. New skills are hard to discover and, therefore, don’t get used. To become a habit, new use cases need to be well understood, simple to use, and solve a problem that voice is uniquely suited to solve. We will continue to refine our test to be sure it addresses these new use cases as they emerge.

Disclaimer: We actively write about the themes in which we invest or may invest: virtual reality, augmented reality, artificial intelligence, and robotics. From time to time, we may write about companies that are in our portfolio. As managers of the portfolio, we may earn carried interest, management fees or other compensation from such portfolio. Content on this site including opinions on specific themes in technology, market estimates, and estimates and commentary regarding publicly traded or private companies is not intended for use in making any investment decisions and provided solely for informational purposes. We hold no obligation to update any of our projections and the content on this site should not be relied upon. We express no warranties about any estimates or opinions we make.

Amazon, Apple, Artificial Intelligence, Google
4 min. read Show less
New EV Entrants Unlikely to Threaten Model 3

New EV Entrants Unlikely to Threaten Model 3

Over the past year, there has been increasing anticipation for the release of new EV models from the likes of Audi, Jaguar, Fiat, Hyundai, and Honda. This stoked concerns that Tesla’s year to date 75% US EV market share will quickly be diluted by established automakers that are already selling cars at scale. We took a closer look at what has changed in the EV landscape over the past year and compared available models by price, range, and delivery time.

There are now 17 EVs in the US market, compared to 11 a year ago. Next year that number should rise to 24. The majority of these models fall into two camps: those with starting prices above $70,000, and those with ranges below 130 miles. These two camps are largely ruled out as a mainstream option given the high price or limited range. We view a more mainstream option as priced below $40,000 with a range above 225 miles.  When filtering for prices below $40,000 with a range above 225 miles, the list narrows to five options, compared to three a year ago. Today, this list includes the Model 3, Chevy Volt/Bolt, Hyundai Kona, and Kia Niro. Within that group, there’s a wide range of design options, software functionality, and charging network availability. Considering those factors, Model 3 is the clear winner in terms of value.

Our conclusion: competition in 2019 and likely 2020 is not a measurable threat to Tesla. The Model 3 is unchallenged in its EV value proposition.

EV Models Available in the US Today

Not surprising, the US electric vehicle market is rapidly growing. In 2016, EVs accounted for less than 0.25% of cars sold in the US. In 2019, EVs will account for 2-3%. We’re likely 15-25 years away from 100% EV adoption. The table below outlines EV models that are currently available in the US sorted by starting price.

EV Models Expected in the US in 2020

The following table outlines expectations from automakers for new EVs in 2020 sorted by starting price.

Long-Term Market Share Expectations

As mentioned, Tesla’s year to date US EV market share is 75%. Over the next 10 years, we expect Tesla’s share to decline to 20-25%. As a point of reference, in 2018, GM had the leading market share in the US at about 17%. Factoring in a stable US auto market of 18m units a year (eventually, all cars will be electric) yields 3.6-4.5m annual Tesla sales in the US. This compares to what will likely be just over 200k US Tesla deliveries (of our estimated 360k global deliveries) in 2019.

Tesla’s 7 Year Head Start

The prevailing wisdom holds that OEMs are quite good at producing cars and switching the drive train to electric will be simple, allowing them to profitably scale EV production and maintain their market position. If this is true, it would dramatically lower Tesla’s market share and make the story less compelling overall, but we believe Tesla’s 7-year head start sets the company up to control a significant share of the market for a long time. Three key benefits from Tesla’s head start:

  • 92% more efficient batteries than four other EV manufacturers, adjusting for differences in range evaluation methods.
  • Vertically integrated Supercharger network is easier to use compared to generic charging stations.
  • More advanced self-driving capabilities.
Disclaimer: We actively write about the themes in which we invest or may invest: virtual reality, augmented reality, artificial intelligence, and robotics. From time to time, we may write about companies that are in our portfolio. As managers of the portfolio, we may earn carried interest, management fees or other compensation from such portfolio. Content on this site including opinions on specific themes in technology, market estimates, and estimates and commentary regarding publicly traded or private companies is not intended for use in making any investment decisions and provided solely for informational purposes. We hold no obligation to update any of our projections and the content on this site should not be relied upon. We express no warranties about any estimates or opinions we make.
Tesla
3 min. read Show less
Nvidia’s Path Forward in Data Centers

Nvidia’s Path Forward in Data Centers

We recently ran through the top-performing stocks in the S&P 500 for the last 15 years. At the very top were some household names like Netflix >12,000% returns, Apple >10,000% returns, revolutionizing the way people experience the world, and Amazon >5000% returns, led by the wealthiest person on the planet today. Little overlooked is the fabless semiconductor design company Nvidia which achieved >5000% share price gains in the period, even off 48% from its 2018 highs. Nvidia achieved this on the back of its Graphics Processor Unit (GPU) architecture originally developed to render PC graphics but has evolved to become the default processing engine in areas like virtual reality (VR), high-performance computing (HPC), and artificial intelligence (AI) including machine learning (ML).

Nvidia’s Next Five Years

Intel tweeted in June that it is on track to release its first dedicated GPU graphics card of this century. The company released its last discrete graphics card, the i740, in 1998 which was not a commercial success, although the technology from the i740 continues to live in Intel’s integrated CPU cores. In the mid-2000s, Intel had a second attempt to enter the discrete graphics processor with a project code-named ‘Larrabee.’ The product was never released due to price/performance issues. Today, with the massive growth of the cloud computing market and GPUs playing a central role in many AI intensive cloud computing applications, there are significant incentives for Intel to continue to invest in this space.

Intel just sold off its modem business to Apple, essentially exiting the smartphone modem market (Intel does not have a cellphone processing chip) and leaving Qualcomm and MediaTek as the only platform System On Chip (SOC) semiconductor players in the space. Nvidia smartly exited the smartphone chip market in 2014 when it ended its Tegra line of products, avoiding a bruising battle with Qualcomm and potentially losing billions of dollars in the process. For Intel and Nvidia exiting these markets was the right move shifting the development focus away from slowing smartphone segment to higher growth opportunities. The growth segment Intel and Nvidia are pursuing is the data center back end processing of everything from gaming, AI, mobility, AR and VR applications which will grow exponentially with the coming wave of 5G and connected devices.

The Threat of New Multibillion-Dollar Silicon Design Startups

While Nvidia is well-positioned to capitalize on the above themes, there are well-funded custom silicon startups that pose a new threat to their dominance. Since Nvidia’s GPUs were never designed to just handle machine learning algorithms, there is room for new silicon design startups like Loup portfolio company Rain Neuromorphics, Wave Computing ($203m raised), Cerebras Systems ($112m raised), Graphcore ($310mraised), SambanNova ($56m raised), Mythic Inc ($55m raised), Cambricon Technology ($110m raised), Horizon Robotics ($700m raised), and Deephi ($40m) to come up with original architectures. If we expect an aggregate 10% returns from well known Silicon Valley VC funds backing these startups, we should expect another multi-billion dollar silicon design competitor in this space in the next 5 years.

The Threat of Incumbent Cloud Platforms

An even greater threat to Nvidia’s ability to maximize its returns comes from the incumbent cloud platforms. In 2016, Google announced their propriety tensor processing units (TPUs), an application-specific integrated circuit (ASIC) chip built specifically for ML tasks. Since then both Microsoft, Amazon, and Facebook have announced their own silicon design efforts for ML for improving everything from natural language processing to image recognition applications. These cloud leaders have a different business model than Nvidia or Intel. Google offers access to their TPU servers on an hourly basis, accessed through the Google cloud platform. Last year Amazon announced their own proprietary ARM-based Graviton processors which are offered through the AWS Nitro System platforms. Amazon’s stated one goal for designing their own processors is to gain bargaining power with Intel and Nvidia when building new data centers. In China, Alibaba, Tencent, and Baidu have announced their own in-house chip design initiatives. What these cloud players excel at is a deep understanding of the software stacks that run on these platforms.

It’s still early days in the development of custom design chips for AI and ML, and the mega internet companies have little hardware DNA to advance a proprietary offering. Hardware is a game of intellectual property and continuous design execution, different than the game of building network effects in an internet platform. It is not evident yet which core competency will result in lasting competitive advantage in capturing shareholder value in the data center space.

Disclaimer: We actively write about the themes in which we invest or may invest: virtual reality, augmented reality, artificial intelligence, and robotics. From time to time, we may write about companies that are in our portfolio. As managers of the portfolio, we may earn carried interest, management fees or other compensation from such portfolio. Content on this site including opinions on specific themes in technology, market estimates, and estimates and commentary regarding publicly traded or private companies is not intended for use in making any investment decisions and provided solely for informational purposes. We hold no obligation to update any of our projections and the content on this site should not be relied upon. We express no warranties about any estimates or opinions we make.

Nvidia
3 min. read Show less