Siri Semester Exam Grade Improves to C From D+

Semester exam results are in. In our latest 800 question test of AI assistant Siri, she was able to understand 99% of queries and correctly answer 75% of them, earning a C grade. In April of 2017, Siri earned a D+ on the same test where she understood 94% of queries and correctly answered 66%. These tests are conducted with the same methodology and question set as our smart speaker tests found here. It involves asking 800 questions divided into five categories (local, commerce, navigation, information, and command) designed to test the full range of abilities and accuracy of an AI assistant.

Performance. Siri’s 9% improvement in correct answers over an 8 month period is more or less in line with the high rate of improvement we are noting with smart speaker-based assistants like Amazon Alexa and Google Home. For comparison, Alexa answered 64% of queries correctly and Google Home 81%. These results, however, can not be compared to one another directly as a smartphone-based digital assistant responds differently to queries, geared more toward calling up information on the device’s screen or controlling the device itself. Nonetheless, Siri’s December performance, compared to our test in April of this year, and to previous tests with a similar methodology and question set we conducted in 2012, 2013, and 2014, shows improvement in all five categories.

We’re tough graders at Loup Ventures. We’re tough graders in that during our testing of Siri, we only counted correct answers when she was able to deliver a single concise answer herself rather than bringing you to search results that might help you find an answer. This means, “I found this on the web for…” is counted as incorrect. Siri improved 9% since our April test but remains far away from the A grade that we expect will drive AI assistant technology to mainstream adoption. Test scores should be in the 90% range (90% of queries answered correctly) to receive an A grade. The takeaway is that Siri does not yet act as the fabric that connects our computing experience as we hope AI assistants one day will. We can live without Siri for now, but at the rate of improvement we are seeing, we expect her to be indispensable in 2 years.

Methodology. Methodology. Just as we have in April of this year, we asked 800 questions to Siri on an iPhone X and 8 Plus the last week of December 2017. The queries covered five categories: Local, Commerce, Navigation, Information, and Command. Siri was graded on two metrics: did she understand what was asked? (this can be seen on the device’s screen), and did she answer or execute correctly? It is important to note that we have slightly modified our question set to be more reflective of the changing abilities of AI assistants. As voice computing becomes more versatile and digital assistants become more capable, we will continue to update our question set to be reflective of those improvements going forward. Our changes included questions around the use of smart home devices. We tested Siri with the Philips Hue smart lighting and Wemo Mini smart plugs.

Disclaimer: We actively write about the themes in which we invest: artificial intelligence, robotics, virtual reality, and augmented reality. From time to time, we will write about companies that are in our portfolio.  Content on this site including opinions on specific themes in technology, market estimates, and estimates and commentary regarding publicly traded or private companies is not intended for use in making investment decisions. We hold no obligation to update any of our projections. We express no warranties about any estimates or opinions we make.

8 Tech Predictions for 2018

Loup Ventures’ predictions of 2018:

  1. AI theme continues and artificial general intelligence takes small step forward through Google’s “Deepmind” initiative.
  2. VR gets untethered but it still takes another year to take off.
  3. Google Home continues to gain market share in smart speaker market.
  4. Tesla Model 3 production ramps from 2,500 in 2017 to greater than 150,000 in 2018.
  5. Major automakers announce expanded electric vehicle line-up, but autonomous driving will not ramp up until 2021.
  6. Bitcoin pulls back.
  7. iPhone ASP’s $740 vs. Street at $710.
  8. Amazon will acquire Target.

AI theme continues and artificial general intelligence takes small step forward through Google’s “Deepmind” initiative. We believe the hype around AI is justified given it’s hard to understate the significance that AI will have on the future. In 2018 we expect the AI momentum to continue. In July of 2017, we counted 11% of Fortune 500 companies mentioned AI on their quarterly conference calls. We expect that number to grow in 2018. As for leaders, it’s clear that Google CEO Sundar Pichai is trying to get his point across: AI is the future of Google. We went back and looked at his opening comments over the last year and found he has led his prepared remarks by asserting Google’s evolution from a mobile to an AI-first company on each of the past four earnings calls. The company is pushing its AI into hardware devices (Google AI hardware note) and seeing its work pay off (Google Home’s the smartest smart speaker). Artificial narrow intelligence (ANI) is being mastered and we expect more news about the next frontier in AI, artificial general intelligence (AGI, the ability of a machine to think like a human), to be top of mind in 2018. We expect Google to play a thought leadership role in AGI with its Deepmind platform, but keep in mind true AGI is likely another 20 years away.

VR gets untethered but it still takes another year to take off.  The Wall Street Journal said it best in their Dec 30 article titled “Virtual Reality Needs to Cut the Cord”.  Oculus Go ($199 untethered) which comes out early in 2018 will be a step forward in ease of VR use, but it won’t be until 2019 until more powerful untethered hardware lay the groundwork for content developers and consumers to fully embrace VR. For those disappointed VR won’t be mainstream in 2018, you can at least look forward to Ready Player One’s release on March 30th. Here’s our Facebook in VR outlook.

Google Home continues to gain market share in smart speaker market.  In 2017 the smart speaker market was led by Amazon with the release of 3 new Echo devices (Echo, Echo Plus, Echo Spot), Alexa for Business, sub $50 pricing, and the Alexa app being the #1 downloaded app on Christmas Day vs. Google Home at #6 across Android and iPhone. That said the market is still up for grabs, as evidenced by Amazon and Google collectively spending more than $70m on TV ads from Thanksgiving through Christmas to push the theme. Despite Alexa owning 75% (Loup Ventures) of the global smart speaker market today, we expect in 2018 Google Home will be a share gainer in smart speakers given its performance lead. Here are the details of our recent face off between the smart speaker players.

Tesla Model 3 production ramps from 2,500 in 2017 to greater than 150,000 in 2018.  Near-term, we expect another miss in Model 3 production, but in 2018 we predict production will turn the corner. We continue to stress that Model 3 production over the next several quarters will be largely a guessing game and that short-term production numbers do not materially affect the long-term story. The last update on Model 3 production calls for “a production rate of 5,000 Model 3 vehicles per week by late Q1 2018,” which we believe is ambitious. That said, we’re encouraged by hundreds of Model 3s have been spotted at delivery centers and at the Fremont factory shown in a video here, along with several suppliers reporting that they are back to delivering Model 3 parts at volume.

The reason we remain upbeat on the Tesla story despite the prolonged Model 3 production problems is that EV and autonomy are the future. Tesla is fighting to gain production scale to create that future. While other car manufacturers build gas-powered vehicles at scale, building autonomous EVs is a vastly different process that will require traditional auto manufacturers to re-engineer their production facilities. That means every automaker that wants to compete in the future needs to go through the production pain Tesla’s experiencing today. Here’s our recent note on Model 3 production outlook.

Major automakers announce expanded electric vehicle line-up, but autonomous driving will not ramp up until 2021. 2018 will likely be the year of announcements around expanded EV lineups from traditional automakers. The Detroit Auto Show (Jan 13-28th) is an obvious window for these announcements and we expect a steady drip of EV model announcements throughout the year. We don’t believe these new vehicles will be available until 2019 or 2020. The autonomy theme will be equally top of mind in 2018, lead by likely updates from Apple around their autonomous shuttle project. We continue to expect 2021 as the year autonomy begins to ramp, and our bet is Waymo, GM’s Cruise, and Tesla will be first to market.  Here’s our autonomous vehicle industry model.

Bitcoin pulls back. We feel cryptocurrencies are in a bubble, but something can be in a bubble and still over time become a foundational technology, just like the internet. Those who jumped on history’s greatest “get rich quick” event enjoyed 1409% increase in value in 2017 as the world watched a coin formerly synonymous with the “Silk Road” break into Wall Street. We believe that blockchain technology and cryptocurrencies are here to stay and represent the future of storing value, however, we anticipate that increased oversight (banking and government), speculation amongst institutional investors along with operational difficulties on trading platforms will trigger a crypto sell-off in 2018.  Here’s our recent Bitcoin outlook note.

iPhone ASP’s $740 vs. Street at $710. We remain optimistic that iPhone units in FY18 will be inline with the Street (~242m up 12% y/y), but the mix of iPhone X will exceed Street expectations and have a higher ASP of $740, up 13% y/y compared to the Street ASP of $710. Our ASP estimate is based on a 30% mix of iPhone X and iPhone 18% mix of iPhone 8 (iPhone 8+ and iPhone 8). In a typical iPhone cycle, the newest phone represents about half of the mix, in line with our FY18 outlook of a 48%new phone mix. Here’s our recent note on why we remain comfortable with our iPhone estimates.

Amazon will acquire Target. We saved our boldest 2018 prediction for last, Amazon acquiring Target. Getting the timing on this is difficult, but seeing the value of the combination is easy. Amazon believe’s the future of retail is a mix of mostly online and some offline. Target is the ideal offline partner for Amazon for two reasons, shared demographic and manageable but comprehensive store count. As for the demographic, Target’s focus on moms is central to Amazon’s approach to win wallet share. Amazon has, over the years, aggressively pursued moms through promotions around Prime along with loading Prime Video with kid-friendly content. As for retail stores, Amazon’s acquisition of Whole Foods 470 stores along with testing of the Amazon Go retail concept is evidence that Amazon sees the future of retail as a combination of mostly online and some offline. Despite gaining Whole Foods, Amazon’s ~470 store presence still dwarf’s Walmart at 11,695 (global).  If Amazon acquires Target’s that would jump its store count to about 2,300.  As for anti-trust, the Trump administration won’t do any favors for Jeff Bezos, but the market share numbers suggest the deal will be approved. Walmart will reach about $315B in U.S. sales in 2017 (total 2017 Walmart is expected to be $500B, up 2.6% y/y), and Amazon North American ($105B in 2017 up 31% y/y) and Target ($71B, up 2.4% YoY) would equal about $176B in U.S. revenue. Looking at the top 18 U.S. retailers (including grocery), Walmart has about 23% share and an Amazon/Target combination would have about 13% share. Lastly, Amazon can afford Target. If you assume they pay a 15% premium to the current TGT trading level would imply a take-out valuation of $41 billion, about 8% of the value of Amazon’s current $564 billion market cap.

Disclaimer: We actively write about the themes in which we invest: artificial intelligence, robotics, virtual reality, and augmented reality. From time to time, we will write about companies that are in our portfolio.  Content on this site including opinions on specific themes in technology, market estimates, and estimates and commentary regarding publicly traded or private companies is not intended for use in making investment decisions. We hold no obligation to update any of our projections. We express no warranties about any estimates or opinions we make.

Face Off Part 3 – Echo vs. Google Home vs. Cortana

We tested the Amazon Echo, Google Home, and Harman Kardon Invoke (powered by Microsoft’s Cortana) smart speakers, each with 800 queries, and found that Google Home answered 81% of the questions correctly vs. Echo at 64% and Cortana at 56%.

Did Santa bring you a smart speaker? Smart speakers are quickly becoming one of the most common consumer uses of AI thanks to aggressive pricing and marketing from companies like Amazon and Google. We’ve conducted our latest round of queries, designed to test the full range of abilities and accuracy of a given AI assistant. Our conclusion: home assistants aren’t just gaining popularity, they’re making drastic improvements quickly. Here is what you can expect from the smart speaker that you unwrapped yesterday.

Methodology. Just as we have in February and August of this year, we asked 800 questions to the Amazon Echo (2nd generation this time around) and Google Home. We added the Harman Kardon Invoke speaker, coupled with Microsoft’s Cortana assistant, to our evaluation. The queries covered five categories: Local, Commerce, Navigation, Information, and Command. The smart speakers were graded on two metrics: did the device understand what was asked? (this can be seen on the device’s companion app), and did it answer or execute correctly? It is important to note that we have slightly modified our question set to be more reflective of the changing abilities of AI assistants. As voice computing becomes more versatile and smart speakers become more capable, we will continue to update our question set to be reflective of those improvements going forward. Our changes included questions around the use of smart home devices. We tested each of the speakers with the Philips Hue smart lighting and Wemo Mini smart plugs.

This round, Google Home was the decisive winner, answering 81% of questions correctly vs. the Echo’s 64% and the Invoke’s 56%.  On average, the speakers understood what was being asked 99% of the time. This is a jump up from 95% in August and means that these assistants’ natural language processing has improved to correctly understand, apart from a few anomalies, everything you say. Whether or not it correctly executes your request varies widely between devices and categories.

Results. Google Home continued its outperformance with the top score in each of the five categories. The Invoke, powered by Microsoft’s Cortana, scored more or less in line with the Amazon Echo – an impressive feat considering its tiny market share and comparatively short time in the hands of users. The Echo made considerable improvements in several categories, most notably navigation, while remaining relatively flat in areas like commerce and local.

Improvement across the board. We continue to be impressed with the rate at which these AI assistants are improving. Since our first test in February, the total number of correct responses has increased by 29% for the Echo and 42% for the Google Home. Cortana, which we tested in April on a Microsoft Surface Book, has improved 8%. In less than a year we have noted remarkable improvement in every category and can only expect this level of progress to continue as adoption grows and user base expands. Google Home, however seems to be on the steepest improvement curve. One would expect Google to dominate the information category with its unprecedented access to search data, but its outperformance in categories like commands, navigation, and even commerce furthers the case, in our minds, for Google Home to steal market share away from Alexa going forward.

New features emerge. During our testing, we noticed several minor, but encouraging enhancements to the voice computing experience, namely connectivity to both other devices and other services. Excluding the Echo, when you ask for navigation information, Google and Cortana will send directions to your phone either through a push notification or the companion app. Hailing a ride through Uber was smooth on both the Echo and Google Home. The Google Home could also hand you off to a designated EBay assistant to shop for used goods. We were impressed with improvements around media services like Spotify and smart home capabilities using both lights and smart switches.

Google takes a commanding lead in commerce. Google Home correctly answered 72% of commerce questions, as opposed to 42% and 15% for the Echo and Invoke respectively. This is an area that is thought to be dominated by Amazon. However, Google’s recent anti-Amazon partnerships with retailers like Target and Walmart to make voice-based commerce more accessible pushed the Google Home ahead of other assistants. This was the Google Home’s largest area of improvement, correctly answering 34% more queries than it did in August. 

Sound quality. Overwhelmingly, smart speaker owners use them to call up and listen to music. According to Quartz data, 74% of owners play music through their smart speakers, more than asking for weather and basic info. Of the devices we tested, the Harman Kardon Invoke sounded the best; the Google Home and Echo are very similar, but are not focused on premium sound quality. For this mid-tier price range (sale prices – Google Home: $79, Echo: $79, Invoke: $99) the Invoke sounds best. Great sounding music, however, is less defensible than the AI itself. A useful assistant can be loaded onto a tiny speaker like the Echo Dot and Home Mini, or a premium speaker as we have seen with the Alexa-enabled Sonos One, Google Home Max, and the incoming Apple HomePod. It becomes clear that you are paying almost exclusively for sound quality.

Still not mainstream. Despite our comprehensive question set, the everyday user rarely extends beyond simple queries like asking for the weather, general knowledge questions, or requesting a song. In fact, 65% of Alexa users have not even enabled a skill. In order for voice computing to take hold in the home and beyond, there needs to be improvement in terms of connectivity with other devices like phones, laptops, and televisions. We did note much better connection to your phone via companion apps, your television via devices like a Chromecast of Fire TV, and better control of smart home applications, which is an encouraging sign. A short list of persistent issues keeps smart speakers in the camp of tech gadget instead of essential device (like a phone or computer), but at the rate of change we are observing, we remain confident that AI assistants, in time, will become an integral part of how we interact with computers.

Disclaimer: We actively write about the themes in which we invest: artificial intelligence, robotics, virtual reality, and augmented reality. From time to time, we will write about companies that are in our portfolio.  Content on this site including opinions on specific themes in technology, market estimates, and estimates and commentary regarding publicly traded or private companies is not intended for use in making investment decisions. We hold no obligation to update any of our projections. We express no warranties about any estimates or opinions we make.

Apple Wearables Drafting on Tech’s Push into Healthcare

Healthcare is in the midst of a dramatic transformation. This may seem obvious, but the culmination of this week’s news – CVS buying Aetna to create a new healthcare platform and Apple partnering with Stanford to carry out a medical study on AFib using the Apple Watch – brought the pace of change into perspective. Healthcare is transforming before our eyes, and new players are moving into the space that accounts for 18% of U.S. GDP.

Taking a step back, healthcare 100 years ago was fundamentally the same as it is today. We go to the doctor for two reasons – we’re sick, or it’s time for an annual checkup. The effectiveness of this approach is dreadful, illustrated by the fact that about half of all Americans have one or more chronic condition, diabetes and heart disease are on the rise, almost 40% of us are obese, and 7 out of 10 deaths are attributed to chronic diseases. All the tech advancements we’ve made have not kept Humpty Dumpty together. The reason for this is that healthcare is generalized, impersonal, and reactive in nature. The individual must fight the day to day battle of preventative care, not the provider who the average American sees only 4 times per year.

Today, we see a shift toward two themes – personalization and prevention – and the future of healthcare will be grounded in the frequency of health monitoring. CVS and Aetna are coming together to create what CVS CEO calls the country’s “front door to healthcare,” because more doors means more frequent access to care. Apple and Stanford aim to collect data on more people more frequently. The concept of increasing health monitoring frequency holds the greatest promise of actually making people healthier and the easiest approach to increasing frequency is through wearables.

Today wearables are seen as a luxury gadget for geeks and health nuts. In the future (7-10 years from now), we will be inseparable from our wearables, similar to our current obsession with smartphones. Today, the smartwatch is the wearable of choice. Soon, however, that could include things like hearables (think AirPods), contact lenses, and connected fabrics.

Driving with your eyes closed. Healthcare monitoring today is comical. Nootrobox CEO Geoffrey Woo on an a16z podcast put it into perspective by saying “imagine that we’re driving cars and we only let ourselves open our eyes every minute. That’s essentially the snapshot of information we get when we go to the doctor.” We go in for a checkup, make a course correction, then drift back into our old habits until the next time we see the doctor. Continuous health measurement is the most effective approach to stay our course corrections. We now have biometric sensors in common devices and the computing power to make sense of that volume of data. The benefit of continuous health measurement is twofold – it allows for large-scale data collection from which AI algorithms can derive insights, and it keeps your health top of mind. And it appears to be working, studies show that 70% of Apple Watch users track their heart health, even weeks after purchasing the device.

Apple’s got a tiger by the tail. Investor opinions on the Apple Watch range from “it’s a rounding error” (4% of overall revenue), to “it’s a dud.” The reason is investors had been spoiled by Apple’s vertical growth in new product categories with the iPod, iPhone, and iPad. Apple Watch simply didn’t live up to its predecessors. While it has been a slow start for Apple Watch, we believe the Watch and future (7-10 years) wearables (notably hearables) will account for a material part of future Apple revenue. As the health advantages of wearables begin to resonate, we foresee Apple selling as many them (Apple Watch and hearables in the future) as they do iPhones. At a wearable ASP of $300 (below current Apple Watch ASP assumption of $450) and 250M units a year, that would equate to $75B in annual revenue (not in our model today).

Apple has been engaged with the concept of healthcare since it introduced the Apple Watch in 2015, releasing ResearchKit that year and CareKit in 2016. While their new Heart Study is technically their first true medical study, the Watch has been used in the past for similar crowdsourcing of biometric data (along with Fitbit and others). So this begs the question, why is Apple interested in healthcare? Their core competencies are well-aligned to benefit from the shift toward personalized and preventative care. They also have a platform in their device user base and software frameworks, the data and AI power to carry out large-scale operations, and the design expertise to integrate sensors into devices that consumers want to use.

A word on hearables. Apple has tipped their hand. Earlier this year, the company filed patents suggesting AirPods may have a future as in biometrics. The patents outlined the addition of a photoplethysmogram, or PPG sensor, that can measure heart rate, VO2, galvanic skin, EKG, impedance cardiography, and temperature. We don’t have enough details to guess when these features might be integrated into a product, but do see a future when these hearables are continuously worn, giving users volume control of the world, as well as next-level, real-time health monitoring.

What about other tech companies. Don’t forget about Google and Amazon. One of Google’s other bets, Verily Life Sciences, is focused squarely on making healthcare more preventative and data-driven. Verily argues, “a new car has up to 400 different sensors. You know the oil pressure and how much air is in your tires, but we don’t do that with people.” They have undertaken an array of different projects from glucose monitors in contact lenses to eradicating vector-borne diseases by engineering and releasing fertile mosquitos. Verily’s efforts are largely complementary to Apple’s health ambitions, and their engagement in the space is confirmation that big tech companies have a place in healthcare. The opportunity here is substantial enough to accommodate more than a few entrants. While it is unlikely that Amazon will be a player in data or wearables, the company has the DNA to reinvent the logistics around how care is delivered.

It will take time to win over the “I don’t want to be monitored” segment. It’s going to take years for widespread adoption of health monitoring wearables, as defined by more than a billion daily users. As a point of reference, we’re at 40-50m today, with about 25-30m of those being Apple Watches. Some people resist continuous monitoring on the grounds of privacy, inconvenience, and anxiety around knowing their true health. That said, the resistance group will shrink over time (some due to poor health).

Disclaimer: We actively write about the themes in which we invest: artificial intelligence, robotics, virtual reality, and augmented reality. From time to time, we will write about companies that are in our portfolio.  Content on this site including opinions on specific themes in technology, market estimates, and estimates and commentary regarding publicly traded or private companies is not intended for use in making investment decisions. We hold no obligation to update any of our projections. We express no warranties about any estimates or opinions we make.

Alexa First to Business but Google AI Close Behind

Bottom Line. Alexa has 70-80% smart speaker market share and is pedal to the metal in expanding the market beyond the consumer today, being the first company to target the business opportunity. While we expect Alexa will continue to be the market share leader, we believe Google will close the gap in the next 3-5 years as AI becomes foundational to the future of smart speakers.

News. Amazon announced Alexa for Business today. The basic idea is Amazon is building skills around everyday business activities like conference calls, scheduling meetings, keeping track of tasks and ordering supplies. This includes integrations with Salesforce, Concur, SuccessFactors, Polycom and Crestron to name a few. They are also selling a hardware starter bundle which includes 3 Echo’s, 2 Dots and 2 Echo Show’s.

Bravo Alexa. Today’s Alexa for Business announcement is further evidence Alexa is leading the smart speaker market. Alexa is the market share leader for good reason. First, Alexa is smart, and now has over 25k skills compared to 12k that we counted in April (Google does not have skills rather actions).  Second, Alexa 3rd party hardware integration is expanding and earlier this year we estimated there were about 100 manufactures with integrated Alexa IP.

AI, the elephant in the smart speaker room. While it may look like Alexa is running away with the smart speaker market, Google is gaining ground. In October at the Google Hardware Event, Google explained how hardware products will facilitate AI first computing. In 2017 CEO Sundar Pichai has opened each of his public remarks stating Google’s goal of becoming an AI first company. This has obvious implications for Google’s advertising, Maps, YouTube, cloud business and now hardware.

Google likely to gain smart speaker share in 3-5 years, but Alexa will still be a share leader. Google’s efforts in the next few years could yield a measurable increase in market share. As mentioned, today we estimate various forms of Alexa account for roughly 70-80% smart speaker share and we envision Google’s share increasing from about 25% today to greater than 35% in the next 3-5 years.

Why Google’s in a good position. While Google is lightyears behind Alexa’s install base, we believe Google has the best AI (see our comparison here), and their more robust product line could catch up quickly. Google is going toe-to-toe with Alexa in terms of hardware pricing (Echo Dot and Google Home Mini starting at $29). More importantly, we expect Google will weave their AI Assistant into the fabric of the device ecosystem. This is important because integrating an array of devices (i.e. the handoff between the home and the road) is what will push users toward the next generation of our interaction with machines.

Disclaimer: We actively write about the themes in which we invest: artificial intelligence, robotics, virtual reality, and augmented reality. From time to time, we will write about companies that are in our portfolio.  Content on this site including opinions on specific themes in technology, market estimates, and estimates and commentary regarding publicly traded or private companies is not intended for use in making investment decisions. We hold no obligation to update any of our projections. We express no warranties about any estimates or opinions we make.