Apple's new mixed reality headset

After owning Apple for much of the past decade+ (2008-2019), I no longer follow the company's financials in depth. However, as a big user of their products, I do track their product events, including their annual Worldwide Developer Conference (WWDC) every June and the focused events that crop up every few months. After a slew of incremental updates in this week's WWDC keynote, CEO Tim Cook had "one more thing" to announce. Apple finally announced its long-rumored AR+VR device, Vision Pro – and it was impressive. Unlike past VR devices, it suddenly feels like we are watching the future unfold in front of us, with features only shown in sci-fi movies.

I don't typically explore consumer tech devices in the HHHYPERGROWTH service, preferring to invest more in cloud-native software platforms that can more easily scale than physical devices with supply chains and inventory concerns. But as an optimistic technologist, I am greatly interested in this space, and believe Apple's moves below will renew attention in these emerging markets of interest (mixed-reality, and, perhaps from there, a coming wave of "the Metaverse"). And, of course, this is of interest to investors in Apple, Meta, Microsoft, and others moving into mixed-reality devices, Unity (XR gaming engine), as well as fans of Apple or XR products.

The keynote and provided videos were fascinating, and the reviewers present were given hands-on demos. [See links at the bottom for the takes I enjoyed the most, and thought captured the experience best.] However, I am also seeing some very off-kilter takes that get the specs or.  experiences wrong (one even called the lens translucent), overly focus on the price (comparing it to lower-end gaming-focused VR rigs, especially market leader Meta Quest), and do not quite get the gravity of the tech combination presented or Apple's vision. Let's walk through it all.

Takeaways

• Vision Pro is an all-new mixed reality device that competes with similarly-priced high-end enterprise-grade devices like Magic Leap 2, Varjo XR-3, and Microsoft HoloLens 2, as well as the mid-level enterprise-focused XR devices Meta Quest Pro and Lenovo ThinkReality VRX.
• It is a standalone device that provides a new computing paradigm Apple dubs "spatial computing". Apple envisions this being the next evolution in computing devices, as a mixed-reality device that can extend to use cases beyond what our current laptops, smartphones, tablets, and smartwatches are capable of – morphing their features into an infinite virtual spatial environment.
• Again – this is a high-end computing device, not a consumer-grade one with solely an entertainment/gaming focus. I think it best to think of Vision Pro as an evolution of the Mac and iPad – where Apple is moving towards the Quest, but not as a direct competitor to the Quest. This is not a product one would expect to be adopted en masse. It is the first step in a multi-year strategy, where other iterations and future products may eventually converge with Meta's "Metaverse"-driven vision. As is typical of Apple, they set the bar high with a top-of-the-line product right out of the gate.
• Think of Vision Pro as combining all of Apple's high-end products in one. It acts as a MacBook Air laptop, iPhone/iPad (minus the cellular connection), dual 4K displays with an AppleTV, and dual AirPods together, all worn on your head.
• This device is the culmination of all of Apple's past decade-plus of work on its various product lines, including its specialized chips, displays, microphones, cameras, sensors, and bands. This extensive expertise allowed Apple to instantly leap into being a top-of-the-line product in its coming release, and Apple clearly expects a fuller ecosystem to form by the time it launches early next year.
• It is built atop their custom M2 chip (which now powers every Mac line), and a new R1 specialty chip for real-time spatial processing over an array of sensors, mics, and cameras (from existing tech across their devices), including a digital crown (from the Watch), spatial audio (from the HomePod and AirPods), and AR & spatial mapping (from ARKit & RealityKit in the iPhone & iPad).
• As Apple is prone to do as it enters new markets (as it has done across the audio player, smartphone, tablet, and smartwatch markets), Vision Pro is now poised to dominate the market in high-end mixed reality. It is priced starting at $3499, with add'l charges for the optional magnetic optical lens from Zeiss for vision correction. And perhaps the "starting price" given hints at flexibility in the amount of compute and memory it has. [To compare it to Apple's other high-end computing products: the new Mac Pro with M2 Ultra starts at $5999, and their Pro Display XDR (6K monitor) at $4999.]
• Like the iPad, it comes with a virtual keyboard and dictation capabilities, plus works with existing Bluetooth devices like Magic Keyboard, Magic Trackpad, AirPods, and game controllers. As they have done with the Apple Watch, expect a variety of headbands and other accessories from Apple and 3rd parties, including fashion-oriented ones. (Some are already appearing.)
• The device is expected in early 2024 (~8-9mo away!), but is announced now to allow partners and developers time to take advantage of this new paradigm, which will help build out the ecosystem at launch. Disney was focused on as a launch partner. They are finalizing the product for release (and readying Apple Stores for the demos and fittings).
• As always, Apple will continue to iterate from here. I expect Vision Pro to be a heavy focus over the next few years as their product flywheel takes effect and drives the acceptance and adoption of this emerging paradigm. I expect other related XR devices to emerge (perhaps consumer-based ones honed on gaming and fitness), and Apple might revitalize its smart glasses ambitions from what it learns.
• Despite its high-end feature set that compares to enterprise-focused competitors like HoloLens 2 and Magic Leap 2, most can't help but compare it to the price of the lower-end consumer-focused Quest line. Yes, it is way more expensive ($3499 vs $399... now $299... now $499 with Quest 3), but this is not just an entertainment and gaming VR environment – it is a high-end feature-rich standalone generalized computing device that can likely replace laptops, external displays, tablets, and (single viewer) TV/audio setups.

This is the first new product line and OS from Apple since the Watch hit in 2015. (No, I'm not counting the HomePod from 2018, as it doesn't have its own OS and App Store.) As they did in the mobile (smartphones and tablets) and wrist (smartwatches) markets, Apple is stepping into a new computing direction as a major new platform – not as the first to market, but as the best hardware and software combined experience. XR is an area that Microsoft has been focused on for over 7 years now (HoloLens first appeared in early 2016, and their current HoloLens 2 is now 3.5 years old), and Magic Leap and Oculus even longer (founded in 2010 and 2012). Countless other startups have emerged from there, like Varjo, Lynx, XREAL, and Vuzix.

I've long been interested in the shifts toward greater adoption of XR and ultimately "the Metaverse", but after a decade-plus of progress, the market hasn't shown up beyond consumer gaming and a handful of enterprise uses. The existing XR market feels like it is losing steam (see the rise and fall of the interest in the Metaverse, rumors that Microsoft halted development of HoloLens, Oculus's elusive profits and mass adoption, Magic Leap's constant struggles) – but we now have Apple entering with a new vision and direction, with an extremely high-end product that vaults to the top of the pack. What we see now is merely the pre-game of a multi-year focus. I feel that Apple's moves will greatly boost interest in the market, followed now by several more years of innovation and iteration that help bring it more and more to the masses.

The Layers of Reality

Let's quickly review the various flavors of AR and VR devices out there.

• Augmented Reality (AR) devices are transparent smart glasses focused on viewing the real world and digitally enhancing it. These can include Heads-Up Display (HUD) features that analyze what you are viewing to inject further virtual details, and integrated cameras for seamless video capture. Examples include Google's now-discontinued Google Glass, as well as Snap, Lenovo, Rayban (helped by Meta), XREAL, and Vuzix. [Meta and Apple have both been rumored to be working on AR smart glasses of their own as well.]
• Virtual Reality (VR) devices are vision-blocking headsets focused on viewing fully immersive virtual worlds. They are typically paired with a set of hand controllers as motion sensors, and earlier higher-end editions needed to be tethered to an external computing system for the massive level of compute needed in rendering it. VR headsets widely available today are typically lower-end consumer-grade gear built for entertainment/gaming use cases – an area that Meta/Oculus has long been focused on, along with HTC, Sony, and Valve. Several have models moving into basic enterprise needs like virtual conferencing and training.
• Mixed Reality (MR) is a combination of AR+VR, with devices that tend to focus on either consumer or enterprise use cases. This is being approached from 2 directions. VR devices began to overlay the physical world via outward-facing cameras being rendered in the virtual view (called Passthrough VR), but these flavors of "digitized AR" have typically had high lag and low-resolution pixelated views – which then leads to a poor user experience, or even eye fatigue, nausea, and motion sickness. From the AR side, capabilities emerged to allow users to overlay and interact with virtual objects within the augmented view of the smart glasses. Examples of MR include Microsoft's HoloLens, Magic Leap, Varjo, Lenovo, Lynx, and Meta's enterprise-focused Quest Pro.
• eXtended Reality (XR) is the umbrella term over all of these. The term is often intermixed with MR.

These flavors typically focus on consumer use cases (entertainment, gaming, social) and/or enterprise use cases (training, group conferencing, augmented skills, virtual work environments, simulations).

The rumor mill was abuzz heading into WWDC on how Apple was releasing an XR headset, including this major praise from the founder of Oculus a few weeks ago. (Oculus released their first VR headset over a decade ago now.)

Vision Pro

What's important to note upfront is that Apple's vision of their new mixed-reality headset is as a high-end standalone computing device. This is an entirely new form factor that Apple is calling a "spatial computer", complete with its own OS (visionOS) and coming App Store.

Pricing starts at $3499, and it will not be available until early next year in the US then expanding globally from there. It will come with Apple's standard core apps, a large set of Apple Arcade games, and will support a long list of compatible iPhone and iPad apps. They announced this at WWDC to get developers on board, as specialized apps are needed to make the new platform compelling beyond using it as an infinite workspace and entertainment/gaming device. Many are balking at the price, however, and most comparisons are putting it up against the current VR market leader, the way lower-spec'd Meta Quest line (Quest 2 and Quest Pro).

Let's first walk through the product's capabilities from head to toe, then circle back to the competition.

• The device requires a custom fitting process at an Apple Store [or a custom iPhone/iPad app to measure face to order online, as we found out this week], to select the right light seal around the lens to fit your face, pick a headband style and size, and adjust the audio straps to your ears properly. It also requires an orientation session to explain the new eye/hand-based controls, and calibrate the eye tracking and spacial audio. Once you get the device, you then set up your authentication and avatar (see further points on those).
• It will not fit over glasses. Those with vision issues must purchase additional corrective lenses that are provided by 3rd party partner Zeiss. [This is typical of modern VR sets. Beyond Zeiss, there are plenty of other 3rd party providers that provide corrective lenses for popular VR brands.]
• It maintains your privacy and settings through a new authentication system via iris recognition called "OpticID". Your eyes serve as your unique fingerprint. (Apple has always led the market here, between Touch ID, Face ID, and unlocking Macs with your Watch.)
• You use the device initially (off your head, pointed at you) to capture your likeness as an AI-driven 3D avatar, which is then used to represent you in features like video conferencing, all in real-time as you move, blink, and speak. You basically set up a digital twin of yourself, which acts similar to the Memojis in Messages, but is way more hyperrealistic (less cartoony). This is a sign that Apple is moving in a completely different direction from where Meta/Oculus is going. One should expect the hyperrealism to continue to improve from here.
• Their new R1 specialty chip is specifically focused on capturing and processing real-time information from sensors (LiDAR, IR), microphones, and cameras (forward, down, and side) as you move your head, eyes, and hands and use your voice. The lens is not translucent – these real-time cameras provide passthrough video of the physical world in AR that is displayed in your VR view. The result is a silky-smooth hyper-realistic AR experience, making it seem as if the lens is translucent. [Some reviewers pointed out that the room lighting seemed dimmer or warmer in AR, but otherwise, it felt that they were looking directly at the room.]  
• There are no external controllers needed to control the device. Similar to Microsoft's HoloLens, you control it through your head, eyes, hand gestures, and voice. You don't need to use exaggerated movements, and can remain standing in place or seated as you control it. Sensors over your eyes allow your gaze to control the focus (say, selecting an app from the home screen or changing focus to another window). You control the click-and-touch actions via hand gestures, which seem similar to existing touch gestures on the Mac trackpad and iPhone/iPad screens. Search bars and other text inputs allow using your voice. [Let's hope Siri really starts improving rapidly from here with all these new modern LLM models.]
• The 2 video monitors for your eyes are both micro-OLED displays the size of postage stamps, with minified pixels that are a mere 7.5 microns wide (64 of which fit in one iPhone pixel!). This resulted in an over 4K OLED display per eye – double the resolution of the HoloLens 2 and the Quest Pro. The refresh rate wasn't mentioned – but given the smoothness, is believed to be high. The two screens combine into 23M pixels overall – a touch more than their high-end Pro Display XDR 6K monitor with 20.4M pixels (which costs an eye-popping $4999).
• Vision Pro intermixes the virtual and physical world well, and allows the user to blend elegantly between overlaid AR and immersive VR via the digital crown, by enveloping you with 360° immersive scenes they call "Environments". [Think of these as the nature-focused high-resolution screensavers you get on Apple TV becoming an enveloping background that surrounds you as you turn the digital crown.]
• This all combines to essentially provide an infinite workspace in your virtual spatial environment, where you can move and resize windows, move them closer or farther, or surround yourself with different screens. Apple stated that individual screens can be made to seem 100 feet wide (from your perspective) if desired – turning your virtual environment into a private IMAX movie theater just for you. Your perspective changes as you turn your head, giving you somewhere between a 180°-360° environment based on how much you turn your head and body. Computation needs are minimized through foveated rendering, a capability used in VR (coupled with eye tracking) where a specific part of the view is brought to the highest resolution only as you focus upon it, keeping the periphery at a lower resolution.
• Connecting the lens housing, the light seal, and the head strap together are "audio straps" that place spatial audio speakers at your temples. These use a lot of the tech that emerged in HomePods and Airpods to allow you to hear sounds in a 3-dimensional space. You can also use AirPods to maintain complete audio privacy.
• To combat the isolation issues typical of VR devices, it will display nearby people in your AR field of view as they walk near you and communicate with you. As for what that person sees, the Vision Pro has an innovative feature called "EyeSight" that shows your "eyes" (via that 3D hyper-realistic avatar mentioned above) on an outward screen on the front of the lens, to show others that you are looking at them. From there, it obscures your "eyes" when you looking at AR content (partial obscure) or immersive VR content (full obscure), as well as when you are using the new spatial camera (blinking).
• The Vision Pro home screen looks like the iPad's when you first boot it up, which appears when you click the digital crown. Standard Apple apps are available like Safari, Photos, Notes, Mail, Messages, FaceTime, Apple TV, and more, plus the device can share content with other Apple devices via iCloud. They also promised 100+ games in Apple Arcade will be supported and available, plus announced Unity is adding support for the platform. Apple will also make a huge number of compatible iPhone/iPad apps available when the App Store launches for visionOS. [This injects a big ecosystem of apps into the computing device, greatly helping kick-start its ecosystem, though will not be ones specifically honed for a spatial experience.]
• It does not require tethering to a device to operate – the cord you see in demos is the power cord to an external battery or power source. Like the iPad, it is entirely standalone in that users can use a virtual keyboard or voice dictation, plus can also connect to external devices like AirPods, Magic Keyboard, Magic Trackpad, and game controllers. I expect accessories (like external batteries, custom headbands, and protective cases) to appear by launch.
• It renders both 2D and 3D content seamlessly in the virtual space from apps, games, or movies. Apple showed off the 3D features by letting reviewers watch a scene from the latest Avatar movie in 3D, then gave several 3D demos that were shot from its own internal Apple Immersive Video system – which shoots high-resolution 3D video in a 180° view. (This system will be used to provide new AppleTV+ content in 3D, and hopefully, others like Disney will adopt it for their own content and sporting events.)
• It also has an all-new 3D camera system that can record spatial photos and videos via a button on the outside of the lens. (And as mentioned before, the EyeSight feature lets others near you know you are recording.) While their canned video showed it being used in a consumer setting (a father recording his children), it is easier to envision this being more ideal for enterprise use (say, for the Vision Pro to be the equipment for a roving cameraman). [I hope to see this new spatial camera also move to the iPhone, which is easier to "whip out" in order to capture the moment when needed – making the Vision Pro as the ideal place to view that content, rather than shoot it.]
• It allows you to remotely control your Mac by looking at its screen, which opens up a window that acts as a remote 4K external display to the Mac – letting you then use Mac apps via the Vision Pro's eye/hand controls. [The demo showed an office worker using Final Cut Pro, Apple's video editing software.]
• It uses wifi. There was no mention of it having a 5G/LTE mode like the iPad or Watch. [I feel this says volumes about how portable the device is really intended to be.]

How it can be used

Microsoft has had a monopoly on enterprise use cases for mixed reality with its HoloLens line, and Magic Leap, Lenovo, and Meta are now focused on enterprise uses as well. Apple let those companies develop and explore the market over the last several years, then leap-frogged them in one single chess move.

It essentially provides a new spatially-based computing platform with infinite display capabilities. At a minimum, this device could potentially replace the need for a Mac with a multi-display setup, as well could replace a high-end TV and spatial audio setup for a single individual. You can run all the same apps as on a Mac or iPad, and share files and content with your other Apple devices through iCloud, plus it can also serve as a private external display for external Macs. But it's much more than just that, as you can now pick up and take it (your Mac and multi-display setup, or private movie theater) to wherever you work.

Ben Thompson at Stratechery highlighted this in his take:

"What is much more compelling to me, though, is simply using a Vision Pro instead of a Mac (or in conjunction with one, by projecting the screen). At the risk of over-indexing on my own experience, I am a huge fan of multiple monitors: I have four at my desk, and it is frustrating to be on the road right now typing this on a laptop screen. I would absolutely pay for a device to have a huge workspace with me anywhere I go, and while I will reserve judgment until I actually use a Vision Pro, I could see it being better at my desk as well."

Consumer use cases include having an enormous (portable) virtual workspace, watching entertainment, gaming, video conferencing, and reviewing photos/videos & taking new spatial ones. Apple suggested you can make videos appear to be 100 feet wide (wider than an IMAX movie screen), and that Panorama photos taken on the iPhone wrap around you as a 180° view. Enterprise use cases include using the device as an enormous (portable) virtual "work" space, and seems especially ideal for traditional Mac use cases like visual design, development, game design, photo/video editing, and much more. Beyond that, the enterprise use cases being explored by existing AR and VR devices also apply, such as augmenting skills, training, simulations, and virtual work environments.

It is clear that Apple doesn't think of this device solely as as being consumer-focused on gaming & entertainment, which is where existing VR makers like Meta have been focused. They are positioning it as a more general computing device good for both enterprise and consumer use cases. And this is clearly high-end with a high price tag ($3499 starting price), as they see it ultimately replacing the need for a Mac or iPad and multi-display setups, with a device that gives you an infinite workspace where you can make screens as large as you want. With “starting price”, they seem to be suggesting it will have different options available to up the compute (perhaps between the M2 Max and new M2 Ultra?) and memory. I assume this will be marketed to the same audience as their MacBook Pro and Mac Studio around design, video editing, content creation, and development. [It is essentially a $2K Mac Studio on your head, and if it uses the new M2 Ultra chip, it is essentially a $4K Mac Studio on your head. However, it is harder for it to compete with the $6K Mac Pro, which allows you to use PCIe cards for digital processing.]

Competition

And, of course, we must circle back to the collective gasp that emerged from the crowd when the price was announced. However, Apple is not focused solely on AR (like Google Glass) or VR (like Meta Quest, now morphing into mixed), and is not limiting itself to being solely a consumer-grade product. The appeal of it seems more in line with where higher-end enterprise-grade units like Microsoft Hololens have gone in mixed reality, and where Magic Leap is taking AR into enterprise use cases. All of these are based on earlier Qualcomm Snapdragon mobile chips (before they added XR-focused versions).

• Microsoft's HoloLens 2 is its high-end mixed-reality device focused on enterprise use cases, which was announced in Feb-19 and released in Nov-19 (3.5 years ago!). It has 2K resolution (aka 1080p), a step up from the original version's 720p. It has a starting price of $3500, with premium versions optimized for certain workplace environments (such as industrial clean rooms and risky worksites), and has even been sold into military applications (to mixed results). Unlike typical VR devices, it does not utilize hand controllers, instead detecting hand gestures based on Microsoft's discontinued Kinect motion-sensing products for Xbox. It is a standalone device that runs on a flavor of Windows 10 called "Windows Holographic" – recently upgraded to Windows 11 in Apr-23. Most recently in May-23, they reiterated their commitment to the platform – likely in response to Apple's coming product, after rumors from early 2022 that they shelved HoloLens 3 efforts.
• Magic Leap was a well-loved startup solely focused on AR smart glasses. After its disappointing first release back in 2018, they pivoted into enterprise use cases as they strived to recover their brand. They released the Magic Leap 2 smart glasses in Sep-22 at a starting price of $3299. These are high-end smart glasses focused on enterprise AR use cases like augmented training and guided work. This greatly improved Magic Leap's feasibility, and they see themselves as competing with XR.
• Varjo is a Finnish company (founded by ex-Microsoft and Nokia execs) that started in VR in 2019 and has evolved into XR. They released their latest VR-3 and XR-3 models in Dec-20 for enterprise use, at starting prices of $3645 and $6495. Like the HoloLens, it is using Windows as the core OS. They have been exploring it in various enterprise and industrial use cases since. In Oct-21, they released their Aero model, a lower-priced consumer-grade VR unit.

The next level down from those high-end enterprise-focused devices are the mid-tier VR devices that are moving into XR. All of these are based on Qualcomm's newer Snapdragon chip focused on XR.

• Meta released Quest Pro in Oct-22, a version of Quest 2 that is focused more on enterprise uses and mixed reality (first priced at $1499, now at $999) – however, the new AR features have been poorly reviewed given the display quality. A few days before the WWDC, Meta lowered Quest 2 prices and announced their coming Quest 3 coming in fall (priced at $499), which improves the passthrough to provide a smoother and higher resolution mixed reality. [I expect a revised Pro edition will follow, to continue their shift towards enterprise use cases.]
• Lenovo has evolved from its A3 AR smart glasses into mixed reality with its new ThinkReality VRX series announced in Sep-22. They had a reactionary announcement during WWDC that it will finally release in June, priced at $1299.

Note that all of these competitors above are based on the Qualcomm Snapdragon system-on-a-chip (SoC) line, which has evolved from mobile to XR. Apple is completely relying upon its own using an M2 chip, which powers the entire Mac lineup now. Beyond that, it uses a newly designed R1 chip designed specifically for its vital real-time sensor/camera processing.  Apple is stating that the R1 chip gives the AR a latency of 12ms or less – well below the lag seen in the Quest Pro and the HoloLens that can lead to nausea after prolonged use. This is cutting-edge performance for a real-time processing engine. Combining that R1 with Apple's massive advancements in compute (M2) and miniaturizing OLED pixels into an overall 4K+ resolution in the headset, the specs of the Vision Pro far outweigh the competition.

My take

First off, Apple's focus feels different than other platforms. It is going after both high-end consumer and enterprise markets as a general computing device – the same cross-applicability it had in its laptop, tablet, and smartphone markets. Most VR devices have a consumer focus on gaming/entertainment – not as a general computing device with a dedicated OS.  Apple is positioning Vision Pro as an everyday computing device that is focused on hyperrealism, not a cartoony vision of a social VR metaverse being chased by lower-spec'd devices from Meta/Oculus. Like in all of its products, Apple has built-in privacy and security features through Optic ID (much less the fact no one can see your virtual workspace). No accessibility features were announced, but I expect a focus on it.

The success of this device ultimately depends on the app ecosystem. Like with the iPhone, iPad, and Watch, Apple is seeding it with its own native apps first, plus is able to boost its initial ecosystem from Apple's existing pool of compatible games and iPhone/iPad apps. From there, it depends on partners and developers to really start taking advantage of this new computing paradigm. Vision Pro provides a spatial environment, and an infinite display, and is controlled with natural gestures, so I believe this device will ultimately be successful in developing a rich app ecosystem, which in turn further drives adoption.

As it did with laptops, smartphones, tablets, and smartwatches, Apple is never one to be first to market. Apple has been developing this internally for years, taking its time to learn from others in order to carve its own path. It typically then leapfrogs long-time players at launch with a top-of-the-line product, which then helps the market more fully mature and take hold. Apple's leadership upon entering these markets typically triggers a flywheel driving further interest in adoption – all while the existing leaders fall behind or start copying features to keep up. I foresee this replaying yet again now with mixed reality.

At launch, Apple will immediately vault to being the top-of-the-line in mixed-reality devices. All of their prior devices have led up this moment, which now culminates in the Vision Pro. This product was built by combining Apple's vast expertise in custom chips, high-res displays, device sensors, cameras, and microphones – with Apple mentioning that they filed 5000 patents along the way as they designed this over the years.

In particular, I think it benefited from a few ongoing focuses over the past few years. First is its focus on multi-dimensionality on iPhones, iPads, HomePods, and AirPods (via LiDAR & IR sensors, ARKit & RealityKit SDK, and advancements like FaceID, 3D emojis that track facial movements, room mapping, spatial awareness, and spatial audio). Apple also has extensive expertise in advancing OLED displays and miniaturizing pixels, which is incredibly important to this device having such high resolution in such a small package. Using Vision Pro is ultimately like having infinite displays attached to a desktop computer.

I feel the biggest key was how smooth and real-time the device's AR is due to their new R1 chip, which controls all the sensor and camera inputs to reduce latency to 12ms or less – 8x faster than the blink of an eye (100ms+). This helps keep the AR smooth, which greatly reduces the eye fatigue and motion sickness issues common in VR headsets. All that experience in creating custom chips over the years has now culminated in them solving some core issues in AR with R1. They have gained a lot of system-on-a-chip (SoC) experience from their past A series (iPhone), W-to-H series (AirPods), S series (Watch), M series (Macs), Neural Engine, and Secure Enclave chips – and R1 is a sign of more to come. [I have to wonder if the R1 emerged from their past attempts at a smart car? Smart cars are another real-time system built atop an array of sensors and cameras.]

Existing enterprise XR devices aren't this seamless and latency-free with their AR, nor have this high of a display resolution. (The Sony PSVR has that level of display resolution, yet requires being tethered to a Playstation for its compute needs.) Per the reviews thus far, Apple seems to have nailed the gesture control via eyes/hands, the AR is seamless and undetectable from the R1 and high-rez displays, and text appears crisp and readable as you browse the web or view content. You definately don't get that in mid-tier XR systems like Oculus's Quest Pro, and the advancements seen in the HoloLens 2 (released 3.5 years ago) are quickly being outpaced.

So to sum up, Apple outpaced the competition in one swoop due to its vast experience from:

• Better core chipsets (its existing specialized M2 and the new R1)
• Better display specs and clarity (years of OLED experience on the iPhone, Watch, Macs, and pro displays)
• Better sensors, cameras, and microphones (from years of adding them to Macs, iPhones, iPads, Watches, HomePods, and AirPods)
• Better passthrough AR, latency, spacial awareness, and fine-tuned controls via gestures and eyes (the new R1 is key!)
• Better sound (spatial audio from the HomePod and AirPods)

Another interesting innovation is the AI-controlled avatar it creates as your digital twin. This allows Apple to help reduce the isolation inherent in VR devices (which hide your eyes) with their new "EyeSight" feature, combined with how it shows others you are communicating with in your AR field of view. They are also focused on making the device comfortable, which matters greatly when wearing this for long periods – however, it remains heavy even with the battery pack moved to an external pack. I expect them to improve upon the weight and portability in future iterations. It has a modular design (lens, light seal, audio straps, headband) to best fit a wide variety of humanity's head shapes and sizes, which also opens up a new accessories market from 3rd party partners.

Apple is starting with an initial ecosystem in place from its existing core apps, as well as a pool of games in Apple Arcade and a wide set of compatible apps from the iPhone/iPad. A more honed app ecosystem will emerge in the coming months as developers gain access to the device, and Apple mentioned apps from partners like Adobe and Disney will be available at launch. It not only has Apple TV+ for entertainment content, but Disney plans to fully support the mixed reality features across ESPN for sports and Disney+ for content from Disney, Star Wars, and Marvel. It initially supports FaceTime for video conferencing, with the promise of other 3rd party apps to be available at launch – like Webex and Zoom, which are both now supported on the Apple TV. Beyond that initial set of 100+ Apple Arcade games supported, Unity has announced support (as it did for Oculus) – so expect a lot more games to come, especially from the pool of games already on Oculus created on Unity that can be easily ported. (I assume Disney will embrace it for gaming as well as the content.)

Concerns

This device is not perfect, of course, as there are some known downsides:

• Users have to fit the device to their face, train it to track their eyes, and position audio straps over their ears. Luckily, Apple Stores are already well-positioned to provide this, along with orientation sessions on using the device via eye/hand gestures.
• It doesn't fit over glasses – corrective lenses will be needed, which are provided by Zeiss at an extra cost. (Or one can wear contacts while wearing it to avoid this.). But this isn't unique to Vision Pro.
• The battery does not (yet) seem a major focus, and was isolated to an external pack to reduce the lens's weight. It seems better suited for sitting in one spot on a constant power feed versus traveling with it. They showed it being used to watch a movie on a plane, but that feels like an outlier given the limited battery life when making it portable. I expect bigger batteries to quickly emerge so you can better use it on the road or in a coffee shop type of setting.
• No mobile (5G/LTE) features were mentioned, only wifi. Again, not really an issue if the device isn't truly that portable. [I would expect this more in smart glasses if they emerge from here.]
• Reviewers noted it is heavy from the lens frame and all the tech it houses. This may not be that much of an issue if the device isn't truly that portable. Also, head fatigue seems less critical than the more immediate issues with mixed reality devices like eye fatigue and motion sickness.
• There are no capabilities [yet] for physical feedback. While they nailed the eye/hand controls, spatial awareness, and spatial audio, there are no haptic sensors installed. This is likely because of their choice to go controller-free. [I wonder if haptics might emerge as after-market Bluetooth accessories.]
• It also seemed that Apple is not focusing on fitness tracking with the Vision Pro as it is the Watch and iPhone – which is a shame, as the immersive VR features would allow for some escapism as you move through virtual environments. While you aren't going to wear it hiking, running, or biking outside, it would seem to be appropriate for relatively-fixed activities on a treadmill, rower, stationary bike, or elliptical machine. Again, the headset is heavy, and I wonder if their testing showed discomfort during fitness activities. [This makes me wonder if they will introduce a lighter, more portable model better suited for "stationary" fitness uses eventually, which could tie more closely into the Watch's features.]

It is easy to envision how the success of they found with iPhone, iPad, and Watch will be repeated now in Vision Pro – however, Apple has not always excelled in its market dominance or triggered a rise in consumer adoption. In particular, its moves into the living room have not worked out as well. Apple TV was a strong product, but didn't capture much market share (beyond all-in Apple fans like myself) given lower-cost competition from Google, Amazon, and Roku. HomePod didn’t sell well and was eventually discontinued in Mar-21 (leaving only the Mini version available) – only to reverse course, bringing it back in a 2nd generation in Jan-23. (However, there were benefits – HomePod had made several advancements in spatial audio and room mapping that are now in use here in the Vision Pro.) And voice control has long seemed a weak point (which also likely hamstrung the success of HomePod) – hopefully, Siri gets renewed focus with Vision Pro using voice so prominently. There were also rumored smart TV and car initiatives that ultimately fizzled and were abandoned.

And what about the XR market itself – is Apple moving into a tired market stuck in a dead-end? We have thus far seen muted adoption of existing AR/VR devices, and a now waning interest in the Metaverse in this rocky macro environment. I believe the answer is no, as Apple tends to enter new markets and dominate with a top-of-the-line product – which then bolsters a new wave of adoption. The Metaverse (whatever that truly means) does not really exist yet – but after this move, Apple is sure to be a big player when it ultimately does arrive. Apple sees the next direction in computing devices being set in hyperrealistic mixed reality, in order to provide an infinite spatial workspace for both consumer and enterprise needs. The Vision Pro feels flexible – if Apple needs to pivot quickly into the cartoony social Metaverse envisioned by Meta, it can easily integrate Memojis from the iPhone, or pull in other tech (such as haptics) from its devices.

They called this device Vision Pro right out of the gate, suggesting it will be their top-of-the-line device (and is already top-of-the-line in the XR market). Apple will continue to iterate from here as prices drop, with a future non-Pro version likely (as we saw with iPhones and Watch SE) to more directly compete with Oculus. And Apple may yet pursue a separate line of smart glasses – which has been rumored for years now as well. These types of future efforts will learn from how Vision Pro ultimately gets used, as well as how Apple will continue to iterate to remain at the forefront of the tech underlying it all.

I have to agree with long-time Apple observer John Gruber:

"Is it a compelling product, though? It’s a famous Steve Jobs axiom that technology is not enough, that you don’t make compelling products — let alone entire platforms — starting from advanced technology and working backward. You start with a vision for the product and platform experience and then create new technology to make it real. I simply can’t say whether Vision Pro is going to be a compelling product. I spent too little time with it, the software as of today is too far from complete, and, most importantly, the whole experience is too entirely new and mind-bending to render any such conclusion.

But the potential for Vision Pro to be a compelling product, across several use cases, is obvious. This might be great. And without question it is interesting, and I think the fundamental conceptual bones Apple has designed for VisionOS lay the groundwork for a long future. The first generation Vision Pro may or may not be a successful product — I simply don’t want to speculate on that front yet. But even just a small taste of VisionOS made me feel confident that it is going to be the next major platform for Apple and Apple developers, alongside MacOS and iOS/iPadOS."

Meta's Response

After writing this, The Verge published a leak yesterday of Meta CEO's response yesterday to the Apple announcement. I recommend reading it, as it draws attention to the wildly different visions. I found his response quite lacking, as it ignored a lot of the ramifications to his own vision.

"I think that their announcement really showcases the difference in the values and the vision that our companies bring to this in a way that I think is really important ... More importantly, our vision for the metaverse and presence is fundamentally social. It’s about people interacting in new ways and feeling closer in new ways. Our device is also about being active and doing things. By contrast, every demo that they showed was a person sitting on a couch by themself. I mean, that could be the vision of the future of computing, but like, it’s not the one that I want. There’s a real philosophical difference in terms of how we’re approaching this."

I completely agree with that last sentence. Meta, a software platform that acquired its way into hardware, is focused on "the Metaverse" – aka it is a social media platform that is striving for XR to become a socially-focused virtual platform. That is not Apple's vision at all. This new product is infused with Apple, from the elegant combination of next-gen hardware and software down to the focus on security and privacy. This again reiterates how Apple is making a high-end computing device for an individual user to use (as I said before, an evolution of a Mac + iPad), while Meta is focused on consumer-grade mass adoption of a "social" type of VR.

But this all washes over the fact that VR is an inherently lonely device! You are shutting out the real world – whether you are moving in a game or workout, or sitting on a couch. This is exactly why Apple was so focused on solving the isolation issues with VR. Zuck focused only on the entertainment/gaming demos, conveniently ignoring the ones that featured a group collaboration session in FaceTime (something the Quest Pro is specifically focused on), the dad working at home on product design while standing at his kitchen island, and the entire segment set in an office where a coworker "broke into" the AR – a part of the demo highlighting Apple's focus on reducing the isolation. It was also silly that he mocked the tethered battery, given how the Quest Pro's battery life matches the Vision Pro's at 2 hours.

Yes, Zuck – these are wildly different visions and philosophical differences. The cartoon version of the Metaverse may ultimately succeed, but that is not at all where Apple is moving in its first iteration of an XR device.

Add'l Reading

• You can watch the whole 2hr keynote, but probably best to skip right to the "one last thing". The Verge had a 26-minute supercut on all the main points, but again, you can skip right to the Vision Pro supercut.
• See Apple's Vision Pro PR and product page for more details, as well as their 9-minute demo video. One easter egg in their demo video was how the office worker browsed a webpage from a design magazine called Azure – which kind of felt like a nod to Microsoft Hololens.
• I wanted to get a sense of what the hands-on demo experience was like. See  this CNBC take as well as the extended take from the always enjoyable John Gruber of DaringFireball.
• If you read any piece, I highly recommend this take from Ben Thompson of Stratechery on Apple's real vision here.
• I also enjoyed the YouTube takes from consumer tech reviewers MrWhoseTheBoss and Marques Brownlee of MKBHD. I also enjoyed this overall WWDC take from that YouTube filmmaker that I showed a tweet from, and his later reactions to Vision Pro as a filmmaker.
• My award for worst take goes to PCWorld. Apparently, Microsoft showed us all these features 7 years ago! (Back in their first 720p resolution Hololens released in 2016, priced at $5000.) So much for those specialized chips, advanced sensors, 4K+ resolution, and that silky smooth AR experience from spacial video and audio in the Vision Pro.
• Second place goes to an opinion piece in Wired with a hugely negative take, which comes off as a big whine fest about the isolation caused by wearing VR devices and how goofy the devices would look in public – something that is in no way specific to Apple, and, like Zuck above, ignores Apple's big focus on features to combat it and other VR pitfalls (like EyeSight and smooth passthrough AR). Yes, some of Apple's promotional demos of it were silly (wearing it to capture family moments or to watch a movie on a plane), but others were spot on to show its versatility (using it in an office or home setting for a private infinite workspace). The piece seems to ignore how multiple big tech firms have all been chasing this vision for years now.

XR seems likely to eventually have its adoption explode in some form or another, and Apple likely kick-started a renewed interest with its announcement. Apple and others are sure to continue to iterate from here for the foreseeable future – giant goggles on your head are not the end game here! But despite that, Apple has created a compelling new high-end spatial computing device that looks to eventually replace a multi-display Mac setup.

As noted above, this was written 7 months ago, right after WWDC – so it missed the release of Quest 3 and other AR/VR announcements out of CES'24. The Vision Pro is now launching February 2 in the US, with pre-sale opening up tomorrow morning.  But I'll take that conversation to the new blog. Come join!

-muji