Apple

Apple Reality is getting real

memoji wwdc20

Meta CEO Mark Zuckerberg told us that in the future we will hold our business meetings in virtual spaces and be represented there by avatars.

He was right about that. But it seems that Apple, rather than Meta, is crafting the more compelling take on avatar-based virtual business meetings. New details have emerged about Apple’s augmented reality (AR), and it looks like something that could really transform the way professionals communicate with each other.

Small warning: the platforms we are talking about did not appear even in beta and therefore cannot be compared. All we have to rely on are company statements, leaks, reports, detectives, speculation – and common sense.

Let’s start with the detective.

Apple is apparently working on an operating system called “realityOS”, which Apple sometimes abbreviates to “rOS”. We know this because references to “realityOS” and “rOS” have been discovered in iOS 13 preview builds, a GitHub repository, and even App Store download logs. The GitHub repository also hints at a realityOS simulator, presumably for developers.

Apple uses “Reality” in the registered trademark for two AR development tools named “RealityKit” and “Reality Composer”.

And Apple generally names its operating systems after the associated hardware platforms. Witness it:

Apple iPhone: iphoneOS (now abbreviated as iOS)

Apple Watch: watch OS

Apple iPad: ipadOS

MacApple: macOS

Apple TV: tvOS

And so:

Apple Reality: reality OS

All clues lead me to believe that Apple’s mixed reality platform will be called Apple Reality. This would be a reference to augmented, virtual and mixed “reality”, but also to the famous “reality distortion field” of the late founder of Apple, Steve Jobs.

I think that’s a pretty good guess, and so for the rest of this column, I’ll refer to Apple’s upcoming AR platform as “Apple Reality”.

Getting to grips with reality

Good reports, a few leaks, and a sprinkling of speculation suggest that Apple will ship (maybe next year, probably 2024) a headset that can be used for both AR and virtual reality (VR). While the hardware will support virtual reality, Apple will emphasize AR applications. Company statements, product launches, patents and acquisitions all indicate that Apple is obsessed with augmented reality and somewhat indifferent to virtual reality.

Apple’s first headset will do AR like an iPhone, but through stereoscopic glasses. The iPhone does AR by capturing real-time video through the camera and then overlaying virtual objects over that video. With Apple’s Reality Glasses, you’ll be able to see the world around you, but only on screens via video.

Apple is also reportedly working on a more advanced product — more like regular eyeglasses — that will overlay AR virtual objects over your natural field of vision.

I predicted a year and a half ago that Apple would use Memojis — Cartoon representation of Apple users, currently used for iMessage and other platforms — as avatars for virtual meetings.

Now Bloomberg reporter Mark Gurman (who has exceptionally good anonymous sources within Apple or its partners) supports my prediction claiming that Memojis “could be central” to the experience of using a future version of FaceTime in mixed reality.

What’s great about Memojis is that the avatar transmits the depicted user’s facial expressions, head tilts, gestures, and other non-verbal communications in real time, while speaking with the voice of the user. The ability to convey non-verbal communication without having to appear on video is more appealing and comfortable for many users than Zoom-like video calls, which can leave people feeling exposed, uncomfortable and drained. (It’s called Zoom fatigue.)

Apple’s virtual meeting technology was originally created by a DreamWorks Animation company called Spaces. He originally wanted to develop customer experience technology for theme parks using breakthrough technology that allowed multiple people to interact through avatars with the same virtual objects. The company then turned to VR conferencing technology. Apple acquired Spaces in August 2020.

Spaces methods and technologies incorporated into future FaceTime over Reality glasses would show a first-person view of participants in a circle or arranged around a virtual table, with all access to shared virtual resources like whiteboards, 3D models, floating graphics and other virtual objects.

Crucial to this experience: Each meeting participant will see other meeting participants as holograms in their own physical space, as opposed to a virtual meeting room. This theoretically reduces VR-related mental fatigue because the environment you see matches the environment you know is actually there. It also means you wouldn’t spill your coffee with a wave of your hand because you might see your mug sitting on the desk.

To talk to the person to your right, you turn your physical head to the right and make eye contact with that person’s Memoji. You would also hear their voice coming from your right. While Person A and Person B are looking into each other’s eyes, Person C would see these Memojis looking at each other.

While today’s Memojis capture non-verbal signals using a camera, the Reality headset would use both cameras and other sensors that could transmit those signals more finely, according to patents. Apple.

Using FaceTime for virtual meetings would be great, as early adopters could appear as 3D Memojis of themselves in the same call as late adopters, who would appear as themselves on video. Business meetings could involve people on iPhones using FaceTime for a Zoomlike experience and in the same meeting people using Apple Reality having the meetings in the “metaverse” (for lack of a better term). Apple Reality users would see other Reality users in the meeting as 3D avatars and old-school FaceTime users on a floating rectangle showing that participant’s video.

Gurman wrote that SharePlay would also be central, allowing business users to share presentations and documents in virtual meetings. Apple Reality could become a hugely compelling collaboration tool, thanks to 3D AR plus SharePlay.

According to Gurman, the Reality headset’s external cameras will capture hand movements, which would not only allow Memojis to display hand gestures in real time, but also allow users to type on virtual keyboards and write on shared virtual whiteboards.

Many of Apple’s recent additions to iOS may actually be designed to prepare the world and major developers for realityOS and the Reality platform – the U1 chip I told you about in 2019, ARKit, spatial audio with dynamic head tracking, etc.

Apple may believe, as I do, that AR will one day rise to replace the smartphone as the world’s most dominant and ubiquitous computing platform. And he wants to lead that future.

While Meta, the company formerly known as Facebook, sees users immersed in virtual reality all day, including for business meetings, Apple sees users join AR meetings and then leave Reality for come back to reality.

I believe Meta is wrong and Apple is right about what people will want and how much virtual world people can really tolerate. Zuckerberg’s vision is that people will live and work all day in the VR metaverse. In reality, I think business people will prefer Reality.

Copyright © 2022 IDG Communications, Inc.