The XR community is buzzing about passthrough camera access, as enthusiasts eagerly await Google’s next move with Android XR. We’ve seen what Meta, Apple, and Pico have up their sleeves, but the big question is: what will Google do? After chatting with the company, I can reveal they’ll offer a solution akin to what’s available on their smartphones. Stick around to find out more!
## The Camera Access Dilemma
Before diving in, let’s clarify what this excitement is all about. Recent standalone VR headsets are essentially MR headsets, providing users with an RGB passthrough view of the world via front cameras. This technology enables intriguing mixed reality applications such as Cubism, Starship Home, and Pencil.
These cameras capture frames that the operating system uses to render the passthrough view. Developers are keen to access these frames too, aiming to enhance user experiences with AI and computer vision algorithms. As I’ve expressed before, the key to unlocking the full potential of mixed reality lies in camera access because it empowers applications to understand their surrounding context seamlessly. For example, by employing a clever trick with Quest, I created an AI+MR app that assists in interior designing. This wouldn’t have been feasible otherwise.
This prospect sounds amazing, but there’s a real concern about privacy. A rogue developer with camera access could discreetly capture and analyze images from users’ environments, potentially extracting personal information like IDs or bank card details. The risk extends to capturing faces or bodies for nefarious purposes.
Balancing user privacy against the immense potential of mixed reality is a critical challenge.
## The Approach of XR Companies
Initially, things were relaxed, with full camera access available. If you’ve been following my journey, you might recall my team’s experiments in 2019 with Vive Focus at NTW, where we played with diminished reality, Aruco marker tracking, and more.
But as mixed reality gained traction, companies like Meta, Pico, HTC, and Apple tightened their security, blocking developers from accessing camera frames due to privacy worries.
This became the norm until the XR developer community made it clear that camera access was essential, prompting manufacturers to reconsider. Champions like Cix Liv, Michael Gschwandtner, and I pushed hard for user-consented camera access, advocating for transparency. Why the apprehension over XR devices when smartphones allow camera access with just a permission request?
This advocacy led to XR firms warming up to the idea. Meta pledged a “Passthrough API,” expected to debut this year. So, what’s Google’s plan with Android XR?
## Android XR Treats Headsets Like Phones
Android, the dominant OS for smartphones, allows apps to access camera streams by merely requesting user permission. Once granted, developers can specify camera IDs to access captured frames.
Google aims for Android XR’s integration with Android apps, offering similar functionality. Though rumors have swirled, no official word had been given—until now. After emailing a Google spokesperson, I got the scoop on camera access in Android XR:
Developers can leverage existing camera frames, with user permission, just like Android apps.
Our detailed developer article covers additional app request permissions:
A developer can access the main world-facing camera stream using camera_id=0, akin to the “rear camera” on Android. Likewise, camera_id=1 fetches the selfie-camera stream, comparable to the Android “front camera.” These streams are accessible through standard Android Camera APIs, namely Camera2 and CameraX.
The world-facing camera, similar to a smartphone’s back camera access, requires user camera permission.
For the selfie-camera, apps receive an avatar-like stream, fueled by user movement data from OpenXR APIs, based on tracking from inward-facing cameras monitoring user gestures and facial expressions.
Android developers can utilize familiar classes like CameraX to manage camera streams on Android XR devices. Since these classes allow frame grabbing and ML analysis, this should extend to headsets and glasses, which is wonderful news!
While developers can access the front camera, the rear stream is essentially a reconstructed avatar, mirroring Apple’s approach with the Vision Pro. This decision maintains Android XR’s consistency with Android on phones: access the “rear camera” to see the user’s view and the “selfie camera” for the user’s avatar.
Google ensures seamless operation for Android apps on Android XR, even those accessing cameras. This is commendably smart. Consistency in permission requests between phones and headsets/glasses is vital, and I wholeheartedly support it.
Some of you might wonder about accessing all raw camera streams. Unfortunately, there’s no encouraging answer yet:
Currently, apps cannot access non-standard (e.g., forward-facing camera, reconstructed inward camera) sensor data.
The response indicates non-standard streams will remain inaccessible, though we can hope for future releases, particularly for enterprise users.
Another query: how about Unity development? Both Camera2 and CameraX are native Android classes, so what’s our approach, as Unity developers? If Android XR parallels Android, the WebcamTexture class might work for frame grabbing. If not, a JNI workaround could yield a Unity-to-native CameraX library, granting access to the community.
## A Note on Android XR
Remember, Android XR is still in preview and no devices running it have officially launched. Therefore, the details shared here could change before its full release. It’s unlikely, but worth keeping in mind.
## Unlocking Camera Access
With Google and Meta set to allow camera access, others are sure to follow. 2025 seems primed for unleashing fresh, innovative avenues in mixed reality. I’m eager to see where this potential takes the developer community!
(Header image inspired by a Samsung image)
Disclaimer: This blog includes advertisements and affiliate links for maintenance. Click an affiliate link, and I’ll earn a small commission on your purchase. Full disclosure is available here.
