At Varjo, our mission is to deliver the world's most realistic training experiences. As part of this mission, lately we have been exploring how A.I. models can be leveraged to push the boundaries of XR simulations. The introduction of People Occlusion (Beta) in Varjo Base 4.16 is another significant step toward truly seamless mixed reality and a showcase of how artificial intelligence can benefit immersive technology.
In multi-crew simulations such as cockpit training, ground vehicle operation, or coordinated mission rehearsal, trainees rarely work alone. A co-pilot is adjusting controls to your right. An instructor is monitoring from the left seat. Another crew member operates a station behind you.
In fully VR setups, these other participants are usually represented with an avatar, but with high resolution passthrough cameras, why not bring the actual person into the virtual world?
Before the introduction of the new People Occlusion mode, there were workarounds to achieve this, but they are not always ideal. Standard depth occlusion rendered everything within range into the passthrough feed, meaning parts of the physical environment bled into the virtual world. And chroma keying the physical environment often requires studio-like conditions with green screens and controlled lighting. Neither approach fully matched the need: a simple way to maintain awareness of the people around you while remaining fully immersed in the virtual environment.
People Occlusion (Beta) is a new mode, available in Varjo Base 4.16, that uses real-time A.I. based object recognition to detect people in the passthrough stream and make them visible within the virtual environment - even in a fully VR scene.
Rather than rendering everything within a depth range into passthrough, the system identifies human figures specifically and composites them into the virtual layer. Your copilot becomes visible. Your instructor appears beside you. The virtual environment remains intact around them.
Requirements: People Occlusion (Beta) is available on XR-4 Series headsets with a Varjo Base Pro license. It requires a high-end GPU for best segmentation performance. Good lighting conditions also improve segmentation accuracy.
You can enable the feature in one of three ways;
A note on beta status:
People Occlusion is shipping as a beta feature, and it's worth being clear about what that means in practice.
The current release does not yet support depth sorting. Real-world people always appear on top of virtual content under the segmentation mask, regardless of their actual position in space. Depth-based occlusion for people is planned for a future release. Additionally, segmentation quality remains an area of active improvement. The feature is not compatible with ChromaKey in Varjo Base 4.16, and some VR applications may experience compatibility limitations during the beta period.
For most multi-crew training setups, these limitations won't block meaningful use of the feature, but they are worth accounting for during integration planning.
The clearest use case is multi-crew simulation. A pilot and co-pilot can now share a fully virtual cockpit environment while remaining aware of each other's physical presence and actions. Instructors can monitor and interact with trainees without either party needing to break immersion. Crew stations that require coordination between real people can be trained at higher fidelity than was previously possible in a purely virtual environment.
Beyond the cockpit, the same capability applies to any training context where real people and virtual environments need to coexist: coordinated vehicle simulation, team-based mission rehearsal, or collaborative design reviews. The through-line is the same in every case: the value of the virtual environment shouldn't come at the cost of awareness of the people around you.
Varjo Base 4.16 reorganizes the occlusion settings into a single, unified section designed to give developers and operators a coherent set of tools for deciding exactly what comes through from the real world:
The right mode depends entirely on what you need to bring through: an object on a workbench, your hands on a physical control, or the crew member in the seat beside you.
People Occlusion (Beta) is available now for XR-4 Series headsets with the Varjo Base 4.16 Release Candidate, released June 9, 2026. You can enable it directly in the Mixed Reality settings or via the Varjo CLI.
If you're working on a multi-crew simulation program and want to explore how People occlusion fits your specific environment and hardware setup, the Varjo Alpha team is available to help. Varjo Alpha specialists can assist with calibration, configuration, and integration planning to ensure the feature works exactly the way your scenario requires.
The ability to see your crew while inside a virtual world changes what multi-crew training can look like. We’re excited to see how teams will use this to push multi-crew training forward—and we're just getting started.
-Fabian