Blend real and virtual seamlessly with new mixed reality features

Imagine sitting in a replica of a plane cockpit and being able to look outside and see yourself flying over Tokyo, New York or Calcutta. Or imagine sitting in a car and being able to replace the interior with designs that are not yet reality, changing them with a click of a button and interacting with your colleagues while doing it. Or holding a 3D printed object in your hand and ‘dressing’ it instantly to look like a material-finished product.

Today, we are excited to roll out a software update that makes it extremely easy for Varjo’s mixed reality users to blend virtual content with the surrounding reality. Through an industry-standard technique known as ‘green-screening’ and used in broadcasting and film, Varjo is the first company to deliver chroma keying in real-time for mixed reality devices. And using visual markers, professional users can instantly anchor any virtual objects to the real world.

Mixed reality applications offer huge cost savings for companies that traditionally train their staff in expensive simulators or build physical prototypes. With Varjo’s pass-through mixed reality device, enterprises are able to perform simulations or design studies with photorealistic virtual replicas. With Varjo’s mixed reality, we’ve solved the ‘hard AR’ problem, which exists on optical see-through devices like HoloLens or Magic Leap that always portray virtual elements as hazy and holographic and can’t display shadows or opaque content.

The two key technical problems are masking and occlusion. Accurate masking refers to rendering the VR content on top of the real world exactly where it’s intended to appear. Occlusion means that when your hands are on top of the VR content, they are actually appearing over the VR content instead of behind it.

One example of the importance of accurate masking is mixed reality flight simulators, where a trainee is sitting in a physical cockpit but expects to see VR scenery outside the windows. This can be achieved by creating an accurately positioned virtual model of the cockpit and using it to mask out the virtual content. The Varjo mixed reality trainer developed by Bohemia Interactive Simulations is an elegant example of this.

Estimating real-world depth is one solution to the occlusion problem. Varjo’s software can estimate the real-world distance of each pixel of the Varjo video pass-through image in real-time by using depth-from-stereo and active infrared imaging. This means we can automatically render pass-through reality over VR content when the virtual content is further away from the user. However, real-time depth estimation also has some challenges, such as pixelation of silhouettes, incorrect depth estimates on low-texture surfaces, and achieving correct depth order when virtual and real surfaces overlap.

Real-time chroma keying solves these challenges.

For designers and engineers, chroma keying for mixed reality makes conducting design reviews of future products possible in real scale. Automotive designers can place green material in areas where virtual content should be rendered. A virtual display or virtual controls are only rendered over the marked areas. Hovering your hand over – or even touching – the physical surface will keep your hand rendered over the virtual content, just as you would intuitively expect.

Chroma keying can also be tremendously useful in rapid product design visualizations where a 3D printed object can be dressed in various designs with the click of a button, enabling intuitive interactions with virtual products. It’s even possible to install touch-enabled screens displaying green or have green 3D printed controls for haptic feedback. In such cases, virtual designs can be prototyped extremely rapidly, saving time, money and effort.