Talk to sales
person standing in front of green screen wearing Varjo headset

Blend real and virtual seamlessly with new mixed reality features

Mixing real and virtual content into a seamless experience is one of the key challenges for developers. For Varjo users, reprogramming reality just got effortless with our new real-time chroma key and marker tracking features.

Watch how Varjo delivers absolute immersion for professional mixed reality.

A whole new range of benefits for industrial mixed reality

Authors:

Ville Timonen
Mixed Reality Lead at Varjo

Saku Tiainen
Mixed Reality Developer at Varjo

 

WATCH OUR IN-DEPTH webinar

 

Imagine sitting in a replica of a plane cockpit and being able to look outside and see yourself flying over Tokyo, New York or Calcutta. Or imagine sitting in a car and being able to replace the interior with designs that are not yet reality, changing them with a click of a button and interacting with your colleagues while doing it. Or holding a 3D printed object in your hand and ‘dressing’ it instantly to look like a material-finished product.

Today, we are excited to roll out a software update that makes it extremely easy for Varjo’s mixed reality users to blend virtual content with the surrounding reality. Through an industry-standard technique known as ‘green-screening’ and used in broadcasting and film, Varjo is the first company to deliver chroma keying in real-time for mixed reality devices. And using visual markers, professional users can instantly anchor any virtual objects to the real world.

 

Watch the video to see how real-time chroma key for mixed reality enables blending virtual and real.

Mixed reality applications offer huge cost savings for companies that traditionally train their staff in expensive simulators or build physical prototypes. With Varjo’s pass-through mixed reality device, enterprises are able to perform simulations or design studies with photorealistic virtual replicas. With Varjo’s mixed reality, we’ve solved the ‘hard AR’ problem, which exists on optical see-through devices like HoloLens or Magic Leap that always portray virtual elements as hazy and holographic and can’t display shadows or opaque content.

 

Why chroma keying makes mixing virtual and real effortless

For mixed reality applications to be realistic, the virtual content needs to blend accurately with reality. Previously, it has taken a lot of development to merge virtual content and pass-through video stream realities into a seamless experience.

Step into the new reality. Watch how you can use Google Earth with Varjo and real-time chroma keying.

The two key technical problems are masking and occlusion. Accurate masking refers to rendering the VR content on top of the real world exactly where it’s intended to appear. Occlusion means that when your hands are on top of the VR content, they are actually appearing over the VR content instead of behind it.

One example of the importance of accurate masking is mixed reality flight simulators, where a trainee is sitting in a physical cockpit but expects to see VR scenery outside the windows. This can be achieved by creating an accurately positioned virtual model of the cockpit and using it to mask out the virtual content. The Varjo mixed reality trainer developed by Bohemia Interactive Simulations is an elegant example of this.

Estimating real-world depth is one solution to the occlusion problem. Varjo’s software can estimate the real-world distance of each pixel of the Varjo video pass-through image in real-time by using depth-from-stereo and active infrared imaging. This means we can automatically render pass-through reality over VR content when the virtual content is further away from the user. However, real-time depth estimation also has some challenges, such as pixelation of silhouettes, incorrect depth estimates on low-texture surfaces, and achieving correct depth order when virtual and real surfaces overlap.

Real-time chroma keying solves these challenges.

 

 

Chroma key enables absolute immersion in mixed reality simulations

Chroma keying is an industry-standard technique where parts of the real world are marked with a coloured canvas or paint, usually using green or blue. In the post-process, these areas are then detected and replaced with synthetic graphics. Varjo demonstrates this technique for the first time in the world in real-time mixed reality applications to mask virtual reality content.

When we first implemented chroma keying, our XR team had endless fun literally turning a piece of green cardboard into a window into another reality. But jaw-dropping demos aside, chroma keying makes building professional mixed reality scenarios extremely cost-effective, quick and easy. In flight simulators, you can place a green canvas around the physical cockpit. Only virtual content will appear over the areas where the green color is visible. If you lift your hand over the window where VR content is displayed, the hand will naturally occlude the VR environment with pixel-perfect precision. Real-time chroma keying also enables multi-user training scenarios where it is possible to interact with co-pilots or other colleagues, as even complex and dynamic shapes occlude accurately.

“With chroma key, Varjo took an industry-standard technique and turned it into a useful new feature for dynamic mixed reality simulations,” said Bob Vaughn, Product Manager at FlightSafety International, a worldwide leader in aviation training. “We look forward to further exploring the feature applied to a variety of simulation opportunities. We highly value our collaborative relationship with Varjo, and are excited to continue to push the boundaries of mixed reality.”

See how pilots can train in mixed reality with real-time chroma keying using DCS.

Design the products of tomorrow instantly

Using real-time chroma keying with Autodesk VRED.

For designers and engineers, chroma keying for mixed reality makes conducting design reviews of future products possible in real scale. Automotive designers can place green material in areas where virtual content should be rendered. A virtual display or virtual controls are only rendered over the marked areas. Hovering your hand over – or even touching – the physical surface will keep your hand rendered over the virtual content, just as you would intuitively expect.

Chroma keying can also be tremendously useful in rapid product design visualizations where a 3D printed object can be dressed in various designs with the click of a button, enabling intuitive interactions with virtual products. It’s even possible to install touch-enabled screens displaying green or have green 3D printed controls for haptic feedback. In such cases, virtual designs can be prototyped extremely rapidly, saving time, money and effort.

 

 

Track objects accurately using visual markers

Our latest software release also introduces visual marker-based object tracking. Using visual markers and Varjo, professionals can anchor virtual objects exactly where they want them in their surroundings. This allows the exact positioning of virtual displays, controls or any objects to the reality around you. Varjo markers support individual tracking of up to 1,000 objects, without the need to use active controllers or tracking pucks.

Get started with the new Varjo features

With our latest software update, merging synthetic and analogue realities becomes easier than ever. It becomes almost impossible to tell where the real world ends and virtual reality begins.

Both chroma keying and marker tracking are now available in early access to all users of Varjo’s mixed reality headsets. See our documentation and start blending real and virtual seamlessly.

Download the new Varjo Base

See developer docs

Read more about Varjo's mixed reality:

FlightSafety International: Varjo Is a Breakthrough Technology for Aviation Training

Testimonial

Feb 17, 2020

Video pass-through mixed reality

Video Pass-Through XR Changes Reality As You Know it

Blog

Jan 2, 2020

Introducing Varjo Workspace, A New Way of Using Computers

Blog

Dec 16, 2019