The world’s first true-to-life mixed reality demo by Varjo and Volvo

Mixed reality means blending virtual content with the real world. So far mixed reality has been accomplished with optical see-through, where the user sees digital objects augmented on top of reality through a pair of glasses. This is fine for portraying infographics or playing games, but for realistic scenes, it offers little value. Optical see-through devices can’t display black or opaque content on top of the real world. Everything appears hazy and holographic.

We at Varjo wanted to get rid of this limitation and be able to render photorealistic, opaque content – where it is impossible to distinguish between what is real and what is virtual. Our mission was to make true-to-life mixed reality possible with video pass-through. Video pass-through means using cameras to digitize the world in real time, and then showing the combined result of real mixed with virtual to the user.

Before we could achieve this, we first needed a VR headset that can display the real world in human-eye resolution. That is why we released our first human-eye resolution headset, targeted for professional users, to the market in February 2019.

And at Augmented World Expo 2019 in May, we showed a glimpse of the magic that can be accomplished with video pass-through. We publicly demonstrated our new mixed reality headset for the first time with a joint demo with Volvo, made with Unity.

With Varjo’s mixed reality, you can blend virtual content seamlessly with reality with extremely low latency and integrated eye-tracking, in superior resolution – being the first product to ever do so.

Here’s how the world’s first photorealistic mixed reality demo was created.

Varjo began working on a video-pass through mixed reality headset already in early 2018. The collaboration between Varjo and Volvo also started in spring 2018, as Volvo outlined the need for an XR headset that would allow them to tests various elements of future cars – such as heads up displays, new materials and UI for infotainment systems – inside a real car while driving on a real test track. The high requirements on readability and low-latency needed to drive a car on a test-track pushed Varjo to succeed in product development.

Given how well Unity already worked for Varjo’s VR, it was a natural choice to try out how the virtual objects would look like in mixed reality. The fact that Unity is easy to integrate and extend with C++ libraries such as our own Varjo plugin made it possible for us to extend our plugin to support Mixed Reality. By simply defining the empty background in a VR scene to be replaced by the video-pass-through signal, we were quickly able to see virtual objects in a real environment.

The close collaboration and fast iterations were made possible by Unity’s ease of use, as our team was developing and improving pass-through simultaneously while working hand in hand with our customer. A year later, the first public demonstration of Varjo’s mixed reality brought to life the capabilities of our technology, combined with Volvo’s superior models and photorealistic Unity graphics.

The demo showcased at AWE illustrates for the power of video pass-through mixed reality as opposed to optical see-through. With Varjo, you can see true-to-life virtual content blending with reality in a full field of view. You can also switch seamlessly from XR to full VR. In this demo, you have the following steps:

1. Experience Real Reality.

You see the real world around you through the Varjo headset. The real world is streamed with <10ms latency via the high-res cameras in the front plate. You see the world in a full field of view, high resolution with a 90Hz framerate, which gives a sensation of not wearing any headset at all, i.e. seeing the real world with your own eyes. You can walk around and explore the real world freely.

2. Enter Photorealistic Mixed Reality.

A beautiful Volvo XC60 builds up in front of you. It first appears as a stylized transparent blue wireframe. The virtual car is anchored to the real floor in the room around you and oriented so that the chair in the booth is aligned with the driver seat of the virtual car. The viewer can take a seat in the real chair and is still able to see the real surroundings through the wireframe.

The car now turns into a solid model, and the surfaces goes from transparent to opaque. The virtual car casts shadows on the floor of the real world, and looking on the car’s surface, it is possible to see that the real world is reflected in the car’s surfaces. The reflections come from an HDR cube map that was taken during setup on the exact spot of the car. The same cube map is also used for ambient lighting.

This is the first time the viewer sees opaque mixed reality, and the effect is stunning. You can still see the real world and your colleagues through windscreens.

How it was accomplished: The car model was provided to Varjo by Volvo. Because the resolution of the headset and the car was so high, we needed to make as much pre-processing as possible. The lighting was baked in the DCC to textures and multiplied in custom shaders. The baked textures only dealt with occlusion and the shading is still affected by the skybox.

Mattias Wilkenmalm from Volvo did the asset creation and wrote custom car paint shaders that deliver superior results, and we simply modified them to get the look and transitions we needed. The final model is around 7 million polygons and has around 150 4k textures.

3. Switch Seamlessly into Virtual Reality – and Back.

The viewer is then asked to step outside the car, and we transition to Venice. The last pieces of the real world around the viewer are now disappearing in a unique transition as the reality transforms into a virtual scene of Venice, where the car is parked in one of the alleys. The reflections in the car are now those of Venice, and the shadows of the car are now landing on the streets of Venice.

After a while, we transition back from virtual to the real world. The user can now go around the virtual car and see all the details and reflections. This shows that Varjo offers ability to still interact with others and select only the parts you want to virtualize.

How it was accomplished:

To make the transitions visually pleasing, Volvo’s Timotei Ghiurau used world space 3D noise with alpha cutouts to bring in the car and the environment. This is fast to do in the fragment and it looks very cool. It was a perfect combination when dealing with tight deadline. Noise functions can be fetched for example from Unity’s Kejiro’s repository.

To get the smooth transition for the car reflections, the Venice environment was added to a separate layer so that the real-time reflection probe was only rendering minimal amount of geometry. The reflection probe was rendered at 30 frames per second while the scene renders at a much higher frame rate. Having the transition visible in the reflection probe added much more immersion to the scene.

The fact that Varjo is a first-of-a-kind headset to offer the ability to seamlessly switch from Real Reality into Mixed Reality onwards into full VR and back to Real Reality again makes for a very impressive demo. It is a Matrix-like moment to see the surrounding reality disappear and replaced by a virtual scene – and then traveling back. This is something that no other device has been able to do previously.

Learn more