Mixed reality driving: Real hands on a virtual wheel
Not being able to see your own hands in a virtual reality racing simulation can break the immersion, sometimes stiff digital representations of your arms just don’t quite cut it. Using passthrough cameras to capture and display real-time footage of your hands solves this issue. Keeping the steering wheel itself virtual though allows for drivers to experience a visual range of different wheels, with working indications. This project aimed to tackle the challenge of seamlessly mixing real footage of a driver’s hands with a virtual steering wheel.
So, the idea for a green steering wheel came as a solution to show well-defined hands on top of a fully virtual wheel that you can touch for real. In the end, we used a combination of the Varjo Lab Tools application, chroma key from Varjo Base, and a can of spray paint to achieve our goal.
The Varjo Lab Tools application is a powerful set of tools that allows intuitive fusing of the real world with a virtual one. One of the features of the tools is video depth testing which allows you to show your hands in the tool on top of any application. Shown below was our first attempt at a mixed reality driving setup, but when using this method, the depth test function also shows peripherals like wheels and joysticks with the hand. Our goal, in this case, was to only see the hands merged onto a digital wheel. As awesome as the depth test function is, it was therefore not ideal for this mission.
So, we went back to the drawing board and thought of a new possible solution. The idea of painting the whole wheel green and using chroma was brought up, so I passed the wheel to our prototype lab where our handy prototype engineer Toni Grönmark took the task of creating a green wheel. The spray paint we used stuck well on the wheel and after letting it dry for a few days, we had a nice even dry green wheel that did not even leave any color on the hands when touched. When combined with a good surrounding light setup you could completely chroma key out the wheel.
The green backplate is removable from the wheel with a single screw and a magnet for easy transportation.
Once we had the green wheel, we created a virtual restricted mask on Lab Tools that covers the green steering wheel and backplate. Using this type of masking, we will see the real world within the mask and virtual world outside of the mask. Then we combined this mask with chroma key by applying chroma key effect from Varjo Base to video feed. Chroma key would let us show a virtual world within the created mask but let us still see our hands.
Varjo Lab Tools provides simple 3D model creation from scratch.
Finally, we had a mask and wheel working like we wanted. The last step was to launch our racing simulator. We used Assetto Corsa with Lotus Exos 125 Formula car. The car had a wheel that was approximately the same size as our wheel controller so we could use the position reset function in the game to match the digital wheel with the real one.
So now we had a perfect setup at the office for racing! Not so fast. There are still things we can improve. If you look at the video carefully you can see that the digital and real wheel do not match 1:1 even after careful position resetting in the game. This is because they are physically different shapes and sizes.
Rune Huse Karldstad from Varjo has modified his personal steering wheel to match the digital one. A setup like that, painted green, would provide an even better result. So, in the future we need to send our green wheel back to our prototype lab to Toni for further modifications!
Off the shelf gaming racing wheels provide a good base for modifications.
If you would like to learn more about Varjo Lab Tools, and the fun you can get up to with real-time masking, depth testing and more, check out this webinar where I live demo all the features.