Introducing Integrated Hand Tracking for Varjo XR-4 Series
We’re excited to announce that native Hand Tracking support is now available for all Varjo XR-4 series headsets as part of the Varjo Base 4.10 software release. This powerful new feature is included free of charge and is automatically enabled through the Varjo Base settings.
⚠️ Please note: Hand Tracking is being introduced as a Beta feature. While it already enables exciting new possibilities, there are some known limitations—including gesture reliability and latency. Your feedback is incredibly valuable and will help us refine the experience in future updates.
Seamless Integration with OpenXR Applications
The integrated hand tracking works across both fixed-focus (FF) and auto-focus (FE) variants of the XR-4 headset. Users can now enjoy immersive interaction in any application that supports hand tracking via OpenXR APIs. Specifically, Varjo’s implementation includes support for the XR_EXT_hand_tracking and XR_EXT_hand_interaction OpenXR extensions.
-
XR_EXT_hand_tracking enables full articulation with 26 hand landmarks, providing detailed skeletal tracking.
-
XR_EXT_hand_interaction supports four intuitive gestures: aiming, pinching, grasping, and poking. For the best user experience, we recommend using applications that leverage this extension.
For those who already use the Ultraleap Leap Motion Controller 2, or prefer to continue using it, it remains fully compatible. In Automatic mode, the system will prioritize the Ultraleap device if it’s connected.
Camera-Based Tracking – No Extra Hardware Needed
This new feature is made possible by leveraging the video-see-through (VST) camera streams of the XR-4 headset. This means no additional hardware is required—your hands are tracked directly within the visible area of the headset.
Hand tracking is supported in both 90 Hz and 75 Hz VST modes, although the tracking frame rate is slightly reduced in the initial release. (For users of the XR-4 FE, note that the 75 Hz mode offers a slightly larger vertical field of view.)
Optimized for Real-World Conditions
The system is optimized for typical office lighting conditions (200–500 lux). To achieve optimal performance, ensure your environment is well lit, allowing the VST cameras to capture high-quality frames.
Hand tracking works independently of the headset’s positional tracking method—whether using Varjo’s inside-out tracking (IoT) or SteamVR tracking.
Powered by Machine Learning
At the core of the feature is a machine learning model trained in-house at Varjo. The system detects and interprets hands in 3D using a combination of:
-
Stereo vision from the headset’s VST cameras
-
Depth data from the built-in time-of-flight sensor
We’ve ensured robust and inclusive performance by training the model on data from individuals across a wide range of ages, skin tones, hand sizes, and genders.
Inference is performed on the CPU, so users with high-performance processors will experience the smoothest tracking. The GPU is not utilized for hand tracking in order to prioritize user’s applications.
What’s Next?
This integrated hand tracking is developed entirely in-house at Varjo, and this release marks just the beginning. Expect continuous enhancements and new capabilities in upcoming Varjo Base updates.
We can’t wait to see what you build with it—and we’re listening closely to your feedback.