Using motion trackers with Unreal Engine projects
Ingredients:
• A freshly installed Varjo Base 3.4 downloaded from https://account.varjo.com/downloads
• Source code of Unreal Engine 4.27
• An OpenXR plugin for Unreal Engine providing HTCX_vive_tracker_interaction, and a patch for Unreal Engine 4.27
• The usual development tools for C++, such as git and Visual Studio 2019
Build Unreal Engine:
Building Unreal Engine from source is not the most enjoyable thing to do, but it is a good skill to learn as this enables taking advantage of the availability of the Unreal Engine source code and applying changes to the engine to extend the engine capabilities. In this specific case, we need to extend the core OpenXR plugin of UE4.27 to enable custom action sets. It is unlikely there will be a 4.28 revision that includes those changes. However, it looks like this change is included in UE5 code base, so most likely UE5, when available, will include this already. But for now, UE5 has not yet been released.
There is a wiki page describing the process to build Unreal Engine on windows. The first thing is to get the source code, which requires signing up for a GitHub account if you do not have one already. You also need an Unreal Engine account, and then you need to associate your GitHub account with your Unreal Engine account as described here.
Now that access is granted, you need to grab the source code using git to clone the project and get the code corresponding to the latest release
git clone https://github.com/EpicGames/UnrealEngine.git git checkout –b release
You are going to need Microsoft Visual Studio to compile the engine, make sure to install the C++ compiler and Windows 10 SDK when installing Visual Studio, as the default settings do not include this. It was reported that Visual 2022 is much faster to compile Unreal Engine, however it is better to stick with 2019 for development purposes.
Before compiling the engine we want to apply the patch. The changes we want to apply are on GitHub in a pull request: using git can be done in a few commands:
git fetch origin pull/8621/head git cherry-pick 1d70d028
Now you are ready to compile the engine, which can be done in a few steps and a lot of patience.
Setup.bat (register unreal file types) GenerateProjectFiles.bat -2019
And then open the UE4.sln solution files with Visual Studio 2019. Right click the UE4 project in the solution tree and build it using the Development Editor Configuration. Note that not everything in the solution will build, just compile the UE4 project.
Create Unreal C++ project
Once the engine is compiled, select the UE4 Project in the Engine folder of the solution explorer, set it as the Startup Project, and run it. Create a new Game VR project, called VarjoTrackers for example. Be patient, Unreal is going to spend a considerable amount of time compiling shaders. Once the project is opened in the editor, you will be adding a C++ component to make it a C++ project.
Open the File menu and select ‘add C++ class. Select None (empty C++ class) and keep the default options. Be patient it is compiling some C++ Code, and eventually will open the project in visual studio. At this point let’s quit Visual Studio, quit Unreal Engine, and set up the trackers.
Alternatively, instead of creating a new project from scratch, you could start with the Varjo sample project, which contains all the additional features you may want to access from Varjo HMD, such as foveated rendering, MR, eye tracking. Please get the special dev branch as the main branch is not compatible with locally build Unreal Engines. Also, for the curious, this branch contains the source code of the Varjo Unreal plugin, which could be quite useful for debugging and extensions.
Setup the motion tracker
These instructions should work just fine with all SteamVR motion trackers, but in this example, we have used a tracker produced by HTC as an example. The HTC Vive tracker has two parts. The crown-shaped motion tracker, and the USB dongle. Each motion tracker must be paired with its own dongle. Note that Varjo HMD includes 2 dongles that are connected to the computer as part of all the signals transported via the USB-c cable(s). So, if you do not use controllers, you can pair up to two HTC trackers without using the additional USB dongles.
Once the USB dongle is plugged into the computer, it is necessary to pair the motion tracker. For this, we can launch SteamVR using the Pair SteamVR device feature in Varjo Base.
Once SteamVR has started, you can right-click on a controller and select ‘Pair Controller’. If SteamVR detects there are enough dongles available, it will open a selection window and you can click on the HTC Vive Tracker image and follow the steps. If you already have paired an HTC tracker, and want to add more, you will see a crown icon on the SteamVR interface and using right-click you can select Pair Tracker from the menu.
Once everything has been paired and visible from the lighthouses, you should see something like this on the SteamVR interface:
One more thing to finalize the setup, you must name each motion tracker. Right-click on the tracker icon, select Manage Trackers, and on the windows that opens, select Manage Vive Trackers. Then select a name for each paired tracker, make sure not to select DISABLED.
Add the HTC tracker plugins to the Unreal project
Create a Plugins folder inside the VarjoTrackers project folder. Download https://github.com/RemiArnaud/openxr_engine_plugins/archive/refs/heads/main.zip. Open the zip and go to openxr_engine_plugins-main\unreal_engine\ue4 and copy the OpenXRViveTracker folder into the Plugins folder created before.
Note that the original plugin is written by Valve and is hosted at https://github.com/ValveSoftware/openxr_engine_plugins, but as of the time of this writing there is an issue that needs to be merged, so in the meantime, we will be using a modified version from my private Github.
Optionally, if you did not start from the Varjo Unreal sample project, download the Varjo OpenXR plugin for Unreal 4.27 (https://downloads.varjo.com/software/plugins/unreal/1.1.0/Varjo_OpenXR_Plugin_for_Unreal_UE4.27_1.1.0.zip) and store it into the project. Inside the Plugins folder create a VarjoOpenXR folder and dump all the content of the Varjo_OpenXR_Plugin_for_Unreal_UE4.27_1.1.0.zip inside.
Right-click on the project file and select ‘Generate Project Files’. Then open the VarjoTrackers.sln file and build the solution. Make sure to open the project with the correct version of Visual Studio (2019). Also, if you did not use the engine you built to create that project, you need to use the ‘Switch Unreal Engine version…’ and select the engine you just built.
Right-click on the VarjoTrackers project in the solution tree and build. Once this is done you can exit Visual Studio and open the VarjoTrackers project in Unreal by double-clicking on VarjoTrackers.uproject. Once in the editor, click on ‘Manage Plugins’ and make sure at least the OpenXR and the OpenXR Vive Tracker plugins are enabled. Enable EyeTracking, HandTracking and Varjo OpenXR as well if you want access to those features.
Use HTC Motion trackers in the Unreal Project
Copy the VarjoTrackers\Plugins\OpenXRViveTracker\Content\RenderModels folder into the VarjoTrackers\Content\Geometry folder. In the engine content browser, you will now have access to several trackers’ geometry and materials.
In the Content -> VRTemplate -> Blueprints, create a new blueprint, call it HTC tracker for example. Edit the blueprint. In components, add the ViveTracker component and use it as the main component of this blueprint. Add a Static Mesh component. Drag the Tracker model and material into the right slots in the Static Mesh component.
Now let’s edit the event graph to update the position of this mesh with the position of the physical tracker. Make sure to select the same name you had used when setting up the HTC tracker previously, here I am using ‘Waist’.
Profit
Make sure controllers and tracker are on, push VR preview play in editor, and look around.
Admittedly, this is not the easiest recipe to setup, not yet plug and play, but considering that the OpenXR extension specification has been released only a couple of months ago (https://github.com/KhronosGroup/OpenXR-Docs/blob/master/CHANGELOG.Docs.md) and was implemented and released in Varjo Open Runtime recently, it is impressive it can already be used in Unreal Engine, without waiting for a new release of the engine.
The extension was created by HTC, implemented by Varjo and Valve independently in the respective run-time after its publication in the OpenXR specification. Valve also provided an extension to Unreal Engine as well as a fix to Unreal Engine to expose the feature at the engine level. And this can now be used by Varjo customers. It is a remarkable collaboration across several companies contributing to OpenXR.