XR-based training devices, those that use head-mounted displays (HMDs) versus traditional dome and flat-panel displays, are becoming more common as XR technologies advance to support more demanding requirements. XR solutions can be broken down into two primary categories: virtual reality solutions, where the trainee is fully immersed in a computer-generated environment, and mixed-reality solutions, where trainees experience an immersive environment consisting of computer generated content and simultaneous access to the physical world.
What are the use cases and benefits of virtual and mixed reality for pilot training? Each option provides features and benefits with associated pros and cons that can impact which option to choose for training deployment.
In pilot training, the adoption of XR-based solutions is often held back by obstacles, such as insufficient resolution and field of view, excessive latency, and ergonomic issues, but when these are addressed with mature headset technology, deployment options abound.
Enterprise-class HMDs are now supporting pilot training programs. For example, the US Navy is reimagining their flight training curriculum to leverage virtual and mixed reality devices to support basic pilot training. The US Airforce is looking to mixed-reality solutions to train for B-52 aerial refueling. In Europe, civilian helicopter pilots can log flight training hours on VR-based training systems.
So what are the benefits of using these XR-based technologies?
There are many benefits in adopting virtual and mixed reality training technologies. They include:
When training organizations successfully implement XR-solutions, they will train more pilots faster, provide more flexibility in the training at lower costs than ever before.
XR-based training solutions are used today to support a wide variety of core commercial and military flight training tasks including the following:
In virtual reality training, all visual content seen by the trainee is entirely replaced with computer-generated 3D scenes representing the virtual environment needed to support the training scenario.
The trainee is completely immersed in the virtual training environment and is, for the most part, cut off from the real world. This enables full freedom to create and customize different training scenarios as needed and, depending on the device, support interaction with the virtual content and objects using hands, controllers or eyes.
Virtual reality experiences can vary from a full photogrammetric capture of the real world to a computer-generated scene that is 3D modeled and animated in a gaming engine. Some VR headsets are tethered to a PC which is typically needed for rich, realistic content that requires the most powerful graphics. Alternatively, some VR headsets are untethered, with content being created by on-board graphics and computer processors. Where this allows users to move around freely without cables, the quality and resolution of the content is dramatically limited and training scenarios are limited by battery life.
While VR-based solutions can provide an effective, fully immersive experience, they can suffer limitations in a user’s ability to interact with real-world objects. Collaboration with real people such as co-pilots or instructors is limited since they cannot be seen while fully immersed. Communication and interaction with the physical world can be difficult when compared to traditional training or mixed reality.
While VR headsets have limitations not present in traditional devices or with mixed reality headsets, VR-based training solutions can be very effective for training certain types of pilot training tasks, especially those that do not relay on the development of muscle memory or complex interactions with physical devices. Where these requirements exist, mixed-reality training solutions may be a better fit.
Mixed reality is all about digitally merging virtual and real-world content inside a headset – enabling a trainee to see and interact with select real-world people and objects while fully immersed in a virtual environment. Ideally, in a mixed reality solution, it’s impossible to discern which elements are real and which are computer generated. Virtual objects appear as a natural part of the real world and will be occluded behind real objects where appropriate. Physical objects around the user can be seen and touched as needed and the real objects can also influence the shadows and lighting of virtual elements.
With mixed reality, a user is not separated from the real-world environment so they can see their hands and body and interact with colleagues and real-world objects, such as physical control sticks and instrument panels. Mixed reality-based devices enable trainees to immediately grasp the point of each lesson and experience the world in proper context as they would in a real aircraft, and they are able to see cause and effect relationships. Most significant is the ability of a trainee to develop true muscle memory, since they can interact with physical hardware as they do in the real world.
For mixed reality solutions to achieve full immersion and suspend disbelief, they must be convincing – blending real and virtual content to the point that it is difficult or impossible to tell where reality ends and the virtual world begins. Mixed reality typically comes with a higher price tag compared to VR solutions due to hardware and development costs, but added fidelity and support for tactile feedback is gained.
The processing power required to collect and process the video and to display these experiences means that mixed reality headsets must currently be tethered to a computer.