How the USAF Trains Pilots of the Future with XR: Your Questions Answered

Answers provided by
John Burwell, Global Lead, Simulation & Training, Varjo
TJ Moser, Aviation Business Unit Lead, Vertex Solutions
Alex Horn, Major, US Air Force Defense Innovation Unit

Q: Do you have any guidance as to specifications/requirements for HMDs to minize the effects of VR induced sickness? E.g. screen refresh rates, GPU specs etc. Also HMD FoV for realistic environment presentation.

A: Specifications to consider:

• Field of view requirement is application specific. Ideally, at least 120×90. 
• Automatic adjustments of the inter-pupillary distance for each user at each session to ensure proper set-up. 
• Human eye resolution (70+ pixels/degree) to reduce eye strain and ensure ability to see and read visible content. 
• Reduce overall system latency to less than 50msec from physical input to visibility of first pixel change in the display. 
• Minimum system update and refresh rates at least 60 Hz or more to support reduced transport delay. If possible, keep the refresh rates at 90FPS.   60 FPS takes less processing, but anything less than 90 FPS seems to contribute to cybersickness.
• GPU specs need to be sufficient to support the run-time requirements listed above. What this means practically is to use the fastest available NVIDIA GPUS available.  
• HMD weight needs to be minimized, especially to support long duration exercises. Read more about comfort.

Q: Flying around the flagpole is great, but XR encounters significant barriers when one tries to use this technology in a secure environment for advanced combat training. Try not to punt: will DIU be a point of contact for industry to not duplicate effort when breaking these cyber barriers. These headset are a known quantity–USAF should have a pre-packaged risk management guidance per COTS hardware.

A: Vertex is working with DIU to get these devices classified.  Initially air gapped, but hopefully fully IL-6 compliant. This will allow for threats and weapons critical to combat flight training. There are plenty of hurdles.  Once solved, an “ATO as a service” would be a great out of the box solution for other flight simulators.   

Q: I’m curious as to what efforts have been taken to get ITDs some kind of FAA certification level?    

A: Vertex is currently pursuing a FTD Level 7 (rotary) cert for our Multi-place Mixed Reality device.   

Q: What game engine is used for building these experiences?

A: Lockheed Martin’s Prepar3D for PTN and PTT (T-6/T38/T-45). Metrea’s NOR for upcoming FBF (T-38/F-16) projects. Aechelon’s PC-Nova for TH-1H.

A: The trend in COTS hardware is to get rid of wires. The USAF seems terrified of anything wireless. What is the USAF doing to mitigate the risk? We shouldn’t be hobbling the advantages of what these solutions bring. RF paint is a thing!

Q: For some applications like maintenance training where movement around physical objects in important, the USAF needs wireless HMDs vs. those that are tethered to PCs. In these applications, wireless, all-in-one commercial devices are often a good fit – as long as the content is not complex, and limited battery life is not an issue. In other applications, where the content that needs to be displayed in the HMD is rich and/or complex, wired devices are likely to be needed for much longer because all in one HMD solutions are limited by battery power. Other wireless solutions are in development where content can be delivered from a cloud-based architecture to an HMD. These solutions will need to be reviewed according to a risk management framework to see what sort of content they can be used to support. The idea of outfitting a facility to prevent any sort of transmissions (Tempest) is not knew – perhaps there are new ways to accomplish this with new technologies.

Q: What are your thoughts on enclosed AR headsets (use cameras to show real environment) versus open AR headsets (project onto lenses but leave side views open)?

A: Feedback from users on optical see-through technology (OST) vs. video pass-through is as follows: OST provides an excellent way to visualize physical objects in the scene, but suffers from major limitations in field of view and lacks that ability to create a truly immersive environment as the computer-generated image does not match the view of the physical world.

Q: Want to know more about Multi Place Mixed reality, are the HMDs connected to same render or different instance is running on multiplayer mode?

A: Different instances connected via multiplayer.

Q: How are Varjo headsets better than other options? What are the main pros for the Varjo headset when it comes to that application?

A: Varjo HMDs provide human-eye resolution so the pilot is able to see and read virtual content such as text as they can in the real world. They use custom optics to dramatically increase scene quality to increase immersion. They support eye tracking which helps to eliminate cyber sickness via automatic IPD adjustment, and to enable foveated rendering to limit pixel rendering requirements as well as support human performance assessment. They are light weight, support operation with glasses, and built with materials that are easy to clear and replace when necessary. Varjo supports special national security requirements including TAA and 889 compliance to support use in classified environments.

Q: What is the getwell if either the VR headset or the actual sled (sim) goes down? In other words, is there a degraded ops?

A: All sleds come with a monitor that’s used as an operational console. However, if the HMD goes down the monitor can be used as an out-the-window display. Where this supports a limited field of view, and even more limited field of regard, it can continue to support some training tasks. Ideally, with the low cost of the HMDs, spare units should be available at training sites so if one goes down, a new unit can be swapped out in a matter of minutes by simply plugging in a new one.

Q: How may this XR training be extended beyond USAF, to other weapon systems?

A: DIU comment: The Vertex solution was designed and sourced to be modular and with an open architecture. The primary challenge with extending to additional platforms (and beyond USAF) is in the quality of available PC-based simulation environments. Until now, PC-based simulation engines generally do not support the quality of training capabilities required to extend beyond basic flight training in the T-6 or T-38. Next gen simulation software, such as Metrea’s prototype NOR environment, may enable the sort of high-fidelity training capabilities desired by those additional flying communities. We intend to test these capabilities with FBF/T-38C, and if successful, begin scaling this and similar solutions.

Vertex immersive training devices currently supporting the USAF are built on a common platform that is easily reconfigurable to other weapon systems, air, land, and sea saving time and money. Vertex is currently providing XR flight simulators to the US Navy in support of the Naval Training Next program in T-45 and T-6 variants. This same concept has been adopted for commercial airlines at Republic Airways’ LIFT Academy where Vertex immersive training devices flying Diamond DA-40 and DA-42 aircraft are integrated into their training program.

Software architecture isn’t a problem with the Vertex ITD Common Interface layer which provides 100% vendor- and simulation-agnostic integration.

Q: What are your results on including motion simulation with visual – is motion simulation mandatory versus nice-to-have in optimal training?

A: For Pilot Training Transformation (PTT) adding motion was explored and Vertex has several devices with motion. It was decided that motion was not necessary for PTT production devices. However, our TH-1H virtual reality Immersive Training Devices at Ft. Novosel have motion. The MPMR will also have motion in future iterations as we progress toward final development. I wouldn’t say it’s a nice to have or mandatory, it depends on the training requirements.

Q: Did you have problem in a Virtual Environment to interact with switches and buttons that are very close each other? There have been some “mis-click” problems? If yes, how did you solve the problem?

A: During the prototyping stages through user experience events all issues with mis-clicks are resolved.

Q: What gap(s) (compared to “conventional” training) or even negative transfer (if any) that you have observed so far?

A: From Vertex Solutions’ perspective, VR is most effective when used as a complement to “conventional” training. The use of VR allows for training in the cognitive realm in ways that cannot be replicated in conventional training. By watching where the students are looking as they practice in VR, instructors can more quickly get to the root causes of mistakes, even on first maneuver attempts. Used effectively, VR will help a student attain proficiency much faster when in the aircraft. What is often not possible in VR is getting physically representative repetitions of maneuvers. This is where the “gaps” and “negative transfer” will appear. Anecdotally, these gaps and negatives almost all disappear as students adapt to the aircraft environment and do not show any signs of meaningful trouble. This is where our mixed reality (MR) devices fill that gap. With MR, students will gain familiarity and develop muscle memory just as fast as in VR only devices, however they get that tactile feel of the physically representative cockpit. Vertex offers different technology, devices, and learning approaches for any customer requirement, it all depends on the needs of the end user to learn faster, retain longer, and perform better.

From the DIU perspective, these solutions are currently best in an “augmenting” role to legacy training systems and curricula. There are still gaps, but much less “negative transfer” than one might imagine. Students are generally quite adept at switching back and forth between different training media (including the aircraft). What these lightweight XR systems provide is a way to reliably access a training medium that allows for immersive, quality, training content and experiences. Think about it as a way to optimize training time. Rather than chair flying, students can fly an XR simulator. Rather than talking with an IP about a maneuver, the IP can go in the next room and demonstrate it in XR, with a student watching. While formal studies will still need to be accomplished there’s no doubt that this will be a net gain for the training ecosystem. The only obvious gaps at the moment with these systems are the lack of ability to accomplish section 2 or emergency procedures because of the lack of interaction available. However, the Vertex MPMR device and the SAIC/Redbird T-1 XR sim are proving that we will soon be able to accomplish such events as EPs in these devices as well.

Q: What protocols are used for integrating with hardware? (like MQTT, OSI?)

A: MOSA (Open SOurce)

Q: What’s the best way to get ATO’s / how do we accelerate this process for these MR trainers?

A: Vertex has received multiple ATOs for the IL-4 environment (unclassified) as well as Interim authorities (IATTs). Vertex is working with DIU to get these devices classified. Initially air- gapped, but hopefully fully IL-6 compliant. This will allow for threats and weapons critical to combat flight training. There are plenty of hurdles. Once solved, an “ATO as a service” would be a great out of the box solution for other flight simulators. Work with the MAJCOM A6 that the devices are going to be given the ATO. There are significantly different requirements for each command.

Q: With regards to MR devices, how much work has been put into instructor monitoring. Would the instructor also wear a XR HMD, or would they be limited to a seeing a repeat of what the student is looking at? As a fast jet instructor, I think that this would make overall mission management difficult.

A: Typically, the instructor does not wear an HMD, though they can, but through our Instructor Operating Station (IOS) software the instructor can see a repeat of the HMD view the student is seeing. Vertex is currently developing instructor operator stations for four different mixed reality and virtual reality training projects to include a remote instructor operating station. We have put a lot of time and effort into the instructor options with a lot more work to come. If interested in following up please contact info@vertexsolutions.com

Q: Are there any documents that I could read that discuss instructor options?

A: No, but there needs to be a specific study and effort placed into optimizing this. PTN did have quite the gambit of options for instruction and have mostly narrowed down their preferences. Those may not be the same for every org.

Q: Are you also considering a system for real inflight and augmented reality?

A: See Red6. They have an AR system in dev (ATARS).

Q: What are you currently using for terrain generation?

A: Vertex is using COTS simulation environments like Prepar3D, NOR, or X-Plane in our flight simulators; Cesium ION in WorldView in our standalone virtual reality devices and in the instructor operator station.

Q: What makes Varjo the best HMD for the US Gov & these types of applications?

A: FOV, frame rates, Focal Edition viewing/focus regions within arm’s reach. TAA compliance will be critical for ATO approval (ensures parts and code not developed in unapproved nations).

Q: How long do we think until FAA approves certification so big commercial airlines can use this type of tech?

A: This technology is available now to commercial airlines. Although it is not yet FAA certified, and you cannot log flight hours in the simulators there are other benefits to adopting this technology now. There has been up to a 50% reduction in time to first solo flight, which would allow for students to move through the pipeline faster. The quality of pilot being generated by integrating VR/MR flight simulators into training curriculum is much higher than what traditional flight training outputs. Airlines using the latest and greatest training technology are seen as the best.

Q: Is DoD using MR for any other applications outside of pilot training?

A: Maintenance trainers, scuba training, Jumpmaster all projects that Vertex is currently working with DoD in MR.

Q: Do these simulators have hand gestures to mimic real world usage in aircraft?

A: Not currently, however that is only because it wasn’t a customer requirement at the time. Vertex is currently using hand gestures to mimic real world usage in aircraft for other simulators in development for other customers.

Q: What do trainee pilots actually fly if T-1’s are going away?

A: Simulators including the Air Mobility Fundamentals (AMF) simulator in XR.

Q: How much difference pilot finds after trained on simulation and getting into real world?

A: At Vertex Solutions we understand that training devices are most effective when they are used properly in your training program. If a student uses VR expecting it to be a perfect replication of the aircraft, then they will often find and complain about the differences. When VR is used to train cue-response and mental processes, then the differences become secondary, and even a part of the training method. Pilots who understand and expect the simulation to be an imperfect representation will focus on practicing skills that effectively transfer between training mediums and will disregard or undertrain in areas of non-transferable difference. VR simulators differ from real planes in many ways, the effect of those differences comes down to the quality of training more than any other factor. Vertex not only develops cutting edge training devices, but we also excel at developing curriculum within the devices and training our customers on how to effectively integrate extended reality training with your current training programs.

Q: Can these systems replicate G forces?

A: No, not really. Motion can be added to give a bit more sensation of Gs, but no ground-based device can effectively simulate Gs. There are haptic sensors included stimulation units that send a low voltage shock to muscles that will fatigue muscles similarly, but not 1 to 1 for pulling Gs considering the effects on heart and brain (blood flow)

Q: To be effective how long are the sim session wearing HMD, bearing in mind the HF aspects?

A: Roughly 1 hour per user at a time with zero reports of HF/sickness from students. Vertex finds that the HMD is an invaluable tool for training and is the single most critical element to making their devices effective for training. The limit on their use in a training session is rarely physiological. Any symptoms of sickness reported were not a VR symptom so much as a general motion sickness symptom which would have appeared in any highly maneuverable medium. Students have spent entire afternoons using VR flight simulators for training with no issues. Vertex encourages any student we work with regarding these devices to not exceed 45 minutes of training time with them, but that is purely from a learning capacity concern. Going beyond 45 minutes of training results in rapidly diminishing returns in retention and understanding, which is a learning factor unrelated to the device. Students can and do use these machines for durations that go beyond their ability to get meaningful training on them, all without physiological issues.

Q: What do you think about the necessity of 6DOF motion platforms on MR/VR training devices? VERTEX devices shown seem to be used without motion. Any plans to add (6 DOF) motion to the XR setup?

A: Vertex builds devices with and without motion. We have several devices in the field that have 6 DOF motion. It is entirely dependent on the customer requirements and if motion is needed to achieve their training goals. The MPMR device will have motion and you can add motion to any of the Vertex devices with ease. It really depends on the level of training the device will be used for.

Q: What was the biggest challenges to choose the way of implementing the haptic/force feedback? How about the challenges of choosing a specific technology for the haptic/force feedback?

A: Basically you want as large and powerful a motor as you can fit into the space and afford. The more watts of power the more effective it is. Other haptics require gloves or special equipment that is not as desirable due to the multiple users sharing the same gloves, etc. Nor is it particularly scalable.

Q: Does it necessary to use base station when working with Varjo Base and SteamVR?

A: Yes, Steam Base Stations and a mounting system.

Q: We use our MR driving simulator to train Traffic Officers and also evaluate road user behaviour and Traffic Officer performance using the eye tracking embedded and biometrics. 2 questions: 1. Did you give any consideration to using chroma and 2. Are you looking to use eye tracking and biometrics in the training of pilots?

A: Varjo headsets track the eyes and that data has only recently been good enough to incorporate in flight training. Particularly in analyzing instrument scan patterns and search outside the cockpit (target/runway fixation, etc).

Q: Those chair setups are pretty slick – can they be sold to companies outside of the Air Force partnership?

A: Yes, you can contact Vertex Solutions for purchasing options. info@vertexsolutions.com

Q: In addition to using TACVIEW, what other forms of Integrated Debrief and Live Instruction (Shared Cockpit, Laser Pointer on Avionics, Eye Tracking the Crosscheck, etc) do you see being most useful and potentially implemented in current or future platforms?

A: Cloud ahoy, some developmental products from DMI called VIPER

Q: How did you assess the simulator physical/functional/psychological fidelity to prove it was sufficiently representative of the aircraft for different training objectives.

A: During Pilot Training Transformation (PTT) prototyping we held rigorous user experience testing events with the end user where we used data from those events to influence how the final production simulators for PTT were designed.

Q: What do you think about moving on a complete Virtual Reality, instead of a mixed one? I mean, you can completely remove the cockpit and just keep the HOTAS for dog-fighting.

A: The over 260 Vertex Solutions devices currently in operation across five USAF bases are all in VR with a stick, HOTAS, and rudders. Mixed reality serves a different purpose and need for pilot training. So we have developed options for every level of training needed. It depends on what is needed by the end user to learn in the most efficient, effective, and quickest way possible.

Q: One of the slides mentioned Embry Riddle’s reduction, is Embry Riddle utilizing similar VR technology for its students? 

A: Explore these resources:
Virtual Reality Flight-Training Program at Embry-Riddle Set for Expansion | Embry-Riddle Aeronautical University – Newsroom (erau.edu) 

Aviation Week Op-Ed: Let’s Prove that Virtual Reality Enhances Pilot Training | Embry-Riddle Aeronautical University – Newsroom (erau.edu) 

Extended Reality (XR) Lab | Embry-Riddle Aeronautical University – Daytona Beach, FL (erau.edu) 

Q: From the USAF perspective, what would be the most audacious “XR”-based simulator goal imaginable, e.g., F-22 combat environment simulation?

A: A big challenge is network and hardware security. These devices use mainly COTS components that will need to be accredited before we can tackle anything like an F-35 or F-22. Also, the program offices that manage those programs are far less likely to quickly or easily adopt these sorts of solutions. Because of these challenges, we are doubling down on the momentum we have gained for UPT, including FBF, and in platforms/MAJCOMS with lower-classification aircraft. We also recommend that our industry partners take a similar approach. We will build up to those higher classification aircraft, but there are other policy decisions and technological advancements that will need to take place first before that becomes a reality.

Q: In addition to things like high resolution and wide FoV, what are other specific things in headsets that pilots are looking for to have an effective/immersive VR/MR simulation training?

A: Audio

Q: Is this technology going to extend down to C-172s, Pipers, or intro trainers??

A: This same concept has been adopted for commercial airlines at Republic Airways’ LIFT Academy where Vertex immersive training devices flying Diamond DA-40 and DA-42 aircraft are integrated into their training program now. Contact Vertex Solutions today for commercial trainer options. Info@vertexsolutions.com

Q: 350M in savings, 50% reduction in solo-flight time, outperforming pilots: with such extraordinary improvements, why is it still going to take a few years before this is the new industry standard? Why aren’t companies battling over this technology?

A: The lack of FAA certification is probably the biggest barrier for widespread adoption in the commercial airline industry. In the military there are several barriers that could be of issue such as money and adoption by the luddites who have always trained this way so why change now.

Q: What do you think about conducting tactical mission training (where the focus is on tactics and/or mission operations, rather than on flying the aircraft) using generic (non-aircraft specific) VR/MR devices?

A: Vertex currently builds VR and MR devices for tactical mission training. We are developing a jumpmaster MR training simulator for training jumpmasters. Vertex recently delivered a six crew position AC-130J combat mission trainer to AFSOC where all six crew members can interact and participate in a virtual mission together. The possibilities are endless, the funding just needs to be available.

Want to learn more about industry leading VR/XR simulation training?

Download free whitepaper

Browse more Varjo Insider posts