Case Explico: How Mixed Reality Helps Forensic Engineers Educate Jurors in the Courtroom

Did the specific lighting conditions and nearby structures affect a driver’s ability to perceive the distance of an oncoming  motorcycle or passing pedestrian? Was a downed powerline open and obvious to a nearby individual walking by? Did a carpeting’s pattern create an optical illusion thereby obscuring a step, which ultimately caused an individual to fall and become injured? Legal matters involving accidents, failures and even serious crimes often hinge on whether one side can effectively communicate critical, complex issues to the jury. 

Answering these questions objectively is not easy. To help prove what happened and effectively communicate it to a jury, 3D animations have long since become an important tool in courtrooms because they can convey very complex issues in a compelling and understandable way. Example use cases include showing a scene of a crime, illustrating the inner workings of a failed mechanical device, and demonstrating the events that lead to an injury.

In these cases, accuracy is paramount. Accurate, high-fidelity representations of objects and environments can be generated from 3D laser scanning (LiDAR) devices. These 3D reconstructions and animations are more informative than traditional photographs or video, but they still have limitations. 

The biggest challenge is that 3D content is still being viewed on traditional 2D screens. This results in a lack of immersion and interactivity. Even the most detailed 3D animation, when viewed on a conventional display, will not be able to provide the sense of space and depth that is available in XR. Consequently, for certain accidents and issues, XR’s ability to provide an immediate understanding of what a person may have observed from their perspective, could provide critical understanding to jurors when reaching a verdict.

Things are about to change, though. Immersive technology is already revolutionizing how 3D scenes and animations are used in mediations, and it won’t be long before virtual and mixed reality headsets make their way to the courtroom. And at the heart of this development are Explico’s Steve Rundell, PhD, PE and Steve Arndt, PhD, CHFP.

Mixed reality also makes the experience less jarring than being immersed in a fully virtual scene right from the start. This is especially useful with people who might not even have heard of virtual reality before and do not necessarily know what to expect when they are put in such a scene.

Ease of use is extremely important once headsets are used in legal proceedings. Jurors, judges and lawyers cannot be expected to be proficient in working with headsets. The technology needs to be intuitive to any kind of user. Once this technology is deployed in jury trials, there simply won’t be time to teach each and every juror to operate controllers, navigate complex settings, or adjust interpupillary distances so they don’t get nauseous. 

This is why accurate and natural hand tracking is a must have: there is no need for learning any kind of controller once a person can simply use their own hands to interact with an environment. Automatic IPD removes the need for manual visual calibration so the device can simply be put on and adjusted with the click of a single button, without any expertise needed from the user. This ensures that there is no eye strain, everything can be seen clearly, and there is no cyber sickness.

Another major benefit of this technology is that it can show events in real-life scale inside the courtroom, something that would be impossible with physical objects and places alone. For example, it is simply not realistic to bring large machines or vehicles into a courtroom, or expect a jury or judge to travel to the actual accident site in every single case.

The technology was also relatively easy to take into use as the Explico team was already using Unity to create 3D animations. Converting them for use with an XR headset was simply a natural step forward.

While not deployed during a jury trial within the United States yet, these immersive solutions have already been used as part of arbitration and mediation proceedings that ended up in a settlement outside the courtroom. Judges and arbitrators have already been able to view scenarios in mixed reality. It’s only a matter of time until jurors do so as well.

However, given how compelling many of these interactive demonstrations are using the Varjo headset, Steve Rundell of Explico anticipates that most cases will settle prior to trial. Specifically, once one side has had an opportunity to experience the other side’s compelling and interactive VR experience, they won’t want to run the risk of having a jury be exposed to the same evidence. Consequently, many lawyers will fight to keep VR out of the courtroom if it doesn’t aid in their case. Fortunately, a combination of Varjo’s incredibly high visual acuity combined with Explico’s previous and future peer-reviewed research will make exclusion of this evidence difficult for attorneys. 

Explico’s Steve Rundell sees that it is only a matter of time before the technology is deployed during a jury trial as well, and thanks to Varjo, Explico is ready to be a part of that monumental shift. He believes the first cases where the technology is going to be used will most likely end up in court fairly soon – perhaps already this year.

“The technology has reached a sufficient level of maturity to become widely adopted. It has the chance to revolutionize court proceedings, especially in tricky cases where you need to depict something that can only be depicted in XR. For example, nobody wanted to touch animations in the 80’s, but they are now a staple in many court cases. I predict a similar shift towards extended reality will happen eventually, because the degree of accuracy and how well it can reflect the real world is simply unmatched, and Varjo is a perfect device for making that happen,” Rundell says.