A high-altitude proto-metaverse experiment was conducted by two fighter pilots on May 10.
They wore special AR headsets to connect to a system that overlay a ghostly, shimmering image of a refueling aircraft flying alongside them in the sky a few thousand feet above the California desert in a pair of Berkut 540 planes. The virtual tanker was then refueled by one of the pilots while the other stood by and watched. Hello and welcome to the nascent military metaverse.
The metaverse craze isn’t limited to Silicon Valley these days. Many defense startups, contractors, and financiers are increasingly talking about the metaverse, even though its definition and utility aren’t always apparent, just as tech companies and corporations scramble to devise strategies for virtual worlds.
The fundamental technologies required for the metaverse—augmented and virtual reality, head-mounted displays, 3D simulations, and artificial intelligence-powered virtual environments—are currently in use in the military industry. The end result is less polished, adorable, and expansive than Mark Zuckerberg’s virtual world concept, but that’s part of the goal. Even if it stutters in the civilian realm, the underlying technology has a strong chance of taking off.
Fighter pilots have been able to practice dogfighting against simulated opponents, such as Chinese and Russian jets, using a combination of augmented reality, artificial intelligence, and video game visuals. The company behind the technology, Red 6, claims that it provides a significantly more realistic examination of a pilot’s ability than a traditional flight simulator. “We can fly against any attack,” says Daniel Robinson, the company’s creator and CEO. “And that threat may be managed remotely by a person or by artificial intelligence.”
Under comparison to consumer AR or VR headsets, Red6’s AR technology must perform in more harsh environments, with reduced latency and higher dependability. Robinson goes on to say that the business is currently developing a platform that will allow a variety of scenarios to be depicted in augmented or virtual reality. He explains, “What we’re developing is actually a military metaverse.” “It’s like a sky-based multiplayer computer game.”
Some of the most recent military systems already use metaverse-related concepts. For example, the high-tech helmet for the new F-35 fighter plane incorporates an augmented reality display that overlays video feeds from around the aircraft with telemetry data and target information. The US Army stated in 2018 that it would pay Microsoft up to $22 billion to construct an augmented reality system for warfighters called the Integrated Visual Augmentation System (IVAS).
In recent years, virtual reality and augmented reality have become commonplace in military training. Project BlueShark, a technology that allowed sailors to control ships and collaborate in a virtual environment, was developed in 2014 by the Office of Naval Research and the Institute for Creative Technologies at the University of Southern California. Project Avenger is now being used to assist in the training of US Navy pilots. Virtual reality is being used by the US Air Force to teach pilots how to control aircraft and missions. VR is also utilized to treat chronic pain and post-traumatic stress in veterans. Boeing has also designed an augmented reality environment that allows mechanics to practice working on planes before boarding a real one.
The US military has recently begun to investigate more complicated virtual worlds. There is also a rising interest in connecting and integrating virtual worlds in a metaverse-like manner. The US Air Force convened a high-level conference through virtual environment in December 2021, comprising approximately 250 persons from places ranging from the United States to Japan.
“The promise is in integrating these technologies,” says Caitlin Dohrman, general manager of Improbable’s defense division, which has created sprawling virtual battlefields with over 10,000 individually controlled characters for the UK’s military wargames and also collaborates with the US Department of Defense (DOD). “It’s an extraordinarily sophisticated form of simulation,” Dohrman says, “particularly given the fidelity that the military requires.” “You can have live players participate in the simulation, or [characters] can be AI-enabled, as the military frequently does.”
Luckey cofounded the defense firm Anduril in 2017. Despite the recent metaverse excitement, he believes there is significant defense potential, mainly due to the importance and cost of military training. However, he claims that the technology does not have to be hyper-realistic to be beneficial, and that Anduril should only employ it when absolutely necessary. He claims that “what we’re doing with VR is something that is uniquely better than any other alternative.” VR may be used to instruct people to fly Anduril’s drones, or to display information about an area based on data from ground sensors, he says.
Newer military systems, like Zuckerberg’s projected metaverse, rely significantly on AI to be effective. Red6’s AR technology was utilized in October 2020 to pit a real fighter pilot against an AI-controlled aircraft produced as part of a Defense Advanced Research Projects Agency (DARPA) AI dogfighting study. Another business, EpiSci, developed the AI top gun, which learnt how to outmaneuver and outgun an opponent through trial and error. The AI pilot finally achieved superhuman abilities and was able to consistently outperform its human opponent.
Another DARPA project, Perceptually-enabled Task Guidance, attempts to develop an AI assistant that monitors what a soldier is doing and provides guidance via speech, sound, or visuals. Unlike Boeing’s augmented reality system, which only functions in a restricted environment, such a system would have to understand the real world. The actual benefit of the technologies being examined by the military, according to Bruce Draper, the DARPA program manager in charge, lies in the melding of the real and the virtual. “The metaverse is primarily virtual, and virtual worlds are important for training,” he argues. “However, we live in the physical world.” “It’s not about an abstract metaverse in the military arena,” says the author.
However, attempts to integrate the virtual and actual worlds have run into difficulties. A leaked Microsoft letter from March 2022 allegedly revealed that staff working on IVAS, the US Army’s equivalent of the HoloLens AR headgear, expected it to be poorly welcomed by consumers. And, according to a DOD audit revealed in April 2022, the US Army may be wasting money as a result. Jason Kuruvilla, a Microsoft senior communications manager, published multiple statements from high-ranking army officers extolling the IVAS’s capabilities. He also cited a DOD report from 2021 that emphasizes the significance of creating IVAS quickly so that issues can be dealt out along the way.
The confidence of those pushing the military metaverse has only grown as a result of such high-profile and costly undertakings. “I believe this is the future of military training,” says Doug Philippone, Palantir’s global defense lead, who has invested in both Anduril and Red6. “But I also see it as the way the military fights and makes decisions in the future.” So it isn’t just about battling; it is also about making choices.”
Anduril, according to Luckey, is already developing technologies that might be used in training missions and combat. “The next big step for us, which I’m really thrilled about,” he says, “is taking data from our core product and feeding it to heads-up displays that front-line troops will be able to wear.”
But it’s unclear how much of this cutting-edge technology makes it to the front lines—or even into training exercises. Sorin Adam Matei, a professor at Purdue University in West Lafayette, Indiana, who has designed virtual warfare training platforms for the US military, believes that the technology deployed will be far simpler than proponents of the metaverse believe.
He speculates that a less complex version of the IVAS headset could be fitted with an AR rifle scope in the future. “The last thing you want to worry about when you’re out there shooting and being shot at is another piece of equipment,” he says. And technology doesn’t have to be as big as the metaverse to be beneficial. “We need to think about this metaverse idea a little bit more—powerful, it’s but it has its limitations.”