Exploring space using virtual reality
COLLABORATING WITH TECH GIANTS INCLUDING MICROSOFT, NASA IS NOW ABLE TO CONQUER THE UNIVERSE WITHOUT NEEDING TO LEAVE EARTH – AND YOU CAN COME ALONG FOR THE RIDE
Collaborating with tech giants including Microsoft, NASA is now able to conquer the cosmos without leaving Earth
Virtual reality (VR) is bringing space closer to armchair astronauts than ever before. Not only is it opening up the galaxy to those of us that will never make it beyond the Kármán line, it’s also helping astronauts train for the harsh environment of microgravity and scientists for their projects.
In March 2017 a virtual recreation of the International Space Station (ISS) was launched for the Oculus Rift VR headset. Mission:ISS, which was made in collaboration with NASA, the European Space Agency (ESA) and the Canadian Space Agency, is the work of visual effects firm Magnopus, based in Los Angeles. “We started with engineering models from NASA at Johnson Space Center,” Ben Grossmann, VR director of the experience and co-founder of Magnopus, tells All About Space.
“Then we scoured the freely available images on NASA’s websites for photo and video references, which our artists used when they added details. It was quite the research project because there’s so much activity on the ISS, and it’s so difficult to ‘take inventory’ regularly and get any confidence that something is where we thought it might be”.
The team made a painstaking effort to get every detail exactly right, interviewing astronauts on whether there was anything missing that they expected to see. “For example, one ISS crew member pointed out to us that there was damage to one of the radiator panels that was catalogued during STS-119 that no one ever seems to acknowledge in recreations, so we got that in there,” explains Grossmann.
Aside from getting legal clearances for all the imagery and videos of astronauts, one of the biggest challenges for the effects firm was conveying the weightless experience of the orbiting lab, and translating that experience so that it would run on minimum-spec VR computers. “We spent a lot of time working on the way people could navigate with the hands by grabbing onto the station and pushing, pulling and twisting themselves through the modules,” says Grossmann. “A little over 560 people have been [to space], and it’s going to be a while before we can do it casually, so we wanted to give people a taste of the overall effect” he added.
While not used for formal astronaut training, Mission:ISS has been used at NASA’s Johnson Space Center. According to Grossmann, when astronaut Cady Coleman was immersed in the experience, her son asked if he could have a turn, to which she replied: “No. I’m treating my homesickness.”
As well as enabling people on Earth to experience the ISS, VR has also made it possible to get a glimpse of the journey back home. London’s Science Museum hosts a ‘Space Descent VR with Tim Peake’ experience. “In response to the landmark acquisition of Tim Peake’s spacecraft, Soyuz
TMA-19M, in late
2016, the Science Museum Group commissioned the award-winning Alchemy VR to develop an experience that would put the visitor inside the spacecraft to experience the thrill of a highspeed descent to Earth,” Mark Cutmore, head of commercial experiences at the Science Museum, tells All About Space.
The VR flight, which is the Science Museum’s first-ever permanent VR fixture, is viewed using Samsung Gear VR headsets. “To achieve the extraordinary level of realism offered in the experience, the team consulted leading experts, including Tim Peake himself, and then employed the rendering power of 100 computers for a full month,” explains Cutmore.
“As well as lending his voice and on-screen presence to the experience, talking users of Space Descent VR through the stages of the flight from the International Space Station to Earth, Tim Peake consulted on the script, sharing his experience of the journey he made in June 2016 at the end of his Principia mission.”
Visitors to the museum have been impressed with the realism of Space Descent VR, says Cutmore, who explains why VR is such a useful educational tool: “The immersive nature of VR means that we can learn in a different way about the scientific story around that object or moment.” Speaking at the time of the opening, Tim Peake said: “It really is breathtaking – and that comes from someone who has spent an awful lot of time using VR systems while training for my first mission.”
As well as bringing recent missions to a wider audience, VR also enables us to relive crucial events from the history of human spaceflight, including the 1969 Moon landing. After reaching its target on crowdfunding site Kickstarter, the Apollo 11 VR experience was launched in 2016 on a number of VR platforms including Oculus Rift and Sony’s PlayStation VR. The virtual experience includes several highlights of the era-defining mission, from the Saturn V rocket down to the lunar lander at Tranquillity Base. Though not officially endorsed by NASA, it uses the American space agency’s archive footage and audio from the Command Module’s cockpit to recreate the historic journey.
There are all kinds of space-based VR apps that can easily be downloaded for viewing on basic headsets like Google’s Cardboard viewer, and even the revamped stereoscopic View-Master viewer is giving children the chance to explore a virtual version of our very own Solar System.
Not only does VR enable those on Earth to catch a glimpse of life in orbit, it’s also used to prepare astronauts for the challenges of space. NASA was an early pioneer of VR, with its first headset dating back to 1991. The astronaut crews from Mercury, Gemini and Apollo spent more than a third of their training time packed into simulators, while crews today use VR simulations to train for their missions. One vital piece of training involves practice with a version of the Simplified Aid For EVA Rescue (SAFER) – a jet pack worn by astronauts that they can use to manoeuvre back to the ISS should they become untethered during a spacewalk.
As well as training astronauts on Earth, VR is also being used in space to carry out vital experiments. The French space agency Centre National d’Etudes Spatiales (CNES) is using one such experiment called GRASP (Gravitational References for Sensorimotor Performance: Reaching and Grasping) to look at how astronauts’ brains adapt to coordinate hand movements in microgravity. French ESA astronaut Thomas Pesquet was tasked with setting up and testing the equipment on the ISS, where it is used for tests by crew members.
The kit includes an Oculus Rift headset which has been modified for use in space to ensure that
“THE IMMERSIVE NATURE OF VR MEANS THAT WE CAN LEARN ABOUT THE SCIENTIFIC STORY IN A DIFFERENT WAY”
Mark cutmore
it is not a fire risk. The headset’s accelerometer cannot be used in the same way in microgravity, so researchers rely on an infrared motion-tracking system developed by the ESA to monitor movement instead. The experiment involves astronauts reaching for virtual objects to enable researchers to understand how important gravity is compared to other senses.
Maurice Marnat from the MEDES Institute of Space Medicine and Physiology in Toulouse, France, which is an affiliate of CNES, explains that on Earth the brain uses sensations caused by gravity as a reference for movement, using a football player as an example. “If I see a football coming towards me, I know it will have a specific weight, and I can predict its trajectory. Children take a long time to learn to make these predictions accurately,” he tells All About Space. “When astronauts first arrive in space, they are like babies again. They have to learn how to move and how to integrate their visual information into their movement.”
Not only could the findings help astronauts aboard the ISS – and during spacewalks where
hand-eye coordination is extremely important – they could also eventually help scientists to better understand patients with conditions that cause a loss of balance on Earth.
Another CNES experiment will investigate the problems astronauts have with time estimation in space, which is known to be connected to microgravity’s effect on the inner ear. “The goal is to quantity this effect. The VR is used only to emerge the subject in a virtual world – to disconnect them from any external information,” explains Marnat.
Meanwhile, a separate experiment from the Canadian Space Agency will look at the effects of reduced gravity on astronauts’ perception of self-motion. “We hope to be able to build a VR community in space,” explains Marnat. “VR opens the door to many more applications other than science, such as on-board training, and helping astronauts on a daily basis as a countermeasure against confinement”.
And it’s not just VR that’s blurring the boundaries between Earth and travelling in space. Augmented reality (AR), where computer graphics are superimposed over a real-world view, is also becoming more popular, but it’s mixed reality (MR) – a combination of VR and AR – that could really be a game changer. One of the best known examples of an MR device is Microsoft’s HoloLens.
In December 2015 NASA launched a pair of HoloLens headsets to the ISS as part of Project Sidekick. The devices were originally intended to arrive earlier, but the first batch were blown up when SpaceX’s CRS-7 Falcon 9 resupply rocket exploded shortly after take-off in June 2015. In ‘Remote Expert Mode’ the HoloLens enables ground operators to be able to see what a crew member sees via messaging platform Skype, providing real-time guidance and annotations to guide the astronaut through a task. Previously they would have had to rely on just vocal and written instructions. In ‘Procedure Mode’ it presents animated illustrations on top of the objects the crew are interacting with. The idea is to cut down the amount of training for future crews.
“The benefit of Project Sidekick and related AR applications is the ability for crew to operate hands-free and see their surroundings comfortably at all times,” Victor Luo of NASA’s Jet Propulsion Laboratory tells us.
In collaboration with Microsoft, NASA has also developed software called OnSight that enables scientists to work virtually on Mars using the HoloLens. “We take the 2D rover images from Mars and reconstruct a 3D terrain mesh using our homegrown photogrammetry pipeline,” explains Luo. “Every day the scientists on the Curiosity mission can load up OnSight and walk on the latest place on Mars. They can gather with their international colleagues in the same session and collectively make strategic science planning decisions for the operations of the rover.
“We have been developing outreach experiences with this technology to allow the general public to experience the journey of the rover as well. The most recent public release is the Destination: Mars experience that we deployed to Kennedy Space Center Visitor Complex in 2016, which captures Buzz Aldrin in holographic form to guide the audience through three real sites on Mars.”
NASA has also developed ProtoSpace, which uses the HoloLens to project images of rover mockups in 3D to help engineers pinpoint any potential problems in future designs.
“These technologies will allow our scientists, astronauts and engineers to immersive themselves in the data that they need at any given time,” says Luo. “When we continue to explore the Solar System via robots and astronauts, we will be able to take everyone along for the ride.”