When NASA scientists want to discuss how to best roll a rover over a Martian hill, they’ll be able to do more than talk. They can strap on a pair of goggles and navigate the Red Planet together, from the comfort of their respective offices.
To create an entirely immersive experience of a different planet, NASA scientists built OnSight, software that works with Microsoft’s HoloLens technology to produce a realistic holographic image of Mars based on data collected by the Curiosity rover and Mars reconnaissance orbiter. HoloLens debuted earlier this year as Microsoft’s answer to technologies like Google Glass and Facebook’s Oculus—the augmented reality hardware overlays digital renderings of objects and experiences on top of your existing area.
Victor Luo, the human interfaces lead engineer at NASA’s Jet Propulsion Laboratory, said scientists will be able to control the Mars rover with OnSight in July.
“They’re wearing this, they’re not only walking on Mars by themselves, they’re walking on Mars together with everyone on the mission,” Luo said in an interview at the Space Apps hackathon at Microsoft New York over the weekend. “So it’s going to represent their avatars, they’re going to be able to talk, be able to see where each of them are looking.”
To create a life-like representation of Mars terrain, NASA takes data from multiple cameras on the Curiosity rover and combines it with satellite data and information gleaned from where the rover previously drove. It can render a complete image in under an hour, so scientists can experience what Mars is like at the exact moment, and use the data to make decisions and drive the rover in real-time.
Right now, it’s a time-consuming process, one that requires massive email threads with attachments of 2D renderings of the surface of the planet. Scientists take screenshots of the images and make notes on the photos to send to colleagues. It can take hours, or sometimes days, for scientists to agree on how to maneuver the rover. But with OnSight, Luo said that time can be drastically reduced.
“As successful as the Curiosity is, it has missed a lot of its milestones,” Luo said. “It’s made the main milestones, but it’s not as performant as we thought it was going to be, because part of this operation process, like I mentioned this 24-hour planning cycle, most of it is spent on debating where to target.”
Luo said NASA hopes that the HoloLens and OnSight can help compress the planning timeframe down to just a few hours. It’s still experimental technology, but it has the potential to change the way scientists execute different missions. Scientists will be able to program the rover to complete multiple activities simply by looking at an object and using gestures to command the robot.
“You can see where we came from the early days of Kinect and really hacking together ideas to a place where we’re actually going to use this on a real mission.”
To say that NASA scientists and engineers are excited about HoloLens, is an understatement. Luo said that when Fred Calef, NASA’s “Keeper of Maps” (he’s in charge of building precise representations of Mars) first tried out OnSight, he started crawling along the floor and moving his head in different angles to experience the entirety of Mars.
“No one expected that from him,” Luo said.
HoloLens isn’t the only technology NASA has experimented with. The organization frequently adopts new ideas and applies them to current missions, robotics, and operations. Years ago, NASA used Leap Motion to control the giant All-Terrain Hex-Limbed Extra-Terrestrial Explorer (ATHLETE) robot by gesturing with one hand, and programmed Microsoft Kinect and Oculus Rift to control a robotic arm.
Virtual reality devices like Oculus also create immersive experiences that could allow scientists to interact like they would on the surface of Mars. However, NASA decided to pursue augmented reality software, in part because virtual reality headsets can be intimidating for people who aren’t used to leaving physical environments for a completely digital space.
“You put these [VR headsets] in front of a scientist, and you have to realize they’re not technologists, not hackers, not gamers; it’s scary,” Luo said. “Suddenly everything goes away and you don’t know where you are.”
With HoloLens, engineers were able to create renderings that didn’t obfuscate the real world—scientists and drivers can walk through the experience of Mars while still seeing their office carpet. The simulation is also capable of cutting out the desk area, so scientists can still use their computer, and move a cursor from one screen to the floor of another planet.
HoloLens will be tested on missions by this summer, and could be used in missions including Mars 2020 that aims to further Mars exploration. And the device adds another successful experiment to NASA’s longstanding history of embracing new technologies to change the way people explore outer space.
“We find [new technologies], we embrace them, and we explore the opportunities, and narrow in on a couple of ideas that we try to polish up,” Luo said. “You can see where we came from the early days of Kinect and really hacking together ideas to a place where we’re at now, where we’re actually going to use this on a real mission.”
Photo via NASA/JPL-Caltech