Project Sidekick: Microsoft’s HoloLens Hitching Ride on SpaceX CRS-7 to Give Astronauts Virtual Aid

NASA and Microsoft engineers test Project Sidekick on NASA’s Weightless Wonder C9 jet. Project Sidekick will use Microsoft HoloLens to provide virtual aid to astronauts working on the International Space Station. Credits: NASA
NASA and Microsoft engineers test Project Sidekick on NASA’s Weightless Wonder C9 jet. Project Sidekick will use Microsoft HoloLens to provide virtual aid to astronauts working on the International Space Station. Credits: NASA

A new project between NASA and Microsoft will be hitching a ride to the International Space Station (ISS) aboard SpaceX’s Dragon resupply ship this Sunday on the CRS-7 mission, aiming to put the first fully untethered, see-through holographic computer in action—in space—to give astronauts living and working on the ISS a virtual aid when and where they need it.

Known as Project Sidekick, the new project will employ Microsoft’s HoloLens to provide a new capability that could reduce crew training requirements and increase the efficiency at which astronauts can work in space. It is part of a larger partnership formed by NASA and Microsoft some time ago to explore applications of holographic computing in space exploration, and a pair is onboard Dragon CRS-7 for the 17,500-mph uphill ride on June 28 to the $100 billion orbiting research laboratory.

NASA and Microsoft engineers tested Project Sidekick and the Microsoft HoloLens aboard NASA’s Weightless Wonder C9 jet to ensure they function as expected in free-fall in advance of their delivery to the microgravity environment of the space station. Image Credit: NASA/Microsoft
NASA and Microsoft engineers tested Project Sidekick and the Microsoft HoloLens aboard NASA’s Weightless Wonder C9 jet to ensure they function as expected in free-fall in advance of their delivery to the microgravity environment of the space station. Image Credit: NASA/Microsoft

READ our in-depth CRS-7 launch and mission preview!

“HoloLens and other virtual and mixed reality devices are cutting edge technologies that could help drive future exploration and provide new capabilities to the men and women conducting critical science on the International Space Station,” said Sam Scimemi, director of the ISS program at NASA Headquarters in Washington. “This new technology could also empower future explorers requiring greater autonomy on the journey to Mars.”

From Microsoft: “We’ve made incredible advances as an industry in the ways we interact with computers. However, we are still constrained because we must conform to the ways computers recognize our commands through mouse clicks or by touching a screen. Using holograms, you can pin your digital content, such as apps, information, and even multi-dimensional videos, in the physical space around you, so you can interact with it in the same ways that you interact with other physical objects. Holographic objects seen with HoloLens can be pinned, or anchored, to physical locations you choose, move according to their own rules, or remain in a specific location within your field of view regardless of where you are or in which direction you are looking.”

Microsoft adds: “The holograms can appear life-like, and can move, be shaped, and change according to interaction with you or the physical environment in which they are visible. There isn’t a screen to touch or a mouse to click. Use gestures to create, shape, and size holograms. Use your gaze to navigate and explore. Use your voice to communicate with your apps. Microsoft HoloLens understands your movements, gaze, and voice, enabling you to interact with content and information naturally. Using holograms, you can pin your digital content, such as apps, information, and even multi-dimensional videos, in the physical space around you, so you can interact with it.”

Both NASA and Microsoft engineers took to the skies recently to test Project Sidekick and the Microsoft HoloLens aboard NASA’s Weightless Wonder C9 jet to ensure they function as expected in free-fall, in advance of their delivery to the microgravity environment of the ISS in the coming days.

Sidekick has two modes of operation: “Remote Expert Mode” and “Procedure Mode.” NASA outlines each as follows:

“Remote Expert Mode uses Skype, part of Microsoft, to allow a ground operator to see what a crew member sees, provide real-time guidance, and draw annotations into the crew member’s environment to coach him or her through a task. Until now, crew members have relied on written and voice instructions when performing complex repair tasks or experiments.

“Procedure Mode augments standalone procedures with animated holographic illustrations displayed on top of the objects with which the crew is interacting. This capability could lessen the amount of training that future crews will require and could be an invaluable resource for missions deep into our solar system, where communication delays complicate difficult operations.”


WATCH: Sidekick Gets Tested on NASA’s C9 “Weightless Wonder” Jet

Dragon will arrive at the ISS Tuesday, June 30, assuming a launch occurs as scheduled on Sunday. Once at the ISS the Expedition 44 crew will unload and (eventually) test the devices to verify the software and hardware functionality in a standalone mode.

A second set of devices will be delivered on a future mission to test and verify Sidekick functionality with network connectivity to test the Remote Expert Mode, and NASA expects astronauts on the ISS will first use Sidekick by the end of this year.

“Microsoft HoloLens is about transforming the ways you create, connect, and explore,” said Alex Kipman, technical fellow, Windows and Devices Group at Microsoft. “Sidekick is a prime example of an application for which we envisioned HoloLens being used – unlocking new potential for astronauts and giving us all a new perspective on what is possible with holographic computing.”

NASA will also put Sidekick to work here on Earth as well, using and evaluating it the during the agency’s upcoming two-week-long NASA Extreme Environment Mission Operations (NEEMO) 20 expedition, which will begin July 21 at Florida International University’s underwater Aquarius Reef Base, located 6.2 miles off the coast of Key Largo. Aquarius is the world’s only undersea research station, providing for a habitat and surroundings that create a convincing analog for space exploration. NEEMO 20 will be commanded by European Space Agency (ESA) astronaut Luca Parmitano, with NASA astronaut Serena Aunon, NASA EVA Management Office engineer David Coan, and Japan Aerospace Exploration Agency (JAXA) astronaut Norishige Kanai joining him.

Sidekick is not the only NASA/Microsoft collaboration at work. Earlier this year, NASA and Microsoft announced a collaboration to develop software called OnSight, a new technology that will enable scientists to work virtually on Mars using the same HoloLens technology.

“Our team is excited to be building virtual and mixed reality tools that will make our explorers more efficient and effective,” said Jeff Norris, project lead for Sidekick and OnSight at JPL.

 

Be sure to “Like” AmericaSpace on Facebook and follow us on Twitter: @AmericaSpace

 

 

Missions » ISS » COTS » CRS-7 »

SLS Development Engine Test Fire #4 Ignites With Longest Duration Firing Yet

‘To Work Co-operatively’: 20 Years Since the First Shuttle-Mir Docking Mission (Part 3)