Abstract

Hands are the primary and most direct way we interact with the world. We use tools all the time, but our hands are always with us. With HoloLens 2, we introduced a new way for users to interact with virtual objects in mixed reality – simply reaching out and touching them. Direct Manipulation lets users grab, move, and adjust digital content in the same ways as they do with real, physical objects. To enable this kind of interaction, we must first track the user’s hands.

In this talk, I will describe how we at Microsoft turned research into product to bring Fully Articulated Hand Tracking to HoloLens. This story starts nine years ago with an internship project about fitting smooth 3D models to pictures of dolphins, and ends today with a hand tracker that can accurately and robustly track the surface and joints of a user’s hands at 45Hz, while also being performant enough to fit within the power and computation constraints of a head-mounted device.

I will present the technical highlights of our journey from research to product, focusing on several key engineering efforts and algorithmic innovations. This includes making our code over 500 times faster through careful low-level optimization, collecting a massive dataset of 3D hand data to understand how human hands look and move, and developing a photo-realistic synthetic data pipeline for preparing vast amounts of noise-free training data for machine learning.

Video recording of this talk is available at:

https://www.cl.cam.ac.uk/seminars/wednesday/video/lt2-190424-wed-1600-t123406-only-ucam.html

(access restricted to the University of Cambridge)