Abstract

Optical flow represents the apparent 2D motion of the 3D world as captured in a sequence of video frames. This flow is related to physical structure in the world and is useful for applications as diverse as robot navigation, human motion understanding, and video compression. This talk will review several recent advances that have led to significant improvements in optical flow accuracy and will explore the principles behind these. Long range spatial correlations in the optical flow prove particularly important to model and several methods for doing so will be presented. Among these, layered models of the scene prove particularly useful because the spatial correlations are captured by the per-pixel layer assignment, the layers capture the occlusion relationships in the scene, and the motion within layers can be fairly simply described. Additionally, temporal coherence is important to understand occlusion boundaries and what is consistent over time is not the motion but the scene structure represented by the layers. The talk will also introduce the MPI -Sintel dataset, which provides a challenging benchmark for flow algorithms and reveals current limitations. Finally, the talk will show that optical flow is now accurate enough to enable applications such as human motion analysis. In particular, the talk will introduce “flowing puppets” that use optical flow to improve 2D body pose estimation.

This is joint work with many co-authors, including Deqing Sun, Stefan Roth, Erik Sudderth, Jonas Wulff, Silvia Zuffi, Javier Romero, Dan Butler, Hans-Peter Pfister, and Cordelia Schmid.

Related papers:

A Quantitative Analysis of Current Practices in Optical Flow Estimation and the Principles behind Them, IJCV ’13 http://link.springer.com/content/pdf/10.1007%2Fs11263-013-0644-x.pdf

A fully-connected layered model of foreground and background flow, CVPR ’13 http://files.is.tue.mpg.de/black/papers/SunCVPR2013.pdf

Layered segmentation and optical flow estimation over time, CVPR ’12 http://www.cs.brown.edu/~dqsun/pubs/cvpr_2012_layer.pdf

A naturalistic open source movie for optical flow evaluation, ECCV ’12 http://files.is.tue.mpg.de/black/papers/ButlerECCV2012-corrected.pdf

Estimating human pose with flowing puppets, ICCV ’13 http://files.is.tue.mpg.de/black/papers/ZuffiICCV2013.pdf