Human Pose, Shape and Action
3D Pose from Images
2D Pose from Images
Beyond Motion Capture
Action and Behavior
Body Perception
Body Applications
Pose and Motion Priors
Clothing Models (2011-2015)
Reflectance Filtering
Learning on Manifolds
Markerless Animal Motion Capture
Multi-Camera Capture
2D Pose from Optical Flow
Body Perception
Neural Prosthetics and Decoding
Part-based Body Models
Intrinsic Depth
Lie Bodies
Layers, Time and Segmentation
Understanding Action Recognition (JHMDB)
Intrinsic Video
Intrinsic Images
Action Recognition with Tracking
Neural Control of Grasping
Flowing Puppets
Faces
Deformable Structures
Model-based Anthropometry
Modeling 3D Human Breathing
Optical flow in the LGN
FlowCap
Smooth Loops from Unconstrained Video
PCA Flow
Efficient and Scalable Inference
Motion Blur in Layers
Facade Segmentation
Smooth Metric Learning
Robust PCA
3D Recognition
Object Detection
Optical Flow and Human Action
] from synthetically generated flow fields and find that this generalizes to real videos of human movement. (Bottom) We fine tune an optical flow algorithm to produce flow that improves action recognition []. (Left columns) SpyNet. (Right columns) FlowNet. In each set, left to right: first image in sequence, original flow, flow when trained on action recognition, differences in the flow are focused on the human action.Understanding human action requires modeling and understanding human movement. While we mostly focus on 3D human movement, what is directly observable in videos is the 2D optical flow. Previous work has shown that flow is useful for action recognition and, consequently, we explore how to better estimate human flow and improve action recognition.
Specifically, we train a neural network to compute single-human [] and multi-human [] optical flow. To enable this we create a new synthetic training database of image sequences with ground-truth human optical flow. For this we use the 3D SMPL body model, motion-capture data, and computer graphics to synthesize realistic flow fields; this effectively extends the SURREAL dataset []. We then train a convolutional neural network (SpyNet []) to estimate human optical flow from pairs of images.
The new network is more accurate than a wide range of top methods on held-out test data and generalizes well to real image sequences. When combined with a person detector/tracker, the approach provides a full solution to the problem of 2D human flow estimation.
Most of the top-performing action-recognition methods use optical flow as a ``black box'' input. In [], we take a deeper look at the combination of flow and action recognition, and find that: 1) optical flow is useful for action recognition because it is invariant to appearance, 2) flow accuracy at boundaries and for small displacements is most correlated with action-recognition performance, 3) training optical flow needs to minimize classification error instead of the popular end-point-error (EPE) to improve action recognition, and 4) optical flow learned for action recognition differs from traditional optical flow mostly inside and at the boundary of human bodies.
Members
Publications