Human Pose, Shape and Action
3D Pose from Images
2D Pose from Images
Beyond Motion Capture
Action and Behavior
Body Perception
Body Applications
Pose and Motion Priors
Clothing Models (2011-2015)
Reflectance Filtering
Learning on Manifolds
Markerless Animal Motion Capture
Multi-Camera Capture
2D Pose from Optical Flow
Body Perception
Neural Prosthetics and Decoding
Part-based Body Models
Intrinsic Depth
Lie Bodies
Layers, Time and Segmentation
Understanding Action Recognition (JHMDB)
Intrinsic Video
Intrinsic Images
Action Recognition with Tracking
Neural Control of Grasping
Flowing Puppets
Faces
Deformable Structures
Model-based Anthropometry
Modeling 3D Human Breathing
Optical flow in the LGN
FlowCap
Smooth Loops from Unconstrained Video
PCA Flow
Efficient and Scalable Inference
Motion Blur in Layers
Facade Segmentation
Smooth Metric Learning
Robust PCA
3D Recognition
Object Detection
Expressive Body Models
Until recently, human pose has often been represented by 10-12 body joints in 2D or 3D. This is inspired by Johannson's moving light displays, which showed that some human actions can be recognized from the motion of the major joints of the body. We have argued that such representations are too impoverished to model human behavior. Humans express their emotions through the surface of their face and manipulate the world through the surface of their bodies.
Consequently, Perceiving Systems has focused on modeling and inferring 3D human pose and shape (HPS) using expressive 3D body models that capture the surface of the body either explicitly as a mesh or implicitly as a neural network. Such 3D shape models allow us to capture human-scene contact and provide information about a person related to their health, age, fitness, and clothing size.
We introduced the SMPL body model in 2015 [
] and made it available for research and commercial licensing. SMPL is realistic, efficient, posable, and compatible with most graphics packages. It is also differentiable and easy to integrate into optimization or deep learning methods. Since its release, it has become the de facto standard in the field and is widely used in industry and academia.
SMPL also has limitations, some of which have been addressed by STAR [], which is learned from thousands more 3D body scans and has local pose corrective blend shapes.
We have steadily improved on SMPL adding hands and faces to create SMPL-X []. Most recently we have combined this with the detailed facial model from DECA [] to increase expressive realism. We always combine these models with methods to estimate them from images. Our most recent neural regression recent method, PIXIE [], uses a moderator to assess the reliability of face and hand regressors before integrating the body, face, and hand features.
Current work is extending these models to include clothing. For example, CAPE [] uses a convolutional mesh VAE to learn a generative model of clothing that is compatible with SMPL. See also our work on learning implicit models of clothed 3D humans.
This work on modeling humans is the foundation for our analysis of human movement, emotion, and behavior.
Members
Publications