In the past 20 years impressive advances have been made in capturing, modeling and tracking the human body in 3D, thanks to the availability of large amounts of 3D body scans and mocap data. Animals have received much less attention, despite many applications in biomechanics, biology, neuroscience, robotics, smart farming, and entertainment. The main reason for the lack of methods for the 3D modeling and tracking of animals is that the methods derived for the human body cannot be easily applied to animals: animals are not cooperative, cannot be brought to the lab in large numbers, and current scanners cannot be taken into the wild. It is also challenging to capture significant motion of animals using motion capture equipment. 

In this project we develop methods to learn 3D articulated statistical shape models that can represent a wide variety of species in the animal kingdom, allowing intra- and inter-species analysis of 3D shape and the automatic and non-invasive assessment of animal shape and pose from images and video.

From scans of toy animals, we learn the SMAL (Skinned Multi Animal Linear) model [], a 3D articulated statistical shape model able to represent animal shapes for different species: big cats, dogs, cows, horses, zebras, and hippos. To capture animals outside the SMAL space, we developed SMALR (SMAL with Refinement) []. SMALR estimates a detailed 3D textured mesh using a small set of uncalibrated, non-simultaneous images of the animal. We are also developing species-specific models for dogs, rats and horses exploiting different modalities.

Today animal motion is mostly captured indoors for domestic species with marker-based systems. We are exploiting our 3D articulated animal shape models to develop markerless motion capture systems that can capture the shape and articulated motion of wild animals in their natural environment. In this context, we have applied our technology to capture the shape, pose, and texture of the Grevy's zebra from in-the-wild images [] using a novel neural-network regressor. The approach learns the zebra shape space during training using a photometric loss.

We have also focused on domestic animal species of particular relevance. Specifically, we have addressed the problem of 3D dog reconstruction from a single image in two works. In BARC [], we explored how incorporating dog breed information for training images enables learning a network that better estimates 3D shapes. In a subsequent work, BITE [], we tackled the challenge of 3D dog reconstruction for complex poses, such as sitting and lying down, by leveraging contact information.

Among domestic animals, horses are arguably the most interesting and widely studied. We have created the first 4D scanner for horses and, in collaboration with the Swedish University of Agricultural Sciences (SLU), have so far scanned more than 150 subjects. Additionally, we developed the first 3D articulated parametric shape model for animals, learned from real subjects. Our model, VAREN [], can also simulate muscle deformation during motion. Furthermore, we captured a unique dataset of horse motion, PFERD [], by employing dense motion capture on a diverse set of horses with varying shapes, performing both common and complex, uncommon poses and motions.

In AWOL [] we have addressed how to control parametric models of animals and trees using language, such that 3D models of species not seen during training can be easily created.