Institute Homepage
Institute Homepage EN Sign In

Perceiving Systems PS:License 1.0 GrabNet: Generating 3D hand grasps for unseen 3D objects There is a significant interest in the community in training models to grasp 3D objects. This is important for example for interacting human avatars, as well as for robotic grasping by imitating human grasping. We use our GRAB dataset (see entry above) of whole-body grasps, and extract hand-only information. We then train on this our deep-net model GrabNet to generate 3D hand grasps, using our hand model MANO, for unseen 3D objects. We provide both the GrabNet model and its training dataset for research purposes.
Thumb ticker sm grab thumb3
Perceiving Systems PS:License 1.0 ExPose: EXpressive POse and Shape rEgression Training models to quickly and accurately estimate expressive humans (SMPL-X) from an RGB image, including the main body, face and hands, is challenging for a number of reasons.  First, there exists no dataset with paired images and ground truth SMPL-X annotations.  Secondly, the face and hands take up much fewer pixels than the main body, making inference harder.  Third, full body images are further downsampled to use with contemporary methods. Here we provide the first dataset of 32.617 pairs of: (1) an in-the-wild RGB image, and (2) an expressive whole-body 3D human reconstruction (SM...
Thumb ticker sm merged2
Perceiving Systems PS:License 1.0 Generating 3D People in Scenes without People Our PSI system aims to generate 3D people in a 3D scene from the view of an agent. The system takes as input the depth and the semantic segmentation from a camera view, and generates plausible SMPL-X body meshes, which are naturally posed in the 3D scene. Scripts of data pre-processing, training, fitting, evaluation and visualization, as well as the data, are incorporated.
Thumb ticker sm yancvpr
Perceiving Systems PS:License 1.0 CAPE: Dressing SMPL CAPE provides a "dressed SMPL" body model. We train CAPE as a conditional Mesh-VAE-GAN to learn the clothing deformation from the SMPL body model, making clothing an additional term on SMPL. CAPE is conditioned on both pose and clothing type, giving the ability to draw samples of clothing to dress different body shapes in a variety of styles and poses.
Thumb ticker sm cape teaser ps website v3
Software Workshop Autonomous Motion The 3-Clause BSD License CityGraph A Python framework for representing a city (real or virtual) and moving around it.
Thumb ticker sm tuebingen altstadt
Software Workshop Haptic Intelligence Haptipedia An online, open-source, visualization of a growing database of haptic devices invented since 1992.
Thumb ticker sm screenshot 2020 04 06 at 08.51.55
Perceiving Systems PS:License 1.0 VIBE: Video Inference for Human Body Pose and Shape Estimation VIBE is a neural network method that takes video of a human in motion as input and outputs the 3D pose and shape of the body in every frame. The output is in SMPL body format and represents the state of the art at time of release. The method runs quickly and can process arbitrary sequence lengths. The trained model is available now and training code will be provided later.
Thumb ticker sm vibe
Software Workshop NuGridPy Python framework for analyzing data produced by stellar astrophysical codes.
Thumb ticker sm abu chart
Perceiving Systems The MIT License Efficient Learning on Point Clouds with Basis Point Sets Basis Point Set (BPS) is a simple and efficient method for encoding 3D point clouds into fixed-length representations. It is based on a simple idea: select k fixed points in space and compute vectors from these basis points to the nearest points in a point cloud; use these vectors (or simply their norms) as features. The basis points are kept fixed for all the point clouds in the dataset, providing a fixed representation of every point cloud as a vector. This representation can then be used as input to arbitrary machine learning methods, in particular it can be used as input to off-...
Thumb ticker sm bps teaser2
Perceiving Systems The MIT License Markerless Outdoor Human Motion Capture Using Multiple Autonomous Micro Aerial Vehicles Capturing human motion in natural scenarios means moving motion capture out of the lab and into the wild. Typical approaches rely on fixed, calibrated, cameras and reflective markers on the body, significantly limiting the motions that can be captured. To make motion capture truly unconstrained, we describe the first fully autonomous outdoor capture system based on flying vehicles. We use multiple micro-aerial-vehicles(MAVs), each equipped with a monocular RGB camera, an IMU, and a GPS receiver module. These detect the person, optimize their position, and localize themselves approximately. ...
Thumb ticker sm content iccv 2 cover crop
Perceiving Systems PS:License 1.0 SPIN: Human pose and shape from an image SPIN is a state-of-the-art deep network for regressing SMPL body shape and pose parameters directly from an image. SPIN uses a novel training method that combines a bottom-up deep network with a top-down, model-based, fitting method. SMPLify model fitting is used in the loop with the DNN training to provide SMPL parameters used in the training loss. Code is available.
Thumb ticker sm spin2
Perceiving Systems PS:License 1.0 AMASS Dataset AMASS is a large dataset of human motions - 45 hours and growing. AMASS enables the training of deep neural networks to model human motion. AMASS unifies multiple datasets by fitting the SMPL body model to mocap markers. The dataset includes SMPL-H body shapes and poses as well as DMPL soft tissue motions. If you want to include your own mocap sequences in the dataset, please contact us. The release includes tutorial code for training DNNs with AMASS. Also the MoSh++ code is now available. We also release SOMA, our complementary tool for automatic mocap labeling.
Thumb ticker sm amass
Perceiving Systems The MIT License Three-D Safari: Learning to Estimate Zebra Pose, Shape, and Texture from Images "In the Wild" We present the first method to perform automatic 3D pose, shape and texture capture of animals from images acquired in-the-wild. In particular, we focus on the problem of capturing 3D information about Grevy's zebras from a collection of images. We integrate the recent SMAL animal model into a network-based regression pipeline, which we train end-to-end on synthetically generated images with pose, shape, and background variation. We couple 3D pose and shape prediction with the task of texture synthesis, obtaining a full texture map of the animal from a single image. The predicted textur...
Thumb ticker sm ps web
Software Workshop Sleep Learning An application for collecting, organizing, and post-processing EEG and EMG data from animal sleep.
Thumb ticker sm screenshot 2020 04 04 at 10.26.06
Perceiving Systems The MIT License Competitive Collaboration Competitive Collaboration is a generic framework in which networks learn to collaborate and compete, thereby achieving specific goals. Competitive Collaboration is a three player game consisting of two players competing for a resource that is regulated by a third player, moderator. This framework is similar in spirit to expectation-maximization (EM) but is formulated for neural network training.
Thumb ticker sm mac
Software Workshop Rationality Enhancement Attention Training A productivity application for tracking and training the ability of the user to focus on a chosen task.
Thumb ticker sm actrac
Perceiving Systems PS:License 1.0 Expressive Body Capture: 3D Hands, Face, and Body from a Single Image SMPL-X is a major update to the SMPL body model that adds an expressive face and fully articulated hands. If you use SMPL, this is a straightforward upgrade that improves realism and allows you to capture facial expressions and gestures. We also provide SMPLify-X to estimate SMPL-X from a single image. This is a major update to SMPlify in several senses: (1) we detect 2D features corresponding to the face, hands, and feet and fit the full SMPL-X model to these; (2) we train a new neural network pose prior using a large MoCap dataset; (3) we define a new interpenetration penalty that i...
Thumb ticker sm thumb small smplex
Perceiving Systems PS:License 1.0 Learning joint reconstruction of hands and manipulated objects Estimating hand-object manipulation is essential for interpreting and imitating human actions. Previous work has made significant progress towards reconstruction of hand poses and object shapes in isolation. Yet, reconstructing hands and objects during manipulation is a more challenging task due to significant occlusions of both the hand and object. While presenting challenges, manipulations may also simplify the problem since the physics of contact restricts the space of valid hand-object configurations. For example, during manipulation, the hand and object should be in contact but not int...
Thumb ticker sm thumb small obman
Perceiving Systems The MIT License RingNet: Learning to Regress 3D Face Shape and Expression from an Image without 3D Supervision Code: We provide the inference code of RingNet. Please check the repository which is self explanatory. NoW Benchmark Dataset and Challenge: Please check the external link to download the data and participate in the challenge.
Thumb ticker sm ringnet teaset
Software Workshop Code Cov A tool to analyze the coverage of python code by extracting and visualizing using an interactive web interface.
Thumb ticker sm coverage
Perceiving Systems The MIT License VOCA: Capture, Learning, and Synthesis of 3D Speaking Styles VOCA (Voice Operated Character Animation) is a framework that takes a speech signal as input and realistically animates a wide range of adult faces. <p><strong>Code: </strong>We provide Python demo code that outputs a 3D head animation given a speech signal and a static 3D head mesh. The codebase further provides animation control to alter the speaking style, identity-dependent facial shape, and head pose (i.e. head rotation around the neck) during animation. The code further demonstrates how to sample 3D head meshes from the publicly available FLAME model, that can then be animated&nbs...
Thumb ticker sm teaser voca
Perceiving Systems PS:License 1.0 The Virtual Caliper: Rapid Creation of Metrically Accurate Avatars from 3D Measurements The Virtual Caliper project provides you with software tools to rapidly generate metrically accurate avatars based on measurements. These avatars can be generated offline in FBX format for later import into game engines or alternatively within the game engine itself. Avatars are based on the SMPL body model and support skeletal mesh animation. Released software includes Unity project files and standalone binaries for Windows/Linux/macOS and Blender Python code for FBX generation.
Thumb ticker sm virtualcaliper
Software Workshop Perceiving Systems PS:License 1.0 Monocle Application used for capturing from the Kinect 2.0 device.
Thumb ticker sm monocle
Software Workshop Movement Generation and Control Robot Data Collector A web application for managing, organizing, and sharing robotic data.
Thumb ticker sm screenshot 2020 04 04 at 16.40.01
Perceiving Systems PS:License 1.0 Deep Inertial Poser: Learning to Reconstruct Human Pose from Sparse Inertial Measurements in Real Time This is the code for our SIGGRAPH Asia 2018 project <Deep Inertial Poser: Learning to Reconstruct Human Pose from Sparse Inertial Measurements in Real Time>. The BiRNN model training and testing parts along with real-time demo are released to facilitate reproductivity and future research. The large-scale synthetic dataset and real DIP-IMU we introduced in the paper are compatible with this code, and can be accessed via the project page.
Thumb ticker sm thumb lg dip final
Perceiving Systems PS:License 1.0 Skinned Multi-Infant Linear Model (SMIL) SMIL is a learned 3D model of infant body shape and pose that can be animated and fit to data. It is based on SMPL but the shape space is adapted to capture the body shape of babies.
Thumb ticker sm sample3 merge black
Perceiving Systems PS:License 1.0 3D Poses in the Wild Dataset The "3D Poses in the Wild dataset" is the first dataset with monocular hand-held video together with accurate 3D human poses for evaluation. Our method combines video and IMU to recover accurate 3D human body models and their projection into the video sequences. The dataset includes: 60 video sequences; 2D pose annotations; 3D poses obtained with our method; Camera poses for every frame in the sequences; 3D body scans; and 18 3D human models with different clothing variations.
Thumb ticker sm eccv18 teaser
Perceiving Systems The MIT License Convolutional Mesh Autoencoders The code allows to build convolutional networks on mesh structures analogous to CNNs on images. The code includes mesh convolutions, and introduces downsampling and upsampling operators that can be directly applied to the mesh structure. The code implements a Convolution Mesh Autoencoder using the above mesh processing operators and achieves state of the art results on generating 3D facial meshes.
Thumb ticker sm thumb lg coma faces
Perceiving Systems Software Workshop PS:License 1.0 Mesh Library When working in 3D graphics, one needs to load raw data, conduct various processing on it, visualize the results to help understanding, then save the output in different kinds of formats. Here we release the Mesh Library to facilitate all these aforementioned operations. This library is built on top of OpenGL and CGAL, with an easy-to-use Python interface. Other than the basic usages like data IO and interactive visualization, it also supports other more complex functionalities like texture rendering, visibility computation, and geometry arithmetic. We hope the release of this tool makes th...
Thumb ticker sm mesh library img
Perceiving Systems The MIT License Learning Human Optical Flow The optical flow of humans is well known to be useful for the analysis of human action. Given this, we devise an optical flow algorithm specifically for human motion and show that it is superior to generic flow methods. Designing a method by hand is impractical, so we develop a new training database of image sequences with ground truth optical flow. For this we use a 3D model of the human body and motion capture data to synthesize realistic flow fields. We then train a convolutional neural network to estimate human flow fields from pairs of images. Since many applications in human motion an...
Thumb ticker sm thumb lg persondetect copy
Perceiving Systems PS:License 1.0 SMALR: Capturing Animal Shape and Texture from Images The SMALR release includes an updated SMAL model of animals and 3D animal models recovered from images. SMALR is the Skinned Multi-Animal Linear Model with Refinement. All the 3D shapes from the CVPR paper are available for download as 3D meshes, which can be posed and animated. As we create new meshes, they will be added here.
Thumb ticker sm smalr
Software Workshop Perceiving Systems PS:License 1.0 Accelerated K-means An efficient and generic implementation of the k-means clustering algoriothm.
Thumb ticker sm norm25
Perceiving Systems PS:License 1.0 HMR: End-to-end Recovery of Human Shape and Pose Trained model to estimate 3D human shape and pose directly from an image. The input is pixels, and the output is a 3D body in SMPL format (shape parameters and pose parameters). Also provided is the code and data needed to train the model.
Thumb ticker sm hmr
Perceiving Systems Apache License, Version 2.0 Towards Accurate Marker-less Human Shape and Pose Estimation over Time Model-based reconstruction of 3D SMPL body shape and pose from multi-view images. 2D joints and silhouettes from multi-view are used in the process. And DCT-based temporal prior is utilized to regularize the recovered 3D joint trajectory.
Thumb ticker sm muvs
Perceiving Systems PS:License 1.0 FLAME: 3D model of facial shape and expression FLAME is a lightweight and expressive generic head model learned from over 33,000 of accurately aligned 3D scans. We provide the trained 3D face models, registrations for the dynamic D3DFACS dataset, and demo code in Chumpy and Tensorflow to load and sample the model, and to fit the model to 3D landmarks.
Thumb ticker sm flame
Perceiving Systems PS:License 1.0 MANO: 3D hand model Data, code and model. This includes over 1000 3D hand scans and aligned meshes, the learned 3D hand shape model, the full articulated hand model with pose-dependent blend shapes. Also included is the SMPL body model with the hands attached to it, providing a realistic hand and body model.
Thumb ticker sm hands2
Software Workshop Probabilistic Numerics Empirical Inference OpenPhd Guiding Algorithm using a learned motion correction based on a Gaussian Process to produce better images from a telescope.
Thumb ticker sm screenshot 2020 04 06 at 09.50.56
Perceiving Systems PS:License 1.0 BUFF: Bodies under Flowing Fashion, 4D dataset High quality 4D dataset of people in clothing with ground truth 3D shape. The BUFF dataset consists of 5 subjects, 3 male and 2 female wearing 2 clothing styles: a) t-shirt and long pants and b) a soccer outfit. They perform 3 different motions i) hips ii) tilt_twist_left iii) shoulders_mill.
Thumb ticker sm buff
Perceiving Systems PS:License 1.0 Dynamic FAUST This dataset is a unique resource containing over 40,000 4D scans of multiple people; 4D means 3D scans over time. Processing 4D data is challenging, so we provide aligned data in which we have registered a common template mesh to all scans. This alignment process takes into account geometry and surface texture to make it accurate. The dataset includes the raw scan data, registered template meshes, and masks that say where the template mesh is sufficiently accurate to be considered ground truth.
Thumb ticker sm teaser
Perceiving Systems PS:License 1.0 SMAL: 3D articulated model of animals shapes We provide the SMAL model of animal shapes and demo code. We also provide all the results from the CVPR paper of animal shapes estimated from images. We do not provide the 3D scans of the toy animals for copyright reasons but do provide a shopping list so that you can purchase the same toys that we used.
Thumb ticker sm smal
Perceiving Systems PS:License 1.0 SURREAL: Synthetic human dataset and trained networks First large-scale person dataset to generate depth, body parts, optical flow, 2D/3D pose, surface normals ground truth for RGB video input. The dataset contains 6M frames of synthetic humans. The images are photo-realistic renderings of people under large variations in shape, texture, view-point and pose. To ensure realism, the synthetic bodies are created using the SMPL body model, whose parameters are fit by the MoSh method given raw 3D MoCap marker data. Trained CNNs are also provided.
Thumb ticker sm surreal
Perceiving Systems PS:License 1.0 Unite the People – Closing the Loop Between 3D and 2D Human Representations The dataset includes annotations of common human pose datasets. These include 3D body pose, 91 surface and joint landmarks, foreground segmentation, and body part segments. Together with the images, these can be used to train neural networks for human pose estimation tasks, including 3D pose estimation. The 3D body is represented by SMPL. Training code is provided.
Thumb ticker sm teaser 2
Perceiving Systems PS:License 1.0 MR-Flow: Optical flow in mostly-rigid scenes Code for the paper "Optical Flow in Mostly Rigid Scenes" by Jonas Wulff, Laura Sevilla-Lara, Michael Black, CVPR 2017. This is one of the best performing methods across different datasets. In rigid parts of the scene, a plane-plus-parallax model is used. The method segments out the non-rigid regions and uses a more generic flow method there.
Thumb ticker sm mrflow
Software Workshop Autonomous Motion Movement Generation and Control Robot Analysis Infrastructure An application for analyzing simultaneously videos and sensors data from robotic experiments.
Thumb ticker sm screenshot 2020 04 04 at 16.30.44
Perceiving Systems PS:License 1.0 A Generative Model of People in Clothing We provide an image-based generative model of people in clothing for the full body. The training dataset is built on top of Chictopia10K. We provide processed annotations as well as the SMPL body model fit to the images. We also provide our trained models for download.
Thumb ticker sm lassner
Perceiving Systems Autonomous Vision PS:License 1.0 Slow Flow: Exploiting High-Speed Cameras for Accurate and Diverse Optical Flow Reference Data Existing optical flow datasets are limited in size and variability due to the difficulty of capturing dense ground truth. In this paper, we tackle this problem by tracking pixels through densely sampled space-time volumes recorded with a high-speed video camera. Our model exploits the linearity of small motions and reasons about occlusions from multiple frames. Using our technique, we are able to establish accurate reference flow fields outside the laboratory in natural environments. Besides, we show how our predictions can be used to augment the input images with realistic motion blur. We ...
Thumb ticker sm joel slow flow crop
Probabilistic Numerics Apache License, Version 2.0 Probabilistic Filtering ODE Solver A numerical ODE solver implemented in Matlab which returns a Gaussian probability distribution over the solution. It's reliable, fast and mostly API compatible.
Thumb ticker sm homepage
Software Workshop Perceiving Systems PS:License 1.0 Scan Manager A web application for being able to manage the increasing amount of body scans.
Thumb ticker sm screenshot 2020 04 06 at 10.19.32
Perceiving Systems PS:License 1.0 SPyNet: Optical Flow Estimation using a Spatial Pyramid Network We learn to compute optical flow by combining a classical spatial-pyramid formulation with deep learning. This estimates large motions in a coarse-to-fine approach by warping one image of a pair at each pyramid level by the current flow estimate and computing an update to the flow. Instead of the standard minimization of an objective function at each pyramid level, we train one deep network per level to compute the flow update. Check the website for updates; we provide code for the original SypNet as well as an end-to-end trainable version.
Thumb ticker sm pyramid
Software Workshop Empirical Inference Distributed brain-computer interface A brain-computer interface (BCI) to assist and interpret thoughts from patients suffering diseases such as amyotrophic lateral sclerosis. This monitoring tool is especially suited for research and for reaching patients living in remote locations.
Thumb ticker sm screenshot 2020 04 04 at 16.12.38