Header logo is

Generating Human Interaction Motions in Scenes with Text Control

2024

Conference Paper

ps


We present TeSMo, a method for text-controlled scene-aware motion generation based on denoising diffusion models. Previous text-to-motion methods focus on characters in isolation without considering scenes due to the limited availability of datasets that include motion, text descriptions, and interactive scenes. Our approach begins with pre-training a scene-agnostic text-to-motion diffusion model, emphasizing goal-reaching constraints on large-scale motion-capture datasets. We then enhance this model with a scene-aware component, fine-tuned using data augmented with detailed scene information, including ground plane and object shapes. To facilitate training, we embed annotated navigation and interaction motions within scenes. The proposed method produces realistic and diverse human-object interactions, such as navigation and sitting, in different scenes with various object shapes, orientations, initial body positions, and poses. Extensive experiments demonstrate that our approach surpasses prior techniques in terms of the plausibility of human-scene interactions, as well as the realism and variety of the generated motions.

Author(s): Yi, Hongwei and Thies, Justus and Black, Michael J. and Peng, Xue Bin and Rempe, Davis
Book Title: European Conference on Computer Vision (ECCV 2024)
Year: 2024
Month: October
Series: LNCS
Publisher: Springer Cham

Department(s): Perceiving Systems
Bibtex Type: Conference Paper (inproceedings)
Paper Type: Conference

Event Place: Milan, Italy

Links: pdf
project

BibTex

@inproceedings{tesmo:2024,
  title = {Generating Human Interaction Motions in Scenes with Text Control},
  author = {Yi, Hongwei and Thies, Justus and Black, Michael J. and Peng, Xue Bin and Rempe, Davis},
  booktitle = {European Conference on Computer Vision (ECCV 2024)},
  series = {LNCS},
  publisher = {Springer Cham},
  month = oct,
  year = {2024},
  doi = {},
  month_numeric = {10}
}