Institute Homepage
Institute Homepage DE Sign In

Back

Embodied Vision Conference Paper 2024

Physics-Based Rigid Body Object Tracking and Friction Filtering From RGB-D Videos

no image
Embodied Vision
Rama Kandukuri
Thumb ticker sm dsc 3874 2
Embodied Vision
Michael Strecke
Research Associate
Thumb ticker sm profilepic stueckler
Embodied Vision
Jörg Stückler
Max Planck Research Group Leader

Physics-based understanding of object interactions from sensory observations is an essential capability in augmented reality and robotics. It enables to capture the properties of a scene for simulation and control. In this paper, we propose a novel approach for real-to-sim which tracks rigid objects in 3D from RGB-D images and infers physical properties of the objects. We use a differentiable physics simulation as state-transition model in an Extended Kalman Filter which can model contact and friction for arbitrary mesh-based shapes and in this way estimate physically plausible trajectories. We demonstrate that our approach can filter position, orientation, velocities, and concurrently can estimate the coefficient of friction of the objects. We analyze our approach on various sliding scenarios in synthetic image sequences of single objects and colliding objects. We also demonstrate and evaluate our approach on a real-world dataset. We make our novel benchmark datasets publicly available to foster future research in this novel problem setting and comparison with our method.

Author(s): Kandukuri, Rama Krishna and Strecke, Michael and Stueckler, Joerg
Book Title: Proceedings of the International Conference on 3D Vision (3DV)
Year: 2024
Project(s):
Bibtex Type: Conference Paper (inproceedings)
DOI: 10.1109/3DV62453.2024.00111
State: Published
URL: https://doi.org/10.1109/3DV62453.2024.00111
Electronic Archiving: grant_archive
Links:

BibTex 

@inproceedings{kandukuri2023ekfphys,
  title = {Physics-Based Rigid Body Object Tracking and Friction Filtering From RGB-D Videos},
  booktitle = {Proceedings of the International Conference on 3D Vision (3DV)},
  abstract = {Physics-based understanding of object interactions from sensory observations is an essential capability in augmented reality and robotics. It enables to capture the properties of a scene for simulation and control. In this paper, we propose a novel approach for real-to-sim which tracks rigid objects in 3D from RGB-D images and infers physical properties of the objects. We use a differentiable physics simulation as state-transition model in an Extended Kalman Filter which can model contact and friction for arbitrary mesh-based shapes and in this way estimate physically plausible trajectories. We demonstrate that our approach can filter position, orientation, velocities, and concurrently can estimate the coefficient of friction of the objects. We analyze our approach on various sliding scenarios in synthetic image sequences of single objects and colliding objects. We also demonstrate and evaluate our approach on a real-world dataset. We make our novel benchmark datasets publicly available to foster future research in this novel problem setting and comparison with our method.},
  year = {2024},
  slug = {kandukuri2023ekfphys},
  author = {Kandukuri, Rama Krishna and Strecke, Michael and Stueckler, Joerg},
  url = {https://doi.org/10.1109/3DV62453.2024.00111}
}