RayNet: Learning Volumetric 3D Reconstruction with Ray Potentials
In this paper, we consider the problem of reconstructing a dense 3D model using images captured from different views. Recent methods based on convolutional neural networks (CNN) allow learning the entire task from data. However, they do not incorporate the physics of image formation such as perspective geometry and occlusion. Instead, classical approaches based on Markov Random Fields (MRF) with ray-potentials explicitly model these physical processes, but they cannot cope with large surface appearance variations across different viewpoints. In this paper, we propose RayNet, which combines the strengths of both frameworks. RayNet integrates a CNN that learns view-invariant feature representations with an MRF that explicitly encodes the physics of perspective projection and occlusion. We train RayNet end-to-end using empirical risk minimization. We thoroughly evaluate our approach on challenging real-world datasets and demonstrate its benefits over a piece-wise trained baseline, hand-crafted models as well as other learning-based approaches.
| Author(s): | Despoina Paschalidou and Ali Osman Ulusoy and Carolin Schmitt and Luc Gool and Andreas Geiger |
| Book Title: | IEEE Conference on Computer Vision and Pattern Recognition (CVPR) |
| Year: | 2018 |
| Publisher: | IEEE Computer Society |
| Project(s): | |
| Bibtex Type: | Conference Paper (inproceedings) |
| Event Name: | IEEE International Conference on Computer Vision and Pattern Recognition (CVPR) 2018 |
| Event Place: | Salt Lake City, USA |
| Electronic Archiving: | grant_archive |
| Links: | |
BibTex
@inproceedings{Paschalidou2018CVPR,
title = {RayNet: Learning Volumetric 3D Reconstruction with Ray Potentials},
booktitle = {IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
abstract = {In this paper, we consider the problem of reconstructing a dense 3D model using images captured from different views. Recent methods based on convolutional neural networks (CNN) allow learning the entire task from data. However, they do not incorporate the physics of image formation such as perspective geometry and occlusion. Instead, classical approaches based on Markov Random Fields (MRF) with ray-potentials explicitly model these physical processes, but they cannot cope with large surface appearance variations across different viewpoints. In this paper, we propose RayNet, which combines the strengths of both frameworks. RayNet integrates a CNN that learns view-invariant feature representations with an MRF that explicitly encodes the physics of perspective projection and occlusion. We train RayNet end-to-end using empirical risk minimization. We thoroughly evaluate our approach on challenging real-world datasets and demonstrate its benefits over a piece-wise trained baseline, hand-crafted models as well as other learning-based approaches.},
publisher = {IEEE Computer Society},
year = {2018},
slug = {paschalidou2018cvpr},
author = {Paschalidou, Despoina and Ulusoy, Ali Osman and Schmitt, Carolin and Gool, Luc and Geiger, Andreas}
}