Institute Homepage
Institute Homepage DE Sign In

Back

Autonomous Learning Conference Paper 2019

Variational Autoencoders Pursue PCA Directions (by Accident)

Thumb ticker sm michal
Autonomous Learning
Michal Rolinek
Thumb ticker sm profile
Empirical Inference
Dominik Zietlow
  • Guest Scientist
Thumb ticker sm georg 2018 crop small
Empirical Inference, Autonomous Learning
Georg Martius
Senior Research Scientist

The Variational Autoencoder (VAE) is a powerful architecture capable of representation learning and generative modeling. When it comes to learning interpretable (disentangled) representations, VAE and its variants show unparalleled performance. However, the reasons for this are unclear, since a very particular alignment of the latent embedding is needed but the design of the VAE does not encourage it in any explicit way. We address this matter and offer the following explanation: the diagonal approximation in the encoder together with the inherent stochasticity force local orthogonality of the decoder. The local behavior of promoting both reconstruction and orthogonality matches closely how the PCA embedding is chosen. Alongside providing an intuitive understanding, we justify the statement with full theoretical analysis as well as with experiments.

Author(s): Michal Rolinek and Dominik Zietlow and Georg Martius
Book Title: Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)
Pages: 12406-12415
Year: 2019
Month: June
Project(s):
Bibtex Type: Conference Paper (inproceedings)
Event Name: IEEE International Conference on Computer Vision and Pattern Recognition (CVPR) 2019
Event Place: Long Beach, USA
URL: http://openaccess.thecvf.com/content_CVPR_2019/papers/Rolinek_Variational_Autoencoders_Pursue_PCA_Directions_by_Accident_CVPR_2019_paper.pdf
Electronic Archiving: grant_archive
Supplement: http://openaccess.thecvf.com/content_CVPR_2019/supplemental/Rolinek_Variational_Autoencoders_Pursue_CVPR_2019_supplemental.pdf
Links:

BibTex 

@inproceedings{RolinekZietlowMartius:VAERecPCA,
  title = {Variational Autoencoders Pursue PCA Directions (by Accident)},
  booktitle = {Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)},
  abstract = {The Variational Autoencoder (VAE) is a powerful architecture capable of representation learning and generative modeling. When it comes to learning interpretable (disentangled) representations, VAE and its variants show unparalleled performance.
  However, the reasons for this are unclear, since a very particular alignment of the latent embedding is needed but the design of the VAE does not encourage it in any explicit way.
  We address this matter and offer the following explanation: the diagonal approximation in the encoder together with the inherent stochasticity force local orthogonality of the decoder. The local behavior of promoting both reconstruction and orthogonality matches closely how the PCA embedding is chosen. Alongside providing an intuitive understanding, we justify the statement with full theoretical analysis as well as with experiments.},
  pages = {12406-12415},
  month = jun,
  year = {2019},
  slug = {rolinekzietlowmartius-vaerecpca},
  author = {Rolinek, Michal and Zietlow, Dominik and Martius, Georg},
  url = {http://openaccess.thecvf.com/content_CVPR_2019/papers/Rolinek_Variational_Autoencoders_Pursue_PCA_Directions_by_Accident_CVPR_2019_paper.pdf},
  month_numeric = {6}
}