Variational Autoencoders Pursue PCA Directions (by Accident)
The Variational Autoencoder (VAE) is a powerful architecture capable of representation learning and generative modeling. When it comes to learning interpretable (disentangled) representations, VAE and its variants show unparalleled performance. However, the reasons for this are unclear, since a very particular alignment of the latent embedding is needed but the design of the VAE does not encourage it in any explicit way. We address this matter and offer the following explanation: the diagonal approximation in the encoder together with the inherent stochasticity force local orthogonality of the decoder. The local behavior of promoting both reconstruction and orthogonality matches closely how the PCA embedding is chosen. Alongside providing an intuitive understanding, we justify the statement with full theoretical analysis as well as with experiments.
| Author(s): | Michal Rolinek and Dominik Zietlow and Georg Martius |
| Book Title: | Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) |
| Pages: | 12406-12415 |
| Year: | 2019 |
| Month: | June |
| Project(s): | |
| Bibtex Type: | Conference Paper (inproceedings) |
| Event Name: | IEEE International Conference on Computer Vision and Pattern Recognition (CVPR) 2019 |
| Event Place: | Long Beach, USA |
| URL: | http://openaccess.thecvf.com/content_CVPR_2019/papers/Rolinek_Variational_Autoencoders_Pursue_PCA_Directions_by_Accident_CVPR_2019_paper.pdf |
| Electronic Archiving: | grant_archive |
| Supplement: | http://openaccess.thecvf.com/content_CVPR_2019/supplemental/Rolinek_Variational_Autoencoders_Pursue_CVPR_2019_supplemental.pdf |
| Links: | |
BibTex
@inproceedings{RolinekZietlowMartius:VAERecPCA,
title = {Variational Autoencoders Pursue PCA Directions (by Accident)},
booktitle = {Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)},
abstract = {The Variational Autoencoder (VAE) is a powerful architecture capable of representation learning and generative modeling. When it comes to learning interpretable (disentangled) representations, VAE and its variants show unparalleled performance.
However, the reasons for this are unclear, since a very particular alignment of the latent embedding is needed but the design of the VAE does not encourage it in any explicit way.
We address this matter and offer the following explanation: the diagonal approximation in the encoder together with the inherent stochasticity force local orthogonality of the decoder. The local behavior of promoting both reconstruction and orthogonality matches closely how the PCA embedding is chosen. Alongside providing an intuitive understanding, we justify the statement with full theoretical analysis as well as with experiments.},
pages = {12406-12415},
month = jun,
year = {2019},
slug = {rolinekzietlowmartius-vaerecpca},
author = {Rolinek, Michal and Zietlow, Dominik and Martius, Georg},
url = {http://openaccess.thecvf.com/content_CVPR_2019/papers/Rolinek_Variational_Autoencoders_Pursue_PCA_Directions_by_Accident_CVPR_2019_paper.pdf},
month_numeric = {6}
}