Empirical Inference Conference Paper 2009

PAC-Bayesian Generalization Bound for Density Estimation with Application to Co-clustering

We derive a PAC-Bayesian generalization bound for density estimation. Similar to the PAC-Bayesian generalization bound for classification, the result has the appealingly simple form of a tradeoff between empirical performance and the KL-divergence of the posterior from the prior. Moreover, the PAC-Bayesian generalization bound for classification can be derived as a special case of the bound for density estimation. To illustrate a possible application of our bound we derive a generalization bound for co-clustering. The bound provides a criterion to evaluate the ability of co-clustering to predict new co-occurrences, thus introducing a supervised flavor to this traditionally unsupervised task.

Author(s): Seldin, Y. and Tishby, N.
Book Title: JMLR Workshop and Conference Proceedings Volume 5: AISTATS 2009
Journal: In the proceedings of the 12th International Conference on Artificial Intelligence and Statistics (AISTATS 2009)
Pages: 472-479
Year: 2009
Month: April
Day: 0
Publisher: MIT Press
Bibtex Type: Conference Paper (inproceedings)
Address: Cambridge, MA, USA
Event Name: 12th International Conference on Artificial Intelligence and Statistics
Event Place: Clearwater Beach, FL, USA
Digital: 0
Electronic Archiving: grant_archive
Language: en
Organization: Max-Planck-Gesellschaft
School: Biologische Kybernetik
Links:

BibTex

@inproceedings{6592,
  title = {PAC-Bayesian Generalization Bound for Density Estimation with Application to Co-clustering},
  journal = {In the proceedings of the 12th International Conference on Artificial Intelligence and Statistics (AISTATS 2009)},
  booktitle = {JMLR Workshop and Conference Proceedings Volume 5: AISTATS 2009},
  abstract = {We derive a PAC-Bayesian generalization bound for density estimation. Similar to the PAC-Bayesian generalization bound for classification, the result has the appealingly simple form of a tradeoff between empirical performance and the KL-divergence of the posterior from the prior. Moreover, the PAC-Bayesian generalization bound for classification can be derived as a special case of the bound for density estimation. To illustrate a possible application of our bound we derive a generalization bound for co-clustering. The bound provides a criterion to evaluate the ability of co-clustering to predict new co-occurrences, thus introducing a supervised flavor to this traditionally unsupervised task.},
  pages = {472-479},
  publisher = {MIT Press},
  organization = {Max-Planck-Gesellschaft},
  school = {Biologische Kybernetik},
  address = {Cambridge, MA, USA},
  month = apr,
  year = {2009},
  slug = {6592},
  author = {Seldin, Y. and Tishby, N.},
  month_numeric = {4}
}