Empirical Inference Conference Paper 2006

Estimating Predictive Variances with Kernel Ridge Regression

In many regression tasks, in addition to an accurate estimate of the conditional mean of the target distribution, an indication of the predictive uncertainty is also required. There are two principal sources of this uncertainty: the noise process contaminating the data and the uncertainty in estimating the model parameters based on a limited sample of training data. Both of them can be summarised in the predictive variance which can then be used to give confidence intervals. In this paper, we present various schemes for providing predictive variances for kernel ridge regression, especially in the case of a heteroscedastic regression, where the variance of the noise process contaminating the data is a smooth function of the explanatory variables. The use of leave-one-out cross-validation is shown to eliminate the bias inherent in estimates of the predictive variance. Results obtained on all three regression tasks comprising the predictive uncertainty challenge demonstrate the value of this approach.

Author(s): Cawley, GC. and Talbot, NLC. and Chapelle, O.
Book Title: MLCW 2005
Journal: Machine Learning Challenges: First PASCAL Machine Learning Challenges Workshop (MLCW 2005)
Pages: 56-77
Year: 2006
Month: April
Day: 0
Editors: Quinonero-Candela, J. , I. Dagan, B. Magnini, F. D‘Alché-Buc
Publisher: Springer
Bibtex Type: Conference Paper (inproceedings)
Address: Berlin, Germany
DOI: 10.1007/11736790_5
Event Name: First PASCAL Machine Learning Challenges Workshop
Event Place: Southampton, United Kingdom
Digital: 0
Electronic Archiving: grant_archive
Language: en
Organization: Max-Planck-Gesellschaft
School: Biologische Kybernetik
Links:

BibTex

@inproceedings{3643,
  title = {Estimating Predictive Variances with Kernel Ridge Regression},
  journal = {Machine Learning Challenges: First PASCAL Machine Learning Challenges Workshop (MLCW 2005)},
  booktitle = {MLCW 2005},
  abstract = {In many regression tasks, in addition to an accurate estimate
  of the conditional mean of the target distribution, an indication of the
  predictive uncertainty is also required. There are two principal sources
  of this uncertainty: the noise process contaminating the data and the
  uncertainty in estimating the model parameters based on a limited sample
  of training data. Both of them can be summarised in the predictive
  variance which can then be used to give confidence intervals. In this paper,
  we present various schemes for providing predictive variances for
  kernel ridge regression, especially in the case of a heteroscedastic regression,
  where the variance of the noise process contaminating the data is
  a smooth function of the explanatory variables. The use of leave-one-out
  cross-validation is shown to eliminate the bias inherent in estimates of
  the predictive variance. Results obtained on all three regression tasks
  comprising the predictive uncertainty challenge demonstrate the value
  of this approach.},
  pages = {56-77},
  editors = {Quinonero-Candela, J. , I. Dagan, B. Magnini, F. D‘Alché-Buc},
  publisher = {Springer},
  organization = {Max-Planck-Gesellschaft},
  school = {Biologische Kybernetik},
  address = {Berlin, Germany},
  month = apr,
  year = {2006},
  slug = {3643},
  author = {Cawley, GC. and Talbot, NLC. and Chapelle, O.},
  month_numeric = {4}
}