Institute Homepage DE Sign In

Back

Empirical Inference Conference Paper 2008

Episodic Reinforcement Learning by Logistic Reward-Weighted Regression

Thumb ticker sm 12009745 10103538825457245 7502907506146263960 n
Empirical Inference
Jan Peters
Research Group Leader

It has been a long-standing goal in the adaptive control community to reduce the generically difficult, general reinforcement learning (RL) problem to simpler problems solvable by supervised learning. While this approach is today’s standard for value function-based methods, fewer approaches are known that apply similar reductions to policy search methods. Recently, it has been shown that immediate RL problems can be solved by reward-weighted regression, and that the resulting algorithm is an expectation maximization (EM) algorithm with strong guarantees. In this paper, we extend this algorithm to the episodic case and show that it can be used in the context of LSTM recurrent neural networks (RNNs). The resulting RNN training algorithm is equivalent to a weighted self-modeling supervised learning technique. We focus on partially observable Markov decision problems (POMDPs) where it is essential that the policy is nonstationary in order to be optimal. We show that this new reward-weighted logistic regression u sed in conjunction with an RNN architecture can solve standard benchmark POMDPs with ease.

Author(s): Wierstra, D. and Schaul, T. and Peters, J. and Schmidhuber, J.
Book Title: ICANN 2008
Journal: Artificial Neural Networks: ICANN 2008
Pages: 407-416
Year: 2008
Month: September
Day: 0
Editors: Kurkova-Pohlova, V. , R. Neruda, J. Koutnik
Publisher: Springer
Bibtex Type: Conference Paper (inproceedings)
Address: Berlin, Germany
DOI: 10.1007/978-3-540-87536-9_42
Event Name: 18th International Conference on Artificial Neural Networks
Event Place: Praha, Czech Republic
Digital: 0
Electronic Archiving: grant_archive
Institution: European Neural Network Society
Language: en
Organization: Max-Planck-Gesellschaft
School: Biologische Kybernetik
Links:

BibTex 

@inproceedings{5168,
  title = {Episodic Reinforcement Learning by Logistic Reward-Weighted Regression},
  journal = {Artificial Neural Networks: ICANN 2008},
  booktitle = {ICANN 2008},
  abstract = {It has been a long-standing goal in the adaptive control community to reduce the generically difficult, general reinforcement learning (RL) problem to simpler problems solvable by supervised learning. While this approach is today’s standard for value function-based methods, fewer approaches are known that apply similar reductions to policy search methods. Recently, it has been shown that immediate RL problems can be solved by reward-weighted regression, and that the resulting algorithm is an expectation maximization (EM) algorithm with strong guarantees. In this paper, we extend this algorithm to the episodic case and show that it can be used in the context of LSTM recurrent neural networks (RNNs). The resulting RNN training algorithm is equivalent to a weighted self-modeling supervised learning technique. We focus on partially observable Markov decision problems (POMDPs) where it is essential that the policy is nonstationary in order to be optimal. We show that this new reward-weighted logistic regression u
  sed in conjunction with an RNN architecture can solve standard benchmark POMDPs with ease.},
  pages = {407-416},
  editors = {Kurkova-Pohlova, V. , R. Neruda, J. Koutnik},
  publisher = {Springer},
  organization = {Max-Planck-Gesellschaft},
  institution = {European Neural Network Society},
  school = {Biologische Kybernetik},
  address = {Berlin, Germany},
  month = sep,
  year = {2008},
  slug = {5168},
  author = {Wierstra, D. and Schaul, T. and Peters, J. and Schmidhuber, J.},
  month_numeric = {9}
}