research-article

On discovery and learning of models with predictive representations of state for agents with continuous actions and observations

Authors:

Satinder SinghAuthors Info & Claims

AAMAS '07: Proceedings of the 6th international joint conference on Autonomous agents and multiagent systems

Article No.: 187, Pages 1 - 8

https://doi.org/10.1145/1329125.1329352

Published: 14 May 2007 Publication History

Abstract

Models of agent-environment interaction that use predictive state representations (PSRs) have mainly focused on the case of discrete observations and actions. The theory of discrete PSRs uses an elegant construct called the system dynamics matrix and derives the notion of predictive state as a sufficient statistic via the rank of the matrix. With continuous observations and actions, such a matrix and its rank no longer exist. In this paper, we show how to define an analogous construct for the continuous case, called the system dynamics distributions, and use information theoretic notions to define a sufficient statistic and thus state. Given this new construct, we use kernel density estimation to learn approximate system dynamics distributions from data, and use information-theoretic tools to derive algorithms for discovery of state and learning of model parameters. We illustrate our new modeling method on two example problems.

References

[1]

T. M. Cover and J. A. Thomas. Elements of Information Theory. Wiley-Interscience, 1991.

Digital Library

[2]

Y. Engel, S. Mannor, and R. Meir. Bayes meets bellman: The gaussian process approach to temporal difference learning. In ICML, 2003.

Digital Library

[3]

K. E. Hild, D. Erdogmus, and J. C. Principe. Blind source separation using renyi's mutual information. IEEE Signal Processing Letters, 8(6):174--176, 2001.

[4]

M. R. James and S. Singh. Learning and discovery of predictive state representations in dynamical systems with reset. In ICML, pages 417--424, 2004.

Digital Library

[5]

J. Kapur. Measures of Information and their Application. John Wiley, 1994.

[6]

M. L. Littman, R. S. Sutton, and S. Singh. Predictive representations of state. In NIPS, pages 1555--1561, 2002.

Digital Library

[7]

P. McCracken and M. Bowling. Online discovery and learning of predictive state representations. In NIPS, pages 875--882, 2006.

[8]

J. C. Principe, D. Xu, and J. W. Fisher. Information theoretic learning. pages 265--319, 1999.

[9]

E. J. Rafols, M. B. Ring, R. S. Sutton, and B. Tanner. Using predictive representations to improve generalization in reinforcement learning. In IJCAI, pages 835--840, 2005.

Digital Library

[10]

M. Rudary and S. Singh. Predictive linear-Gaussian models of controlled stochastic dynamical systems. In ICML, 2006.

Digital Library

[11]

M. Rudary, S. Singh, and D. Wingate. Predictive linear-Gaussian models of stochastic dynamical systems. In UAI, pages 501--508, 2005.

[12]

S. Singh, M. R. James, and M. R. Rudary. Predictive state representations: A new theory for modeling dynamical systems. In UAI, pages 512--519, 2004.

Digital Library

[13]

K. Torkkola. Feature extraction by non-parametric mutual information maximization. Journal of Machine Learning Research, (3):1415--1438, 2003.

Digital Library

[14]

E. Wiewiora. Learning predictive representations from a history. In ICML, pages 964--971, 2005.

Digital Library

[15]

D. Wingate and S. Singh. Kernel predictive linear Gaussian models for nonlinear stochastic dynamical systems. In ICML, 2006.

Digital Library

[16]

D. Wingate and S. Singh. Mixtures of predictive linear Gaussian models for nonlinear stochastic dynamical systems. In AAAI, 2006.

Digital Library

[17]

D. Wingate, V. Soni, B. Wolfe, and S. Singh. Relational knowledge with predictive representations of state. In IJCAI, 2007.

Digital Library

[18]

B. Wolfe, M. R. James, and S. Singh. Learning predictive state representations in dynamical systems without reset. In ICML, pages 980--987, 2005.

Digital Library

Cited By

Dick JLadosz PBen-Iwhiwhu EShimadzu HKinnell PPilly PKolouri SSoltoggio A(2020)Detecting Changes and Avoiding Catastrophic Forgetting in Dynamic Partially Observable EnvironmentsFrontiers in Neurorobotics10.3389/fnbot.2020.57867514Online publication date: 23-Dec-2020
https://doi.org/10.3389/fnbot.2020.578675
Boots BSiddiqi SGordon G(2019)Closing the learning-planning loop with predictive state representationsInternational Journal of Robotics Research10.1177/027836491140409230:7(954-966)Online publication date: 17-Jan-2019
https://dl.acm.org/doi/10.1177/0278364911404092
Yun-Long LRen-Hou L(2018)Discovery and learning of models with predictive state representations for dynamical systems without resetKnowledge-Based Systems10.1016/j.knosys.2009.01.00122:8(557-561)Online publication date: 31-Dec-2018
https://dl.acm.org/doi/10.1016/j.knosys.2009.01.001
Show More Cited By

Index Terms

On discovery and learning of models with predictive representations of state for agents with continuous actions and observations
1. Computing methodologies
  1. Machine learning

Recommendations

Learning and discovery of predictive state representations in dynamical systems with reset
ICML '04: Proceedings of the twenty-first international conference on Machine learning

Predictive state representations (PSRs) are a recently proposed way of modeling controlled dynamical systems. PSR-based models use predictions of observable outcomes of tests that could be done on the system as their state representation, and have model ...
Approximate predictive state representations
AAMAS '08: Proceedings of the 7th international joint conference on Autonomous agents and multiagent systems - Volume 1

Predictive state representations (PSRs) are models that represent the state of a dynamical system as a set of predictions about future events. The existing work with PSRs focuses on trying to learn exact models, an approach that cannot scale to complex ...
Predictive representations of state
NIPS'01: Proceedings of the 15th International Conference on Neural Information Processing Systems: Natural and Synthetic

We show that states of a dynamical system can be usefully represented by multi-step, action-conditional predictions of future observations. State representations that are grounded in data in this way may be easier to learn, generalize better, and be less ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

AAMAS '07: Proceedings of the 6th international joint conference on Autonomous agents and multiagent systems

May 2007

1585 pages

ISBN:9788190426275

DOI:10.1145/1329125

Conference Chairs:
Edmund Durfee
University of Michigan
,
Makoto Yokoo
Kyushu University
,
Program Chairs:
Michael Huhns
University of South Carolina
,
Onn Shehory
IBM Haifa Research Lab, Israel

Copyright © 2007 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

IFAAMAS

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 May 2007

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Division of Information and Intelligent Systems

Conference

AAMAS07

Sponsor:

AAMAS07: International Conference on Autonomous Agents and Mulitagent Systems

May 14 - 18, 2007

Hawaii, Honolulu

Acceptance Rates

Overall Acceptance Rate 1,155 of 5,036 submissions, 23%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
255
Total Downloads

Downloads (Last 12 months)1
Downloads (Last 6 weeks)0

Reflects downloads up to 01 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Dick JLadosz PBen-Iwhiwhu EShimadzu HKinnell PPilly PKolouri SSoltoggio A(2020)Detecting Changes and Avoiding Catastrophic Forgetting in Dynamic Partially Observable EnvironmentsFrontiers in Neurorobotics10.3389/fnbot.2020.57867514Online publication date: 23-Dec-2020
https://doi.org/10.3389/fnbot.2020.578675
Boots BSiddiqi SGordon G(2019)Closing the learning-planning loop with predictive state representationsInternational Journal of Robotics Research10.1177/027836491140409230:7(954-966)Online publication date: 17-Jan-2019
https://dl.acm.org/doi/10.1177/0278364911404092
Yun-Long LRen-Hou L(2018)Discovery and learning of models with predictive state representations for dynamical systems without resetKnowledge-Based Systems10.1016/j.knosys.2009.01.00122:8(557-561)Online publication date: 31-Dec-2018
https://dl.acm.org/doi/10.1016/j.knosys.2009.01.001
Stork JEk CKragic D(2015)Learning Predictive State Representations for planning2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)10.1109/IROS.2015.7353855(3427-3434)Online publication date: Sep-2015
https://doi.org/10.1109/IROS.2015.7353855
Böhmer WSpringenberg JBoedecker JRiedmiller MObermayer K(2015)Autonomous Learning of State Representations for Control: An Emerging Field Aims to Autonomously Learn State Representations for Reinforcement Learning Agents from Their Real-World Sensor ObservationsKI - Künstliche Intelligenz10.1007/s13218-015-0356-129:4(353-362)Online publication date: 19-Mar-2015
https://doi.org/10.1007/s13218-015-0356-1
Boots BByravan AFox D(2014)Learning predictive models of a depth camera & manipulator from raw execution traces2014 IEEE International Conference on Robotics and Automation (ICRA)10.1109/ICRA.2014.6907443(4021-4028)Online publication date: May-2014
https://doi.org/10.1109/ICRA.2014.6907443
Still SPrecup D(2012)An information-theoretic approach to curiosity-driven reinforcement learningTheory in Biosciences10.1007/s12064-011-0142-z131:3(139-148)Online publication date: 12-Jul-2012
https://doi.org/10.1007/s12064-011-0142-z
Wingate D(2012)Predictively Defined Representations of StateReinforcement Learning10.1007/978-3-642-27645-3_13(415-439)Online publication date: 2012
https://doi.org/10.1007/978-3-642-27645-3_13
Bush KPineau J(2009)Manifold embeddings for model-based reinforcement learning under partial observabilityProceedings of the 23rd International Conference on Neural Information Processing Systems10.5555/2984093.2984115(189-197)Online publication date: 7-Dec-2009
https://dl.acm.org/doi/10.5555/2984093.2984115
Zhao MJaeger HThon M(2009)A Bound on Modeling Error in Observable Operator Models and an Associated Learning AlgorithmNeural Computation10.1162/neco.2009.01-08-68721:9(2687-2712)Online publication date: Sep-2009
https://doi.org/10.1162/neco.2009.01-08-687

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten