Abstract
A hidden Markov model (HMM) comprises a state with Markovian dynamics that can only be observed via noisy sensors. This paper considers three problems connected to HMMs, namely, inverse filtering, belief estimation from actions, and privacy enforcement in such a context. First, the authors discuss how HMM parameters and sensor measurements can be reconstructed from posterior distributions of an HMM filter. Next, the authors consider a rational decision-maker that forms a private belief (posterior distribution) on the state of the world by filtering private information. The authors show how to estimate such posterior distributions from observed optimal actions taken by the agent. In the setting of adversarial systems, the authors finally show how the decision-maker can protect its private belief by confusing the adversary using slightly sub-optimal actions. Applications range from financial portfolio investments to life science decision systems.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Wahlberg B, Hjalmarsson H, and Annergren M, On optimal input design in system identification for control, 49th IEEE Conference on Decision and Control (CDC), pages 2010, 5548–5553. DOI: https://doi.org/10.1109/CDC.2010.5717863.
Annergren M, Larsson C A, Hjalmarsson H, et al., Application-oriented input design in system identification: optimal input design for control [applications of control], IEEE Control Systems Magazine, 2017, 37(2): 31–56, DOI: https://doi.org/10.1109/MCS.2016.2643243.
Kalman R E, When is a linear control system optimal?, Journal of Basic Engineering, 1964, 86(1): 51–60.
Zhang H, Umenberger J, and Hu X, Inverse optimal control for discrete-time finite-horizon Linear quadratic regulators, Automatica, 2019, 110: 108593.
Zhang H, Li Y, and Hu X, Discrete-time inverse linear quadratic optimal control over finite time-horizon under noisy output measurements, Control Theory and Technology, 2021.
Li Y, Yao Y, and Hu X, Continuous-time inverse quadratic optimal control problem, Automatica, 2020, 117: 108977.
Ng A Y, Russell S J, and others, Algorithms for inverse reinforcement learning, Proccedings of the International Conference on Machine Learning (ICML), 2000, 1: 2.
Abbeel P and Ng A Y, Apprenticeship learning via inverse reinforcement learning, Proceedings of the Twenty-First International Conference on Machine Learning, 2004, 1.
Mattila R, Rojas C, Krishnamurthy V, et al., Inverse filtering for hidden Markov models, Advances in Neural Information Processing Systems (NIPS) 2017, 2017, 30.
Mattila R, Rojas C R, Krishnamurthy V, et al., Inverse filtering for hidden Markov models with applications to counter-adversarial autonomous systems, IEEE Transactions on Signal Processing, 2020, DOI: https://doi.org/10.1109/TSP.2020.3019177.
Mattila R, Rojas C R, Krishnamurthy V, et al., Inverse filtering for linear Gaussian state-space models, 2018 IEEE Conference on Decision and Control (CDC), pages 5556–5561. IEEE, 2018.
Mattila R, Lourenço I, Rojas C R, et al., Estimating private beliefs of bayesian agents based on observed decisions, IEEE Control Systems Letters, 2019, 3(3): 523–528.
Lourenço I, Mattila R, Rojas C R, et al., How to protect your privacy? A framework for counter-adversarial decision making, Proceedings of the 59th IEEE Conference in Decision and Control (CDC), 2020, 1785–1791.
Mattila R, Lourenço I, Krishnamurthy V, et al., What did your adversary believe? Optimal filtering and smoothing in counter-adversarial autonomous systems, Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020, 5495–5499.
Norris J R, Markov Chains, Cambridge University Press, Cambridge, 1998.
Krishnamurthy V, Partially Observed Markov Decision Processes: From Filtering to Controlled Sensing, Cambridge University Press, Cambridge, 2016.
Anderson B D O and Moore J B, Optimal Filtering, Prentice-Hall, Upper Saddle River, New Jersey, 1979.
Cappé O, Moulines E, and Rydén T, Inference in Hidden Markov Models, Springer, New York, 2005.
Hsu D, Kakade S M, and Zhang T, A spectral algorithm for learning hidden Markov models, Journal of Computer and System Sciences, 2012, 78(5): 1460–1480.
Buchta C, Kober M, Feinerer I, et al., Spherical k-means clustering, Journal of Statistical Software, 2012, 50(10): 1–22.
Krishnamurthy V and Rangaswamy M, How to calibrate your adversary’s capabilities? Inverse filtering for counter-autonomous systems, IEEE Transactions on Signal Processing, 2019, 67(24): 6511–6525
Kuptel A, Counter unmanned autonomous systems (CUAxS): priorities, policy, future capabilities, Multinational Capability Development Campaign (MCDC), Social Science Electronic Publishing, 2017, 15–16.
Mas-Colell A, Whinston M D, and Green J R, Microeconomic Theory, volume 1, Oxford University Press, New York, 1995.
Luenberger D G, Microeconomic Theory, Mcgraw-Hill College, New York, 1995.
Machina M J, Choice under uncertainty: Problems solved and unsolved, Journal of Economic Perspectives, 1987, 1(1): 121–154.
Varian H R, Revealed preference, Samuelsonian Economics and the Twenty-First Century, 2006, 99–115.
Varian H R, Microeconomic Analysis, volume 3, Norton, New York, 1992.
Ahuja R K and Orlin J B, Inverse optimization, Operations Research, 2001, 49(5): 771–783.
Iyengar G and Kang W, Inverse conic programming with applications, Operations Research Letters, 2005, 33: 319–330
Zhang J and Xu C, Inverse optimization for linearly constrained convex separable programming problems, European Journal of Operational Research, 2010, 200(3): 671–679.
Keshavarz A, Wang Y, and Boyd S, Imputing a convex objective function, IEEE International Symposium on Intelligent Control, 2011, 613–619.
Boyd S and Vandenberghe L, Convex Optimization, Cambridge University Press, Cambridge, 2004
Rockafellar R T, Convex Analysis, Princeton University Press, Princeton, 1970.
Yin G G and Zhou X Y, Markowitz’s mean-variance portfolio selection with regime switching: From discrete-time models to their continuous-time limits, IEEE Transactions on Automatic Control, 2004, 49(3): 349–360.
Elliott R J, Siu T K, and Badescu A, On mean-variance portfolio selection under a hidden Markovian regime-switching model, Economic Modelling, 2010, 27(3): 678–686.
Nystrup P, Madsen H, and Lindstrm E, Dynamic portfolio optimization across hidden market regimes, Quantitative Finance, January 2018, 18(1): 83–95
Puterman M L, Markov Decision Processes: Discrete Stochastic Dynamic Programming, John Wiley & Sons, Inc., 1994.
McKinsey J C C, Introduction to the Theory of Games, Courier Corporation, 2003.
Davis P J and Rabinowitz P, Methods of Numerical Integration, Courier Corporation, 2007.
Author information
Authors and Affiliations
Corresponding authors
Additional information
This work was supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP), the Swedish Research Council and the Swedish Research Council Research Environment NewLEADS under contract 2016-06079.
Rights and permissions
About this article
Cite this article
Lourenço, I., Mattila, R., Rojas, C.R. et al. Hidden Markov Models: Inverse Filtering, Belief Estimation and Privacy Protection. J Syst Sci Complex 34, 1801–1820 (2021). https://doi.org/10.1007/s11424-021-1247-1
Received:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11424-021-1247-1