Abstract
In this paper, we model the interference management problem in heterogeneous femto and macro networks, through a stochastic game. We claim that in a realistic wireless scenario this game cannot be analytically solved, so that we propose a solution based on a Reinforcement Learning (RL,) scheme. We present a taxonomy of RL, approaches, and we propose to select the most appropriate one to find a solution to our problem. Once we select the most adequate learning method, we study how to optimally design it in order to maximize the system performances.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
3GPP TSG RAN WG4 (Radio) Meeting 51: Simulation assumptions and parameters for FDD HeNB RF requirements. Technical report, 4–8 May 2009
Altman E, Koole G (1993) Stochastic scheduling games with Markov decision arrival processes. Comput Math Appl 26(6):141–148
Bellman R (1957) Dynamic programming. Princeton University Press, Princeton
Chandrasekhar V, Andrews JG, Gatherer A (2008) Femtocell networks: a survey. IEEE Commun Mag 46(9):59–67
Claussen H (2007) Performance of macro- and co-channel femtocells in a hierarchical cell structure. In: Proc. of the 18th annual IEEE international symposium on personal, indoor and mobile radio communications, IEEE PIMRC, Sept. 2007
Fudenberg D, Levine DK (1998) The theory of learning in games, vol 1. MIT press books. MIT Press, Cambridge
Galindo-Serrano A, Giupponi L (2010) Distributed Q-learning for interference control in OFDMA-based femtocell networks. In: Proc of the IEEE 71th vehicular technology conference VTC spring, May, 2010
Galindo-Serrano A, Giupponi L, Dohler M (2010) Cognition and docition in OFDMA-based femtocell networks. In: Proc of IEEE global communications conference (IEEE GLOBECOM), Dec., 2010
Galindo-Serrano A, Giupponi L, Auer G (2011) Distributed femto-to-macro interference management in multiuser OFDMA networks. In: Proc of IEEE 73rd vehicular technology conference (VTC2011-spring), workshop on broadband femtocell technologies, May, 2011
Galindo-Serrano A, Giupponi L, Majoral M (2011) On implementation requirements and performances of Q-learning for self-organized femtocells. In: Proc of the IEEE global communications conference (IEEE Globecom 2011), second workshop on femtocell networks (FEMnet), 5–9 December 2011
Harmon ME, Harmon SS (2000) Reinforcement learning: a tutorial
He G, Betz S, Debbah M (2009) Game-theoretic deployment design of small-cell OFDM networks. In: 3rd ICST/ACM international workshop on game theory in communication networks, Oct. 2009
Salo J, Del Galdo G, Salmi J, Kyösti P, Milojevic M, Laselva D, Schneider C (2005) MATLAB implementation of the 3GPP Spatial Channel Model (3GPP TR 25.996). Online, Jan 2005
Sutton RS (1988) Learning to predict by the methods of temporal differences. Mach Learn 3:9–44
Sutton RS, Barto AG (1998) Reinforcement learning: an introduction. MIT Press, Cambridge
Sycara KP (1998) Multiagent systems. AI Mag 19(2):79–92
Watkins CJCH, Dayan P (1992) Technical note: Q-learning. Mach Learn 8:279–292
Zhang J, de la Roche G (2010) Femtocells—technologies and deployment. Wiley, New York
Acknowledgements
This work was made possible by NPRP grant No. 5-1047-2-437 from the Qatar National Research Fund (a member of The Qatar Foundation). The statements made herein are solely the responsibility of the authors. The work has also been partially funded by SOFOCLES grant grant (TEC2010-21100).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Galindo-Serrano, A., Giupponi, L. Designing Time Difference Learning for Interference Management in Heterogeneous Networks. Dyn Games Appl 3, 105–123 (2013). https://doi.org/10.1007/s13235-013-0074-y
Published:
Issue Date:
DOI: https://doi.org/10.1007/s13235-013-0074-y