Skip to main content
SpringerLink
Log in

A personalized QoE-aware handover decision based on distributed reinforcement learning

  • Published:
Wireless Networks Aims and scope Submit manuscript

Abstract

Recent developments in heterogeneous mobile networks and growing demands for variety of real-time and multimedia applications have emphasized the necessity of more intelligent handover decisions. Addressing the context knowledge of mobile devices, users, applications, and networks is the subject of context-aware handoff decision as a recent effort to this aim. However, user perception has not been attended adequately in the area of context-aware handover decision making. Mobile users may have different judgments about the Quality of Service (QoS) depending on their environmental conditions, and personal and psychological characteristics. This reality has been exploited in this paper to introduce a personalized user-centric handoff decision method to decide about the time and target of handover based on User Perceived Quality (UPQ) feedbacks. The UPQ degradations are mainly for the sake of (1) exiting the coverage of the serving Point of Attachment (PoA) or (2) QoS degradation of serving access network. Using UPQ metric, the proposed method obviates the necessity of being aware about rapidly varying network QoS parameters and overcomes the complexity and overhead of gathering and managing some other context information. Moreover, considering the underlying network and geographical map, the proposed method is able to inherently exploit the trajectory information of mobile users for handover decision. UPQ degradation is not only due to the user behaviour, but also due to the behaviours of others users. As such, multi-agent reinforcement learning paradigm has been considered for target PoA selection. The employed decision algorithm is based on WoLF-PHC learning method where UPQ is used as a delayed reward for training. The proposed handoff decision has been implemented under IEEE 802.21 framework using NS2 network simulator. The results have shown better performance of the proposed method comparing to conventional methods assuming regular movement of mobile users.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19

Similar content being viewed by others

Abbreviations

AHD:

Adaptive handover decision

AHP:

Analytic hierarchy process

IS:

Information server

MADM:

Multi attribute decision making

MARL:

Multi agent reinforcement learning

MCHO:

Mobile controlled handover

MICS:

MIH independent command service

MIES:

Media independent event service

MIH:

Media independent handover

MIIS:

MIH independent information service

MIP:

Mobile IP

MN:

Mobile node

MOS:

Mean opinion score

PHC:

Policy hill climbing

PoA:

Point of attachment

PQE:

Perceived quality evaluator

PSNR:

Peak signal to noise ratio

QoCE:

Quality of customer experience

QoE:

Quality of experience

QoS:

Quality of service

QoUE:

Quality of user experience

RSS:

Received signal strength

SAW:

Simple additive weighting

SCM:

Spatial conceptual map

SG:

Stochastic game

SSNR:

Segmental signal to noise ratio

TLV:

Type-length-value

UPQ:

User perceived quality

WEA:

Way elementary areas

WoLF:

Win or learn fast

References

  1. Prehofer, C., Nafisi, N., & Wei, Q. (2003). A framework for context-aware handover decisions. In 14th IEEE proceedings on personal, indoor and mobile radio communications, PIMRC, 2003, Vol. 3, pp. 2794–2798.

  2. Nguyen-Vuong, Q.-T., Agoulmine, N., & Ghamri-Doudane, Y. (2008). A user-centric and context-aware solution to interface management and access network selection in heterogeneous wireless environments. Computer Networks, 52(18), 3358–3372.

    Article  MATH  Google Scholar 

  3. Jingjing, Z., & Ansari, N. (2011). On assuring end-to-end QoE in next generation networks: Challenges and a possible solution. Communications Magazine, 49(7), 185–191.

    Article  Google Scholar 

  4. Kilkki, K. (2008). Quality of experience in communications ecosystem. Journal of Universal Computer Science, 14(5), 615–624.

    Google Scholar 

  5. Stankiewicz, R., & Jajszczyk, A. (2011). A survey of QoE assurance in converged networks. Computer Networks, 55(7), 1459–1473.

    Article  Google Scholar 

  6. Herman, H., Rahman, A. A., Syahbana, Y. A., & Bakar, K. A. (2011). Nonlinearity modelling of QoE for video streaming over wireless and mobile network. In Second international conference on intelligent systems, modelling and simulation (ISMS), 2011, pp. 313–317.

  7. Mitra, K., Zaslavsky, A., & Aahlund, C. (2011). A probabilistic context-aware approach for quality of experience measurement in pervasive systems. In Proceedings of the ACM symposium on applied computing, 2011, pp. 419–424.

  8. Winkler, S., & Mohandas, P. (2008). The evolution of video quality measurement: from PSNR to hybrid metrics. IEEE Transactions on Broadcasting, 54, 660–668.

    Article  Google Scholar 

  9. Saliba, J., Beresford, A., Ivanovich, M., & Fitzpatrick, P. (2005). User-perceived quality of service in wireless data networks. Personal Ubiquitous Computing, 9(6), 413–422.

    Article  Google Scholar 

  10. Magoulas, G. D., & Ghinea, G. (2001). Neural network-based interactive multicriteria decision making in a quality of perception-oriented management scheme. In International joint conference on neural networks, 2001, Vol. 4, pp. 2536–2541, vol. 2534.

  11. Ghahfarokhi, B. S., & Movahhedinia, N. (2011). A contextaware handover decision based on user perceived quality of service trigger. Wireless Communication and Mobile Computing, 11, 723–741.

    Article  Google Scholar 

  12. Piamrat, K. et al. (2011). QoE-aware vertical handover in wireless heterogeneous networks, In Wireless communications and mobile computing conference (IWCMC), 2011, pp. 95–100.

  13. Hasswa, A., Nasser, N., & Hassanein, H. (2007). A seamless context-aware architecture for fourth generation wireless networks. Wireless Personal Communications, 43(3), 1035–1049.

    Article  Google Scholar 

  14. Ahmed, T., Kyamakya, K., & Ludwig, M. (2007). Architecture of a context-aware vertical handover decision model and its performance analysis for GPRS–WiFi handover. In 11th IEEE symposium on computers and communications, 2007, pp. 795–801.

  15. Kang, J. -M., Ju, H. -T., & Hong, J. (2006). Towards autonomic handover decision management in 4G networks. In Autonomic management of mobile multimedia services, pp. 145–157.

  16. Prehofer, C., Nafisi, N., & Wei, Q. (2003). A framework for context-aware handover decisions. In IEEE international symposium on personal, indoor and mobile radio communications, 2003, pp. 2794-2798.

  17. Wei, Q., Farkas, K., Prehofer, C., Mendes, P., & Plattner, B. (2006). Context-aware handover using active network technology. Computer Networks, 50(15), 2855–2872.

    Article  Google Scholar 

  18. Ghahfarokhi, B. S., & Movahhedinia, N. (2012). Context-aware handover decision in an enhanced media independent handover framework. Wireless Personal Communications. Published online, doi:10.1007/s11277-012-0543-4.

  19. Neves, P., Soares, J., Sargento, S., Pires, H., & Fontes, F. (2011). Context-aware media independent information server for optimized seamless handover procedures. Computer Networks, 55(7), 1498–1519.

    Article  Google Scholar 

  20. Bowling, M., & Veloso, M. (2002). Multiagent learning using a variable learning rate. Artificial Intelligence, 136(2), 215–250.

    Article  MathSciNet  MATH  Google Scholar 

  21. IEEE 802.21. (2008). IEEE standard for local and metropolitan area networks: Media independent handover services.

  22. Cacace, F., & Vollero, L. (2006). Managing mobility and adaptation in upcoming 802.21 enabled devices. In 4th international workshop on wireless mobile applications and services on WLAN hotspots, 2006, pp. 1–10.

  23. Oliva, A. d. l., Melia, T., Vidal, A., Bernardos, C. J., Soto, I., & Banchs, A. (2007). IEEE 802.21 enabled mobile terminals for optimized WLAN/3G handovers: a case study. SIGMOBILE Mobile Computing and Communications Review, 11(2), 29–40.

    Google Scholar 

  24. ITU-T Rec. P.800. (1996). Methods for subjective determination of transmission quality.

  25. ITU-T Rec. J.246. (2008). Perceptual audiovisual quality measurement techniques for multimedia services over digital cable television networks in presence of reduced bandwidth reference.

  26. ITU-T Rec. G.1070. (2007). Opinion model for videophone applications.

  27. ITU-T Rec. J.247. (2008). Objective perceptual multimedia video quality measurement in the presence of a full reference.

  28. ITU-T Rec. p.910. (1999). Subjective video quality assessment method for multimedia applications.

  29. Reves, X. (2006). User perceived quality evaluation in a B3G network testbed. 15th IST mobile and wireless summit, 2006.

  30. ITU-T Rec. P.861. (1998). Objective quality measurement of telephone-band speech codecs.

  31. ITU-T Rec. p.862. (2001). Perceptual evaluation of speech quality (PESQ): An objective method for end to end speech quality assessment of narrowband telephone networks and speech codecs.

  32. ITU-T Rec. J.144. (2004). Objective perceptual video quality measurement techniques for digital cable television in presence of full reference.

  33. Menkovski, V., Exarchakos, G., & Liotta, A. (2010). Machine learning approach for quality of experience aware networks. In 2nd international conference on intelligent networking and collaborative systems, 2010, pp. 461–466.

  34. Fiedler, M., Hossfeld, T., & Tran-Gia, P. (2010). A generic quantitative relationship between quality of experience and quality of service. IEEE Network, 24(2), 36–41.

    Article  Google Scholar 

  35. Brooks, P., & Hestnes, B. r. (2010). User measures of quality of experience: why being objective and quantitative is important. IEEE Network, 24(2), 8–13.

    Article  Google Scholar 

  36. Mahdi, A., & Picovici, D. (2010). New single-ended objective measure for non-intrusive speech quality evaluation. Signal, Image and Video Processing, 4(1), 23–38.

    Article  Google Scholar 

  37. Rein, S., Fitzek, F. H. P., & Reisslein, M. (2005). Voice quality evaluation in wireless packet communication systems: A tutorial and performance results for ROHC. Wireless Communications, 12(1), 60–67.

    Article  Google Scholar 

  38. Jiang, X., Wang, Y., & Wang, C. (2009). No-reference video quality assessment for MPEG-2 video streams using BP neural networks. In 2nd international conference on interaction sciences: Information technology, culture and human, 2009, pp. 307–311.

  39. Klaue, J., Rathke, B., & Wolisz, A. (2003). EvalVid—A framework for video transmission and quality evaluation. In Computer performance, pp. 255–272.

  40. Takahashi, A., Hands, D., & Barriac, V. (2008). Standardization activities in the ITU for a QoE assessment of IPTV. Communications Magazine, 46, 78–84.

    Article  Google Scholar 

  41. Mitchell, T. M. (1997). Machine learning. New Delhi: McGraw-Hill.

    MATH  Google Scholar 

  42. Busoniu, L., Babuska, R., & De Schutter, B. (2008). A Comprehensive Survey of Multiagent Reinforcement Learning. IEEE Transactions on Systems, Man, and Cybernetics Part C: Applications and Reviews, 38(2), 156–172.

    Article  Google Scholar 

  43. Weiss, G. (1999). Multiagent systems: A modern approach to distributed artificial intelligence. Cambridge, MA: MIT Press.

    Google Scholar 

  44. Sleem, A., & Kumar, A. (2005). Handoff management in wireless data networks using topography-aware mobility prediction. Journal of Parallel and Distributed Computing, 65(8), 963–982.

    Article  Google Scholar 

  45. Ghahfarokhi, B. S., & Movahhedinia, N. (2007). QoS provisioning by EFuNNs-based handoff planning in cellular MPLS networks. Computer Communications, 30(13), 2676–2685.

    Article  Google Scholar 

  46. Onel, T., Ersoy, C., Cayirci, E., & Parr, G. (2004). A multicriteria handoff decision scheme for the next generation tactical communications systems. Computer Networks, 46(5), 695–708.

    Article  Google Scholar 

  47. Edwards, G., & Sankar, R. (1998). Microcellular handoff using fuzzy techniques. Wireless Networks, 4(5), 401–409.

    Article  Google Scholar 

  48. Sharma, S., Baek, I., & Chiueh, T.-C. (2007). OmniCon: A mobile IP-based vertical handoff system for wireless LAN and GPRS links. Softw. Pract. Exper., 37(7), 779–798.

    Article  Google Scholar 

  49. Zhu, F., & McNair, J. (2006). Multiservice vertical handoff decision algorithms. EURASIP Journal on Wireless Communications and Networks, 2006(2), 52.

    Google Scholar 

  50. Stevens-Navarro, E., & Wong, V. W. S. (2006). Comparison between vertical handoff decision algorithms for heterogeneous wireless networks. In IEEE 63rd vehicular technology conference, VTC, 2006, Vol. 2, pp. 947–951.

  51. Zhang, W. (2004). Handover decision using fuzzy MADM in heterogeneous networks. In IEEE wireless communications and networking conference, 2004, Vol. 2, pp. 653–658.

  52. Hongyan, B., Chen, H., & Lingge, J. (2003). Intelligent signal processing of mobility management for heterogeneous networks. In Proceedings of the international conference on neural networks and signal processing, 2003, Vol. 2, pp. 1578–1581.

  53. Chan, P. M. L., Sheriff, R. E., Hu, Y. F., Conforto, P., & Tocci, C. (2002). Mobility management incorporating fuzzy logic for heterogeneous IP environment. Communications Magazine, 39(12), 42–51.

    Article  Google Scholar 

  54. Inoue, M., Mahmud, K., Murakami, H., Hasegawa, M., & Morikawa, H. (2005). Context-based network and application management on seamless networking platform. Wireless Personal Communications, 35(1–2), 53–70.

    Article  Google Scholar 

  55. Wang, C. -Y., Huang, H. -Y., & Hwang, R. -H. (2011). Mobility management in ubiquitous environments. Personal Ubiquitous Computings, 15(3), 235–251.

    Article  Google Scholar 

  56. Indulska, J., & Balasubramaniam, S. (2004). Context-aware vertical handovers between WLAN and 3G networks. In IEEE 59th vehicular technology conference, 2004, Vol. 5, pp. 3019–3023.

  57. Kassar, M., Kervella, B., & Pujolle, G. (2008). An overview of vertical handover decision strategies in heterogeneous wireless networks. Computer Communications, 31(10), 2607–2620.

    Article  Google Scholar 

  58. Hong, C.-P., Weems, C. C., & Kim, S.-D. (2008). An effective vertical handoff scheme based on service management for ubiquitous computing. Computer Communications, 31(9), 1739–1750.

    Article  Google Scholar 

  59. Wang, Y., Zhang, P., Zhou, Y., Yuan, J., Liu, F., & Li, G. (2010). Handover management in enhanced MIH framework for heterogeneous wireless networks environment. Wireless Personal Communications, 52(3), 615–636.

    Article  Google Scholar 

  60. Mussabbir, Q. B., Wenbing, Y., Zeyun, N., & Xiaoming, F. (2007). Optimized FMIPv6 using IEEE 802.21 MIH services in vehicular networks. IEEE Transactions on Vehicular Technology, 56(6), 3397–3407.

    Article  Google Scholar 

  61. Samaan, N., & Karmouch, A. (2005). A mobility prediction architecture based on contextual knowledge and spatial conceptual maps. IEEE Transactions on Mobile Computing, 4(6), 537–551.

    Article  Google Scholar 

  62. Sen, S., Sekaran, M., & Hale, J. (1994). Learning to coordinate without sharing information. In Twelfth national conference on artificial intelligence, 1994, Vol. 1, pp. 426–431.

  63. Kettani, D., & Moulin, B. (1999). A spatial model based on the notions of spatial conceptual map and of object’s influence areas. Paper presented at the Spatial Information Theory. Cognitive and Computational Foundations of Geographic Information Science, 1999, pp. 401–416.

  64. National Institute of Standards and Technology. http://w3.antd.nist.gov/seamlessandsecure/pubtool.shtml, visited on July 2011.

  65. Balasubramaniam, S., & Indulska, J. (2004). Vertical handover supporting pervasive computing in future wireless networks. Computer Communications, 27(8), 708–719.

    Article  Google Scholar 

  66. Rovatsos, M., & Belesiotis, A. (2007). Advice taking in multiagent reinforcement learning, In Proceedings of the 6th international joint conference on Autonomous agents and multiagent systems (AAMAS), 2007, pp. 1342–1344.

Download references

Author information

Authors and Affiliations

  1. Department of Information Technology Engineering, University of Isfahan, Isfahan, Iran

    Behrouz Shahgholi Ghahfarokhi

  2. Department of Computer Engineering, University of Isfahan, Isfahan, Iran

    Naser Movahhedinia

Authors
  1. Behrouz Shahgholi Ghahfarokhi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Naser Movahhedinia
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Behrouz Shahgholi Ghahfarokhi.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Ghahfarokhi, B.S., Movahhedinia, N. A personalized QoE-aware handover decision based on distributed reinforcement learning. Wireless Netw 19, 1807–1828 (2013). https://doi.org/10.1007/s11276-013-0572-2

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11276-013-0572-2

Keywords

Search

Navigation