Skip to main content
Springer Nature Link
Log in

A Q-learning approach for machine-type communication random access in LTE-Advanced

  • Published:
Telecommunication Systems Aims and scope Submit manuscript

Abstract

Due to the wide proliferation of the 3GPP long term evolution (LTE) and LTE-Advanced systems as the air interface for 4th generation (4G) wireless systems and beyond, telecommunication operators became more interested in using the LTE infrastructure to meet the projected surge in demand for machine-to-machine (M2M) and IoT communications. As a result, the LTE-Advanced system must evolve to provide these services as an overlay over the original network that is originally designed for human-to-human (H2H) communication. In this setup, M2M communication devices share the same random access channel (RACH) with the H2H communication devices. Since M2M communication is expected to be massive, consequently the RACH is a new bottleneck in the LTE-A system. This triggered the need to enhance the operation of the RACH to meet the needs to support the low power and low data rates of M2M devices. The current standardized scheme to request access to the system is known to suffer from congestion and overloading in the presence of a huge number of devices. Accordingly, recent research pointed to the need of designing more efficient ways to manage the random access channel in such setups. Most of previous works focused on solving the congestion problem in RACH in the presence of M2M solely, but not with the existence of both M2M and H2H services. In this work we propose a new random access channel scheme based on Q-learning approach to reduce the congestion problem. The scheme adaptively divides the available preambles between both M2M and H2H devices in a way that provides an acceptable service for the H2H devices and maximizes the number of active M2M devices. The adaptation is done based on current demand levels from both the H2H and M2M devices and the observed service levels. The results indicate that the proposed approach provides high random access channel success probability for both M2M and H2H devices even with the huge number of M2M devices.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  1. Wu, G., Talwar, S., Johnsson, K., Himayat, N., & Johnson, K. D. (2011). M2M: From mobile to embedded internet. IEEE Communications Magazine, 49(4), 36–43.

    Article  Google Scholar 

  2. Biral, A., Centenaro, M., Zanella, A., Vangelista, L., & Zorzi, M. (2015). The challenges of M2M massive access in wireless cellular networks. Digital Communications and Networks, 1(1), 1–19.

    Article  Google Scholar 

  3. Ratasuk, R., Vejlgaard, B., Mangalvedhe, N., & Ghosh, A. (2016). NB-IoT system for M2M communication. In Wireless communications and networking conference workshops (WCNCW) (pp. 428–432). IEEE.

  4. Ratasuk, R., Mangalvedhe, N., & Ghosh, A. (2015). Overview of LTE enhancements for cellular IoT. In IEEE 26th annual international symposium on personal indoor and mobile radio communications (PIMRC) (pp. 2293–2297).

  5. Cisco Blogs, Cisco Visual Networking Index Complete Forecast Update. (2015). Key trends include mobility, M2M, and multimedia content.

  6. Gotsis, A., Lioumpas, A., & Alexiou, A. (2012). M2M scheduling LTE: Challenges and new perspectives. IEEE Vehicular Technology Magazine, 7(3), 34–39.

    Article  Google Scholar 

  7. GPP. (2012). System improvements for machine-type communications. 3rd Generation Partnership Project (3GPP), TR 23.888 V11.0.0, Sept 2012.

  8. Ksentini, A., Aoul, Y. H., & Taleb, T. (2012). Cellular-based machine-to-machine: Overload control. IEEE Network, 26(6), 54–60.

    Article  Google Scholar 

  9. Wang, G., Zhong, X., Mei, S., & Wang, J. (2010). An adaptive medium access control mechanism for cellular based machine to machine (M2M) communication. In Proceedings of IEEE international conference on wireless information technology and systems (ICWITS), Hawaii, HI.

  10. GPP. (2011). Study on RAN improvements for machine-type communications. 3rd Generation Partnership Project (3GPP), TR 37.868 V11.0.0, Oct 2011.

  11. GPP. (2010). MTC simulation results with specific solutions. 3rd Generation Partnership Project (3GPP), TSG RAN WG2 #71, R2-104662, Aug 2010.

  12. GPP. (2010). MTC LTE simulations. 3rd Generation Partnership Project (3GPP), TSG RAN WG2 #71, R2-104663, Aug 2010.

  13. Duan, S., Shah-Mansouri, V., & Wong, V. W. S. (2013). Dynamic access class barring for M2M communications in LTE networks. In 2013 IEEE Global Communications Conference (GLOBECOM), Atlanta, GA (pp. 4747–4752).

  14. Lin, T. M., Lee, C. H., Cheng, J. P., & Chen, W. T. (2014). PRADA: Prioritized random access with dynamic access barring for MTC in 3GPP LTE-A networks. IEEE Transactions on Vehicular Technology, 63(5), 2467–2472.

    Article  Google Scholar 

  15. Tsong, S. S., Chiu, C. H., Cheng, Y. C., & Kuo, K. H. (2012). Self-adaptive persistent contention scheme for scheduling based machine type communications in LTE system. In 2012 International conference on selected topics in mobile and wireless networking (pp. 77–82).

  16. Wu, H., Zhu, C., La, R., Liu, X., & Zhang, Y. (2012). Fast adaptive S-ALOHA scheme. In Vehicular technology conference (VTC-Fall), Quebec City, Canada.

  17. GPP. (2012). Further performance evaluation of EAB information update mechanisms. 3rd Generation Partnership Project (3GPP), RAN WG2#77, R2-120270, Intel, Germany, Feb. 2012.

  18. Park, J., & Lim, Y. (2016). Adaptive access class barring method for machine generated communications. Mobile Information Systems, 2016, Article ID 6923542, p. 6.

  19. Du, Q., Li, W., Liu, L., Ren, P., Wang, Y., & Sun, L. (2016). Dynamic RACH partition for massive access of differentiated M2M services. Sensors, 16(4), 455.

    Article  Google Scholar 

  20. Kim, J., & Lee, J. (2017). Exploiting the capture effect to enhance RACH performance in cellular-based M2M communications. Sensors, 17(10), 2169.

    Article  Google Scholar 

  21. David, A., Dario, V., Castro, F., & Miguel. (2016). Dynamic RACH distribution for M2M massive access in LTE-A. In 7th International conference on the network of the future (NOF) (pp. 1–3).

  22. Oquendo, L., Paramo, D., Pla, V., & Bauset, J. (2018). Reinforcement learning-based ACB in LTE-A networks for handling massive M2M and H2H communications. In IEEE ICC 2018.

  23. Yu, Y., Wang, T., & Luis, S. C. (2018). Deep-reinforcement learning multiple access for heterogeneous wireless networks. In IEEE ICC 2018.

  24. Li, Y. (2010). E-band radios for LTE/LTE-Advanced mobile backhaul. In 2010 Workshop on integrated nonlinear microwave and milli-meter-wave circuits (INMMIC).

  25. Choi, S., Lee, W., Kim, D., Park, K.-J., Choi, S., & Han, K.-Y. (2011). Automatic configuration of random access channel parameters in LTE systems. In Wireless days (WD) IFIP (pp. 1–6).

  26. Kim, D., Kim, W., & An, S. (2013). Adaptive random access preamble split in LTE. In 9th International wireless communications and mobile computing conference (IWCMC) (pp. 814–819).

  27. Batabyal, S., & Das, S. S. (2012). Distance dependent call blocking probability, and area erlang efficiency of cellular networks. In IEEE 75th vehicular technology conference (VTC Spring) (pp. 1–5).

  28. Kaelbling, L., Littman, M. L., & Moore, A. W. (1996). Reinforcement learning: A survey. Journal of Artificial Intelligence Research, 4, 237–285.

    Article  Google Scholar 

  29. Watkins, C. J. (1989). Learning from delayed reward, Ph.D. Thesis, Cambridge University.

  30. Watkins, C. J. (1992). Q learning. Machine Learning, 8, 279–292.

    Google Scholar 

  31. Sutton, R., & Barto, A. (1998). Reinforcement learning: An introduction. Cambridge: MIT Press.

    Google Scholar 

  32. Mnih, V., Kavukcuoglu, K., & Silver, D. (2015). Human-level control through deep reinforcement learning. Nature, 518, 529–533.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Electronics and Communications Engineering, Faculty of Engineering, Cairo University, Giza, 12613, Egypt

    Amaal S. A. El-Hameed & Khaled M. F. Elsayed

Authors
  1. Amaal S. A. El-Hameed
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Khaled M. F. Elsayed
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Khaled M. F. Elsayed.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

El-Hameed, A.S.A., Elsayed, K.M.F. A Q-learning approach for machine-type communication random access in LTE-Advanced. Telecommun Syst 71, 397–413 (2019). https://doi.org/10.1007/s11235-018-0509-2

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11235-018-0509-2

Keywords