Skip to main content

Advertisement

Springer Nature Link
Log in

Resource orchestration in network slicing using GAN-based distributional deep Q-network for industrial applications

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

The Industrial Internet of Things (IIoT) is an emerging and promising concept that allows intelligent manufacturing through the connectivity of 5G/6G and the interaction of industrial production units. The introduction of network slicing in 5G and beyond has made it possible to manage and allocate resources to various applications according to their requirements. In this paper, we study network slicing within a radio access network containing IIoT devices which include base stations that share the same physical infrastructure. We use deep reinforcement learning-based resource orchestration technique to achieve variable service demands of environment state-value and resource allocation as environment action-value. We describe the cognitive decision objectives to maximise the optimal policy for IIoT reward by achieving higher system throughput, spectral efficiency (SE), service level agreement (SLA), transmission packet rate with low power consumption and transmission delay. We use generative adversarial network-based deep distributional noisy Q-networks (GAN–NoisyNet) to learn the action-value distribution. Furthermore, we introduce dueling GAN–NoisyNet, which employs a duel generator that estimates the action advantage function and state-value distribution. Finally, we conduct extensive simulations to verify the performance of the proposed GAN–NoisyNet and dueling GAN–NoisyNet.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

We’re sorry, something doesn't seem to be working properly.

Please try refreshing the page. If that doesn't work, please contact support so we can address the problem.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Data availability

All the used codes/data in this research will be available from the corresponding author on reasonable request.

References

  1. Attaran M (2021) The impact of 5G on the evolution of intelligent automation and industry digitization. J Ambient Intell Humanized Comput. https://doi.org/10.1007/s12652-020-02521-x

    Article  Google Scholar 

  2. Popovski P, Trillingsgaard KF, Simeone O, Durisi G (2018) 5G wireless network slicing for eMBB, URLLC, and mMTC: A communication-theoretic view. IEEE Access 6:55765–55779

    Article  Google Scholar 

  3. Jiang W, Han B, Habibi MA, Schotten HD (2021) The road towards 6G: A comprehensive survey. IEEE Open J Commun Soc 2:334–366

    Article  Google Scholar 

  4. Mahmood A, Beltramelli L, Abedin SF, Zeb S, Mowla NI, Hassan SA, Sisinni E, Gidlund M (2021) Industrial IoT in 5G-and-beyond networks: vision, architecture, and design trends. IEEE Trans Ind Inform

  5. Xu Y, Gui G, Gacanin H, Adachi F (2021) A survey on resource allocation for 5G heterogeneous networks: current research, future trends and challenges. IEEE Commun Surv Tutorials

  6. Gupta RK, Sahoo B (2018) Security issues in software-defined networks. IUP J Inf Technol 14(2):72–82

    Google Scholar 

  7. Al-Surmi I, Raddwan B, Al-Baltah I (2021) Next generation mobile core resource orchestration: comprehensive survey. Challenges Perspect Wirel Personal Commun 120(2):1341–1415

    Article  Google Scholar 

  8. 3GPP TS 38.306. 3rd generation partnership project; Technical specification group radio access network; NR; User Equipment (UE) radio access capabilities

  9. Wu J, Zhang Z, Hong Y, Wen Y (2015) Cloud radio access network (C-RAN): a primer. IEEE Netw 29(1):35–41

    Article  Google Scholar 

  10. Khalaf OI, Ogudo KA, Singh M (2020) A fuzzy-based optimization technique for the energy and spectrum efficiencies trade-off in cognitive radio-enabled 5G network. Symmetry 13(1):47

    Article  Google Scholar 

  11. Henry S, Alsohaily A, Sousa ES (2020) 5G is real: Evaluating the compliance of the 3GPP 5G new radio system with the ITU IMT-2020 requirements. IEEE Access 8:42828–42840

    Article  Google Scholar 

  12. Akeela R, Dezfouli B (2018) Software-defined Radios: Architecture, state-of-the-art, and challenges. Comput Commun 128:106–125

    Article  Google Scholar 

  13. Vitturi S, Zunino C, Sauter T (2019) Industrial communication systems and their future challenges: Next-generation Ethernet, IIoT, and 5G. Proc IEEE 107(6):944–961

    Article  Google Scholar 

  14. Jiang T, Zhang J, Tang P, Tian L, Zheng Y, Dou J, Jämsä T (2021) 3GPP standardized 5G channel model for IIoT scenarios: A survey. IEEE Int Things J 8(11):8799–8815

    Article  Google Scholar 

  15. Wang T, Zhang Y, Xiong N, Wan S, Shen S, Huang S (2021) An Effective Edge-Intelligent Service Placement Technology for 5G-and-beyond Industrial IoT. IEEE Trans Ind Inf 18(6):4148–4157

    Article  Google Scholar 

  16. Kutuzov D, Osovsky A, Stukach O, Starov D (2021) May) Modeling of IIoT traffic processing by intra-chip NoC routers of 5G/6G networks. 2021 International Siberian Conference on Control and Communications (SIBCON). IEEE, pp 1–5

    Google Scholar 

  17. Chen Y, Liu Y, Zeng M, Saleem U, Lu Z, Wen X, Li Y (2020) Reinforcement learning meets wireless networks: A layering perspective. IEEE Int Things J 8(1):85–111

    Article  Google Scholar 

  18. Gupta RK, Misra R (2019) December) Machine learning-based slice allocation algorithms in 5G networks. 2019 international Conference on Advances in Computing, Communication and Control (ICAC3). IEEE, pp 1–4

    Google Scholar 

  19. Wijethilaka S, Liyanage M (2021) Survey on network slicing for internet of things realization in 5G networks. IEEE Commun Surv Tutor 23(2):957–994

    Article  Google Scholar 

  20. Li R, Wang C, Zhao Z, Guo R, Zhang H (2020) The LSTM-based advantage actor-critic learning for resource management in network slicing with user mobility. IEEE Commun Lett 24(9):2005–2009

    Article  Google Scholar 

  21. Liu Y, Ding J, Zhang ZL, Liu X (2021) CLARA: A constrained reinforcement learning based resource allocation framework for network slicing. arXiv preprint arXiv:2111.08397

  22. Yang H, Xiong Z, Zhao J, Niyato D, Yuen C, Deng R (2020) Deep reinforcement learning based massive access management for ultra-reliable low-latency communications. IEEE Trans Wireless Commun 20(5):2977–2990

    Article  Google Scholar 

  23. Shao Y, Li R, Hu B, Wu Y, Zhao Z, Zhang H (2021) Graph attention network-based multi-agent reinforcement learning for slicing resource management in dense cellular network. IEEE Trans Veh Technol 70(10):10792–10803

    Article  Google Scholar 

  24. Dong T, Zhuang Z, Qi Q, Wang J, Sun H, Yu FR, Sun T, Zhou C, Liao J (2021) Intelligent joint network slicing and routing via GCN-powered multi-task deep reinforcement learning. IEEE Trans Cogn Commun Netw 8(2):1269–1286. https://doi.org/10.1109/TCCN.2021.3136221

    Article  Google Scholar 

  25. Alsuhli G, Banawan K, Attiah K, Elezabi A, Seddik K, Gaber A, Zaki M, Gadallah Y (2021) Mobility load management in cellular networks: A deep reinforcement learning approach. IEEE Trans Mobile Comput. https://doi.org/10.1109/TMC.2021.3107458

    Article  Google Scholar 

  26. Naderializadeh N, Sydir JJ, Simsek M, Nikopour H (2021) Resource management in wireless networks via multi-agent deep reinforcement learning. IEEE Trans Wireless Commun 20(6):3507–3523

    Article  Google Scholar 

  27. Hua Y, Li R, Zhao Z, Chen X, Zhang H (2019) GAN-powered deep distributional reinforcement learning for resource management in network slicing. IEEE J Sel Areas Commun 38(2):334–349

    Article  Google Scholar 

  28. Vilà I, Pèrez-Romero J, Sallent O, Umbert A (2021) A multi-agent reinforcement learning approach for capacity sharing in multi-tenant scenarios. IEEE Trans Veh Technol 70(9):9450–9465

    Article  Google Scholar 

  29. Gupta RK, Choubey A, Jain S, Greeshma RR, Misra R (2020) July) Machine learning based network slicing and resource allocation for electric vehicles (EVs). International Conference on Internet of Things and Connected Technologies. Springer, Cham, pp 333–347

    Google Scholar 

  30. Kanani CS, Gill KS, Behera S, Choubey A, Gupta RK, Misra R (2020) July) Shallow over deep neural networks: a empirical analysis for human emotion classification using audio data. International Conference on Internet of Things and Connected Technologies. Springer, Cham, pp 134–146

    Google Scholar 

  31. Gupta RK, Ranjan A, Moid MA, Misra R (2020) July) Deep-learning based mobile-traffic forecasting for resource utilization in 5G network slicing. International Conference on Internet of Things and Connected Technologies. Springer, Cham, pp 410–424

    Google Scholar 

  32. Van Hasselt H, Guez A, Silver D (2016, March) Deep reinforcement learning with double q-learning. In: Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 30, No. 1)

  33. Hessel M, Modayil J, Van Hasselt H, Schaul T, Ostrovski G, Dabney W, Horgan D, Piot B, Azar M, Silver D (2018, April) Rainbow: Combining improvements in deep reinforcement learning. In: Thirty-Second AAAI Conference on Artificial Intelligence

  34. Lillicrap TP, Hunt JJ, Pritzel A, Heess N, Erez T, Tassa Y, Silver D, Wierstra D (2015) Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971

  35. Mnih V, Badia AP, Mirza M, Graves A, Lillicrap T, Harley T, Silver D, Kavukcuoglu K (2016) June) Asynchronous methods for deep reinforcement learning. International Conference on Machine Learning. PMLR, pp 1928–1937

    Google Scholar 

  36. Fortunato M, Azar MG, Piot B, Menick J, Osband I, Graves A, Mnih V, Munos R, Hassabis D, Pietquin O, Blundell C (2017) Noisy networks for exploration. arXiv preprint arXiv:1706.10295

  37. Ji L, He S, Wu W, Gu C, Bi J, Shi Z (2021) Dynamic network slicing orchestration for remote adaptation and configuration in industrial IoT. IEEE Trans Ind Inf1 8(6):4297–4307

    Google Scholar 

  38. Liang F, Yu W, Liu X, Griffith D, Golmie N (2021) Towards deep Q-network based resource allocation in industrial internet of things. IEEE Int Things J 9(12):9138–9150. https://doi.org/10.1109/JIOT.2021.3093346

    Article  Google Scholar 

  39. Mei J, Wang X, Zheng K, Boudreau G, Sediq AB, Abou-Zeid H (2021) Intelligent radio access network slicing for service provisioning in 6G: A hierarchical deep reinforcement learning approach. IEEE Trans Commun 69(9):6063–6078

    Article  Google Scholar 

  40. 3GPP TS 38.306: “5G; NR; User Equipment (UE) radio access capabilities”

  41. Shi Y, Sagduyu YE, Erpek T (2020, September) Reinforcement learning for dynamic resource optimization in 5G radio access network slicing. In: 2020 IEEE 25th international workshop on computer aided modeling and design of communication links and networks (CAMAD), pp 1–6. IEEE

  42. Awais Jadoon M, Pastore A, Navarro M, Perez-Cruz F (2022) Deep reinforcement learning for random access in machine-type communication. arXiv e-prints, arXiv-2201

  43. Yang H, Xie X (2019) An actor-critic deep reinforcement learning approach for transmission scheduling in cognitive internet of things systems. IEEE Syst J 14(1):51–60

    Article  Google Scholar 

  44. Zhao L, Wang H, Zhong X (2018) Interference graph based channel assignment algorithm for D2D cellular networks. IEEE Access 6:3270–3279

    Article  Google Scholar 

  45. Kitindi EJ, Fu S, Jia Y, Kabir A, Wang Y (2017) Wireless network virtualization with SDN and C-RAN for 5G networks: requirements, opportunities, and challenges. IEEE Access 5:19099–19115

    Article  Google Scholar 

  46. Service requirements for next generation new services and markets, Release 15, document 3GPP TS 22.261

  47. Evolved Universal Terrestrial Radio Access (E-UTRA); Further advancements for E-UTRA physical layer aspects, Release 9, document 3GPP TR 36.814

  48. Zheng S, Chen S, Qi P, Zhou H, Yang X (2020) Spectrum sensing based on deep learning classification for cognitive radios. China Commun 17(2):138–148

    Article  Google Scholar 

Download references

Funding

The preparation of the manuscript is not funded.

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, Indian Institute of Technology Patna, 801106, Bihta, Bihar, India

    Rohit Kumar Gupta, Shashwat Mahajan & Rajiv Misra

Authors
  1. Rohit Kumar Gupta
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Shashwat Mahajan
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Rajiv Misra
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Rohit Kumar Gupta.

Ethics declarations

Conflict of interest

There are no competing interests declared by the authors.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gupta, R.K., Mahajan, S. & Misra, R. Resource orchestration in network slicing using GAN-based distributional deep Q-network for industrial applications. J Supercomput 79, 5109–5138 (2023). https://doi.org/10.1007/s11227-022-04867-9

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-022-04867-9

Keywords