Abstract
In the internet of things (IoT) environment consisting of various devices the traffic condition dynamically changes. Failure to process them in complying with the QoS requirement can significantly degrade the reliability and quality of the system. Therefore, the gateway collecting the data needs to quickly establish a new scheduling policy according to the changing traffic condition. The traditional packet scheduling schemes are not effective for IoT since the data transmission pattern is not identified in advance. Q-learning is a type of reinforcement learning that can establish a dynamic scheduling policy without any prior knowledge on the network condition. In this paper a novel Q-learning scheme is proposed which updates the Q-table and reward table based on the condition of the queues in the gateway. Computer simulation reveals that the proposed scheme significantly increases the number of packets satisfying the delay requirement while decreasing the processing time compared to the existing scheme based on Q-learning with stochastic learning automaton. And the processing time is also minimized by omitting unnecessary computation steps in selecting the action in the iterative Q-learning operations.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Ahmad S, Malik S, Kim D-H (2018) Comparative analysis of simulation tools with visualization based on real-time task scheduling algorithms for IoT embedded applications. Int J Grid Distrib Comput 11:1–10
Alsheikh MA, Hoang DT, Niyato D, Tan H-P, Lin S (2015) Markov decision processes with applications in wireless sensor networks: a survey. IEEE Commun Surv Tutor 17:1239–1267. https://doi.org/10.1109/COMST.2015.2420686
Babar M, Arif F (2018) Real-time data processing scheme using big data analytics in internet of things based smart transportation environment. J Ambient Intell Humaniz Comput. https://doi.org/10.1007/s12652-018-0820-5
Cao B, Li Y, Wang C, Feng G, Oin S, Zhou Y (2017) Resource allocation in software defined wireless networks. IEEE Netw 31:44–51. https://doi.org/10.1109/MNET.2016.1500273NM
Chekired DA, Khoukhi L, Mouftah HT (2018) Industrial IoT data scheduling based on hierarchical fog computing: a key for enabling smart factory. IEEE Trans Ind Inf 14:1–12. https://doi.org/10.1109/TII.2018.2843802
Chen H, Zhu X, Guo H, Zhu J, Qin X, Wu J (2015) Towards energy-efficient scheduling for real-time tasks under uncertain cloud computing environment. J Syst Softw 99:20–35. https://doi.org/10.1016/j.jss.2014.08.065
Desai R, Patil BP (2015) Cooperative reinforcement learning approach for routing in ad hoc networks. In: International conference on pervasive computing, pp 1–5. https://doi.org/10.1109/PERVASIVE.2015.7086962
Dey N, Hassanien AE, Bhatt C, Ashour AS (2018) Internet of things and big data analytics toward next-generation intelligence. Springer, Berlin. https://doi.org/10.1007/978-3-319-60435-0
Diro AA, Reda HT, Chilamkurti N (2018) Differential flow space allocation scheme in SDN based fog computing for IoT applications. J Ambient Intell Humaniz Comput. https://doi.org/10.1007/s12652-017-0677-z
Dong Z, Liu N, Rojas-Cessa R (2015) Greedy scheduling of tasks with time constraints for energy-efficient cloud-computing data centers. J Cloud Comput Adv Syst Appl. https://doi.org/10.1186/s13677-015-0031-y
Ferra H, Lau K, Leckie C, Tang A (2003) Applying reinforcement learning to packet scheduling in routers. In: Proceeding of the fifteenth innovative applications of artificial intelligence conference, pp 79–84
Hall J, Mars P (1998) Satisfying QoS with a learning based scheduling algorithm. In: 6th international workshop on quality of service, pp 171–173. https://doi.org/10.1109/IWQOS.1998.675236
He X, Wang K, Huang H, Miyazaki T, Wang Y, Guo S (2018) Green resource allocation based on deep reinforcement learning in content-centric IoT. IEEE Trans Emerg Top Comput. https://doi.org/10.1109/TETC.2018.2805718
Itoh N, Kaneko H, Kohiga A, Iwai T, Shimonishi H (2017) Novel packet scheduling for supporting various real-time IoT applications in LTE networks. In: IEEE international workshop technical committee on communications quality and reliability, pp 1–6. http://dx.doi.org/10.1109/CQR.2017.8289445
Kim D, Lee T, Kim S, Lee B, Youn HY (2018) Adaptive packet scheduling in IoT environment based on Q-learning. In: International conference on emerging ubiquitous systems and pervasive networks. https://doi.org/10.1016/j.procs.2018.10.178
Kumar RA, Varshini KM (2014) Multilevel priority packet scheduling scheme for wireless networks. Int J Distrib Parallel Syst 5:69–76
Li L, Li S, Zhao S (2014) QoS-aware scheduling of services-oriented internet of things. IEEE Trans Ind Inform 10:1497–1505. https://doi.org/10.1109/TII.2014.2306782
Narman HS, Hossain S, Atizuzzaman M, Schen H (2017) Scheduling internet of things applications in cloud computing. Ann Telecommun 72:79–83. https://doi.org/10.1007/s12243-016-0527-6
Nasser N, Karim L, Taleb T (2013) Dynamic multilevel priority packet scheduling scheme for wireless sensor network. IEEE Trans Wirel Commun 12:1448–1459. https://doi.org/10.1109/TWC.2013.021213.111410
Orhean AI, Florin Pop, Ioan Raicu (2017) New scheduling approach using reinforcement learning for heterogeneous distributed systems. J Parallel Distrib Comput 117:292–302. https://doi.org/10.1016/j.jpdc.2017.05.001
Peng Z, Cui D, Zuo J, Li Q, Xu B, Lin W (2015) Random task scheduling based on reinforcement learning in cloud computing. Cluster Comput 18:1595–1607. https://doi.org/10.1007/s10586-015-0484-2
Shahrabi J, Adibi MA, Mahootchi M (2017) A reinforcement learning approach to parameter estimation in dynamic job shop scheduling. Comput Ind Eng 110:75–82. https://doi.org/10.1016/j.cie.2017.05.026
Sharma R, Kumar N, Gowda NB, Srinivas T (2018) Packet scheduling scheme to guarantee QoS in internet of things. Wirel Pers Commun 100:557–569. https://doi.org/10.1007/s11277-017-5218-8
Sledge IJ, Principe JC (2017) Balancing Exploration and Exploitation in Reinforcement Learning using a Value of Information Criterion. In: IEEE international conference on acoustics, speech and signal processing, pp 2816–2820. https://doi.org/10.1109/ICASSP.2017.7952670
Sombolestan SM, Rasooli A, Khodaygan (2018) Optimal path-planning for mobile robots to find a hidden target in an unknown environment based on machine learning. J Ambient Intell Humaniz Comput. https://doi.org/10.1007/s12652-018-0777-4
Tijsma AD, Drugan MM, Wiering MA (2016) Comparing Exploration Strategies for Q-learning in Random Stochastic Mazes. In: IEEE Symposium series on computational intelligence, pp 1–8
Yau K-LA, Komisarczuk P, Teal PD (2012) Reinforcement learning for context awareness and intelligence in wireless networks: review, new features and open issues. J Netw Comput Appl 35:253–267. https://doi.org/10.1016/jjnca.2011.08.007
Zhang J, Liu X (2018) Evaluation and optimization of QoS-aware network management framework based on process synergy and resource allocation. J Ambient Intell Humaniz Comput. https://doi.org/10.1007/s12652-018-0689-3
Acknowledgements
This work was partly supported by Institute for Information and Communications Technology Promotion (IITP) Grant funded by the Korea government(MSIT) (No. 2016-0-00133, Research on Edge computing via collective intelligence of hyperconnection IoT nodes), Korea, under the National Program for Excellence in SW supervised by the IITP (Institute for Information and communications Technology Promotion) (2015-0-00914), Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education, Science and Technology (2016R1A6A3A119 31385, Research of key technologies based on software defined wireless sensor network for realtime public safety service, 2017R1A2B2009095, Research on SDN-based WSN Supporting Real-time Stream Data Processing and Multiconnectivity), the second Brain Korea 21 PLUS project.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Kim, D., Lee, T., Kim, S. et al. Adaptive packet scheduling in IoT environment based on Q-learning. J Ambient Intell Human Comput 11, 2225–2235 (2020). https://doi.org/10.1007/s12652-019-01351-w
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12652-019-01351-w