Abstract
The Fog Computing (FC) paradigm is rapidly becoming an appropriate framework for the infrastructure related to the Internet of Things (IoT). FC can be a good framework for mobile applications in the IoT. This architecture is referred to as the Mobile Fog Computing (MFC). Modules in the applications can be sent to the Fog or Cloud layer in the event of the lack of resources or increased runtime on the mobile. This increases the efficiency of the whole system. As data is entered sequentially, and the input is given to the modules, the number of executable modules increases. So, this research was conducted to find the best place in order to run the modules that can be on the mobile, Fog, or Cloud. According to the proposed method, first, the Fog Devices (FDs) were locally evaluated using a greedy technique; namely, the sibling nodes followed by the parent and in the second step, a Deep Reinforcement Learning (DRL) algorithm found the best destination to execute the module so as to create a compromise between the power consumption and execution time of the modules. The evaluation results obtained regarding the parameters of the power consumption, execution cost, delay, and network resource usage showed that the proposed method on average is better than the local execution, First-Fit (FF), and standard DRL by 18, 6, and 2%, respectively.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Alam MGR, Hassan MM, Uddin MZ, Almogren A, Fortino G (2019) Autonomic computation offloading in mobile edge for IoT applications. Future Gen Comput Syst 90:149–157. https://doi.org/10.1016/j.future.2018.07.050
Boucherie RJ, Van Dijk NM (2017) Markov decision processes in practice vol 248. Springer. https://doi.org/10.1007/978-3-319-47766-4
Boukerche A, Guan S, Grande RED (2019) Sustainable offloading in Mobile cloud computing: algorithmic design and implementation. ACM Comput Surveys (CSUR) 52:1–37. https://doi.org/10.1145/3286688
Cao B, Zhang L, Li Y, Feng D, Cao W (2019) Intelligent offloading in multi-access edge computing: a state-of-the-art review and framework. IEEE Commun Mag 57:56–62. https://doi.org/10.1109/MCOM.2019.1800608
Chang Z, Zhou Z, Ristaniemi T, Niu Z (2017) Energy efficient optimization for computation offloading in fog computing system. In: GLOBECOM 2017–2017 IEEE Global Communications Conference, 2017. IEEE, pp 1–6. https://doi.org/10.1109/GLOCOM.2017.8254207
Chen J, Chen S, Wang Q, Cao B, Feng G, Hu J (2019) iRAF: A deep reinforcement learning approach for collaborative mobile edge computing IoT networks. IEEE Internet Things J 6:7011–7024. https://doi.org/10.1109/JIOT.2019.2913162
Davis MH (1993) Markov models and optimization vol 49. CRC Press
Dinesh S, Veerappa E, Valarmathi K (2020) A novel energy estimation model for constraint based task offloading in mobile cloud computing. J Ambient Intell Human Comput. https://doi.org/10.1007/s12652-020-01903-5
Etemadi M, Ghobaei-Arani M, Shahidinejad A (2020) Resource provisioning for IoT services in the fog computing environment: an autonomic approach. Comput Commun 161:109–131. https://doi.org/10.1016/j.comcom.2020.07.028
Ghobaei-Arani M, Souri A, Rahmanian AA (2019) Resource management approaches in fog computing: a comprehensive review. J Grid Comput 18:1–42. https://doi.org/10.1007/s10723-019-09491-1
Gupta B, Agrawal DP, Yamaguchi S (2019) Deep learning models for human centered computing in fog and mobile edge networks. Springer. https://doi.org/10.1007/s12652-018-0919-8
Gupta H, Vahid Dastjerdi A, Ghosh SK, Buyya R (2017) iFogSim: a toolkit for modeling and simulation of resource management techniques in the Internet of Things Edge and Fog computing environments. Softw Pract Exp 47:1275–1296. https://doi.org/10.1002/spe.2509
Hao P, Hu L, Jiang J, Che X, Li T, Zhao K (2019) Framework for replica placement over cooperative edge networks. J Ambient Intell Human Comput 10:3011–3021. https://doi.org/10.1007/s12652-018-0776-5
Hong C-H, Varghese B (2019) Resource management in fog/edge computing: a survey on architectures, infrastructure, and algorithms. ACM Comput Surveys (CSUR) 52:1–37. https://doi.org/10.1145/3326066
Jia Q, Xie R, Tang Q, Li X, Huang T, Liu J, Liu Y (2019) Energy-efficient computation offloading in 5G cellular networks with edge computing and D2D communications. IET Commun 13:1122–1130. https://doi.org/10.1049/iet-com.2018.5934
Jiang C, Cheng X, Gao H, Zhou X, Wan J (2019) Toward computation offloading in edge computing: a survey. IEEE Access 7:131543–131558. https://doi.org/10.1109/ACCESS.2019.2938660
Khune A, Pasricha S (2018) Mobile network-aware middleware framework for cloud offloading: using reinforcement learning to make reward-based decisions in smartphone applications. IEEE Consumer Electron Mag 8:42–48. https://doi.org/10.1109/MCE.2018.2867972
Kowsigan M, Balasubramanie P (2019) An efficient performance evaluation model for the resource clusters in cloud environment using continuous time Markov chain and Poisson process. Cluster Comput 22:12411–12419. https://doi.org/10.1007/s10586-017-1640-7
Kumar K, Lu Y-H (2010) Cloud computing for mobile users: Can offloading computation save energy? Computer 43:51–56. https://doi.org/10.1109/MC.2010.98
Liu L, Chang Z, Guo X, Mao S, Ristaniemi T (2017) Multiobjective optimization for computation offloading in fog computing. IEEE Internet Things J 5:283–294. https://doi.org/10.1109/JIOT.2017.2780236
Liu N et al. A hierarchical framework of cloud resource allocation and power management using deep reinforcement learning. In: 2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS), 2017b. IEEE, pp 372–382. https://doi.org/10.1109/ICDCS.2017.123
Lu H, Gu C, Luo F, Ding W, Liu X (2020) Optimization of lightweight task offloading strategy for mobile edge computing based on deep reinforcement learning. Future Gen Comput Syst 102:847–861. https://doi.org/10.1016/j.future.2019.07.019
Luong NC, Hoang DT, Gong S, Niyato D, Wang P, Liang Y-C, Kim DI (2019) Applications of deep reinforcement learning in communications and networking: a survey. IEEE Commun Surv Tutor 21:3133–3174. https://doi.org/10.1109/COMST.2019.2916583
Mao H, Alizadeh M, Menache I, Kandula S Resource management with deep reinforcement learning. In: Proceedings of the 15th ACM Workshop on Hot Topics in Networks, 2016. pp 50–56. https://doi.org/https://doi.org/10.1145/3005745.3005750
Martin JP, Kandasamy A, Chandrasekaran K (2020) Mobility aware autonomic approach for the migration of application modules in fog computing environment. J Ambient Intell Human Comput 1:1–20. https://doi.org/10.1007/s12652-020-01854-x
Meng X, Wang W, Zhang Z (2017) Delay-constrained hybrid computation offloading with cloud and fog computing. IEEE Access 5:21355–21367. https://doi.org/10.1109/ACCESS.2017.2748140
Nawrocki P, Sniezynski B (2018) Adaptive service management in mobile cloud computing by means of supervised and reinforcement learning. J Netw Syst Manage 26:1–22. https://doi.org/10.1007/s10922-017-9405-4
Nawrocki P, Sniezynski B, Slojewski H (2019) Adaptable mobile cloud computing environment with code transfer based on machine learning. Pervas Mob Comput 57:49–63. https://doi.org/10.1016/j.pmcj.2019.05.001
Ning Z et al (2019a) Deep reinforcement learning for intelligent Internet of vehicles: an energy-efficient computational offloading scheme. IEEE Trans Cognit Commun Netw 5:1060–1072. https://doi.org/10.1109/TCCN.2019.2930521
Ning Z, Dong P, Wang X, Rodrigues JJ, Xia F (2019b) Deep reinforcement learning for vehicular edge computing: an intelligent offloading system. ACM Trans Intell Syst Technol (TIST) 10:1–24. https://doi.org/10.1145/3317572
Ning Z, Huang J, Wang X, Rodrigues JJ, Guo L (2019c) Mobile edge computing-enabled Internet of vehicles: toward energy-efficient scheduling. IEEE Network 33:198–205. https://doi.org/10.1109/MNET.2019.1800309
Noor TH, Zeadally S, Alfazi A, Sheng QZ (2018) Mobile cloud computing: challenges and future research directions. J Netw Comput Appl 115:70–85. https://doi.org/10.1016/j.jnca.2018.04.018
Qi Q, Wang J, Ma Z, Sun H, Cao Y, Zhang L, Liao J (2019) Knowledge-driven service offloading decision for vehicular edge computing: a deep reinforcement learning approach. IEEE Trans Veh Technol 68:4192–4203. https://doi.org/10.1109/TVT.2019.2894437
Ramírez W, Masip-Bruin X, Marin-Tordera E, Souza VBC, Jukan A, Ren G-J, de Dios OG (2017) Evaluating the benefits of combined and continuous Fog-to-Cloud architectures. Comput Commun 113:43–52. https://doi.org/10.1016/j.comcom.2017.09.011
Ren J, Zhang D, He S, Zhang Y, Li T (2019) A survey on end-edge-cloud orchestrated network computing paradigms: transparent computing, mobile edge computing, fog computing, and Cloudlet. ACM Comput Surv (CSUR) 52:1–36. https://doi.org/10.1145/3362031
Samie F, Bauer L, Henkel J (2019) From cloud down to things: An overview of machine learning in internet of things. IEEE Internet Things J. 6:4921–4934. https://doi.org/10.1109/JIOT.2019.2893866
Shahidinejad A, Ghobaei-Arani M (2020) Joint computation offloading and resource provisioning for edge-cloud computing environment: A machine learning-based approach. Softw Pract Exp 1:1. https://doi.org/10.1002/spe.2888
Shahidinejad A, Ghobaei-Arani M, Esmaeili L (2020a) An elastic controller using Colored Petri Nets in cloud computing environment. Cluster Comput. 23:1045–1071. https://doi.org/10.1007/s10586-019-02972-8
Shahidinejad A, Ghobaei-Arani M, Masdari M (2020b) Resource provisioning using workload clustering in cloud computing environment: a hybrid approach. Cluster Comput. https://doi.org/10.1007/s10586-020-03107-0
Shakarami A, Ghobaei-Arani M, Masdari M, Hosseinzadeh M (2020a) A survey on the computation offloading approaches in mobile edge/cloud computing environment: a stochastic-based perspective. J. Grid Comput. https://doi.org/10.1007/s10723-020-09530-2
Shakarami A, Ghobaei-Arani M, Shahidinejad A (2020b) A survey on the computation offloading approaches in mobile edge computing: a machine learning-based perspective. Comput Netw 182:107496. https://doi.org/10.1016/j.comnet.2020.107496
Shakarami A, Shahidinejad A, Ghobaei-Arani M (2020c) A review on the computation offloading approaches in mobile edge computing: a game-theoretic perspective. Softw Pract Exp 50:1719–1759. https://doi.org/10.1002/spe.2839
Shu Y, Zhu F (2020) An edge computing offloading mechanism for mobile peer sensing and network load weak balancing in 5G network. J Ambient Intell Human Comput 11:503–510. https://doi.org/10.1007/s12652-018-0970-5
Sittón-Candanedo I, Alonso RS, Corchado JM, Rodríguez-González S, Casado-Vara R (2019) A review of edge computing reference architectures and a new global edge proposal. Future Gen Comput Syst 99:278–294. https://doi.org/10.1016/j.future.2019.04.016
Srinivasan PD et al. (2019) Distributed training of reinforcement learning systems. Google Patents,
Sutton RS, Barto AG (2018) Reinforcement learning: an introduction. MIT press,
Talaat FM, Saraya MS, Saleh AI, Ali HA, Ali SH (2020) A load balancing and optimization strategy (LBOS) using reinforcement learning in fog computing environment. J Ambient Intell Human Comput 1:1–16. https://doi.org/10.1007/s12652-020-01768-8
Tan LN (2017) Omnidirectional-vision-based distributed optimal tracking control for mobile multirobot systems with kinematic and dynamic disturbance rejection. IEEE Trans Ind Electron 65:5693–5703. https://doi.org/10.1109/TIE.2017.2782245
Tan LN (2018) Distributed H∞ optimal tracking control for strict-feedback nonlinear large-scale systems with disturbances and saturating actuators. IEEE Trans Syst Man Cybern Syst. https://doi.org/10.1109/TSMC.2018.2861470
Tian Z, Si X, Zheng Y, Chen Z, Li X (2020) Multistep medical image segmentation based on reinforcement learning. J Ambient Intell Human Comput. https://doi.org/10.1007/s12652-020-01905-3
Tran DH, Tran NH, Pham C, Kazmi SA, Huh E-N, Hong CS (2017) OaaS: offload as a service in fog networks. Computing 99:1081–1104. https://doi.org/10.1007/s00607-017-0551-z
Vázquez-Canteli JR, Nagy Z (2019) Reinforcement learning for demand response: a review of algorithms and modeling techniques. Appl Energy 235:1072–1089. https://doi.org/10.1016/j.apenergy.2018.11.002
Venticinque S, Amato A (2019) A methodology for deployment of IoT application in fog. J Ambient Intel Human Comput 10:1955–1976. https://doi.org/10.1007/s12652-018-0785-4
Wang Y, Jin H A boosting-based deep neural networks algorithm for reinforcement learning. In: 2018 Annual American Control Conference (ACC), 2018. IEEE, pp 1065–1071. https://doi.org/10.23919/ACC.2018.8431647
Wang Y, Wang K, Huang H, Miyazaki T, Guo S (2018) Traffic and computation co-offloading with reinforcement learning in fog computing for industrial applications. IEEE Trans Industr Inf 15:976–986
Wei Y, Yu FR, Song M, Han Z (2018) Joint optimization of caching, computing, and radio resources for fog-enabled IoT using natural actor–critic deep reinforcement learning. IEEE Internet Things J. 6:2061–2073. https://doi.org/10.1109/JIOT.2018.2878435
Xu J, Chen L, Ren S (2017) Online learning for offloading and autoscaling in energy harvesting mobile edge computing. IEEE Trans Cognit Commun Netw 3:361–373. https://doi.org/10.1109/TCCN.2017.2725277
Zhao X, Zhao L, Liang K An energy consumption oriented offloading algorithm for fog computing. In: International Conference on Heterogeneous Networking for Quality, Reliability, Security and Robustness, 2016. Springer, pp 293–301. https://doi.org/10.1007/978-3-319-60717-7_29
Zhou B, Song Q, Zhao Z, Liu T (2020) A reinforcement learning scheme for the equilibrium of the in-vehicle route choice problem based on congestion game. Appl Math Comput 371:124895. https://doi.org/10.1016/j.amc.2019.124895
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Jazayeri, F., Shahidinejad, A. & Ghobaei-Arani, M. Autonomous computation offloading and auto-scaling the in the mobile fog computing: a deep reinforcement learning-based approach. J Ambient Intell Human Comput 12, 8265–8284 (2021). https://doi.org/10.1007/s12652-020-02561-3
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12652-020-02561-3