Skip to main content
SpringerLink
Log in

Neighborhood search-based job scheduling for IoT big data real-time processing in distributed edge-cloud computing environment

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

Cloud-edge collaboration architecture, which combines edge processing and centralized cloud processing, is suitable for placement and caching of streaming media. A cache-aware scheduling model based on neighborhood search is proposed. The model is divided into four sub-problems: job classification, node resource allocation, node clustering, and cache-aware job scheduling. Firstly, jobs are categorized into three categories, and then different resources are allocated to nodes according to different job execution conditions. Secondly, the nodes with similar capabilities are clustered, and the jobs are cached by delay-waiting. For jobs that do not satisfy the data locality, the jobs are scheduled to the nodes with similar capabilities according to the neighborhood search results. Meanwhile, a cache-aware scheduling algorithm based on neighborhood search is proposed. Experiments show that the proposed algorithm can effectively minimize the delay of content transmission and the cost of content placement, the job execution time is shortened and the processing capacity of the cloud data center is improved.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Gao Z et al (2019) A light-weight trust mechanism for cloud-edge collaboration framework. In: 2019 IEEE 27th International Conference on Network Protocols (ICNP), Chicago. IEEE, pp 1–6

  2. Zhang H, Chen S, Zou P, Xiong G, Zhao H, Zhang Y (2019) Research and application of industrial equipment management service system based on cloud-edge collaboration. In: 2019 Chinese Automation Congress (CAC), Hangzhou. IEEE, pp 5451–5456

  3. Hao Y, Jiang Y, Chen T, Cao D, Chen M (2019) iTaskOffloading: intelligent task offloading for a cloud-edge collaborative system. IEEE Netw 33(5):82–88

    Article  Google Scholar 

  4. Liang W, Huang J (2018) Research on streaming media adaptive congestion control technology. In: 2018 International Conference on Sensor Networks and Signal Processing (SNSP), Xi’an. IEEE, pp 482–485

  5. Cho J, Ko H, Ko I (2016) Adaptive service selection according to the service density in multiple Qos aspects. IEEE Trans Serv Comput 9(6):883–894

    Article  Google Scholar 

  6. Keshanchi B, Souri A, Navimipour NJ (2017) An improved genetic algorithm for task scheduling in the cloud environments using the priority queues: formal verification, simulation, and statistical testing. J Syst Softw 124(2):1–21

    Article  Google Scholar 

  7. Chen H, Zhu X, Liu G et al (2018) Uncertainty-aware online scheduling for real-time workflows in cloud service environment. IEEE Trans Serv Comput. https://doi.org/10.1109/TSC.2018.2866421

    Article  Google Scholar 

  8. Zhang PY, Zhou MC (2018) Dynamic cloud task scheduling based on a two-stage strategy. IEEE Trans Autom Sci Eng 15(2):772–783

    Article  Google Scholar 

  9. Chen CH, Lin JW, Kuo SY (2018) MapReduce scheduling for deadline-constrained jobs in heterogeneous cloud computing systems. IEEE Trans Cloud Comput 6(1):127–140

    Article  Google Scholar 

  10. Ahani G, Yuan D (2020) Optimal scheduling of content caching subject to deadline. IEEE Open J Commun Soc 1:293–307

    Article  Google Scholar 

  11. Kang L, Tang B, Zhang L, Tang L (2019) Mobility-aware and data caching-based task scheduling strategy in mobile edge computing. In: 2019 IEEE Intl Conf on Parallel & Distributed Processing with Applications, Big Data & Cloud Computing, Sustainable Computing & Communications, Social Computing & Networking (ISPA/BDCloud/SocialCom/SustainCom), Xiamen, pp 1071–1077

  12. Akhavanbitaghsir S, Khonsari A (2018) Cooperative caching for content dissemination in vehicular networks. Int J Commun Syst 31(3):1–22

    Google Scholar 

  13. Gopalan NP, Suresh S (2015) Modified delay scheduling: a heuristic approach for hadoop scheduling to improve fairness and response time. Parallel Process Lett 25(04):1550–1559

    Article  MathSciNet  Google Scholar 

  14. Lim B, Kim JW, Chung YD (2017) CATS: cache-aware task scheduling for hadoop-based systems. Cluster Comput 20(1):1–15

    Article  Google Scholar 

  15. Ying C, Sun L, Chong H et al (2018) Improved side information generation algorithm based on naive bayesian theory for distributed video coding. IET Image Process 12(3):354–360

    Article  Google Scholar 

  16. Mathiya BJ, Desai VL (2016) Apache Hadoop Yarn MapReduce job classification based on cpu utilization and performance evaluation on multi-cluster heterogeneous environment. In: Proceedings of 9th International Conference on ICT for Sustainable Development. Springer, Singapore, pp 35–44

  17. Zhang X, Hu B, Jiang J (2014) An optimized algorithm for reduce task scheduling. J Comput 9(4):794–802

    Google Scholar 

  18. Mccuaig W, Shepherd B (2010) Domination in graphs with minimum degree two. J Graph Theory 13(6):749–762

    Article  MathSciNet  Google Scholar 

  19. Alibaba cloud. IT services. [2020-1-2]. https://www.aliyun.com/ysc/1895756.html

  20. Yu H, Zheng D, Zhao BY et al (2006) Understanding user behavior in large-scale video-on-demand systems. ACM SIGOPS Operat Syst Rev 40(4):333–344

    Article  Google Scholar 

  21. Online Github Google. Youtube-8m. [2019-07-01]. https://github.com/google/youtube-8m

  22. Online Github Intel-Hadoop. HiBench. [2019-07-1]. https://github.com/intel-Hadoop/HiBench

  23. Chunlin Li, Hezhi Sun, Chen Yi, Youlong Luo (2019) Edge cloud resource expansion and shrinkage based on workload for minimizing the cost. Future Gener Comput Syst 101:327–340

    Article  Google Scholar 

Download references

Acknowledgements

The work was supported by Open Fund Project of Key Laboratory of Modern Measurement and Control Technology, Ministry of Education, Beijing Information Science & Technology University (KF20201123205), Application Foundation Frontier Project of Wuhan (No. 2018010401011290). Any opinions, findings, and conclusions are those of the authors and do not necessarily reflect the views of the above agencies.

Author information

Authors and Affiliations

  1. Key Laboratory of Modern Measurement and Control Technology, Ministry of Education, Beijing Information Science and Technology University, Beijing, 100192, People’s Republic of China

    Chunlin Li

  2. Department of Computer Science, Wuhan University of Technology, Wuhan, 430063, People’s Republic of China

    Chunlin Li, YiHan Zhang & Youlong Luo

Authors
  1. Chunlin Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. YiHan Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Youlong Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Youlong Luo.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, C., Zhang, Y. & Luo, Y. Neighborhood search-based job scheduling for IoT big data real-time processing in distributed edge-cloud computing environment. J Supercomput 77, 1853–1878 (2021). https://doi.org/10.1007/s11227-020-03343-6

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-020-03343-6

Keywords

Search

Navigation