Skip to main content
SpringerLink
Log in

StreamTune: dynamic resource scheduling approach for workload skew in video data center

  • Research Article
  • Published:
Frontiers of Computer Science Aims and scope Submit manuscript

Abstract

Video surveillance applications need video data center to provide elastic virtual machine (VM) provisioning. However, the workloads of the VMs are hardly to be predicted for online video surveillance service. The unknown arrival workloads easily lead to workload skew among VMs. In this paper, we study how to balance the workload skew on online video surveillance system. First, we design the system framework for online surveillance service which consists of video capturing and analysis tasks. Second, we propose StreamTune, an online resource scheduling approach for workload balancing, to deal with irregular video analysis workload with the minimum number of VMs. We aim at timely balancing the workload skew on video analyzers without depending on any workload prediction method. Furthermore, we evaluate the performance of the proposed approach using a traffic surveillance application. The experimental results show that our approach is well adaptive to the variation of workload and achieves workload balance with less VMs.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Hu H, Wen Y G, Chua T S, Li X L. Toward scalable systems for big data analytics: a technology tutorial. IEEE Access, 2014, 2: 652–687

    Article  Google Scholar 

  2. Ma H D, Liu L, Zhou A F, Zhao D. On networking of Internet of things: explorations and challenges. IEEE Internet of Things Journal, 2016, 3(4): 441–452

    Article  Google Scholar 

  3. Ma H D. Internet of things: objectives and scientific challenges. Journal of Computer Science and Technology, 2011, 26(6): 919–924

    Article  Google Scholar 

  4. Zhu W W, Luo C, Wang J F, Li S P. Multimedia cloud computing. IEEE Signal Processing Magazine, 2011, 28(3): 59–69

    Article  Google Scholar 

  5. Ahlgren B, Aranda P A, Chemouil P, Oueslati S, Correia L M, Karl H, Sollner M, Welin A. Content, connectivity and cloud: ingredients for the network of the future. IEEE Communication Magazine, 2011, 49(7): 62–70

    Article  Google Scholar 

  6. Yang L, Cao J N, Yuan Y, Li T, Han A, Chan A. A framework for partitioning and execution of data stream applications in mobile cloud computing. ACM SIGMETRICS Performance Evaluation Review, 2013, 40(4): 23–32

    Article  Google Scholar 

  7. Gao Y H, Ma H D, Zhang H T, Kong X Q, Wei W Y. Concurrency optimized task scheduling for workflows in cloud. In: Proceedings of the 6th IEEE International Conference on Cloud Computing. 2013, 709–716

    Google Scholar 

  8. Qian Z P, He Y, Su C Z, Wu Z J, Zhu H Y, Zhang T Z, Zhou L D, Yu Y, Zhang Z. Time Stream: reliable stream computation in the cloud. In: Proceedings of the 8th ACM European Conference on Computer Systems. 2013, 1–14

    Google Scholar 

  9. Zhao X M, Ma H D, Zhan H T, Tang Y, Kou Y. HVPI: extending Hadoop to support video analytic applications. In: Proceedings of the 8th IEEE International Conference on Cloud Computing. 2014, 789–796

    Google Scholar 

  10. Liu W, Mei T, Zhang Y D, Che C, Luo J B. Multi-task deep visualsemantic embedding for video thumbnail selection. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2015, 3707–3715

    Google Scholar 

  11. Liu W, Zhang Y D, Tang S, Tang J H, Hong R, Li J T. Accurate estimation of human body orientation from RGB-D sensors. IEEE Transactions on Cybernetics, 2013, 43(5): 1442–1452

    Article  Google Scholar 

  12. Liu W, Mei T, Zhang Y D. Instant mobile video search with layered audio-video indexing and progressive transmission. IEEE Transactions on Multimedia, 2014, 16(8): 2242–2255

    Article  Google Scholar 

  13. Saini M, Wang X, Atrey P K, Kankanhalli M. Adaptive workload equalization in multi-camera surveillance systems. IEEE Transactions on Multimedia, 2012, 14(3): 555–562

    Article  Google Scholar 

  14. Gao G Y, Zhang W W, Wen Y G, Wang Z, Zhu WW, Tan Y P. Cost optimal video transcoding in media cloud: insights from user viewing pattern. In: Proceedings of IEEE International Conference on Multimedia & Expo. 2014, 1–6

    Google Scholar 

  15. Ren S L, Van der Schaar M. Efficient resource provisioning and rate selection for stream mining in a community cloud. IEEE Transactions on Multimedia, 2013, 15(4): 723–734

    Article  Google Scholar 

  16. Kwon Y C, Balazinska M, Rolia J. Skew-resistant parallel processing of feature-extracting scientific user-defined functions. In: Proceedings of the 1st ACM Symposium on Cloud Computing. 2010, 75–86

    Chapter  Google Scholar 

  17. Pavlo A, Curino C, Zdonik S. Skew-aware automatic database partitioning in shared-nothing, parallel OLTP systems. In: Proceedings of ACM SIGMOD International Conference on Management of Data. 2012, 61–72

    Google Scholar 

  18. Ramakrishnan S R, Swart G, Urmanov A. Balanceing reducer skew in Map Reduce workloads using progressive smapling. In: Proceedings of the 3rd ACM Symposium on Cloud Computing. 2012, 1–13

    Google Scholar 

  19. Kwon Y C, Balazinska M, Howe B, Rolia J. Skew Tune: mitigating skew in Map Reduce applications. In: Proceedings of ACM SIGMOD International Conference on Management of Data. 2012, 25–36

    Google Scholar 

  20. Chen Q, Liu C, Xiao Z. Improving Map Reduce performance using smart speculative execution strategy. IEEE Transactions on Computers, 2014, 63(4): 954–967

    Article  MathSciNet  MATH  Google Scholar 

  21. Ananthanarayanan G, Agarwal S, Kandula S, Greenberg A, Stoica I, Harlan D, Harris E. Scarlett: coping with skewed content popularity in Map Reduce clusters. In: Proceedings of the 6th ACM European Conference on Computer Systems. 2011, 287–300

    Google Scholar 

  22. Le Y F, Liu J C, Ergün F. Wang D. Online load balancing for Map Reduce with skewed data input. In: Proceedings of the 33rd Annual IEEE International Conference on Computer Communications. 2014, 2004–2012

    Google Scholar 

  23. Tang J H, Tay W P, Wen Y G. Dynamic request redirection and elastic service scaling in cloud-centric media networks. IEEE Transactions on Multimedia, 2014, 16(5): 1434–1445

    Article  Google Scholar 

  24. Zhao X M, Ma H D, Zhang H T, Tang Y, Fu G P. Metadata extraction and correction for large-scale traffic surveillance videos. In: Proceedings of IEEE International Conference on Big Data. 2014, 412–420

    Google Scholar 

  25. Neely M J. Stochastic network optimization with application to communication and queueing system. Synthesis Lectures on Communication Networks, 2010 3(1): 1–211

    Article  MATH  Google Scholar 

  26. Feris R S, Siddiquie B, Petterson J, Zhai Y, Datta A, Brown L M, Pankanti S. Large-scale vehicle detection, indexing, and search in urban surveillance videos. IEEE Transactions on Multimedia, 2012, 14(1): 28–42

    Article  Google Scholar 

  27. Dean J, Ghemawat S. Map Reduce: simplified data processing on large clusters. In: Proceedings of the USENIX Symposium on Operating Systems Design and Implementation. 2004, 137–150

    Google Scholar 

  28. Zaharia M, Das T, Li H Y, Hunter T, Shenker S, Stoica I. Discretized streams: fault-tolerant streaming computation at scale. In: Proceedings of the 23th ACM Symposium on Operating Systems Principles. 2013, 423–438

    Google Scholar 

  29. Maguluri S T, Strikant R. Scheduling jobs with unkonwn duration in clouds. In: Proceedings of the 33rd Annual IEEE International Conference on Computer Communications. 2013, 1935–1943

    Google Scholar 

  30. Huang F, Anandkumar A. FCD: fast-concurrent-distributed load balanceing under switch costs and imperfect observation. In: Proceedings of the 33rd Annual IEEE International Conference on Computer Communications. 2013, 1944–1952

    Google Scholar 

  31. Khayyat Z, Awara K, Alonazi A, Jamjoom H, Williams D, Kalnis P. Mizan: a system for dynamic load balancing in large-scale graph processing. In: Proceedings of the 8th ACM European Conference on Computer Systems. 2013, 169–182

    Google Scholar 

Download references

Acknowledgements

The research reported in this paper was supported by the National High-Tech R&D Program (863 Program) (2015AA01A705), the NSFC-Guangdong Joint Found (U1501254), the Cosponsored Project of Beijing Committee of Education, and the Beijing Training Project for the Leading Talents in S&T (ljrc 201502).

Author information

Authors and Affiliations

  1. Beijing Key Lab of Intelligent Telecommunications Software and Multimedia, School of Computer Science, Beijing University of Posts and Telecommunications, Beijing, 100876, China

    Yihong Gao & Huadong Ma

Authors
  1. Yihong Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Huadong Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yihong Gao.

Additional information

Yihong Gao is now a PhD candidate of School of Computer Science, Beijing University of Posts and Telecommunications, China. His current research mainly focuses on resource scheduling approach, video data center, and cloud computing.

Huadong Ma is a Chang Jiang Scholar professor and director of Beijing Key Lab of Intelligent Telecommunications Software and Multimedia, and also the executive dean of School of Computer Science, Beijing University of Posts and Telecommunications, China. He received his PhD degree in computer science from Institute of Computing Technology, Chinese Academy of Sciences, China in 1995. He was an awardee of the National Science Funds for Distinguished Young Scholars in 2009. His current research focuses on multimedia system and networking, sensor networks and Internet of things, and he has published over 180 papers and four books on these fields. He is a member of IEEE and ACM.

Electronic supplementary material

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gao, Y., Ma, H. StreamTune: dynamic resource scheduling approach for workload skew in video data center. Front. Comput. Sci. 12, 669–681 (2018). https://doi.org/10.1007/s11704-016-5438-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11704-016-5438-1

Keywords

Search

Navigation