Skip to main content
SpringerLink
Log in

A dynamic assignment scheduling algorithm for big data stream processing in mobile Internet services

  • Original Article
  • Published:
Personal and Ubiquitous Computing Aims and scope Submit manuscript

Abstract

As the huge number of mobile devices (e.g., smart phones, tablets and netbooks) increases, more and more people choose to use the Internet services financed by mobile Internet service providers (MISPs). To provide better services, it is quite necessary for MISPs to analyze the information hidden in the big data stream generated by users. Therefore, processing the real-time big data stream efficiently has become increasingly important. However, traditional static data storage technology fails to meet the demands of real-time data processing. To improve processing capacity, many parallel processing structures are proposed, which brings up the problem about how the parallel devices can be scheduled to maximize their efficiency. Accordingly, a dynamic assignment scheduling algorithm for big data stream processing in mobile Internet services is proposed, and a stream query graph is built to calculate the weight of every edge. The edge with the minimum weight is selected to send tuples. Simulation results show that the proper number of the logic devices can dramatically reduce system response time. Furthermore, system context switching is reduced by increasing the number of tuples sent each time.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17

Similar content being viewed by others

References

  1. Almenares F, Arias P, Marin A et al (2013) Overhead of using secure wireless communications in mobile computing. IEEE Trans Consum Electron 59(2):335–342

    Article  Google Scholar 

  2. Shin S, Ko J, Shin DH et al (2013) Semantic search for smart mobile devices. In: Proceedings of ACM international conference on intelligent user interfaces companion, pp 95–96

  3. Kwon HJ, Hong KS (2012) Personalized real-time location-tagged contents recommender system based on mobile social networks. In: Proceedings of IEEE international conference on consumer electronics (ICCE), pp 558–559

  4. Jang SB, Kim YG, Na HS et al (2009) Embedded system architecture for an WLAN-based dual mode mobile phone. IEEE Trans Consum Electron 55(3):1623–1630

    Article  Google Scholar 

  5. Wang K, Lu H, Shu L et al (2014) A context-aware system architecture for leak point detection in the large-scale petrochemical industry. IEEE Commun Mag 52(6):62–69

    Article  Google Scholar 

  6. Wang K, Shao Y, Shu L et al (2015) LDPA: a local data processing architecture in ambient assisted living communications. IEEE Commun Mag 53(1):56–63

    Article  Google Scholar 

  7. Fong ACM, Zhou B, Hui SC et al (2011) Web content recommender system based on consumer behavior modeling. IEEE Trans Consum Electron 57(2):962–969

    Article  Google Scholar 

  8. Zhang W, Xu L, Duan P et al (2015) A video cloud platform combing online and offline cloud computing technologies. Pers Ubiquit Comput 19(7):1099–1110

    Article  Google Scholar 

  9. Zhou ZB, Zhao D, Shu L et al (2015) A novel two-tier cooperative caching mechanism for the optimization of multi-attribute periodic queries in wireless sensor networks. Sensors 15(7):15033–15066

    Article  Google Scholar 

  10. De Andrade TPC, da Fonseca NLS, Oliveira LB et al (2012) MAC protocols for wireless sensor networks over radio-over-fiber links. In: Proceedings of IEEE international conference on communications (ICC), pp 254–259

  11. He L, Liu G (2012) Optimal cross layer design for video transmission over OFDMA system. In: Proceedings of IEEE international conference on communications (ICC), pp 1154–1159

  12. Zhou Y, Aberer K, Tan KL (2008) Toward massive query optimization in large-scale distributed stream systems. In: Proceedings of the 9th ACM/IFIP/USENIX international conference on middleware. Springer, New York, Inc., pp 326–345

  13. Gopalan A, Caramanis C, Shakkottai S (2012) Low-delay wireless scheduling with partial channel-state information. In: Proceedings of IEEE international conference on computer communications (INFOCOM), pp 1071–1079

  14. Wang G, Gong W, DeRenzi B et al (2007) Ant colony optimizations for resource-and timing-constrained operation scheduling. IEEE Trans Comput Aided Des Integr Circuits Syst 26(6):1010–1029

    Article  Google Scholar 

  15. Mishra A, Venkitasubramaniam P (2012) Source anonymity in fair scheduling: A case for the proportional method. In: Proceedings of 2012 IEEE international conference on communications (ICC), pp 1118–1122

  16. Wang T, Shen Y, Mazuelas S et al (2012) Distributed scheduling for cooperative localization based on information evolution. In: Proceedings of IEEE international conference on communications (ICC), pp 576–580

  17. Wang K, Shao Y, Shu L et al (2016) Mobile big data fault-tolerant processing for eHealth networks. IEEE Network 30(1):36–42

    Article  Google Scholar 

  18. Li Y, Ghamri-Doudane Y (2011) Channel-hole based cooperative scheduling in multiple relay systems. In: Proceedings of IEEE international conference on communications (ICC), pp 1–6

  19. Ashraf A, Jokhio F, Deneke T et al (2013) Stream-based admission control and scheduling for video transcoding in cloud computing. In: Proceedings of the 13th IEEE/ACM international symposium on cluster, cloud and grid computing (CCGrid), pp 482–489

  20. Kang Y, Lin Y (2011) A recursive algorithm for scheduling of tasks in a heterogeneous distributed environment. In: Proceedings of 2011 4th international conference on biomedical engineering and informatics (BMEI), vol 4, pp 2099–2103

  21. Golab L, Johnson T, Shkapenyuk V (2012) Scalable scheduling of updates in streaming data warehouses. IEEE Trans Knowl Data Eng 24(6):1092–1105

    Article  Google Scholar 

  22. Zhou ZB, Tang J, Zhang LJ et al (2014) EGF-tree: an energy-efficient index tree for facilitating multi-region query aggregation in the internet of things. Pers Ubiquit Comput 18(4):951–966

    Article  Google Scholar 

  23. Zhou ZB, Xing R, Gaaloul W et al (2015) A three-dimensional sub-region query processing mechanism in underwater WSNs. Pers Ubiquit Comput 19(7):1075–1086

    Article  Google Scholar 

  24. Wen Y, Zhang G, Zhu X (2011) Lightweight packet scheduling algorithms for content uploading from mobile devices to media cloud. In: Proceedings of IEEE GLOBECOM workshops (GC Wkshps), pp 45–50

  25. Heinze T, Pappalardo V, Jerzak Z et al (2014) Auto-scaling techniques for elastic data stream processing. In: Proceedings of the 30th IEEE international conference on data engineering workshops (ICDEW), pp 296–302

  26. Gopalan A, Caramanis C, Shakkottai S (2012) Low-delay wireless scheduling with partial channel-state information. In: Proceedings of IEEE international conference on computer communications (INFOCOM), pp 1071–1079

  27. Zhang L, Li Z, Wu C (2014) Dynamic resource provisioning in cloud computing: a randomized auction approach. In: Proceedings of IEEE international conference on computer communications (INFOCOM), pp 433–441

  28. Qiaoyu L, Jianwei L, Yubin X (2010) Performance analysis of data organization of the real-time memory database based on red-black tree. In: Proceedings of international conference on computing, control and industrial engineering (CCIE), pp 428–430

  29. Steinmaurer T, Traxler P, Zwick M et al (2014) Combining stream processing engines and big data storages for data analysis. Foundations of Intelligent Systems. Springer International Publishing, pp 476–485

  30. Wang G, Gong W, DeRenzi B et al (2007) Ant colony optimizations for resource-and timing-constrained operation scheduling. IEEE Trans Comput Aided Des Integr Circuits Syst 26(6):1010–1029

    Article  Google Scholar 

  31. Mingsheng S, Shixin S, Qingxian W (2003) An efficient parallel scheduling algorithm of dependent task graphs. In: Proceedings of the 4th IEEE international conference on parallel and distributed computing, applications and technologies (PDCAT), pp 595–598

  32. Shang P, Wu J, Zhu X (2011) Oblique projection based linear precoding for downlink multi-user multiple-input multiple-output communications. In: Proceedings of IEEE GLOBECOM workshops (GC Wkshps), pp 560–564

  33. Huang W, Ding L (2011) Project-scheduling problem with random time-dependent activity duration times. IEEE Trans Eng Manage 58(2):377–387

    Article  Google Scholar 

  34. Chen Q, Zhang D, Guo M et al (2010) Samr: A self-adaptive mapreduce scheduling algorithm in heterogeneous environment. In: Proceedings of the 10th IEEE international conference on computer and information technology (CIT), pp 2736–2743

  35. Zhou Z, Sellami M, Gaaloul W et al (2013) Data providing services clustering and management for facilitating service discovery and replacement. IEEE Trans Autom Sci Eng 10(4):1131–1146

    Article  Google Scholar 

  36. Yang L, Cao J, Yuan Y et al (2013) A framework for partitioning and execution of data stream applications in mobile cloud computing. ACM Sigmetrics Perform Eval Rev 40(4):23–32

    Article  Google Scholar 

  37. Thomas H, Charles E (2009) Introduction to algorithm. MIT Press, ISBN: 9780262533058

  38. Ikeda Y, Shimada S, Miura T (2012) Well formed PetriNet for reachablility. In: Proceedings of the 8th international conference on signal image technology and internet based systems (SITIS), pp 595–602

  39. Kalita M, Bezboruah T (2011) Investigation on performance testing and evaluation of PReWebD: a. NET technique for implementing web application. IET Softw 5(4):357–365

    Article  Google Scholar 

  40. Anicic D, Fodor P, Stühmer R et al (2009) Event-driven approach for logic-based complex event processing. Proc IEEE Int Conf Comput Sci Eng 1:56–63

    Google Scholar 

  41. Wang K, Yu Y (2013) A query-matching mechanism over out-of-order event stream in IOT. Int J Ad Hoc Ubiquitous Comput 13(3/4):197–208

    Article  Google Scholar 

  42. Wang K, Zhuo L, Shao Y et al (2016) Towards distributed data processing on intelligent leakpoints prediction in petrochemical industries. IEEE Trans Ind Inf. doi:10.1109/TII.2016.2537788

    Google Scholar 

Download references

Acknowledgments

This work is supported by NSFC (61572262, 61100213, 61571233, 61373135 and 61572172); SFDPH (20113223120007); NSF of Jiangsu Province (BK20141427), NUPT (NY214097); and Open research fund of Key Lab of Broadband Wireless Communication and Sensor Network Technology (Nanjing University of Posts and Telecommunications), Ministry of Education (NYKL201507).

Author information

Authors and Affiliations

  1. College of Electrical Engineering and Control Science, Nanjing Tech University, Nanjing, 210009, Jiangsu, China

    Yan Liu

  2. Key Lab of Broadband Wireless Communication and Sensor Network Technology (NUPT), Nanjing, 210003, Jiangsu, China

    Kun Wang, Jin Qi & Yanfei Sun

  3. China Development Lab, IBM Research Center, Ningbo, 315100, China

    Yue Yu

Authors
  1. Yan Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kun Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yue Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jin Qi
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yanfei Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kun Wang.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, Y., Wang, K., Yu, Y. et al. A dynamic assignment scheduling algorithm for big data stream processing in mobile Internet services. Pers Ubiquit Comput 20, 373–383 (2016). https://doi.org/10.1007/s00779-016-0914-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00779-016-0914-z

Keywords

Search

Navigation