Skip to main content
SpringerLink
Log in

Efficient flow detection and scheduling for SDN-based big data centers

  • Original Research
  • Published:
Journal of Ambient Intelligence and Humanized Computing Aims and scope Submit manuscript

Abstract

In Software defined networking (SDN) based datacenters, flow-level management seriously limits system scalability due to large amount of control messages between data and control planes; and mice flows often are blocked by elephant flows because of the indiscriminate flow scheduling. To improve management efficiency and system performance, it is prerequisite to schedule elephant and mice flows respectively. Unfortunately, existing flow scheduling approaches in SDN consider only elephant flows. In this paper, we firstly propose an efficient flow detection mechanism. Then, we propose a novel DIFFERENtiated sChEduling (DIFFERENCE) approach that dynamically sets up paths for elephant and mice flows separately, based on current link workload. Our DIFFERENCE schedules mice flows with proactively installed weighted multipath routing algorithm and adjusts path weight according to link utilization. Instead, we propose a blocking island based path setup algorithm for elephant flows, which find the least congested path with shorter searching space. To balance traffic in a SDN networks, we design an algorithm to dynamically reschedule data flows in terms of current link utilization ratio. Experiment results on real public datacenter traces demonstrate that our approach outperforms related proposals in terms of various system performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  • Afek Y, Bremler-Barr A, Landau Feibish S et al (2015) Sampling and large flow detection in SDN. ACM SIGCOMM Comput Commun Rev 45(4):345–346

    Article  Google Scholar 

  • Al-Fares M, Loukissas A, Vahdat A (2008) A scalable, commodity data center network architecture. ACM SIGCOMM Comput Commun Rev 38(4):63–74

    Article  Google Scholar 

  • Al-Fares M, Radhakrishnan S, Raghavan B, Huang N, Vadhat A (2010) Hedera: dynamic flow scheduling for data center networks. Proc. NSDI 10:19

    Google Scholar 

  • Benson T, Akella A, Maltz DA (2010) Network traffic characteristics of data centers in the wild. In: Proc. the 10th ACM SIGCOMM conference on internet measurement, Melbourne, pp 267–280

  • Cao Y, Xu M, Fu X, Dong E (2013) Explicit multipath congestion control for data center networks. In: Proc. 9th ACM conf. emerging netw. exp. technol. (CoNEXT), Santa Barbara, CA, pp 73–84

  • Cao ZZ, Kodialam M, Lakshman TV (2014) Joint static and dynamic traffic scheduling in data center networks. In: Proc. IEEE INFOCOM, Toronto, pp 2445–2553

  • Curtis AR, Mogul JC, Tourrilhes J.et al (2011a) DevoFlow: scaling flow management for high-performance networks. ACM SIGCOMM Comput Commun Rev 41(4):254–265

    Article  Google Scholar 

  • Curtis AR, Kim W, Yalagandula P (2011b) Mahout: low-overhead datacenter traffic management using end-host-based elephant detection. In: Proc. IEEE INFOCOM, Shanghai, pp 1629–1637

  • Data set for imc 2010 data center measurement. http://pages.cs.wisc.edu/tbenson/IMC10 Data.html

  • Estan C, Varghese G (2003) New directions in traffic measurement and accounting: focusing on the elephants, ignoring the mice. ACM Trans Comput Syst TOCS 21(3):270–313

    Article  Google Scholar 

  • Gill P, Jain N, Nagappan N (2011) Understanding network failures in data centers: measurement, analysis, and implications. In: Proc. ACM SIGCOMM, Toronto

  • Greenberg A, Hamilton JR, Jain N.et al (2009) VL2: a scalable and flexible data center network. ACM SIGCOMM Comput Commun Rev 39(4):51–62

    Article  Google Scholar 

  • Guo B, Chen HH, Han Q, Yu ZW, Zhang DQ, Wang Y (2016) Worker-contributed data utility measurement for visual crowdsensing systems. IEEE Trans Mob Comput 16(8):2379–2391

    Article  Google Scholar 

  • Handigol N, Heller B, Jeyakumar V, Lantz B, McKeown N (2012) Reproducible network experiments using container-based emulation. In: Proc. the 8th international conference on emerging networking experiments and technologies. ACM, Nice, pp 253–264

  • Higashino WA, Capretz MAM, Toledo MBFD., Bittencourt LF (2016) A hybrid particle swarm optimisation-genetic algorithm applied to grid scheduling. Int J Grid Util Comput 7(2):113–129

    Article  Google Scholar 

  • Hong C-Y, Caesar M, Godfrey PB (2012) Finishing flows quickly with preemptive scheduling. In: Proc. ACM SIGCOMM, Helsinki

  • Kandula S, Sengupta S, Greenberg A et al (2009) The nature of data center traffic: measurements & analysis. In: Proc. 9th ACM SIG-COMM conference on internet measurement conference, Chicago, IL, pp 202–208

  • Kuzuno H, Magata K (2016) Detecting and characterising of mobile advertisement network traffic using graph modelling. Int J Space Based Situat Comput 6(2):90–101

    Article  Google Scholar 

  • Li ZT, Xiao F, Wang SG, Pei TR, Li J (2018) Achievable rate maximization for cognitive hybrid satellite-terrestrial networks with AF-relays. IEEE J Sel Areas Commun Spec Issue Adv Satell Commun PP(99):1

    Google Scholar 

  • Lin CY, Chen C, Chang JW et al (2014) Elephant flow detection in datacenters using OpenFlow-based hierarchical statistics pulling. In: Proc. 2014 IEEE global communications conference, Austin, pp 2264–2269

  • Liu CY, He L, Li ZT, Li J (2017) Feature-driven active learning for hyperspectral image classification. IEEE Trans Geosci Rem Sens PP(99):1–14

    Google Scholar 

  • Luo CZ, Li ZT, Huang KZ, Feng JS, Wang M (2017) Zero-shot learning via attribute regression and class prototype rectification. IEEE Trans Image Process PP(99):1–1

    MATH  Google Scholar 

  • McKeown N, Anderson T, Balakrishnan H et al (2008) OpenFlow: enabling innovation in campus networks. ACM CCR

  • Mori T, Uchida M, Kawahara R et al (2004) Identifying elephant flows through periodically sampled packets. In: Proc. the 4th ACM SIGCOMM conference on internet measurement, Taormina, Sicily, pp 115–120

  • Nakamura S, Duolikun D, Enokido T, Takizawa M (2016) A read–write abortion protocol to prevent illegal information flow in role-based access control systems. Int J Space Based Situat Comput 6(1):43–53

    Article  Google Scholar 

  • Popal L, Raiciu C, Stoica I, Rosenblum D (2006) Reducing congestion effects in wireless networks by multipath routing. In: Proc. IEEE int’l conf. network protocols (ICNP), Santa Barbara, CA

  • Ryu. https://osrg.github.io/ryu/

  • sFlow. http://www.sFlow

  • Su Z, Wang T, Xia Y, Hamdi M (2014) CheetahFlow: towards low latency software-defined network. In: Proc. 2014 IEEE international conference on communications (ICC), Sydney, pp 3076–3081

  • Tang FL, Li J (2017) Joint rate adaptation, channel assignment and routing to maximize social welfare in multi-hop cognitive radio networks. IEEE Trans Wirel Commun 16(4):2097–2110

    Article  Google Scholar 

  • Tang FL, Yang LT, Tang C, Li J, Guo MY (2016) A dynamical and load-balanced flow scheduling approach for big data centers in clouds. IEEE Trans Cloud Comput PP(99):1

    Google Scholar 

  • Tang FL, Yang YQ, Yang LT, Zhou T, Guo MY (2017) Delay-minimized routing in mobile cognitive networks for time-critical applications. IEEE Trans Ind Inf 13(3):1398–1409

    Article  Google Scholar 

  • Tang FL, Zhang HT, Li J (2018) Joint topology control and stable routing based on pu prediction for multi-hop mobile cognitive networks. IEEE Trans Wirel Commun 17(3):1713–1726

    Article  Google Scholar 

  • Thaman J, Singh M (2017) Cost-effective task scheduling using hybrid approach in cloud. Int J Grid Util Comput 8(3):241–253

    Article  Google Scholar 

  • Vasudevan V, Phanishayee A, Shah H et al (2009) Safe and effective fine-grained TCP retransmissions for datacenter communication. In: Proc. SIGCOMM, Barcelona, pp 303–314

  • Wang SG, Ruby R, Leung VCM, Yao ZQ, Liu XR, Li ZT (2017) Sum-power minimization problem in multi-source single-AF-relay networks: a new revisit to study the optimality. IEEE Trans Veh Technol 66(11):9958–9971

    Article  Google Scholar 

  • Wette P, Karl H (2013) Which flows are hiding behind my wildcard rule? Adding packet sampling to OpenFlow. ACM SIGCOMM Comput Commun Rev 43(4):541–542

    Article  Google Scholar 

  • Yu Z, Xu H, Yang Z, Guo B (2016) Personalized travel package with multi-point-of-interest recommendation based on crowdsourced user footprints. IEEE Trans Hum Mach Syst 46(1):151–158

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China Projects under Grant 91438121, Grant 61373156 and Grant 61672351, in part by the National Basic Research Program under Grant 2015CB352403, and in part by Huawei Technologies Co. Ltd. Projects under Grant YBN2017090053 and Grant YBN2017050015.

Author information

Authors and Affiliations

  1. School of Software, Shanghai Jiao Tong University, Shanghai, 200240, China

    Heteng Zhang

  2. Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, 200240, China

    Feilong Tang

  3. Faculty of Information Engineering, Fukuoka Institute of Technology, Fukuoka, 811-0295, Japan

    Leonard Barolli

Authors
  1. Heteng Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Feilong Tang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Leonard Barolli
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Feilong Tang.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, H., Tang, F. & Barolli, L. Efficient flow detection and scheduling for SDN-based big data centers. J Ambient Intell Human Comput 10, 1915–1926 (2019). https://doi.org/10.1007/s12652-018-0783-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12652-018-0783-6

Keywords

Search

Navigation