Skip to main content
SpringerLink
Log in

Adaptive component composition and load balancing for distributed stream processing applications

  • Published:
Peer-to-Peer Networking and Applications Aims and scope Submit manuscript

Abstract

Providing real-time and QoS support to stream processing applications running on top of large-scale overlays is challenging due to the inherent heterogeneity and resource limitations of the nodes and the multiple QoS demands of the applications that must concurrently be met. In this paper we propose an integrated adaptive component composition and load balancing mechanism that (1) allows the composition of distributed stream processing applications on the fly across a large-scale system, while satisfying their QoS demands and distributing the load fairly on the resources, and (2) adapts dynamically to changes in the resource utilization or the QoS requirements of the applications. Our extensive experimental results using both simulations as well as a prototype deployment illustrate the efficiency, performance and scalability of our approach.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18

Similar content being viewed by others

Notes

  1. http://www.freepastry.org.

References

  1. Abadi D, Ahmad Y, Balazinska M, Cetintemel U, Cherniack M, Hwang J, Lindner W, Maskey A, Rasin A, Ryvkina E, Tatbul N, Xing Y, Zdonik S (2005) The design of the Borealis stream processing engine. In: CIDR, Asilomar, 4–7 January 2005

  2. Arasu A, Babcock B, Babu S, Cieslewicz J, Datar M, Ito K, Motwani R, Srivastava U, Widom J (2005) STREAM: the Stanford data stream management system

  3. Bavier A, Bowman M, Chun B, Culler D, Karlin S, Muir S, Peterson L, Roscoe T, Spalink T, Wawrzoniak M (2004) Operating system support for planetary-scale network services. In: Proc of NSDI, San Francisco, 29–31 March 2004

  4. Bestavros A (1997) Load profiling: a methodology for scheduling real-time tasks in a distributed system. In: ICDCS, Baltimore, 27–30 May 1997

  5. Brinkschulte U, Schneider E, Picioroaga F (2005) Dynamic real-time reconfiguration in distributed systems: timing issues and solutions. In: ISORC, Seattle, 18–20 May 2005, pp 174–181

  6. Castro M, Druschel P, Kermarrec AM, Nandi A, Rowstron A, Singh A (2003) SplitStream: high-bandwidth multicast in a cooperative environment. In: SOSP, Lake George, 19–22 October 2003

  7. Chandrasekaran S, Cooper O, Deshpande A, JM Hellerstein MF, Hong W, Krishnamurthy S, Madden S, Raman V, Reiss F, Shah M (2003) Telegraphcq: continuous dataflow processing for an uncertain world. In: CIDR, Asilomar, 5–8 January 2003

  8. Chen F, Kalogeraki V (2004) RUBEN: a technique for scheduling multimedia applications in overlay networks. In: Globecom, Dallas, November 2004

  9. Chen F, Trepantis, Kalogeraki V (2005) Coordinated media streaming and transcoding in peer-to-peer systems. In: IPDPS, Denver, 3–8 April 2005

  10. Drougas Y, Kalogeraki V (2005) A fair resource allocation algorithm for peer-to-peer overlays. In: 8th global internet symposium, Miami, 18–19 March 2005

  11. Fry G, West R (2004) Adaptive routing of QoS-constrained media streams over scalable overlay topologies. In: IEEE RTAS, Toronto, 25–28 May 2004

  12. Gerard S, Babau JP, Champeau J (2005) Mode driven engineering for distributed real-time embedded systems. Hermes, Lyon

    Google Scholar 

  13. Ghosh S, Rajkumar R, Hansen JP, Lehoczky JP (2005) Scalable QoS-based resource allocation in hierarchical networked environment. In: IEEE RTAS, San Francisco, 7–10 March 2005

  14. Gu X, Nahrstedt K (2005) Distributed multimedia service composition with statistical QoS assurances. IEEE Trans Multimedia 8:141–151

    Google Scholar 

  15. Gu X, Yu PS, Nahrstedt K (2005) Optimal component composition for scalable stream processing. In: ICDCS, Columbus, 6–10 June 2005

  16. Guo L, Chen S, Ren S, Chen X, Jiang S (2004) PROP: a scalable and reliable P2P assisted proxy streaming system. In: ICDCS, Tokyo, 23–26 March 2004

  17. Hefeeda M, Habib A, Botev B, Xu D, Bhargava B (2003) PROMISE: peer-to-peer media streaming using collectcast. In: ACM multimedia, Berkeley, November 2003

  18. Jain RK, Chiu DMW, Have WR (1984) A quantitive measure of fairness and discrimination for resource allocation in shared computer systems. Tech Rep DEC-TR-301, Digital Equipment Corporation

  19. Kumar V, Cooper B, Cai Z, Eisenhauer G, Schwan K (2005) Resource-aware distributed stream management using dynamic overlays. In: ICDCS, Columbus, 6–10 June 2005

  20. Lynch N (1996) Distributed algorithms. Morgan Kaufmann, San Francisco

    MATH  Google Scholar 

  21. Microsoft Corporation (2006) COM: component object model technologies. http://www.microsoft.com/com

  22. Milojicic D, Kalogeraki V, Lukose R, Nagaraja K, Pruyne J, Richard B, Rollins S, Xu Z (2002) Peer-to-peer computing. HP Technical Report, HPL-2002-57

  23. Nahrstedt K, Wichadakul D, Xu D (2000) Distributed qos compilation and runtime instantiation. In: IWQoS, Pittsburgh, 5–7 June 2000

  24. Object Management Group (2000) The common object request broker: architecture and specification, edition 2.4, formal/00-10-01

  25. Pietzuch P, Ledlie J, Shneidman J, Roussopoulos M, Welsh M, Seltzer M (2006) Network-aware operator placement for stream-processing systems. In: ICDE, Atlanta, 3–7 April 2006

  26. Raftopoulou P (2003) Fair resource allocation in P2P systems: theoretical and experimental results. Master’s thesis, Department of Electronic and Computer Engineering, Technical University of Crete, Greece

  27. Repantis T, Drougas Y, Kalogeraki V (2005) Adaptive resource management in peer-to-peer middleware. In: WPDRTS, Denver, 4–5 April 2005

  28. Repantis T, Gu X, Kalogeraki V (2006) Synergy: sharing-aware component composition for distributed stream processing systems. In: ACM/IFIP/USENIX Middleware, Melbourne, 27 November–1 December 2006

  29. Sharma PK, Loyall JP, G TH, Schantz RE, Shapiro R, Duzan G (2004) Component-based dynamic qos adaptations in distributed real-time and embedded systems. In: IEEE DOA, Agia Napa, 25–29 October 2004

  30. Sun Microsystems (2006) Enterprise JavaBeans Technology. http://java.sun.com/products/ejb/

  31. Wang S, Merrick J, Shin K (2004) Component allocation with multiple resource constraints for large embedded real-time software design. In: IEEE RTAS, Toronto, 25–28 May 2004

  32. Wang S, Rho S, Mai Z, Bettati R, Zhao W (2005) Real-time component-based systems. In: IEEE RTAS, San Francisco, 7–10 March 2005

  33. Watson P, Fowler C, Kubicek C, Mukherjee A, Colquhoun J, Hewitt M, Parastatidis S (2006) Dynamically deploying web services on a grid using Dynasoar, ISORC, Gyeongju, pp 151–158. doi:10.1109/ISORC.2006.32

  34. Xing Y, Zdonik S, Hwang J (2005) Dynamic load distribution in the Borealis stream processor. In: ICDE, Tokyo, 5–8 April 2005

  35. Zegura E, Calvert K, Bhattacharjee S (1996) How to model an internetwork. In: IEEE INFOCOM, San Francisco, 24–28 March 1996

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science & Engineering, University of California, Riverside, Riverside, CA, 92521, USA

    Thomas Repantis, Yannis Drougas & Vana Kalogeraki

  2. Department of Informatics, Athens University of Economics and Business, Athens, Greece

    Vana Kalogeraki

Authors
  1. Thomas Repantis
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yannis Drougas
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Vana Kalogeraki
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Vana Kalogeraki.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Repantis, T., Drougas, Y. & Kalogeraki, V. Adaptive component composition and load balancing for distributed stream processing applications. Peer-to-Peer Netw. Appl. 2, 60–74 (2009). https://doi.org/10.1007/s12083-008-0020-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12083-008-0020-8

Keywords

Search

Navigation