Skip to main content
SpringerLink
Log in

Middleware for enterprise scale data stream management using utility-driven self-adaptive information flows

  • Published:
Cluster Computing Aims and scope Submit manuscript

Abstract

We consider enterprise-wide information flows that are responsible for acquiring, processing and delivering operational information across the business units. Middleware that enables such aggregation of data-streams must not only support scalable and efficient self-management to deal with changes in the operating conditions, but should also have an embedded business-sense to appreciate the business critical nature of some updates. In this paper, we present a novel self-adaptation algorithm that has been designed to scale efficiently for thousands of streams and aims to maximize the overall business utility attained from running middleware-based applications. The outcome is that the middleware not only deals with changing network conditions or resource requirements, but also responds appropriately to changes in business policies. An important feature of the algorithm is a hierarchical node-partitioning scheme that decentralizes reconfiguration and suitably localizes its impact. Extensive simulation experiments and benchmarks attained with actual enterprise operational data corroborate this paper’s claims.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. TRADERBOT: Real-time financial search engine. http://www.traderbot.com

  2. Wal-Mart backs RFID Technology. http://www.computerworld.com/softwaretopics/erp/story/0,10801,82155,00.html

  3. Gavrilovska, A., Schwan, K., Oleson, V.: A practical approach for ‘zero’ downtime in an operational information system. In: Proceedings of International Conference on Distributed Computing Systems, Austria, July 2002

  4. Oleson, V., Schwan, K., Eisenhauer, G., Plale, B., Pu, C., Amin, D.: Operation information system—an example from the airline industry. In: Proceedings of First Workshop on Industrial Experiences with Systems Software, October 2000

  5. Kephart, J.O., Chess, D.M.: The vision of autonomic computing. IEEE Computer 36(1), 41–50 (2003)

    Google Scholar 

  6. White, S.R., Hanson, J.E., Whalley, I., Chess, D.M., Kephart, J.O.: An architectural approach to autonomic computing. In: Proceedings of International Conference on Autonomic Computing, pp. 2–9 (2004)

  7. Kravets, R., Calvert, K.L., Schwan, K.: Payoff adaptation of communication for distributed interactive applications. J. High Speed Netw.: Special Issue on Multimedia Communications 7(3–4), 301–317 (1999)

    Google Scholar 

  8. Walsh, W.E., Tesauro, G., Kephart, J.O., Das, R.: Utility functions in autonomic systems. In: Proceedings of International Conference on Autonomic Computing, pp. 70–77 (2004)

  9. Mas-Colell, A., Whinston, M.D., Green, J.R.: Microeconomic Theory. Oxford University Press, Oxford (1995)

    Google Scholar 

  10. Russel, S., Norvig, P.: Artificial Intelligence: A Modern Approach, 2nd edn. Prentice Hall, New York (2003)

    Google Scholar 

  11. Aiber, S., Gilat, D., Landau, A., Razinkov, N., Sela, A., Wasserkrug, S.: Autonomic self-optimization according to business objectives. In: Proceedings of International Conference on Autonomic Computing, pp. 206–213 (2004)

  12. Peakstone Corporation. http://www.peakstone.com

  13. Manasce, D., Almeida, J., Fonseca, R., Mendes, M.: Business-oriented resource management policies for ecommerce servers. Perform. Eval. 42(2–3), 223–239 (2000)

    Article  Google Scholar 

  14. Content based pub-sub middleware. http://www.research.ibm.com/gryphon/

  15. Eisenhauer, G., Bustamente, F., Schwan, K.: Event services for high performance computing. In: Proceedings of High Performance Distributed Computing, HPDC (2000)

  16. Abdelzaher, T., Dawson, S., Feng, W., Jahanian, F., Johnson, S., Mehra, A., Mitton, T., Shaikh, A., Shin, K., Wang, Z., Zou, H., Bjorkland, M., Marron: ARMADA middleware and communication services. Real-Time Syst. 16(2–3), 127–153 (1999)

    Article  Google Scholar 

  17. Pietzuch, P.R., Bacon, J.M.: Hermes: a distributed event-based middleware architecture. In: Proceedings of the 1st International Workshop on Distributed Event-Based Systems (DEBS’02), pp. 611–618, Vienna, Austria, July 2002

  18. Wolf, M., Cai, Z., Huang, W., Schwan, K.: Smart pointers: personalized scientific data portals in your hand. In: Proceedings of Supercomputing 2002, Baltimore, Maryland, November 2002

  19. Cai, Z., Eisenhauer, G., Poellabauer, C., Schwan, K., Wolf, M.: IQ-services: resource-aware middleware for heterogeneous applications. In: IPDPS/HCW 2004, Santa Fe, NM, April 2004

  20. Babu, S., Widom, J.: Continuous queries over data streams. SIGMOD Rec. 30(3), 109–120

  21. Carney, D., Cetintemel, U., Cherniack, M., Convey, C., Lee, S., Seidman, G., Stonebraker, M., Tatbul, N., Zdonik, S.: Monitoring streams: a new class of data management applications. In: Proceedings of the Twenty Seventh International Conference on Very Large Databases, Hong Kong, August 2002

  22. Koster, R., Black, A., Huang, J., Walpole, J., Pu, C.: Infopipes for composing distributed information flows. In: Proceedings of the 2001 International Workshop on Multimedia Middleware, ON, Canada (2001)

  23. Szalay, A.S., Gray, J: Virtual observatory: the world wide telescope (MS-TR-2001-77). Sci. Mag. 293, 2037–2038 (2001)

    Google Scholar 

  24. Allcock, W., Bester, J., Bresnahan, J., Chervenak, A., Foster, I., Kesselman, C., Meder, S., Nefedova, V., Quesnel, D., Tuecke, S.: Data management and transfer in high-performance computational grid environments. J. Parallel Comput. 28(5), 749–771 (2002)

    Article  Google Scholar 

  25. Stoica, I., Morris, R., Karger, D., Kaashoek, M.F., Balakrishnan, H.: Chord: a scalable peer-to-peer lookup service for Internet applications. In: Proceedings of the ACM SIGCOMM ’01 Conference. ACM, New York (2001)

    Google Scholar 

  26. Zhao, B.Y., Huang, L., Rhea, S.C., Stribling, J., Joseph, A.D., Kubiatowicz, J.D.: Tapestry: a global-scale overlay for rapid service deployment. In: IEEE J-SAC 22, 1, pp. 41–53 (2004)

  27. Zhuang, S.Q., Zhao, B.Y., Joseph, A.D., Katz, R.H., Kubiatowicz, J.: Bayeux: an architecture for scalable and fault-tolerant wide-area data dissemination. In: Proceedings of ACM NOSSDAV (2001)

  28. Chen, Y., Schwan, K., Zhou, D.: Opportunistic channels: mobility-aware event delivery. In: Proceedings of the 4th ACM/USENIX International Middleware Conference (Middleware 2003), June 2003

  29. Bustamante, F., Widener, P., Schwan, K.: Scalable directory services using proactivity. In: Proceedings of Supercomputing 2002, Baltimore, MD, November 2002

  30. Zegura, E., Calvert, K., Bhattacharjee, S.: How to model an internetwork. In: Proceedings of IEEE Infocom ’96, San Francisco, CA

  31. The network simulator ns-2. http://www.isi.edu/nsnam/ns/

  32. Emulab- network emulation testbed home. http://www.emulab.net

  33. The TCP/UDP bandwidth measurement tool. http://dast.nlanr.net/Projects/Iperf/

  34. Delta Technology. http://www.deltadt.com

Download references

Author information

Authors and Affiliations

  1. Center for Experimental Research in Computer Science, Georgia Institute of Technology, Atlanta, GA, 30332, USA

    Vibhore Kumar, Brian F. Cooper, Zhongtang Cai, Greg Eisenhauer & Karsten Schwan

Authors
  1. Vibhore Kumar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Brian F. Cooper
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhongtang Cai
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Greg Eisenhauer
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Karsten Schwan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Vibhore Kumar.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Kumar, V., Cooper, B.F., Cai, Z. et al. Middleware for enterprise scale data stream management using utility-driven self-adaptive information flows. Cluster Comput 10, 443–455 (2007). https://doi.org/10.1007/s10586-007-0040-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10586-007-0040-9

Keywords

Search

Navigation