Skip to main content
SpringerLink
Log in

A self-managing wide-area data streaming service

  • Published:
Cluster Computing Aims and scope Submit manuscript

Abstract

Efficient and robust data streaming services are a critical requirement of emerging Grid applications, which are based on seamless interactions and coupling between geographically distributed application components. Furthermore the dynamism of Grid environments and applications requires that these services be able to continually manage and optimize their operation based on system state and application requirements. This paper presents a design and implementation of such a self-managing data-streaming service based on online control strategies. A Grid-based fusion workflow scenario is used to evaluate the service and demonstrate its feasibility and performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Abdelwahed, S., Kandasamy, N., Neema, S.: A control-based framework for self-managing distributed computing systems. In: Workshop on Self-Managed Systems (WOSS’04), Newport Beach, CA, USA, 2004

  2. Abdelwahed, S., Kandasamy, N., Neema, S.: Online control for self-management in computing systems. In: 10th IEEE Real-Time and Embedded Technology and Applications Symposium, Le Royal Meridien, King Edward, Toronto, Canada, 2004, pp. 368–376

  3. Abdelzaher, T.F., Shin, K.G., Bhatti, N.: Performance guarantees for web server end-systems: a control theoretic approach. IEEE Trans. Parallel Distributed Syst. 13(1), 80–96 (2002)

    Article  Google Scholar 

  4. Abrahams, A., Eyers, D., Bacon, J.: An asynchronous rule-based approach for business process automation using obligations. In: Third ACM SIGPLAN Workshop on Rule-Based Programming (RULE’02), Pittsburgh, PA, pp. 323–345. ACM, New York (2002)

    Google Scholar 

  5. Agarwal, M., Parashar, M.: Enabling autonomic compositions in grid environments. In: Fourth International Workshop on Grid Computing (Grid ’03), Phoenix, Arizona, USA, pp. 34–41. IEEE Computer Society, Los Alamitos (2003)

    Google Scholar 

  6. Bertsekas, D.P.: Dynamic Programming and Optimal Control, vol. 1, 3rd edn. Athena Scientific, Nashua (2005)

    MATH  Google Scholar 

  7. Bhat, V., Klasky, S., Atchley, S., Beck, M., McCune, D., Parashar, M.: High performance threaded data streaming for large scale simulations. In: 5th IEEE/ACM International Workshop on Grid Computing (Grid 2004), Pittsburgh, PA, USA, 2004, pp. 243–250

  8. Bhat, V., Parashar, M., Liu, H., Khandekar, M., Kandasamy, N., Abdelwahed, S.: Enabling self-managing applications using model-based online control strategies. In: 3rd IEEE International Conference on Autonomic Computing, Dublin, Ireland, 2006

  9. Bihari, T.E., Schwan, K.: Dynamic adaptation of real-time software. ACM Trans. Comput. Syst. 9(2), 143–174 (1991)

    Article  Google Scholar 

  10. Capra, L., Emmerich, W., Mascolo, C.: A micro-economic approach to conflict resolution in mobile computing. In: Workshop on Self-healing Systems (SIGSOFT’02), Charleston, SC, USA, 2002, pp. 31–40

  11. Cervin, A., Eker, J., Bernhardsson, B., Arzen, K.: Feedback-feedforward scheduling of control tasks. Real-Time Syst. 23(1-2), 25–53 (2002)

    Article  MATH  Google Scholar 

  12. Chen, J.: M3D Home, http://w3.pppl.gov/%7ejchen, 2005

  13. Cheng, J.J., Flaxer, D., Kapoor, S.: RuleBAM: a rule-based framework for business activity management. In: IEEE International Conference on Services Computing(SCC’04), Shanghai, China, 2004, pp. 262–270

  14. Christensen, E., Curbera, F., Meredith, G., Weerawarana, S.: Web Services Description Language (WSDL) 1.1, http://www.w3.org/TR/wsdl, 15 March 2001

  15. Hellerstein, J.L., Diao, Y., Parekh, S., Tilbury, D.M.: Feedback Control of Computing Systems. Wiley-IEEE Press, Hoboken (2004)

    Google Scholar 

  16. Hellerstein, J.L., Diao, Y., Parekh, S.S.: Applying control theory to computing systems. Technical report, December 7, 2004

  17. Kandasamy, N., Abdelwahed, S., Hayes, J.P.: Self-optimization in computer systems via online control: application to power management. In: 1st IEEE International Conference on Autonomic Computing (ICAC’04), New York, NY, USA, 2004, pp. 54–61

  18. Klasky, S., Beck, M., Bhat, V., Feibush, E., Ludäscher, B., Parashar, M., Shoshani, A., Silver, D., Vouk, M.: Data management on the fusion computational pipeline. J. Phys. Conf. Ser. 16(2005), 510–520 (2005)

    Article  Google Scholar 

  19. Lawrence-Berkeley-National-Laboratory: The Energy Sciences Network (ESnet), http://www.es.net/ (2004)

  20. Lin, Z., Hahm, T.S., Lee, W.W., Tang, W.M., White, R.B.: Turbulent transport reduction by zonal flows: massively parallel simulations. Science 281(5384), 1835–1837 (1998)

    Article  Google Scholar 

  21. Liu, H.: Accord: a programming system for autonomic self-managing applications. PhD thesis, Rutgers University (2005)

  22. Liu, H., Bhat, V., Parashar, M., Klasky, S.: An autonomic service architecture for self-managing grid applications. In: 6th International Workshop on Grid Computing (Grid 2005), Seattle, WA, USA, 2005, pp. 132–139

  23. Liu, H., Parashar, M., Hariri, S.: A component-based programming framework for autonomic applications. In: 1st IEEE International Conference on Autonomic Computing (ICAC-04), New York, NY, USA, 2004, pp. 10–17

  24. Lu, C., Alvarez, G.A., Wilkes, J.: Aqueduct: online data migration with performance guarantees. In: USENIX Conference on File Storage Technologies (FAST’02), Monterey, CA, 2002, pp. 219–230

  25. Lu, C., Stankovic, J.A., Son, S.H., Tao, G.: Feedback control real-time scheduling: framework, modeling, and algorithms. Real-Time Syst. 23(1-2), 85–126 (2002)

    Article  MATH  Google Scholar 

  26. Lu, Z., Hein, J., Humphrey, M., Stan, M., Lach, J., Skadron, K.: Control-theoretic dynamic frequency and voltage scaling for multimedia workloads. In: International Conference on Compilers, Architectures, & Synthesis Embedded Systems (CASES), Grenoble, France, pp. 156–163. ACM Press, New York (2002)

    Chapter  Google Scholar 

  27. Lupu, E.C., Sloman, M.: Conflicts in policy-based distributed systems management. IEEE Trans. Softw. Eng. 25(6), 852–869 (1999)

    Article  Google Scholar 

  28. Ma, X.: Hiding periodic I/O costs in parallel applications. PhD thesis, University of Illinois at Urbana-Champaign (2003)

  29. Ma, X., Lee, J., Winslett, M.: High-level buffering for hiding periodic output cost in scientific simulations. IEEE Trans. Parallel Distributed Syst. 17(3), 193–204 (2006)

    Article  Google Scholar 

  30. Mascolo, S.: Classical control theory for congestion avoidance in high-speed Internet. In: 38th IEEE Conference on Decision and Control, Phoenix, Arizona, USA, 1999, vol. 3, pp. 2709–2714

  31. Nichols, B., Buttlar, D., Farrell, J.P.: PThreads Programming. A POSIX Standard for Better Multiprocessing, 1st edn. O’Reilly, Sebastopol (1996)

    Google Scholar 

  32. NLANR/DAST: Iperf 1.7.0: the TCP/UDP bandwidth measurement tool, http://dast.nlanr.net/Projects/Iperf/ (2005)

  33. Parashar, M., Browne, J.C.: Conceptual and implementation models for the grid. Proc. IEEE 93, 653–668 (2005)

    Article  Google Scholar 

  34. Parekh, S., Gandhi, N., Hellerstein, J., Tilbury, D., Jayram, T., Bigus, J.: Using control theory to achieve service level objectives in performance management. Real-Time Syst. 23(1-2), 127–141 (2002)

    Article  MATH  Google Scholar 

  35. Plank, J.S., Beck, M., Elwasif, W.R., Moore, T., Swany, M., Wolski, R.: The Internet backplane protocol: storage in the network. In: NetStore99: The Network Storage Symposium, Seattle, WA, USA, 1999

  36. Plank, J.S., Beck, M.: The logistical computing stack—a design for wide-area, scalable, uninterruptible computing. In: Dependable Systems and Networks, Workshop on Scalable, Uninterruptible Computing (DNS 2002), Bethesda, Maryland, USA, 2002

  37. Sharma, V., Thomas, A., Abdelzaher, T., Skadron, K., Lu, Z.: Power-aware QoS management in web servers. In: Real-Time Systems Symposium, Cancun, Mexico, 2003, pp. 63–72

  38. Srikant, R.: Control of communication networks. In: T. Samad (ed.) Perspectives in Control Engineering: Technologies, Applications, New Directions, pp. 462–488. Wiley-IEEE Press, New York (2000)

    Google Scholar 

  39. van-der Aalst, W.M.P., ter Hofstede, A.H.M., Kiepuszewski, B., Barros, A.P.: Workflow patterns. Distributed Parallel Databases 14(1), 5–51 (2003)

    Article  Google Scholar 

  40. Vilalta, R., Apte, C., Hellerstein, J.L., Ma, S., Weiss, S.M.: Predictive algorithms in the management of computer systems. IBM Syst. J. 41(3), 461–474 (2002)

    Article  Google Scholar 

  41. W3C. OWL web ontology language overview, http://www.w3.org/TR/owl-features (2004)

  42. Wu, X., Hellerstein, J.L.: Control theory in log processing systems. In: Summer 2005 RADS (Reliable Adaptive Distributed Systems Laboratory) Retreat, 2005

Download references

Author information

Authors and Affiliations

  1. Department of Electrical and Computer Engineering, Rutgers University, 94 Brett Road, Piscataway, NJ, 08854, USA

    Viraj Bhat & Manish Parashar

  2. Xerox Innovation Group, 800 Phillips Rd, Mailstop 128-30E, Webster, NY, 14580, USA

    Hua Liu

  3. Department of Electrical and Computer Engineering, Drexel University, 3141 Chestnut Street, Philadelphia, PA, 19104, USA

    Nagarajan Kandasamy & Mohit Khandekar

  4. Oak Ridge National Laboratory, P.O. Box 2008, Oak Ridge, TN, 37831, USA

    Scott Klasky

  5. Institute for Software Integrated Systems, 2015 Terrace Place, Nashville, TN, 37235, USA

    Sherif Abdelwahed

Authors
  1. Viraj Bhat
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Manish Parashar
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hua Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Nagarajan Kandasamy
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Mohit Khandekar
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Scott Klasky
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Sherif Abdelwahed
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Viraj Bhat.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Bhat, V., Parashar, M., Liu, H. et al. A self-managing wide-area data streaming service. Cluster Comput 10, 365–383 (2007). https://doi.org/10.1007/s10586-007-0023-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10586-007-0023-x

Keywords

Search

Navigation