Greg Eisenhauer
Matthew Wolf
ABSTRACT
Streaming data models have been shown to be useful in many applications requiring high-performance data exchange. Application-level overlay networks are a natural way to realize these applications' data flows and their internal computations, but existing middleware is not designed to scale to the data rates and low overhead computations necessary for the high performance domain. This paper describes EVPath, a middleware infrastructure that supports the construction and management of overlay networks that can be customized both in topology and in the data manipulations being performed. Extending from a previous high-performance publish-subscribe system, EVPath not only provides for the low overhead movement and in-line processing of large data volumes, but also offers the flexibility needed to support the varied data flow and control needs of alternative higher-level streaming models. We explore some of the challenges of high performance event systems, including those experienced when operating an event infrastructure used to transport IO events at the scale of hundred+ thousand nodes. Specifically, when transporting output data from a large-scale simulation running on the ORNL Cray Jaguar petascale machine, a surprising new issue seen in experimentation at scale is the potential for strong perturbation of running applications from inappropriate speeds at which IO is performed. This requires the IO system's event transport to be explicitly scheduled to constrain resource competition, in addition to dynamically setting and changing the topologies of event delivery.
- H. Abbasi, M. Wolf, F. Zheng, G. Eisenhauer, S. Klasky, and K. Schwan. Scalable data staging services for petascale applications. In hpdc2009, 2009. Google Scholar
Digital Library
- S. Agarwala, G. Eisenhauer, and K. Schwan. Lightweight morphing support for evolving data exchanges in distributed applications. In Proc. of the 25th International Conference on Distributed Computer Systems (ICDCS-25), June 2005. Google ScholarDigital Library
- K. Birman, A. Schiper, and P. Stephenson. Lightweight Causal and Atomic Group Multicast. ACM Transactions on Computer Systems, 9(3), Aug. 1991. Google ScholarDigital Library
- D. Box, L. Cabrera, C. Critchley, F. Curbera, D. Ferguson, A. Geller, S. Graham, D. Hull, G. Kakivaya, A. Lewis, et al. Web Services Eventing (WS-Eventing). W3C Member Submission, 2006.Google Scholar
- L. Brenna, A. Demers, J. Gehrke, M. Hong, J. Ossher, B. Panda, M. Riedewald, M. Thatte, and W. White. Cayuga: a high-performance event processing engine. In SIGMOD '07: Proceedings of the 2007 ACM SIGMOD international conference on Management of data, pages 1100--1102, New York, NY, USA, 2007. ACM. Google ScholarDigital Library
- Z. Cai, G. Eisenhauer, Q. He, V. Kumar, K. Schwan, and M. Wolf. Iq-services: Network-aware middleware for interactive large-data applications. Concurrency&Computation. Practice and Exprience Journal, 2005.Google Scholar
- A. Carzaniga, D. S. Rosenblum, and A. L. Wolf. Challenges for distributed event services: Scalability vs. expressiveness. In Proc. of Engineering Distributed Objects (EDO '99), ICSE 99 Workshop, May 1999.Google Scholar
- A. Demers, J. Gehrke, M. Hong, M. Riedewald, and W. White. A general algebra and implementation for monitoring event streams. Technical Report TR2005-1997, Cornell University, 2005.Google Scholar
- A. Demers, J. Gehrke, M. Hong, M. Riedewald, and W. White. Towards expressive publish/subscribe systems. Lecture Notes in Computer Science, 3896:627, 2006. Google ScholarDigital Library
- G. Eisenhauer, F. Bustamante, and K. Schwan. Publish-subscribe for high-performance computing. IEEE Internet Computing - Asynchronous Middleware and Services, 10(1):8--25, January 2006. Google ScholarDigital Library
- G. Eisenhauer and L. K. Daley. Fast heterogenous binary data interchange. In Proc. of the Heterogeneous Computing Workshop (HCW2000), May 3--5 2000. Google ScholarDigital Library
- P. Eugster. Type-based publish/subscribe: Concepts and experiences. ACM Trans. Program. Lang. Syst., 29(1):6, 2007. Google ScholarDigital Library
- P. Eugster, P. Felber, R. Guerraoui, and A.-M. Kerrmarec. The many faces of publish/subscribe. Tech. Report DSC-ID:200104, École Polythechnique Fédérale de Lausanne, Lausanne, France, January 2001.Google Scholar
- I. Foster, K. Czajkowski, D. Ferguson, J. Frey, S. Graham, T. Maguire, D. Snelling, and S. Tuecke. Modeling and managing state in distributed systems: The role of OGSI and WSRF. Proceedings of the IEEE, 93(3):604--612, 2005.Google ScholarCross Ref
- M. K. Gardner, W.-c. Feng, J. S. Archuleta, H. Lin, and X. Ma. Parallel Genomic Sequence-Searching on an Ad-Hoc Grid: Experiences, Lessons Learned, and Implications. In ACM/IEEE SC\06: The International Conference on High-Performance Computing, Networking, Storage, and Analysis, Tampa, FL, November 2006. Best Paper Nominee. Google ScholarDigital Library
- A. Grimshaw, M. Morgan, D. Merrill, H. Kishimoto, A. Savva, D. Snelling, C. Smith, and D. Berry. An open grid services architecture primer. Computer, 42(2):27--34, 2009. Google ScholarDigital Library
- Hewlet-Packard. The netperf network performance benchmark, http://www.netperf.org.Google Scholar
- Y. Huang, A. Slominski, C. Herath, and D. Gannon. Ws-messenger: A web services based messaging system for service-oriented grid computing. In 6th IEEE International Symposium on Cluster Computing and the Grid (CCGrid06), 2006. Google ScholarDigital Library
- N. Jiang, A. Quiroz, C. Schmidt, and M. Parashar. Meteor: a middleware infrastructure for content-based decoupled interactions in pervasive grid environments. Concurr. Comput.: Pract. Exper., 20(12):1455--1484, 2008. Google ScholarDigital Library
- V. Kumar, B. F. Cooper, Z. Cai, G. Eisenhauer, and K. Schwan. Resource-aware distributed stream management using dynamic overlays. In Proceedings of the 25th IEEE International Conference on Distributed Computing Systems (ICDCS-2005), 2005. Google ScholarDigital Library
- J. Lofstead, K. Schwan, S. Klasky, N. Podhorszki, and C. Jin. Flexible io and integration for scientific codes through the adaptable io system (adios). In Challenges of Large Applications in Distributed Environments (CLADE), 2008. Google ScholarDigital Library
- L. Oliker, J. Carter, michael Wehner, A. Canning, S. Ethier, A. Mirin, G. Bala, D. parks, patrick Worley Shigemune Kitawaki, and Y. Tsuda. Leading computational methods on scalar and vector hec platforms. In Proceedings of SuperComputing 2005, 2005. Google ScholarDigital Library
- O. M. G. (OMG). Notification service specification 1.0. ftp://www.omg.org/pub/doc/formal/00-06-20.pdf, June 2000.Google Scholar
- O. M. G. (OMG). Event service specification 1.1. ftp://www.omg.org/pub/docs/formal/01-03-01.pdf, March 2001.Google Scholar
- B. Segall and D. Arnold. Elvin has left the building: A publish/subscribe notification service with quenching. In Proc. of the AUUG (Australian users group for Unix and Open Systems) 1997 Conference, September 1997.Google Scholar
- D. Skeen. The enterprise-capable publish-subscribe server. http://www.vitria.com.Google Scholar
- R. Strom, G. Banavar, T. Chandra, M. Kaplan, K. Miller, B. Mukherjee, D. Sturman, and M. Ward. Gryphon: An information flow based approach to message brokering. In International Symposium on Software Reliability Engineering '98 Fast abstract, 1998.Google Scholar
- Tibco. TIB/rendezvous. http://www.rv.tibco.com/rvwhitepaper.html.Google Scholar
- M. Wolf, H. Abbasi, B. Collins, D. Spain, and K. Schwan. Service Augmentation for High End Interactive Data Services. In Proceedings of Cluster 2005, 2005.Google ScholarCross Ref
- M. Wolf, Z. Cai, W. Huang, and K. Schwan. Smartpointers: Personalized scientific data portals in your hand. In Proceedings of the Proceedings of the IEEE/ACM SC2002 Conference, page 20. IEEE Computer Society, 2002. Google ScholarDigital Library
Index Terms
- Event-based systems: opportunities and challenges at exascale
Recommendations
On Event-Based Middleware for Location-Aware Mobile Applications
As mobile applications become more widespread, programming paradigms and middleware architectures designed to support their development are becoming increasingly important. The event-based programming paradigm is a strong candidate for the development ...
Event Services in High Performance Systems
The Internet and the Grid are changing the face of high performance computing. Rather than tightly-coupled SPMD-style components running in a single cluster, on a parallel machine, or even on the Internet programmed in MPI, applications are evolving ...
Efficient message transport interface between agent framework and event service
Multi-agent techniques have been continuously evolving as ubiquitous and pervasive computing emerges as a key post-internet paradigm. In the ubiquitous environment, the applications need to exchange the requests asynchronously using an event-based ...
Comments