Skip to main content
SpringerLink
Log in

Benchmarking Publish/Subscribe-Based Messaging Systems

  • Conference paper
Database Systems for Advanced Applications (DASFAA 2010)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 6193))

Included in the following conference series:

Abstract

Publish/subscribe-based messaging systems are used increasingly often as a communication mechanism in data-oriented web applications. Such applications often pose serious performance and scalability challenges. To address these challenges, it is important that systems are tested using benchmarks to evaluate their performance and scalability before they are put into production. In this paper, we present jms2009-PS, a new benchmark for publish/subscribe-based messaging systems built on top of the SPECjms2007 standard workload. We introduce the benchmark and discuss its configuration parameters showing how the workload can be customized to evaluate various aspects of publish/subscribe communication. Finally, we present a case study illustrating how the benchmark can be used for performance analysis of messaging servers.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Hinze, A., Sachs, K., Buchmann, A.: Event-Based Applications and Enabling Technologies. In: Proceedings of the International Conference on Distributed Event-Based Systems, DEBS 2009 (2009)

    Google Scholar 

  2. Sun Microsystems, Inc.: Java Message Service (JMS) Specification - Ver. 1.1 (2002)

    Google Scholar 

  3. Happe, J., Friedrich, H., Becker, S., Reussner, R.H.: A pattern-based Performance Completion for Message-oriented Middleware. In: Proc. of the ACM WOSP (2008)

    Google Scholar 

  4. Sachs, K., Kounev, S., Bacon, J., Buchmann, A.: Performance evaluation of message-oriented middleware using the SPECjms2007 benchmark. Performance Evaluation 66(8), 410–434 (2009)

    Article  Google Scholar 

  5. Sachs, K., Kounev, S., Buchmann, A.: Performance Modeling of Message-Oriented Middleware - A Case Study (2009) (in review)

    Google Scholar 

  6. Sachs, K., Kounev, S., Appel, S., Buchmann, A.: A Performance Test Harness For Publish/Subscribe Middleware. In: SIGMETRICS/Performance 2009 Demo Competition, June 2009. ACM, New York (2009)

    Google Scholar 

  7. Eugster, P.T., Felber, P.A., Guerraoui, R., Kermarrec, A.M.: The Many Faces of Publish/Subscribe. ACM Computing Surveys 35(2), 114–131 (2003)

    Article  Google Scholar 

  8. Hohpe, G., Woolf, B.: Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions. Addison-Wesley, Reading (2003)

    Google Scholar 

  9. Sonic Software Corporation: Sonic Test Harness (2005), http://communities.progress.com/pcom/docs/DOC-29828

  10. IBM Hursley: Performance Harness for Java Message Service (2005), http://www.alphaworks.ibm.com/tech/perfharness

  11. ActiveMQ: JMeter performance test (2006), http://incubator.apache.org/activemq/jmeter-performance-tests.html

  12. JBoss: JBoss JMS New Performance Benchmark (2006), http://wiki.jboss.org/wiki/Wiki.jsp?page=JBossJMSNewPerformanceBenchmark

  13. Crimson Consulting Group: High-Performance JMS Messaging - A Benchmark Comparison of Sun Java System Message Queue and IBM WebSphere MQ (2003), http://www.sun.com/software/products/message_queue/wp_JMSperformance.pdf

  14. Krissoft Solutions: JMS Performance Comparison (2006), http://www.fiorano.com/comp-analysis/jms_perf_report.htm

  15. Sonic Software Corporation: Benchmarking E-Business Messaging Providers. White Paper (January 2004)

    Google Scholar 

  16. Carter, M.: JMS Performance with WebSphere MQ for Windows V6.0 (2005), http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg24010028

  17. Fiorano Software Inc.: JMS Performance Comparison - Performance Comparison for Publish Subscribe Messaging (2010), http://www.fiorano.com/whitepapers/fmq/jms_performance_comparison.php

  18. Rindos, A., Loeb, M., Woolet, S.: A performance comparison of IBM MQseries 5.2 and Microsoft Message Queue 2.0 on Windows 2000. IBM SWG Competitive Technical Assessment, Research Triangle Park, NC (2001)

    Google Scholar 

  19. Maheshwari, P., Pang, M.: Benchmarking message-oriented middleware: TIB/RV versus SonicMQ. Concurrency Computat.: Pract. and Exper. 17(12) (2005)

    Google Scholar 

  20. Menth, M., Henjes, R., Zepfel, C., Gehrsitz, S.: Throughput performance of popular JMS servers. SIGMETRICS Perform. Eval. Rev. 34(1), 367–368 (2006)

    Article  Google Scholar 

  21. Kounev, S., Sachs, K.: Benchmarking and Performance Modeling of Event-Based Systems. IT - Information Technology 51(5 ), 262–269 (2009)

    Article  Google Scholar 

  22. Sachs, K., Kounev, S., Appel, S., Buchmann, A.: Benchmarking of Message-Oriented Middleware. In: Proc. of the DEBS 2009 (2009)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Databases and Distributed System Group, TU Darmstadt, Germany

    Kai Sachs, Stefan Appel & Alejandro Buchmann

  2. Descartes Research Group, Karlsruhe Institute of Technology,  

    Samuel Kounev

Authors
  1. Kai Sachs
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Stefan Appel
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Samuel Kounev
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Alejandro Buchmann
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Graduate School of Informatics, Kyoto University, Yoshida Honmachi, Sakyo, 606-8501, Kyoto, Japan

    Masatoshi Yoshikawa

  2. Information School, Renmin University of China, 100872, Beijing, China

    Xiaofeng Meng

  3. Graduate School of Engineering, University of Hyogo, 2167 Shosha, Himeji, 671-2280, Hyogo, Japan

    Takayuki Yumoto

  4. Graduate School of Informatics, Kyoto University, Yoshidahonmachi, Sakyo, 606-8501, Kyoto, Japan

    Qiang Ma

  5. Institute of HCI and Media Integration, Tsinghua University, 100084, Bejing, China

    Lifeng Sun

  6. Department of Information Science, Ochanomizu University, 2-1-1, Otsuka, Bunkyo-ku, 112-8610, Tokyo, Japan

    Chiemi Watanabe

Rights and permissions

Reprints and permissions

Copyright information

© 2010 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Sachs, K., Appel, S., Kounev, S., Buchmann, A. (2010). Benchmarking Publish/Subscribe-Based Messaging Systems. In: Yoshikawa, M., Meng, X., Yumoto, T., Ma, Q., Sun, L., Watanabe, C. (eds) Database Systems for Advanced Applications. DASFAA 2010. Lecture Notes in Computer Science, vol 6193. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-14589-6_21

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-14589-6_21

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-14588-9

  • Online ISBN: 978-3-642-14589-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation