Skip to main content
SpringerLink
Log in

How Fast Is MQTT?

Statistical Model Checking and Testing of IoT Protocols

  • Conference paper
  • First Online:
Quantitative Evaluation of Systems (QEST 2018)

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 11024))

Included in the following conference series:

Abstract

MQTT is one of the major messaging protocols in the Internet of things (IoT). In this work, we investigate the expected performance of MQTT implementations in various settings. We present a model-based performance testing approach that allows a fast simulation of specific usage scenarios in order to perform a quantitative analysis of the latency. Out of automatically generated log-data, we learn the distributions of latencies and apply statistical model checking to analyse the functional and timing behaviour. The result is a novel testing and verification technique for analysing the performance of IoT protocols. Two well-known open source MQTT implementations are evaluated and compared.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://github.com/schumi42/mqttCheck

  2. 2.

    https://fscheck.github.io/FsCheck

  3. 3.

    https://www.r-project.org/

  4. 4.

    https://m2mqtt.wordpress.com

References

  1. Aichernig, B.K., Schumi, R.: Property-based testing with FsCheck by deriving properties from business rule models. In: ICSTW, pp. 219–228. IEEE (2016)

    Google Scholar 

  2. Aichernig, B.K., Schumi, R.: Property-based testing of web services by deriving properties from business-rule models. Softw. Syst. Model. (2017)

    Google Scholar 

  3. Aichernig, B.K., Schumi, R.: Statistical model checking meets property-based testing. In: ICST, pp. 390–400. IEEE (2017)

    Google Scholar 

  4. Alur, R., Dill, D.L.: A theory of timed automata. Theor. Comput. Sci. 126(2), 183–235 (1994)

    Article  MathSciNet  Google Scholar 

  5. Arts, T.: On shrinking randomly generated load tests. In: Erlang 2014, pp. 25–31. ACM (2014)

    Google Scholar 

  6. Ballarini, P., Bertrand, N., Horváth, A., Paolieri, M., Vicario, E.: Transient analysis of networks of stochastic timed automata using stochastic state classes. In: Joshi, K., Siegle, M., Stoelinga, M., D’Argenio, P.R. (eds.) QEST 2013. LNCS, vol. 8054, pp. 355–371. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-40196-1_30

    Chapter  Google Scholar 

  7. Banks, A., Gupta, R.: MQTT version 3.1.1. OASIS Standard, December 2014

    Google Scholar 

  8. Becker, S., Koziolek, H., Reussner, R.H.: The Palladio component model for model-driven performance prediction. J. Syst. Softw. 82(1), 3–22 (2009)

    Article  Google Scholar 

  9. Book, M., Gruhn, V., Hülder, M., Köhler, A., Kriegel, A.: Cost and response time simulation for web-based applications on mobile channels. In: QSIC, pp. 83–90. IEEE (2005)

    Google Scholar 

  10. Bulychev, P.E., et al.: UPPAAL-SMC: statistical model checking for priced timed automata. In: QAPL, EPTCS, vol. 85, pp. 1–16. Open Publishing Association (2012)

    Google Scholar 

  11. Chen, X., Mohapatra, P., Chen, H.: An admission control scheme for predictable server response time for web accesses. In: WWW, pp. 545–554. ACM (2001)

    Google Scholar 

  12. Claessen, K., Hughes, J.: QuickCheck: a lightweight tool for random testing of Haskell programs. In: ICFP, pp. 268–279. ACM (2000)

    Google Scholar 

  13. Collina, M., Corazza, G.E., Vanelli-Coralli, A.: Introducing the QEST broker: scaling the IoT by bridging MQTT and REST. In: PIMRC, pp. 36–41. IEEE (2012)

    Google Scholar 

  14. Draheim, D., Grundy, J.C., Hosking, J.G., Lutteroth, C., Weber, G.: Realistic load testing of web applications. In: CSMR, pp. 57–70. IEEE (2006)

    Google Scholar 

  15. Govindarajulu, Z.: Sequential Statistics. World Scientific (2004)

    Google Scholar 

  16. Hoeffding, W.: Probability inequalities for sums of bounded random variables. J. Am. Stat. Assoc. 58(301), 13–30 (1963)

    Article  MathSciNet  Google Scholar 

  17. Houimli, M., Kahloul, L., Benaoun, S.: Formal specification, verification and evaluation of the MQTT protocol in the Internet of Things. In: ICMIT, pp. 214–221. IEEE, December 2017

    Google Scholar 

  18. Hughes, J.: QuickCheck testing for fun and profit. In: Hanus, M. (ed.) PADL 2007. LNCS, vol. 4354, pp. 1–32. Springer, Heidelberg (2006). https://doi.org/10.1007/978-3-540-69611-7_1

    Chapter  Google Scholar 

  19. Kalaji, A.S., Hierons, R.M., Swift, S.: Generating feasible transition paths for testing from an extended finite state machine. In: ICST, pp. 230–239. IEEE (2009)

    Google Scholar 

  20. Lee, S., Kim, H., Hong, D., Ju, H.: Correlation analysis of MQTT loss and delay according to QoS level. In: ICOIN, pp. 714–717. IEEE (2013)

    Google Scholar 

  21. Legay, A., Delahaye, B., Bensalem, S.: Statistical model checking: an overview. In: Barringer, H., et al. (eds.) RV 2010. LNCS, vol. 6418, pp. 122–135. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-16612-9_11

    Chapter  Google Scholar 

  22. Legay, A., Sedwards, S.: On statistical model checking with PLASMA. In: TASE, pp. 139–145. IEEE (2014)

    Google Scholar 

  23. Lu, Y., Nolte, T., Bate, I., Cucu-Grosjean, L.: A statistical response-time analysis of real-time embedded systems. In: RTSS, pp. 351–362. IEEE (2012)

    Google Scholar 

  24. Nagelkerke, N.J.: A note on a general definition of the coefficient of determination. Biometrika 78(3), 691–692 (1991)

    Article  MathSciNet  Google Scholar 

  25. Papadakis, M., Sagonas, K.: A PropEr integration of types and function specifications with property-based testing. In: Erlang 2011, pp. 39–50. ACM (2011)

    Google Scholar 

  26. Pearson, K.: Note on regression and inheritance in the case of two parents. Proc. Roy. Soc. Lond. 58, 240–242 (1895)

    Article  Google Scholar 

  27. Rencher, A., Christensen, W.: Methods of Multivariate Analysis. Wiley (2012)

    Google Scholar 

  28. Tyagi, R.S.: A comparative study of performance testing tools. Int. J. Adv. Res. Comput. Sci. Softw. Eng. IJARCSSE 3(5), 1300–1307 (2013)

    Google Scholar 

  29. Schumi, R., Lang, P., Aichernig, B.K., Krenn, W., Schlick, R.: Checking response-time properties of web-service applications under stochastic user profiles. In: Yevtushenko, N., Cavalli, A.R., Yenigün, H. (eds.) ICTSS 2017. LNCS, vol. 10533, pp. 293–310. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-67549-7_18

    Chapter  Google Scholar 

  30. Tappler, M., Aichernig, B.K., Bloem, R.: Model-based testing IoT communication via active automata learning. In: ICST, pp. 276–287. IEEE (2017)

    Google Scholar 

  31. Thangavel, D., Ma, X., Valera, A.C., Tan, H., Tan, C.K.: Performance evaluation of MQTT and CoAP via a common middleware. In: ISSNIP, pp. 1–6. IEEE (2014)

    Google Scholar 

  32. Wald, A.: Sequential analysis. Courier Corporation (1973)

    Google Scholar 

Download references

Acknowledgements

This work was supported by the TU Graz LEAD project “Dependable Internet of Things in Adverse Environments”. We are grateful to our colleague Martin Tappler and to the anonymous reviewers for their excellent feedback that helped in improving the quality of the paper.

Author information

Authors and Affiliations

  1. Institute of Software Technology, Graz University of Technology, Graz, Austria

    Bernhard K. Aichernig & Richard Schumi

Authors
  1. Bernhard K. Aichernig
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Richard Schumi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Richard Schumi .

Editor information

Editors and Affiliations

  1. Macquarie University , Sydney, New South Wales, Australia

    Annabelle McIver

  2. University of Turin , Turin, Italy

    Andras Horvath

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Aichernig, B.K., Schumi, R. (2018). How Fast Is MQTT?. In: McIver, A., Horvath, A. (eds) Quantitative Evaluation of Systems. QEST 2018. Lecture Notes in Computer Science(), vol 11024. Springer, Cham. https://doi.org/10.1007/978-3-319-99154-2_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-99154-2_3

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-99153-5

  • Online ISBN: 978-3-319-99154-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation