Skip to main content
SpringerLink
Log in

Predicting and Testing Latencies with Deep Learning: An IoT Case Study

  • Conference paper
  • First Online:
Tests and Proofs (TAP 2019)

Part of the book series: Lecture Notes in Computer Science ((LNPSE,volume 11823))

Included in the following conference series:

Abstract

The Internet of things (IoT) is spreading into the everyday life of millions of people. However, the quality of the underlying communication technologies is still questionable. In this work, we are analysing the performance of an implementation of MQTT, which is a major communication protocol of the IoT. We perform model-based test-case generation to generate log data for training a neural network. This neural network is applied to predict latencies depending on different features, like the number of active clients. The predictions are integrated into our initial functional model, and we exploit the resulting timed model for statistical model checking. This allows us to answer questions about the expected performance for various usage scenarios. The benefit of our approach is that it enables a convenient extension of a functional model with timing aspects using deep learning. A comparison to our previous work with linear regression shows that deep learning needs less manual effort in data preprocessing and provides significantly better predictions.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://fscheck.github.io/FsCheck

  2. 2.

    https://m2mqtt.wordpress.com

References

  1. Agha, G., Palmskog, K.: A survey of statistical model checking. ACM Trans. Model. Comput. Simul. (TOMACS) 28(1), 6:1–6:39 (2018)

    Article  MathSciNet  Google Scholar 

  2. Aichernig, B.K., et al.: Learning and statistical model checking of system response times. Softw. Qual. J. 27, 757–795 (2019)

    Article  Google Scholar 

  3. Aichernig, B.K., Kann, S., Schumi, R.: Statistical model checking of response times for different system deployments. In: Feng, X., Müller-Olm, M., Yang, Z. (eds.) SETTA 2018. LNCS, vol. 10998, pp. 153–169. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-99933-3_11

    Chapter  Google Scholar 

  4. Aichernig, B.K., Schumi, R.: Property-based testing with FsCheck by deriving properties from business rule models. In: ICSTW, pp. 219–228. IEEE (2016)

    Google Scholar 

  5. Aichernig, B.K., Schumi, R.: Property-based testing of web services by deriving properties from business-rule models. Softw. Syst. Model. 18, 889–911 (2019)

    Article  Google Scholar 

  6. Aichernig, B.K., Schumi, R.: Statistical model checking meets property-based testing. In: ICST, pp. 390–400. IEEE (2017)

    Google Scholar 

  7. Aichernig, B.K., Schumi, R.: How fast is MQTT? In: McIver, A., Horvath, A. (eds.) QEST 2018. LNCS, vol. 11024, pp. 36–52. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-99154-2_3

    Chapter  Google Scholar 

  8. Alur, R., Dill, D.L.: A theory of timed automata. Theor. Comput. Sci. 126(2), 183–235 (1994)

    Article  MathSciNet  MATH  Google Scholar 

  9. Arts, T.: On shrinking randomly generated load tests. In: Erlang 2014, pp. 25–31. ACM (2014)

    Google Scholar 

  10. Ballarini, P., Bertrand, N., Horváth, A., Paolieri, M., Vicario, E.: Transient analysis of networks of stochastic timed automata using stochastic state classes. In: Joshi, K., Siegle, M., Stoelinga, M., D’Argenio, P.R. (eds.) QEST 2013. LNCS, vol. 8054, pp. 355–371. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-40196-1_30

    Chapter  Google Scholar 

  11. Banks, A., Gupta, R.: MQTT version 3.1.1. OASIS Standard, December 2014

    Google Scholar 

  12. Becker, S., Koziolek, H., Reussner, R.H.: The Palladio component model for model-driven performance prediction. J. Syst. Softw. 82(1), 3–22 (2009)

    Article  Google Scholar 

  13. Bishop, C.M.: Pattern Recognition and Machine Learning. Information Science and Statistics. Springer, New York (2006)

    MATH  Google Scholar 

  14. Book, M., Gruhn, V., Hülder, M., Köhler, A., Kriegel, A.: Cost and response time simulation for web-based applications on mobile channels. In: QSIC, pp. 83–90. IEEE (2005)

    Google Scholar 

  15. Bulychev, P.E., et al.: UPPAAL-SMC: statistical model checking for priced timed automata. In: QAPL. EPTCS, vol. 85, pp. 1–16. Open Publishing Association (2012)

    Google Scholar 

  16. Chen, X., Mohapatra, P., Chen, H.: An admission control scheme for predictable server response time for web accesses. In: WWW, pp. 545–554. ACM (2001)

    Google Scholar 

  17. Cho, K., et al.: Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078 (2014)

  18. Claessen, K., Hughes, J.: QuickCheck: a lightweight tool for random testing of Haskell programs. In: ICFP, pp. 268–279. ACM (2000)

    Google Scholar 

  19. Collina, M., Corazza, G.E., Vanelli-Coralli, A.: Introducing the QEST broker: scaling the IoT by bridging MQTT and REST. In: PIMRC, pp. 36–41. IEEE (2012)

    Google Scholar 

  20. Deng, L., Yu, D.: Deep learning: methods and applications. Found. Trends Sig. Process. 7(3–4), 197–387 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  21. Draheim, D., Grundy, J.C., Hosking, J.G., Lutteroth, C., Weber, G.: Realistic load testing of web applications. In: CSMR, pp. 57–70. IEEE (2006)

    Google Scholar 

  22. Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: AISTATS. JMLR Proceedings, vol. 9, pp. 249–256. JMLR.org (2010)

    Google Scholar 

  23. Glorot, X., Bordes, A., Bengio, Y.: Deep sparse rectifier neural networks. In: AISTATS. JMLR Proceedings, vol. 15, pp. 315–323. JMLR.org (2011)

    Google Scholar 

  24. Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press, Cambridge (2016)

    MATH  Google Scholar 

  25. Grinchtein, O.: Learning of Timed Systems. Ph.D. thesis, Uppsala University (2008)

    Google Scholar 

  26. Hawkins, D.M.: The problem of overfitting. J. Chem. Inf. Model. 44(1), 1–12 (2004)

    MathSciNet  Google Scholar 

  27. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)

    Article  Google Scholar 

  28. Hoeffding, W.: Probability inequalities for sums of bounded random variables. J. Am. Stat. Assoc. 58(301), 13–30 (1963)

    Article  MathSciNet  MATH  Google Scholar 

  29. Houimli, M., Kahloul, L., Benaoun, S.: Formal specification, verification and evaluation of the MQTT protocol in the Internet of Things. In: ICMIT, pp. 214–221. IEEE, December 2017

    Google Scholar 

  30. Hughes, J.: QuickCheck testing for fun and profit. In: Hanus, M. (ed.) PADL 2007. LNCS, vol. 4354, pp. 1–32. Springer, Heidelberg (2006). https://doi.org/10.1007/978-3-540-69611-7_1

    Chapter  Google Scholar 

  31. Kalaji, A.S., Hierons, R.M., Swift, S.: Generating feasible transition paths for testing from an extended finite state machine. In: ICST, pp. 230–239. IEEE (2009)

    Google Scholar 

  32. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

  33. LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015)

    Article  Google Scholar 

  34. Lee, S., Kim, H., Hong, D., Ju, H.: Correlation analysis of MQTT loss and delay according to QoS level. In: ICOIN, pp. 714–717. IEEE (2013)

    Google Scholar 

  35. Legay, A., Delahaye, B., Bensalem, S.: Statistical model checking: an overview. In: Barringer, H., et al. (eds.) RV 2010. LNCS, vol. 6418, pp. 122–135. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-16612-9_11

    Chapter  Google Scholar 

  36. Legay, A., Sedwards, S.: On statistical model checking with PLASMA. In: TASE, pp. 139–145. IEEE (2014)

    Google Scholar 

  37. Lu, Y., Nolte, T., Bate, I., Cucu-Grosjean, L.: A statistical response-time analysis of real-time embedded systems. In: RTSS, pp. 351–362. IEEE (2012)

    Google Scholar 

  38. Nagelkerke, N.J.: A note on a general definition of the coefficient of determination. Biometrika 78(3), 691–692 (1991)

    Article  MathSciNet  MATH  Google Scholar 

  39. Papadakis, M., Sagonas, K.: A PropEr integration of types and function specifications with property-based testing. In: Erlang 2011, pp. 39–50. ACM (2011)

    Google Scholar 

  40. Pearson, K.: Note on regression and inheritance in the case of two parents. Proc. R. Soc. London 58, 240–242 (1895)

    Article  Google Scholar 

  41. Tyagi, R.S.: A comparative study of performance testing tools. Int. J. Adv. Res. Comput. Sci. Softw. Eng. IJARCSSE 3(5), 1300–1307 (2013)

    Google Scholar 

  42. Schmidt, J., Ghorbani, A., Hapfelmeier, A., Kramer, S.: Learning probabilistic real-time automata from multi-attribute event logs. Intell. Data Anal. 17(1), 93–123 (2013)

    Article  Google Scholar 

  43. Schumi, R.: Predicting and testing system response-times with statistical model checking and property-based testing. Ph.D. thesis, Graz University of Technology (2018)

    Google Scholar 

  44. Schumi, R., Lang, P., Aichernig, B.K., Krenn, W., Schlick, R.: Checking response-time properties of web-service applications under stochastic user profiles. In: Yevtushenko, N., Cavalli, A.R., Yenigün, H. (eds.) ICTSS 2017. LNCS, vol. 10533, pp. 293–310. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-67549-7_18

    Chapter  Google Scholar 

  45. Tappler, M., Aichernig, B.K., Bloem, R.: Model-based testing IoT communication via active automata learning. In: ICST, pp. 276–287. IEEE (2017)

    Google Scholar 

  46. Thangavel, D., Ma, X., Valera, A.C., Tan, H., Tan, C.K.: Performance evaluation of MQTT and CoAP via a common middleware. In: ISSNIP, pp. 1–6. IEEE (2014)

    Google Scholar 

  47. Verwer, S., de Weerdt, M., Witteveen, C.: A likelihood-ratio test for identifying probabilistic deterministic real-time automata from positive data. In: Sempere, J.M., García, P. (eds.) ICGI 2010. LNCS (LNAI), vol. 6339, pp. 203–216. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15488-1_17

    Chapter  Google Scholar 

  48. Wald, A.: Sequential Analysis. Courier Corporation, North Chelmsford (1973)

    MATH  Google Scholar 

  49. Wurm, A.: Predicting the latency of MQTT brokers using deep learning. Master’s thesis, Graz University of Technology (2018)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Institute of Software Technology, Graz University of Technology, Graz, Austria

    Bernhard K. Aichernig & Richard Schumi

  2. Signal Processing and Speech Communication Lab, Graz University of Technology, Graz, Austria

    Franz Pernkopf & Andreas Wurm

Authors
  1. Bernhard K. Aichernig
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Franz Pernkopf
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Richard Schumi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Andreas Wurm
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Richard Schumi .

Editor information

Editors and Affiliations

  1. Ludwig-Maximilians-Universität München, Munich, Germany

    Dirk Beyer

  2. University of Paris-Sud, Orsay, France

    Chantal Keller

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Aichernig, B.K., Pernkopf, F., Schumi, R., Wurm, A. (2019). Predicting and Testing Latencies with Deep Learning: An IoT Case Study. In: Beyer, D., Keller, C. (eds) Tests and Proofs. TAP 2019. Lecture Notes in Computer Science(), vol 11823. Springer, Cham. https://doi.org/10.1007/978-3-030-31157-5_7

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-31157-5_7

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-31156-8

  • Online ISBN: 978-3-030-31157-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation