Skip to main content
SpringerLink
Log in
Book cover

International Conference on Model-Driven Engineering and Software Development

MODELSWARD 2018: Model-Driven Engineering and Software Development pp 406–429Cite as

A Methodology for Generating Tests for Evaluating User-Centric Performance of Mobile Streaming Applications

  • Conference paper
  • First Online:

  • 532 Accesses

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 991))

Abstract

Compared to other platforms, mobile apps’ quality assurance is more challenging, since their functionality is affected by the surrounding environment. In literature, a considerable volume of research has been devoted to develop frameworks that facilitate conducting performance analysis during the development life cycle. However, less attention has been given to test generation and test selection criteria for performance evaluation. In this work, a model based test generation methodology is proposed to evaluate the impact of the interaction of the environment, the wireless network, and the app configurations on the performance of a mobile streaming app and thereby on the experience of the end user. The methodology steps, inputs, and outputs are explained using an app example. The methodology assumes that the app has a network access through a WiFi access point. We evaluate the effectiveness of the methodology by comparing the time cost to design a test suite with random testing. The obtained results are very promising.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

References

  1. Advanced combinatorial testing system (acts) (2016). http://csrc.nist.gov/groups/SNS/acts/

  2. Abbors, F., Ahmad, T., Truscan, D., Porres, I.: Model-based performance testing in the cloud using the mbpet tool. In: Proceedings of the 4th ACM/SPEC International Conference on Performance Engineering, pp. 423–424. ACM (2013)

    Google Scholar 

  3. Abbors, F., Can, D.T.: Approaching performance testing from a model-based testing perspective. In: 2010 Second International Conference on Advances in System Testing and Validation Lifecycle (VALID), pp. 125–128. IEEE (2010)

    Google Scholar 

  4. Adan, I., Resing, J.: Queueing Theory. Department of Mathematics and Computing Science, Eindhoven University of Technology, Eindhoven (2001)

    Google Scholar 

  5. Al-tekreeti, M., Naik, K., Abdrabou, A., Zaman, M., Srivastava, P.: Test generation for performance evaluation of mobile multimedia streaming applications. In: Proceedings of the 6th International Conference on Model-Driven Engineering and Software Development, vol. 1, pp. 225–236. SciTePress (2018)

    Google Scholar 

  6. Avritzer, A., Kondek, J., Liu, D., Weyuker, E.J.: Software performance testing based on workload characterization. In: Proceedings of the 3rd International Workshop on Software and Performance, pp. 17–24. ACM (2002)

    Google Scholar 

  7. Balsamo, S., Marco, A.D., Inverardi, P., Simeoni, M.: Model-based performance prediction in software development: a survey. IEEE Trans. Softw. Eng. 30(5), 295–310 (2004)

    Article  Google Scholar 

  8. Balsamo, S., Marzolla, M.: A simulation-based approach to software performance modeling. In: ACM SIGSOFT Software Engineering Notes, vol. 28, pp. 363–366. ACM (2003)

    Google Scholar 

  9. Barna, C., Litoiu, M., Ghanbari, H.: Autonomic load-testing framework. In: Proceedings of the 8th ACM International Conference on Autonomic Computing, pp. 91–100. ACM (2011)

    Google Scholar 

  10. Barna, C., Litoiu, M., Ghanbari, H.: Model-based performance testing (NIER track). In: Proceedings of the 33rd International Conference on Software Engineering, pp. 872–875. ACM (2011)

    Google Scholar 

  11. Briand, L., Nejati, S., Sabetzadeh, M., Bianculli, D.: Testing the untestable: model testing of complex software-intensive systems. In: Proceedings of the 38th International Conference on Software Engineering Companion, pp. 789–792. ACM (2016)

    Google Scholar 

  12. Brosig, F., Meier, P., Becker, S., et al.: Quantitative evaluation of model-driven performance analysis and simulation of component-based architectures. IEEE Trans. Softw. Eng. 41(2), 157–175 (2015)

    Article  Google Scholar 

  13. Canfora, G., Mercaldo, F., Visaggio, C.A., DAngelo, M., Furno, A., Manganelli, C.: A case study of automating user experience-oriented performance testing on smartphones. In: 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation, pp. 66–69. IEEE (2013)

    Google Scholar 

  14. Charnes, J.M.: Analyzing multivariate output. In: Proceedings of the 27th Conference on Winter Simulation, pp. 201–208. IEEE Computer Society (1995)

    Google Scholar 

  15. Cortellessa, V., Di Marco, A., Inverardi, P.: Model-Based Software Performance Analysis. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-13621-4

    Book  Google Scholar 

  16. Costa, L.T., Czekster, R.M., de Oliveira, F.M., Rodrigues, E.d.M., da Silveira, M.B., Zorzo, A.F.: Generating performance test scripts and scenarios based on abstract intermediate models. In: SEKE, pp. 112–117 (2012)

    Google Scholar 

  17. Cox, D.R.: The analysis of non-Markovian stochastic processes by the inclusion of supplementary variables. In: Mathematical Proceedings of the Cambridge Philosophical Society, vol. 51, pp. 433–441. Cambridge University Press (1955)

    Google Scholar 

  18. Cox, D.R., Miller, H.D.: The Theory of Stochastic Processes. CRC Press, Boca Raton (1977)

    MATH  Google Scholar 

  19. Da Silveira, M.B., Rodrigues, E.d.M., Zorzo, A.F., Costa, L.T., Vieira, H.V., De Oliveira, F.M.: Generation of scripts for performance testing based on UML models. In: SEKE, pp. 258–263 (2011)

    Google Scholar 

  20. Dantas, V.L.L., Marinho, F.G., da Costa, A.L., Andrade, R.M.: Testing requirements for mobile applications. In: 24th International Symposium on Computer and Information Sciences, ISCIS 2009, pp. 555–560. IEEE (2009)

    Google Scholar 

  21. Di Penta, M., Canfora, G., Esposito, G., Mazza, V., Bruno, M.: Search-based testing of service level agreements. In: Proceedings of the 9th Annual Conference on Genetic and Evolutionary Computation, pp. 1090–1097. ACM (2007)

    Google Scholar 

  22. Diaz, A., Merino, P., Rivas, F.J.: Mobile application profiling for connected mobile devices. IEEE Pervasive Comput. 9(1), 54–61 (2010)

    Article  Google Scholar 

  23. Doerner, K., Gutjahr, W.J.: Extracting test sequences from a Markov software usage model by ACO. In: Cantú-Paz, E., et al. (eds.) GECCO 2003. LNCS, vol. 2724, pp. 2465–2476. Springer, Heidelberg (2003). https://doi.org/10.1007/3-540-45110-2_150

    Chapter  Google Scholar 

  24. Fiedler, M., Hossfeld, T., Tran-Gia, P.: A generic quantitative relationship between quality of experience and quality of service. IEEE Netw. 24(2), 36–41 (2010)

    Article  Google Scholar 

  25. German, R.: Performance Analysis of Communication Systems with Non-Markovian Stochastic Petri Nets. Wiley, Hoboken (2000)

    MATH  Google Scholar 

  26. Gias, A.U., Sakib, K.: An adaptive Bayesian approach for URL selection to test performance of large scale web-based systems. In: Companion Proceedings of the 36th International Conference on Software Engineering, pp. 608–609. ACM (2014)

    Google Scholar 

  27. Gosavi, A.: Simulation-Based Optimization: Parametric Optimization Techniques and Reinforcement. ORSIS, vol. 55. Springer, Boston (2015). https://doi.org/10.1007/978-1-4899-7491-4

    Book  MATH  Google Scholar 

  28. Grechanik, M., Fu, C., Xie, Q.: Automatically finding performance problems with feedback-directed learning software testing. In: 2012 34th International Conference on Software Engineering (ICSE), pp. 156–166. IEEE (2012)

    Google Scholar 

  29. Grindal, M., Offutt, J., Andler, S.F.: Combination testing strategies: a survey. Softw. Test. Verif. Reliab. 15(3), 167–199 (2005)

    Article  Google Scholar 

  30. Gu, Y., Ge, Y.: Search-based performance testing of applications with composite services. In: International Conference on Web Information Systems and Mining, WISM 2009, pp. 320–324. IEEE (2009)

    Google Scholar 

  31. Guderlei, R., Mayer, J.: Statistical metamorphic testing: testing programs with random output by means of statistical hypothesis tests and metamorphic testing. In: 7th International Conference on Quality Software (QSIC), pp. 404–409. IEEE (2007)

    Google Scholar 

  32. Guderlei, R., Mayer, J., Schneckenburger, C., Fleischer, F.: Testing randomized software by means of statistical hypothesis tests. In: Fourth International Workshop on Software Quality Assurance: in Conjunction with the 6th ESEC/FSE Joint Meeting, pp. 46–54. ACM (2007)

    Google Scholar 

  33. Ivanovici, M., Beuran, R.: Correlating quality of experience and quality of service for network applications, chap. 15. In: Adibi, S. (ed.) Quality of Service Architectures for Wireless Networks: Performance Metrics and Management, pp. 326–351. IGI Global, Hershey (2010)

    Chapter  Google Scholar 

  34. Jiang, Z., Hassan, A.: A survey on load testing of large-scale software systems. IEEE Trans. Softw. Eng. 41(11), 1091–1118 (2015)

    Article  Google Scholar 

  35. Joorabchi, M.E., Mesbah, A., Kruchten, P.: Real challenges in mobile app development. In: 2013 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, pp. 15–24. IEEE (2013)

    Google Scholar 

  36. Kim, Y., Choi, O., Kim, M., Baik, J., Kim, T.H.: Validating software reliability early through statistical model checking. IEEE Softw. 30(3), 35–41 (2013)

    Article  Google Scholar 

  37. Koziolek, H.: Performance evaluation of component-based software systems: a survey. Perform. Eval. 67(8), 634–658 (2010)

    Article  Google Scholar 

  38. Kumar, R., Tomkins, A., Vassilvitskii, S., Vee, E.: Inverting a steady-state. In: Proceedings of the Eighth ACM International Conference on Web Search and Data Mining, pp. 359–368. ACM (2015)

    Google Scholar 

  39. Law, A.M.: Simulation Modeling and Analysis, 5th edn. McGraw-Hill, New York (2015)

    Google Scholar 

  40. Li, M., Claypool, M., Kinicki, R.: Playout buffer and rate optimization for streaming over IEEE 802.11 wireless networks. ACM Trans. Multimed. Comput. Commun. Appl. (TOMM) 5(3), 26 (2009)

    Google Scholar 

  41. Liu, Y., Xu, C., Cheung, S.C.: Diagnosing energy efficiency and performance for mobile internetware applications. IEEE Softw. 32(1), 67–75 (2015)

    Article  Google Scholar 

  42. Ma, K.J., Bartos, R., Bhatia, S., Nair, R.: Mobile video delivery with HTTP. IEEE Commun. Mag. 49(4), 166–175 (2011)

    Article  Google Scholar 

  43. Matinnejad, R., Nejati, S., Briand, L.C., Bruckmann, T.: Automated test suite generation for time-continuous Simulink models. In: Proceedings of the 38th International Conference on Software Engineering, pp. 595–606. ACM (2016)

    Google Scholar 

  44. Mok, R.K., Chan, E.W., Chang, R.K.: Measuring the quality of experience of HTTP video streaming. In: 12th IFIP/IEEE International Symposium on Integrated Network Management (IM 2011) and Workshops, pp. 485–492. IEEE (2011)

    Google Scholar 

  45. Nayebi, F., Desharnais, J.M., Abran, A.: The state of the art of mobile application usability evaluation. In: CCECE, pp. 1–4 (2012)

    Google Scholar 

  46. Nie, C., Leung, H.: A survey of combinatorial testing. ACM Comput. Surv. (CSUR) 43(2), 11 (2011)

    Article  Google Scholar 

  47. Prowell, S.J.: Using Markov chain usage models to test complex systems. In: Proceedings of the 38th Annual Hawaii International Conference on System Sciences, HICSS 2005, p. 318c. IEEE (2005)

    Google Scholar 

  48. Rios, L.M., Sahinidis, N.V.: Derivative-free optimization: a review of algorithms and comparison of software implementations. J. Glob. Optim. 56(3), 1247–1293 (2013)

    Article  MathSciNet  Google Scholar 

  49. Satoh, I.: Software testing for wireless mobile computing. IEEE Wirel. Commun. 11(5), 58–64 (2004)

    Article  Google Scholar 

  50. Sebih, N., Weitl, F., Artho, C., Hagiya, M., Tanabe, Y., Yamamoto, M.: Software model checking of UDP-based distributed applications. In: 2014 Second International Symposium on Computing and Networking, pp. 96–105. IEEE (2014)

    Google Scholar 

  51. Siavashi, F., Truscan, D.: Environment modeling in model-based testing: concepts, prospects and research challenges: a systematic literature review. In: Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering, p. 30. ACM (2015)

    Google Scholar 

  52. Utting, M., Pretschner, A., Legeard, B.: A taxonomy of model-based testing approaches. Softw. Test. Verif. Reliab. 22(5), 297–312 (2012)

    Article  Google Scholar 

  53. Walls, R.J., Brun, Y., Liberatore, M., Levine, B.N.: Discovering specification violations in networked software systems. In: 2015 IEEE 26th International Symposium on Software Reliability Engineering (ISSRE), pp. 496–506. IEEE (2015)

    Google Scholar 

  54. Walton, G.H., Poore, J.H.: Generating transition probabilities to support model-based software testing. Softw.: Pract. Exp 30(10), 1095–1106 (2000)

    Google Scholar 

  55. Weyuker, E.J., Vokolos, F.I.: Experience with performance testing of software systems: issues, an approach, and case study. IEEE Trans. Softw. Eng. 12, 1147–1156 (2000)

    Article  Google Scholar 

  56. Xu, Q., Mehrotra, S., Mao, Z., Li, J.: PROTEUS: network performance forecast for real-time, interactive mobile applications. In: Proceedings of the 11th Annual International Conference on Mobile Systems, Applications, and Services, pp. 347–360. ACM (2013)

    Google Scholar 

  57. Yang, C.S.D., Pollock, L.L.: Towards a structural load testing tool. In: ACM SIGSOFT Software Engineering Notes, vol. 21, pp. 201–208. ACM (1996)

    Google Scholar 

  58. Yılmaz, C., Fouche, S., Cohen, M.B., Porter, A., Demiröz, G., Koç, U.: Moving forward with combinatorial interaction testing. Computer 47(2), 37–45 (2014)

    Article  Google Scholar 

  59. Zhang, J., Cheung, S.C.: Automated test case generation for the stress testing of multimedia systems. Softw.: Pract. Exp. 32(15), 1411–1435 (2002)

    MATH  Google Scholar 

  60. Zhang, J., Cheung, S.C., Chanson, S.T.: Stress testing of distributed multimedia software systems. In: Proceedings of the IFIP TC6 WG6.1 Joint International Conference on Formal Description Techniques for Distributed Systems and Communication Protocols and Protocol Specification, Testing and Verification, pp. 119–133. Kluwer, BV (1999)

    Google Scholar 

  61. Zhang, P., Elbaum, S., Dwyer, M.B.: Automatic generation of load tests. In: Proceedings of the 2011 26th IEEE/ACM International Conference on Automated Software Engineering, pp. 43–52. IEEE Computer Society (2011)

    Google Scholar 

  62. Zhang, P., Elbaum, S., Dwyer, M.B.: Compositional load test generation for software pipelines. In: Proceedings of the 2012 International Symposium on Software Testing and Analysis, pp. 89–99. ACM (2012)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Waterloo, Waterloo, ON, Canada

    Mustafa Al-tekreeti & Kshirasagar Naik

  2. UAE University, Al-Ain, UAE

    Atef Abdrabou

  3. Technologie Sanstream, Gatineau, QC, Canada

    Marzia Zaman & Pradeep Srivastava

Authors
  1. Mustafa Al-tekreeti
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kshirasagar Naik
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Atef Abdrabou
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Marzia Zaman
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Pradeep Srivastava
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mustafa Al-tekreeti .

Editor information

Editors and Affiliations

  1. Université d’Angers/ESEO, Angers, France

    Slimane Hammoudi

  2. University of Twente, Enschede, The Netherlands

    Luís Ferreira Pires

  3. Malina Software Corp., Nepean, ON, Canada

    Bran Selic

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Al-tekreeti, M., Naik, K., Abdrabou, A., Zaman, M., Srivastava, P. (2019). A Methodology for Generating Tests for Evaluating User-Centric Performance of Mobile Streaming Applications. In: Hammoudi, S., Pires, L., Selic, B. (eds) Model-Driven Engineering and Software Development. MODELSWARD 2018. Communications in Computer and Information Science, vol 991. Springer, Cham. https://doi.org/10.1007/978-3-030-11030-7_18

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-11030-7_18

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-11029-1

  • Online ISBN: 978-3-030-11030-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation