Skip to main content
SpringerLink
Log in

Model-based testing of apps in real network scenarios

  • SPIN 2018
  • Published:
International Journal on Software Tools for Technology Transfer Aims and scope Submit manuscript

Abstract

Traditional testing methods for mobile apps focus on detecting execution errors. However, the evolution of mobile networks toward 5G will require additional support for app developers to also ensure good performance and user experience. Manual testing in a number of scenarios is not enough to satisfy the expectations of the apps’ end users. This paper presents the testing framework developed in the TRIANGLE project (https://www.triangle-project.eu/), which integrates a complete mobile network testbed to test, benchmark and certify mobile apps. In this paper, we focus on a recent extension of the TRIANGLE framework that uses model-based testing based on model checking to support the automatic generation of user interactions. We introduce the complete testing framework and the basis of the model-based extension. Finally, we use the testing framework to evaluate the performance of the ExoPlayer app in different network scenarios. ExoPlayer is a video streaming app for Android that implements different adaptive streaming protocols.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Notes

  1. https://wireless.dekra-product-safety.com/

  2. http://quamotion.mobi/Mwc

References

  1. 3GPP: TR37.901:User Equipment (UE) application layer data throughput (Rel. 15). Technical report (2018)

  2. 5GENESIS project consortium: Deliverable D2.3: Initial planning of tests and experimentation. Pu (2018)

  3. Amalfitano, D., Amatucci, N., Memon, A.M., Tramontana, P., Fasolino, A.R.: A general framework for comparing automatic testing techniques of Android mobile apps. J. Syst. Softw. 125, 322–343 (2017). https://doi.org/10.1016/j.jss.2016.12.017

    Article  Google Scholar 

  4. Amalfitano, D., Fasolino, A.R., Tramontana, P., Ta, B.D., Memon, A.M.: MobiGUITAR: automated model-based testing of mobile apps. IEEE Softw. 32(5), 53–59 (2015). https://doi.org/10.1109/MS.2014.55

    Article  Google Scholar 

  5. Baek, Y.M., Bae, D.H.: Automated model-based android GUI testing using multi-level GUI comparison criteria. In: Proceedings of the 31st IEEE/ACM international conference on automated software engineering, ASE 2016, pp. 238–249. ACM (2016). https://doi.org/10.1145/2970276.2970313

  6. Broy, M., Jonsson, B., Katoen, J.P., Leucker, M., Pretschner, A.: Model-Based Testing of Reactive Systems: Advanced Lectures. Springer, Berlin (2005)

    Book  Google Scholar 

  7. Cattoni, A.F., Corrales-Madueño, G., Dieudonne, M., Merino, P., Díaz-Zayas, A., Salmerón, A., Carlier, F., Saint-Germain, B., Morris, D., Figueiredo, R., Caffrey, J., Baos, J., Cárdenas, C., Roche, N., Moore, A.: An end-to-end testing ecosystem for 5G. In: European conference on networks and communications (EuCNC 2016), pp. 307–312 (2016). https://doi.org/10.1109/EuCNC.2016.7561053

  8. Cellular Telecommunications Industry Association (CTIA): Battery Life Test Plan v.1.2. Technical report (2018). https://api.ctia.org/wp-content/uploads/2018/04/CTIA-Battery-Life-Test-Plan-Version-1.2.pdf

  9. Díaz-Zayas, A., Panizo, L., Baños, J., Cárdenas, C., Dieudonne, M.: QoE Evaluation: The TRIANGLE testbed approach. Wirel. Commun. Mob. Comput. p. 12 (2018). https://doi.org/10.1155/2018/6202854

  10. Espada, A.R., Gallardo, M.M., Salmerón, A., Merino, P.: Performance analysis of spotify® for android with model based testing. Mob. Inf. Syst. 2017, 14 (2017). https://doi.org/10.1155/2017/2012696

    Article  Google Scholar 

  11. Global System for Mobile Communications Association (GSMA): Smartphone performance test case guideline v3.0. Technical report (2017). https://www.gsma.com/newsroom/wp-content/uploads//TS.29_v3.0.pdf

  12. Holzmann, G.: The SPIN Model Checker: Primer and Reference Manual. Addison-Wesley Professional, Boston (2003)

    Google Scholar 

  13. ISO/IEC: Information technology Dynamic adaptive streaming over HTTP (DASH).http://standards.iso.org/ittf/PubliclyAvailableStandards/c065274_ISO_IEC_23009-1_2014.zip (2014)

  14. Koumaras, H., Tsolkas, D., Gardikis, G., Merino-Gómez, P., Frascolla, V., Triantafyllopoulou, D., Emmelmann, M., Koumaras, V., Garcia-Osma, M.L., Munaretto, D., Atxutegi, E., de Puga, J.S., Alay, O., Brunstrom, A., Bosneag, A.M.C.: 5GENESIS: The genesis of a flexible 5G Facility. In: IEEE international workshop on computer-aided modeling analysis and design of communication links and networks (CAMAD-2018 ) (2018)

  15. Kozamernik, F., Steinmann, V., Sunna, P., Wyckens, E.: SAMVIQA new EBU methodology for video quality evaluations in multimedia. SMPTE Motion Imaging J. 114(4), 152–160 (2005). https://doi.org/10.5594/J11535

    Article  Google Scholar 

  16. Liu, Y., Xu, C., Cheung, S.: Characterizing and detecting performance bugs for smartphone applications. In: Proceedings of the 36th international conference on software engineering, ICSE 2014, pp. 1013–1024. ACM (2014). https://doi.org/10.1145/2568225.2568229

  17. Massol, V., Husted, T.: JUnit in Action. Manning Publications Co., Greenwich (2003)

    Google Scholar 

  18. Mehmood, M.A., Wundsam, A., Uhlig, S., Levin, D., Sarrar, N., Feldmann, A.: QoE-Lab: towards evaluating quality of experience for future internet conditions. In: Korakis, T., Li, H., Tran-Gia, P., Park, H.S. (eds.) Testbeds and Research Infrastructure. Development of Networks and Communities, pp. 286–301. Springer, Berlin (2012)

    Chapter  Google Scholar 

  19. Microsoft Corporation: Smooth streaming protocol. https://winprotocoldoc.blob.core.windows.net/productionwindowHrBsarchives/MS-SSTR/[MS-SSTR].pdfHrB (2009)

  20. NeoLoad by Neotys: The load testing platform accelerating DevOps. http://www.neotys.com/. Last accessed: 10/26/2018

  21. NGMN Alliance: Definition of the testing framework for the NGMN 5G pre-commecial networks trails v1.0. Technical report (2018).https://www.ngmn.org/fileadmin/ngmn/content/downloads/Technical/2018/180220__PreCommTrials_Framework_definition_v1_0.pdf

  22. Panizo, L., Díaz-Zayas, A., García, B.: An extension of TRIANGLE testbed with model-based testing. In: M.d.M. Gallardo, P. Merino (eds.) Proceedings of the 25th international symposium on model checking software (SPIN2018), pp. 190–195. Springer International Publishing (2018). https://doi.org/10.1007/978-3-319-94111-0_11

  23. Panizo, L., Salmerón, A., Gallardo, M.M., Merino, P.: Guided Test Case Generation for Mobile Apps in the TRIANGLE Project: Work in Progress. In: Proc. of the 24th International SPIN Symposium on Model Checking of Software, pp. 192–195. ACM (2017). https://doi.org/10.1145/3092282.3092298

  24. Pantos, R., May, W.: HTTP Live Streaming. RFC 8216 (2017). https://doi.org/10.17487/RFC8216

  25. Perfecto: The cloud-based platform for continuous testing in a DevOps environment. https://www.perfecto.io/. Last accessed: 10/26/2018

  26. Solera, M., Toril, M., Palomo, I., Gomez, G., Poncela, J.: A testbed for evaluating video streaming services in LTE. Wirel. Pers. Commun. 98(3), 2753–2773 (2018). https://doi.org/10.1007/s11277-017-4999-0

    Article  Google Scholar 

  27. TRIANGLE project consortium: deliverable D2.1: initial report on the testing scenarios, requirements and use cases. Public (2016)

  28. TRIANGLE project consortium: Deliverable D2.6: Final test scenario and test specifications. Public (2018)

  29. Yang, S., Yan, D., Rountev, A.: Testing for poor responsiveness in android applications. In: Proceedings of the 1st international workshop on the engineering of mobile-enabled systems (MOBS), pp. 1–6 (2013). https://doi.org/10.1109/MOBS.2013.6614215

  30. Yang, W., Prasad, M.R., Xie, T.: A grey-box approach for automated GUI-model generation of mobile applications. In: V. Cortellessa, D. Varró (eds.) Proceedings of the 16th international conference on fundamental approaches to software engineering (FASE 2013), pp. 250–265. Springer,Berlin (2013). https://doi.org/10.1007/978-3-642-37057-1_19

  31. Zein, S., Salleh, N., Grundy, J.: A systematic mapping study of mobile application testing techniques. J. Syst. Softw. 117, 334–356 (2016). https://doi.org/10.1016/j.jss.2016.03.065

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Departamento de Lenguajes y Ciencias de la Computación, Universidad de Málaga, Andalucía Tech, Málaga, Spain

    Laura Panizo & Bruno García

  2. Universidad de Málaga, Andalucía Tech, Málaga, Spain

    Almudena Díaz

Authors
  1. Laura Panizo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Almudena Díaz
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bruno García
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Laura Panizo.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This work is funded by the European Union Horizon 2020 research and innovation programme, grant agreement No 688712 (TRIANGLE project) and 815178 (5GENESIS project).

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Panizo, L., Díaz, A. & García, B. Model-based testing of apps in real network scenarios. Int J Softw Tools Technol Transfer 22, 105–114 (2020). https://doi.org/10.1007/s10009-019-00518-2

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10009-019-00518-2

Keywords

Search

Navigation