Abstract
Traditional testing methods for mobile apps focus on detecting execution errors. However, the evolution of mobile networks toward 5G will require additional support for app developers to also ensure good performance and user experience. Manual testing in a number of scenarios is not enough to satisfy the expectations of the apps’ end users. This paper presents the testing framework developed in the TRIANGLE project (https://www.triangle-project.eu/), which integrates a complete mobile network testbed to test, benchmark and certify mobile apps. In this paper, we focus on a recent extension of the TRIANGLE framework that uses model-based testing based on model checking to support the automatic generation of user interactions. We introduce the complete testing framework and the basis of the model-based extension. Finally, we use the testing framework to evaluate the performance of the ExoPlayer app in different network scenarios. ExoPlayer is a video streaming app for Android that implements different adaptive streaming protocols.
Similar content being viewed by others
References
3GPP: TR37.901:User Equipment (UE) application layer data throughput (Rel. 15). Technical report (2018)
5GENESIS project consortium: Deliverable D2.3: Initial planning of tests and experimentation. Pu (2018)
Amalfitano, D., Amatucci, N., Memon, A.M., Tramontana, P., Fasolino, A.R.: A general framework for comparing automatic testing techniques of Android mobile apps. J. Syst. Softw. 125, 322–343 (2017). https://doi.org/10.1016/j.jss.2016.12.017
Amalfitano, D., Fasolino, A.R., Tramontana, P., Ta, B.D., Memon, A.M.: MobiGUITAR: automated model-based testing of mobile apps. IEEE Softw. 32(5), 53–59 (2015). https://doi.org/10.1109/MS.2014.55
Baek, Y.M., Bae, D.H.: Automated model-based android GUI testing using multi-level GUI comparison criteria. In: Proceedings of the 31st IEEE/ACM international conference on automated software engineering, ASE 2016, pp. 238–249. ACM (2016). https://doi.org/10.1145/2970276.2970313
Broy, M., Jonsson, B., Katoen, J.P., Leucker, M., Pretschner, A.: Model-Based Testing of Reactive Systems: Advanced Lectures. Springer, Berlin (2005)
Cattoni, A.F., Corrales-Madueño, G., Dieudonne, M., Merino, P., Díaz-Zayas, A., Salmerón, A., Carlier, F., Saint-Germain, B., Morris, D., Figueiredo, R., Caffrey, J., Baos, J., Cárdenas, C., Roche, N., Moore, A.: An end-to-end testing ecosystem for 5G. In: European conference on networks and communications (EuCNC 2016), pp. 307–312 (2016). https://doi.org/10.1109/EuCNC.2016.7561053
Cellular Telecommunications Industry Association (CTIA): Battery Life Test Plan v.1.2. Technical report (2018). https://api.ctia.org/wp-content/uploads/2018/04/CTIA-Battery-Life-Test-Plan-Version-1.2.pdf
Díaz-Zayas, A., Panizo, L., Baños, J., Cárdenas, C., Dieudonne, M.: QoE Evaluation: The TRIANGLE testbed approach. Wirel. Commun. Mob. Comput. p. 12 (2018). https://doi.org/10.1155/2018/6202854
Espada, A.R., Gallardo, M.M., Salmerón, A., Merino, P.: Performance analysis of spotify® for android with model based testing. Mob. Inf. Syst. 2017, 14 (2017). https://doi.org/10.1155/2017/2012696
Global System for Mobile Communications Association (GSMA): Smartphone performance test case guideline v3.0. Technical report (2017). https://www.gsma.com/newsroom/wp-content/uploads//TS.29_v3.0.pdf
Holzmann, G.: The SPIN Model Checker: Primer and Reference Manual. Addison-Wesley Professional, Boston (2003)
ISO/IEC: Information technology Dynamic adaptive streaming over HTTP (DASH).http://standards.iso.org/ittf/PubliclyAvailableStandards/c065274_ISO_IEC_23009-1_2014.zip (2014)
Koumaras, H., Tsolkas, D., Gardikis, G., Merino-Gómez, P., Frascolla, V., Triantafyllopoulou, D., Emmelmann, M., Koumaras, V., Garcia-Osma, M.L., Munaretto, D., Atxutegi, E., de Puga, J.S., Alay, O., Brunstrom, A., Bosneag, A.M.C.: 5GENESIS: The genesis of a flexible 5G Facility. In: IEEE international workshop on computer-aided modeling analysis and design of communication links and networks (CAMAD-2018 ) (2018)
Kozamernik, F., Steinmann, V., Sunna, P., Wyckens, E.: SAMVIQA new EBU methodology for video quality evaluations in multimedia. SMPTE Motion Imaging J. 114(4), 152–160 (2005). https://doi.org/10.5594/J11535
Liu, Y., Xu, C., Cheung, S.: Characterizing and detecting performance bugs for smartphone applications. In: Proceedings of the 36th international conference on software engineering, ICSE 2014, pp. 1013–1024. ACM (2014). https://doi.org/10.1145/2568225.2568229
Massol, V., Husted, T.: JUnit in Action. Manning Publications Co., Greenwich (2003)
Mehmood, M.A., Wundsam, A., Uhlig, S., Levin, D., Sarrar, N., Feldmann, A.: QoE-Lab: towards evaluating quality of experience for future internet conditions. In: Korakis, T., Li, H., Tran-Gia, P., Park, H.S. (eds.) Testbeds and Research Infrastructure. Development of Networks and Communities, pp. 286–301. Springer, Berlin (2012)
Microsoft Corporation: Smooth streaming protocol. https://winprotocoldoc.blob.core.windows.net/productionwindowHrBsarchives/MS-SSTR/[MS-SSTR].pdfHrB (2009)
NeoLoad by Neotys: The load testing platform accelerating DevOps. http://www.neotys.com/. Last accessed: 10/26/2018
NGMN Alliance: Definition of the testing framework for the NGMN 5G pre-commecial networks trails v1.0. Technical report (2018).https://www.ngmn.org/fileadmin/ngmn/content/downloads/Technical/2018/180220__PreCommTrials_Framework_definition_v1_0.pdf
Panizo, L., Díaz-Zayas, A., García, B.: An extension of TRIANGLE testbed with model-based testing. In: M.d.M. Gallardo, P. Merino (eds.) Proceedings of the 25th international symposium on model checking software (SPIN2018), pp. 190–195. Springer International Publishing (2018). https://doi.org/10.1007/978-3-319-94111-0_11
Panizo, L., Salmerón, A., Gallardo, M.M., Merino, P.: Guided Test Case Generation for Mobile Apps in the TRIANGLE Project: Work in Progress. In: Proc. of the 24th International SPIN Symposium on Model Checking of Software, pp. 192–195. ACM (2017). https://doi.org/10.1145/3092282.3092298
Pantos, R., May, W.: HTTP Live Streaming. RFC 8216 (2017). https://doi.org/10.17487/RFC8216
Perfecto: The cloud-based platform for continuous testing in a DevOps environment. https://www.perfecto.io/. Last accessed: 10/26/2018
Solera, M., Toril, M., Palomo, I., Gomez, G., Poncela, J.: A testbed for evaluating video streaming services in LTE. Wirel. Pers. Commun. 98(3), 2753–2773 (2018). https://doi.org/10.1007/s11277-017-4999-0
TRIANGLE project consortium: deliverable D2.1: initial report on the testing scenarios, requirements and use cases. Public (2016)
TRIANGLE project consortium: Deliverable D2.6: Final test scenario and test specifications. Public (2018)
Yang, S., Yan, D., Rountev, A.: Testing for poor responsiveness in android applications. In: Proceedings of the 1st international workshop on the engineering of mobile-enabled systems (MOBS), pp. 1–6 (2013). https://doi.org/10.1109/MOBS.2013.6614215
Yang, W., Prasad, M.R., Xie, T.: A grey-box approach for automated GUI-model generation of mobile applications. In: V. Cortellessa, D. Varró (eds.) Proceedings of the 16th international conference on fundamental approaches to software engineering (FASE 2013), pp. 250–265. Springer,Berlin (2013). https://doi.org/10.1007/978-3-642-37057-1_19
Zein, S., Salleh, N., Grundy, J.: A systematic mapping study of mobile application testing techniques. J. Syst. Softw. 117, 334–356 (2016). https://doi.org/10.1016/j.jss.2016.03.065
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
This work is funded by the European Union Horizon 2020 research and innovation programme, grant agreement No 688712 (TRIANGLE project) and 815178 (5GENESIS project).
Rights and permissions
About this article
Cite this article
Panizo, L., Díaz, A. & García, B. Model-based testing of apps in real network scenarios. Int J Softw Tools Technol Transfer 22, 105–114 (2020). https://doi.org/10.1007/s10009-019-00518-2
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10009-019-00518-2