Misael C. Júnior
Domenico Amalfitano
Anna Rita Fasolino
Márcio Delamaro
Skip Abstract Section
Abstract
Context: The mobile app market is continually growing offering solutions to almost all aspects of people’s lives, e.g., healthcare, business, entertainment, as well as the stakeholders’ demand for apps that are more secure, portable, easy to use, among other non-functional requirements (NFRs). Therefore, manufacturers should guarantee that their mobile apps achieve high-quality levels. A good strategy is to include software testing and quality assurance activities during the whole life cycle of such solutions.
Problem: Systematically warranting NFRs is not an easy task for any software product. Software engineers must take important decisions before adopting testing techniques and automation tools to support such endeavors.
Proposal: To provide to the software engineers with a broad overview of existing dynamic techniques and automation tools for testing mobile apps regarding NFRs.
Methods: We planned and conducted a Systematic Mapping Study (SMS) following well-established guidelines for executing secondary studies in software engineering.
Results: We found 56 primary studies and characterized their contributions based on testing strategies, testing approaches, explored mobile platforms, and the proposed tools.
Conclusions: The characterization allowed us to identify and discuss important trends and opportunities that can benefit both academics and practitioners.
- [1] . 2018. Characterization and detection of tail energy bugs in smartphones. IEEE Access 6 (2018), 65098–65108.
DOI: Google Scholar
Cross Ref
- [2] . 2015. Non-functional Requirements in Systems Analysis and Design. Vol. 28. Springer, Cham.
DOI: Google ScholarCross Ref - [3] . 2019. Systematic literature review on penetration testing for mobile cloud computing applications. IEEE Access 7 (2019), 173524–173540.
DOI: Google ScholarCross Ref - [4] . 2019. Testing tools for Android context-aware applications: A systematic mapping. J. Brazil. Comput. Societ. 25, 12 (2019), 1–22.
DOI: Google ScholarCross Ref - [5] . 2020. Mobile app energy consumption: A study of known energy issues in mobile applications and their classification schemes–summary plan. In Proceedings of the International Conference on Software Maintenance and Evolution (ICSME’20). IEEE, New York, NY, 854–854.
DOI: Google ScholarCross Ref - [6] . 2020. Do memories haunt you? An automated black box testing approach for detecting memory leaks in Android apps. IEEE Access 8 (2020), 12217–12231.
DOI: Google ScholarCross Ref - [7] . 2019. AndroShield: Automated Android applications vulnerability detection, a hybrid static and dynamic analysis approach. Information 10, 10 (2019), 326.
DOI: Google ScholarCross Ref - [8] . 2019. Identifying, categorizing and mitigating threats to validity in software engineering secondary studies. Inf. Softw. Technol. 106 (2019), 201–230.
DOI: Google ScholarCross Ref - [9] . 2013. Security testing of the communication among Android applications. In Proceedings of the 8th International Workshop on Automation of Software Test (AST’13). IEEE, New York, NY, 57–63.
DOI: Google ScholarCross Ref - [10] . 2017. EnergyPatch: Repairing resource leaks to improve energy-efficiency of Android apps. IEEE Trans. Softw. Eng. 44, 5 (2017), 470–490.
DOI: Google ScholarCross Ref - [11] . 2019. Automating mockup-based usability testing on the mobile device. In Proceedings of the International Conference on Green, Pervasive, and Cloud Computing (GPC’19). Springer, Cham, 128–143.
DOI: Google ScholarCross Ref - [12] . 2018. Analysing data security requirements of Android mobile banking application. In Proceedings of the International Conference on Intelligent, Secure, and Dependable Systems in Distributed and Cloud Environments (ISDDC’18). Springer, Cham, 30–37.
DOI: Google ScholarCross Ref - [13] . 2018. Mobile application usability testing in quasi-real conditions—The synergy of using different methods. In Proceedings of the 11th International Conference on Human System Interaction (HSI’18). IEEE, New York, NY, 362–368.
DOI: Google ScholarCross Ref - [14] . 2014. Guide to the Software Engineering Body of Knowledge (SWEBOK(R)): Version 3.0 (3rd ed.). IEEE Computer Society Press, Washington, DC.Google ScholarDigital Library
- [15] . 2019. A large-scale study of application incompatibilities in Android. In Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis. ACM, New York, NY, 216–227.
DOI: Google ScholarCross Ref - [16] . 1994. The goal question metric approach. Encyc. Softw. Eng.ineering 2 (1994), 528–532.Google Scholar
- [17] . 2020. Smartphone Market Share. Retrieved from https://www.idc.com/promo/smartphone-market-share/os.Google Scholar
- [18] . 2016. The mobile app usability inspection (MAUi) framework as a guide for minimal viable product (MVP) testing in lean development cycle. In Proceedings of the 2nd International Conference in HCI and UX. ACM, New York, NY, 1–11.
DOI: Google ScholarCross Ref - [19] . 2016. GreenOracle: Estimating software energy consumption with energy measurement corpora. In Proceedings of the 13th Working Conference on Mining Software Repositories (MSR’16). ACM, New York, NY, 49–60.
DOI: Google ScholarCross Ref - [20] . 2017. ErgoMobile: A software to support usability evaluations in mobile devices using observation techniques. In Proceedings of the International Conference of Design, User Experience, and Usability (DUXU’17). Springer, Cham, 363–378.
DOI: .Google ScholarCross Ref - [21] . 2017. ZIPT: Zero-integration performance testing of mobile app designs. In Proceedings of the 30th Annual Symposium on User Interface Software and Technology (UIST’17). ACM, New York, NY, 727–736.
DOI: Google ScholarCross Ref - [22] . 2016. Introduction to Software Testing (2nd ed.). Elsevier Brasil.Google Scholar
- [23] . 2007. Applying systematic reviews to diverse study types: An experience report. In Proceedings of the 1st International Symposium on Empirical Software Engineering and Measurement (ESEM’07). IEEE, New York, NY, 225–234.
DOI: Google ScholarCross Ref - [24] . 2017. Performance analysis of Spotify® for Android with model-based testing. Mob. Inf. Syst. 2017 (2017), 1–14.
DOI: Google ScholarCross Ref - [25] . 2017. Extending mobile app analytics for usability test logging. In Proceedings of the International Conference on Human-Computer Interaction (HCI’17). Springer, Cham, 114–131.
DOI: Google ScholarCross Ref - [26] . 2015. Automated usability tests for mobile devices through live emotions logging. In Proceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct. ACM, New York, NY, 636–643.
DOI: Google ScholarCross Ref - [27] . 2013. ISO/IEC/IEEE International Standard—Software and systems engineering – Software testing - -Part 2:Test processes. https://ieeexplore.ieee.org/document/6588543.Google Scholar
- [28] . 2014. An automated testing approach for inter-application security in Android. In Proceedings of the 9th International Workshop on Automation of Software Test (AST’14). ACM, New York, NY, 8–14.
DOI: Google ScholarCross Ref - [29] . 2015. Dynamic detection of inter-application communication vulnerabilities in Android. In Proceedings of the International Symposium on Software Testing and Analysis (ISSTA’15). ACM, New York, NY, 118–128.
DOI: Google ScholarCross Ref - [30] . 2021. Matt Lacey on mobile app usability. IEEE Softw. 38, 2 (2021), 134–136.
DOI: Google ScholarCross Ref - [31] . 2001. ISO/IEC 9126-1, Software Engineering – Product Quality. ISO, Geneva, Switzerland. https://www.iso.org/standard/22749.html.Google Scholar
- [32] . 2011. ISO/IEC 25010:2011, Systems and software engineering – Systems and software Quality Requirements and Evaluation (SQuaRE) – System and software quality models. ISO, Geneva, Switzerland. https://www.semanticscholar.org/paper/ISO-%2F-IEC-25010-%3A-2011-Systems-and-software-%E2%80%94-and-%28/ad7102b6bd0842fe1b46ce6a9246b1.f00f51948f.Google Scholar
- [33] . 2013. ISO/IEC/IEEE 29119-1: Software and systems engineering-Software testing-Part 1: Concepts and definitions. https://ieeexplore.ieee.org/document/6588537.Google Scholar
- [34] . 2019. Search-based energy testing of Android. In Proceedings of the 41st International Conference on Software Engineering (ICSE’19). IEEE, New York, NY, 1119–1130.
DOI: Google ScholarCross Ref - [35] . 2017. \( \mu \)Droid: An energy-aware mutation testing framework for Android. In Proceedings of the 11th Joint Meeting on Foundations of Software Engineering (ESEC/FSE’17). ACM, New York, NY, 208–219.
DOI: Google ScholarCross Ref - [36] . 2020. Automated construction of energy test oracles for Android. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ACM, New York, NY, 927–938.
DOI: Google ScholarCross Ref - [37] . 2012. Obstacles and opportunities in deploying model-based GUI testing of mobile software: A survey. Softw. Test., Verif. Reliab. 22, 5 (
Aug. 2012), 313–341.DOI: Google ScholarCross Ref - [38] . 2020. Detecting usability problems in mobile applications on the basis of dissimilarity in user behavior. Int. J. Hum.-Comput. Stud. 139 (2020), 102364.
DOI: Google ScholarCross Ref - [39] . 2021. Dataset on Dynamic Testing Techniques of Non-Functional Requirements on Mobile Applications. Retrieved from https://data.mendeley.com/datasets/gswvb2s2ht/3.Google Scholar
- [40] . 2018. Systematic literature review of mobile application development and testing effort estimation. J. King Saud Univ. Comput. Inf. Sci. 1, 1 (2018), 1–22.
DOI: Google ScholarCross Ref - [41] . 2016. Graph-aided directed testing of Android applications for checking runtime privacy behaviours. In Proceedings of the 11th International Workshop on Automation of Software Test (AST’16). ACM, New York, NY, 57–63.
DOI: Google ScholarCross Ref - [42] . 2019. Mimic: UI compatibility testing system for Android apps. In Proceedings of the 41st International Conference on Software Engineering (ICSE’19). IEEE, New York, NY, 246–256.
DOI: Google ScholarCross Ref - [43] . 2009. Performance testing of mobile applications at the unit test level. In Proceedings of the 3rd International Conference on Secure Software Integration and Reliability Improvement (SSIRI’09). Springer, Cham, 171–180.
DOI: Google ScholarCross Ref - [44] . 2010. The value of mapping studies: A participant-observer case study. In Proceedings of the 14th International Conference on Evaluation and Assessment in Software Engineering (EASE’10). 25–33.
DOI: Google ScholarCross Ref - [45] . 2007. Guidelines for performing Systematic Literature Reviews in Software Engineering. https://www.bibsonomy.org/bibtex/23f4b30c0fe1435b642467af4cca120ef/jpmor.Google Scholar
- [46] . 2014. Automated usability testing for mobile applications. In Proceedings of the Web Information Systems and Technologies (WEBIST). 149–156.
DOI: Google ScholarCross Ref - [47] . 2015. Security testing for Android mHealth apps. In Proceedings of the 8th International Conference on Software Testing, Verification and Validation Workshops (ICSTW’15). IEEE, New York, NY, 1–8.
DOI: Google ScholarCross Ref - [48] . 2019. Automated testing of Android apps: A systematic literature review. IEEE Trans. Reliab. 68, 1 (2019), 45–66.Google ScholarCross Ref
- [49] . 2012. Smartphone applications usability evaluation: A hybrid model and its implementation. In Proceedings of the International Conference on Human-Centred Software Engineering (HCSE’12). Springer, Berlin, 146–163.
DOI: Google ScholarCross Ref - [50] . 2019. A cloud-based solution for testing applications compatibility and portability on fragmented Android platform. In Proceedings of the 26th Asia-Pacific Software Engineering Conference (APSEC’19). IEEE, New York, NY, 158–164.
DOI: Google ScholarCross Ref - [51] . 2016. QDroid: Mobile application quality analyzer for app market curators. Mob. Inf. Syst. 2016 (2016), 1–11.
DOI: Google ScholarCross Ref - [52] . 2012. Automated and unsupervised user interaction logging as basis for usability evaluation of mobile applications. In Proceedings of the 10th International Conference on Advances in Mobile Computing & Multimedia (MoMM’12). ACM, New York, NY, 118–127.
DOI: Google ScholarCross Ref - [53] . 2020. Detecting and diagnosing energy issues for mobile applications. In Proceedings of the 29th International Symposium on Software Testing and Analysis. ACM, New York, NY, 115–127.
DOI: Google ScholarCross Ref - [54] . 2018. AppLance: A lightweight approach to detect privacy leak for packed applications. In Proceedings of the Nordic Conference on Secure IT Systems. Springer, Cham, 54–70.
DOI: Google ScholarCross Ref - [55] . 2019. Automated testing of energy hotspots and defects for Android applications. In Proceedings of the International Conference on Energy Internet (ICEI’19). IEEE, New York, NY, 374–379.
DOI: Google ScholarCross Ref - [56] . 2019. A compatibility testing platform for Android multimedia applications. Multimedia Tools Applic. 78, 4 (2019), 4885–4904.
DOI: Google ScholarCross Ref - [57] . 2018. NavyDroid: An efficient tool of energy inefficiency problem diagnosis for Android applications. Sci. China Inf. Sci. 61, 5 (2018), 1–20.
DOI: Google ScholarCross Ref - [58] . 2018. An automatically vetting mechanism for SSL error-handling vulnerability in Android hybrid Web apps. World Wide Web 21 (2018), 127–150.
DOI: Google ScholarCross Ref - [59] . 2020. A survey of context simulation for testing mobile context-aware applications. ACM Comput. Surv. 53, 1 (
Feb. 2020).DOI: Google ScholarCross Ref - [60] . 2013. Design and implementation of a toolkit for usability testing of mobile apps. Mob. Netw. Applic. 18 (2013), 81–97.
DOI: Google ScholarCross Ref - [61] . 2018. Semi-automated usability analysis through eye tracking. South Afric. Comput. J. 30 (2018), 66–84.
DOI: Google ScholarCross Ref - [62] . 2015. Automated testing of mobile applications: A systematic map and review. In Proceedings of the 28th Ibero-American Conference on Software Engineering (CIBSE’15). 1–14.Google Scholar
- [63] . 2012. Software testing of mobile applications: Challenges and future research directions. In Proceedings of the 7th International Workshop on Automation of Software Test (AST’12). IEEE, New York, NY, 29–35.Google ScholarCross Ref
- [64] . 2011. The Art of Software Testing (3rd ed.). Wiley Publishing, New York, NY.Google Scholar
- [65] . 2014. Categorizing configuration parameters of smartphones for energy performance testing. In Proceedings of the 9th International Workshop on Automation of Software Test (AST’14). ACM, New York, NY, 15–21.
DOI: Google ScholarCross Ref - [66] . 2020. Smartphone users worldwide 2016–2021. Retrieved from https://www.statista.com/statistics/330695/number-of-smartphone-users-worldwide/.Google Scholar
- [67] . 2015. Guidelines for conducting systematic mapping studies in software engineering: An update. Inf. Softw. Technol. 64 (2015), 1–18.
DOI: Google ScholarCross Ref - [68] . 2016. Software Engineering: A Practitioner’s Approach (9th ed.). McGraw-Hill, Inc., USA.Google Scholar
- [69] . 2017. Reliability testing using hybrid exploratory basis of tour and fuzzy Inference System Tsukamoto. In Proceedings of the International Conference on Sustainable Information Engineering and Technology (SIET’17). IEEE, New York, NY, 176–183.
DOI: Google ScholarCross Ref - [70] . 2013. AppsPlayground: Automatic security analysis of smartphone applications. In Proceedings of the 3rd Conference on Data and Application Security and Privacy (CODASPY’13). ACM, New York, NY, 209–220.
DOI: Google ScholarCross Ref - [71] . 2015. Mobile application verification: A systematic mapping study. In Proceedings of the International Computational Science and Its Applications (ICCSA’15). Springer, Cham, 147–163.
DOI: Google ScholarCross Ref - [72] . 2015. APSET, an Android aPplication SEcurity testing tool for detecting intent-based vulnerabilities. Int. J. Softw. Tools Technol. Transf. 17, 2 (2015), 201–221.
DOI: Google ScholarCross Ref - [73] . 2020. Attack and system modeling applied to IoT, cloud, and mobile ecosystems: Embedding security by design. ACM Comput. Surv. 53, 2 (2020), 25:1–25:32.
DOI: Google ScholarCross Ref - [74] . 2014. Testing of memory leak in Android applications. In Proceedings of the 15th International Symposium on High-Assurance Systems Engineering. IEEE, New York, NY, 176–183.
DOI: Google ScholarCross Ref - [75] . 2019. MoSSOT: An automated blackbox tester for single sign-on vulnerabilities in mobile applications. In Proceedings of the Asia Conference on Computer and Communications Security (ASIACCS’19. ACM, New York, NY, 269–282.
DOI: Google ScholarCross Ref - [76] . 2020. Model driven engineering for performance testing in mobile applications. In Proceedings of the 5th South-East Europe Design Automation, Computer Engineering, Computer Networks and Social Media Conference (SEEDA-CECNSM’20). IEEE, New York, NY, 1–7.
DOI: Google ScholarCross Ref - [77] . 2011. Software Engineering (9th ed.). Addison-Wesley Publishing Company, USA.Google Scholar
- [78] . 2015. Testing-as-a-Service for Mobile Applications: State-of-the-Art Survey. Vol. 307. Springer, Cham.
DOI: Google ScholarCross Ref - [79] . 2018. Estimating quality ratings from touch interactions in mobile games. In Proceedings of the 10th International Conference on Quality of Multimedia Experience (QoMEX’18). IEEE, New York, NY, 1–6.
DOI: Google ScholarCross Ref - [80] . 2019. Automated functional testing of mobile applications: A systematic mapping study. Softw. Qual. J. 27, 1 (2019), 149–201.
DOI: Google ScholarCross Ref - [81] . 2020. An automated model-based approach for unit-level performance test generation of mobile applications. J. Softw.: Evol. Process 32 (2020), e2215. DOI: .Google ScholarCross Ref
- [82] . 2018. Vulnerability detection on mobile applications using state machine inference. In Proceedings of the European Symposium on Security and Privacy Workshops (EuroS&PW’18). IEEE, New York, NY, 1–10.
DOI: Google ScholarCross Ref - [83] . 2015. Detecting display energy hotspots in Android apps. In Proceedings of the 8th International Conference on Software Testing, Verification and Validation (ICST’15). IEEE, New York, NY, 1–10.
DOI: Google ScholarCross Ref - [84] . 2020. Identifying vulnerabilities of SSL/TLS certificate verification in Android apps with static and dynamic analysis. J. Syst. Softw. 167 (2020), 110609.
DOI: Google ScholarCross Ref - [85] . 2014. IntentFuzzer: Detecting capability leaks of Android applications. In Proceedings of the 9th Symposium on Information, Computer and Communications Security. ACM, New York, NY, 531–536.
DOI: Google ScholarCross Ref - [86] . 2019. A systematic mapping study on cloud-based mobile application testing. J. Inf. Commun. Technol. 18, 4 (2019), 485–527.
DOI: Google ScholarCross Ref - [87] . 2021. Layout and image recognition driving cross-platform automated mobile testing. In Proceedings of the IEEE/ACM 43rd International Conference on Software Engineering (ICSE’21). IEEE, New York, NY, 1561–1571.
DOI: Google ScholarCross Ref - [88] . 2016. Automated support to capture and validate security requirements for mobile apps. In Proceedings of the Asia Pacific Requirements Engineering Conference (APSEC’16). Springer, Singapore, 97–112.
DOI: Google ScholarCross Ref - [89] . 2016. A systematic mapping study of mobile application testing techniques. J. Syst. Softw. 117, C (
July 2016), 334–356.DOI: Google ScholarCross Ref - [90] . 2016. Automated test generation for detection of leaks in Android applications. In Proceedings of the 11th International Workshop on Automation of Software Test. ACM, New York, NY, 64–70.
DOI: Google ScholarCross Ref - [91] . 2015. Compatibility testing service for mobile applications. In Proceedings of the Symposium on Service-Oriented System Engineering (SOSE’15). IEEE, New York, NY, 179–186.
DOI: Google ScholarCross Ref - [92] . 2019. Evaluation of machine learning approaches for Android energy bugs detection with revision commits. IEEE Access 7 (2019), 85241–85252.
DOI: Google ScholarCross Ref
Index Terms
- Dynamic Testing Techniques of Non-functional Requirements in Mobile Apps: A Systematic Mapping Study
Recommendations
A systematic mapping study of mobile application testing techniques
We performed a systematic mapping study on mobile application testing techniques.79 empirical studies are mapped to a classification schema.Classification scheme includes, structure the topic, evaluation type and contribution facets.Several research ...
Testing of Mobile Applications in the Wild: A Large-Scale Empirical Study on Android Apps
ICPC '20: Proceedings of the 28th International Conference on Program ComprehensionNowadays, mobile applications (a.k.a., apps) are used by over two billion users for every type of need, including social and emergency connectivity. Their pervasiveness in today's world has inspired the software testing research community in devising ...
Toward improved traceability of non-functional requirements
TEFSE '05: Proceedings of the 3rd international workshop on Traceability in emerging forms of software engineeringThis position paper examines current practices and challenges for tracing non-functional requirements (NFRs). Anecdotal evidence suggests that many organizations do not effectively trace NFRs and that functional changes are often implemented with very ...
Comments