skip to main content
10.1145/2639108.2639131acmconferencesArticle/Chapter ViewAbstractPublication PagesmobicomConference Proceedingsconference-collections
research-article

Caiipa: automated large-scale mobile app testing through contextual fuzzing

Published: 07 September 2014 Publication History

Abstract

Scalable and comprehensive testing of mobile apps is extremely challenging. Every test input needs to be run with a variety of contexts, such as: device heterogeneity, wireless network speeds, locations, and unpredictable sensor inputs. The range of values for each context, e.g. location, can be very large. In this paper we present Caiipa, a cloud service for testing apps over an expanded mobile context space in a scalable way. It incorporates key techniques to make app testing more tractable, including a context test space prioritizer to quickly discover failure scenarios for each app. We have implemented Caiipa on a cluster of VMs and real devices that can each emulate various combinations of contexts for tablet and phone apps. We evaluate Caiipa by testing 265 commercially available mobile apps based on a comprehensive library of real-world conditions. Our results show that Caiipa leads to improvements of 11.1x and 8.4x in the number of crashes and performance bugs discovered compared to conventional UI-based automation (i.e., monkey-testing).

References

[1]
D. Amalfitano, et al. Using GUI Ripping for Automated Testing of Android Applications. In ASE 2012.
[2]
S. Anand, M. Naik, H. Yang, and M. Harrold. Automated Concolic Testing of Smartphone Apps. In FSE 2012.
[3]
T. Azim et al. Targeted and Depth-first Exploration for Systematic Testing of Android Apps. In OOPSLA 2013.
[4]
C. M. Bishop. Pattern Recognition and Machine Learning (Information Science and Statistics). Springer, August 2006.
[5]
Crashlytics. http://www.crashlytics.com.
[6]
Fortune. http://fortune.com/2009/08/22/40-staffers-2-reviews-8500-iphone-apps-per-week/.
[7]
K. Glerum et al. Debugging in the (Very) Large: Ten Years of Implementation and Experience. In SOSP 2009.
[8]
P. Godefroid, N. Klarlund, and K. Sen. Dart: Directed Automated Random Testing. In PLDI 2005.
[9]
R. Chandra et al. Towards Scalable Automated Mobile App Testing. Technical Report MSR-TR-2014-44, 2014.
[10]
C. Liang et al. Contextual Fuzzing: Automated Mobile App Testing Under Dynamic Device and Environment Conditions. Technical Report MSR-TR-2013-100, 2013.
[11]
P. Godefroid et al. Sage: Whitebox Fuzzing for Security Testing. Communications of the ACM, 55(3):40--44, 2012.
[12]
A. Gonzalez-Sanchez, et al. Prioritizing Tests for Fault Localization through Ambiguity Group Reduction. In Proceedings of the IEEE/ACM ASE 2011.
[13]
Google. UI/Application Exerciser Monkey. http://developer.android.com/tools/help/monkey.html.
[14]
F. Gross, G. Fraser, and A. Zeller. Search-based System Testing: High Coverage, No False Alarms. In ISSTA 2012.
[15]
H. Guo, et al. Practical Software Model Checking Via Dynamic Interface Reduction. In SOSP 2011.
[16]
V. Hautamaki, I. Karkkainen, and P. Franti. Outlier Detection Using k-nearest Neighbour Graph. In ICPR 2004.
[17]
J. Huang, et al. Uncovering Cellular Network Characteristics: Performance, Infrastructure, and Policies. Technical Report MSU-CSE-00-2, 2013.
[18]
Flurry. http://www.flurry.com/bid/91911/Electric-Technology-Apps-and-The-New-Global-Village.
[19]
B. Jiang, et al. Mobiletest: A Tool Supporting Automatic Black Box Test for Software on Smart Mobile Devices. In Proceedings of AST 2007.
[20]
Simulation Dashboard for Windows Phone. http://msdn.microsoft.com/library/windowsphone/develop/jj206953.
[21]
UI Automation Verify. http://msdn.microsoft.com/en-us/library/windows/desktop/hh920986.
[22]
R. Mittal, et al. Empowering Developers to Estimate App Energy Consumption. In Mobicom 2012.
[23]
R. Natella et al. On Fault Representativeness of Software Fault Injection. IEEE Trans. on Software Eng., 39(1), 2013.
[24]
A. J. Oliner, et al. Carat: Collaborative Energy Diagnosis for Mobile Devices. In SenSys 2013.
[25]
Open Signal. http://opensignal.com.
[26]
Open Signal. Signal Reports. http://opensignal.com/reports/fragmentation.php.
[27]
V. Rastogi et al. Appsplayground: Automatic Security Analysis of Smartphone Applications. In CODASPY 2013.
[28]
L. Ravindranath, et al. AppInsight: Mobile App Performance Monitoring in the Wild. In OSDI 2012.
[29]
L. Ravindranath, et. al. Automatic and Scalable Fault Detection for Mobile Applications. In MobiSys 2014.
[30]
Techcrunch. http://techcrunch.com/users-have-low-tolerance-for-buggy-apps-only-16-will-try-a-failing-app-more-than-twice.
[31]
Crittercism. http://pages.crittercism.com/rs/crittercism/images/crittercism-mobile-benchmarks.pdf.
[32]
Telerik. JustDecompile. http://www.telerik.com.
[33]
TestFlight. http://testflightapp.com/.
[34]
Bit9. https://www.bit9.com/download/reports/Pausing-Google-Play-October2012.pdf.
[35]
A. I. Wasserman. Software Engineering Issues for Mobile Application Development. In FSE - FoSER workshop, 2010.
[36]
L. Yan and H. Yin. DroidScope: Seamlessly Reconstructing the OS and Dalvik Semantic Views for Dynamic Android Malware Analysis. In USENIX Security Symposium, 2012.
[37]
T. Boksasp, et al. Android Apps and Permissions: Security and Privacy Risks. NTNU Trondheim TR, June 2012.

Cited By

View all
  • (2024)AutoDroid: LLM-powered Task Automation in AndroidProceedings of the 30th Annual International Conference on Mobile Computing and Networking10.1145/3636534.3649379(543-557)Online publication date: 29-May-2024
  • (2023)CONAN: Statically Detecting Connectivity Issues in Android ApplicationsProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3613097(2182-2186)Online publication date: 30-Nov-2023
  • (2023)Transparent Runtime Change Handling for Android AppsProceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 310.1145/3582016.3582060(756-770)Online publication date: 25-Mar-2023
  • Show More Cited By

Index Terms

  1. Caiipa: automated large-scale mobile app testing through contextual fuzzing

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    MobiCom '14: Proceedings of the 20th annual international conference on Mobile computing and networking
    September 2014
    650 pages
    ISBN:9781450327831
    DOI:10.1145/2639108
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 07 September 2014

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. contextual fuzzing
    2. mobile app testing

    Qualifiers

    • Research-article

    Conference

    MobiCom'14
    Sponsor:

    Acceptance Rates

    MobiCom '14 Paper Acceptance Rate 36 of 220 submissions, 16%;
    Overall Acceptance Rate 440 of 2,972 submissions, 15%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)34
    • Downloads (Last 6 weeks)3
    Reflects downloads up to 14 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)AutoDroid: LLM-powered Task Automation in AndroidProceedings of the 30th Annual International Conference on Mobile Computing and Networking10.1145/3636534.3649379(543-557)Online publication date: 29-May-2024
    • (2023)CONAN: Statically Detecting Connectivity Issues in Android ApplicationsProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3613097(2182-2186)Online publication date: 30-Nov-2023
    • (2023)Transparent Runtime Change Handling for Android AppsProceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 310.1145/3582016.3582060(756-770)Online publication date: 25-Mar-2023
    • (2023)Preference-wise Testing of Android Apps via Test AmplificationACM Transactions on Software Engineering and Methodology10.1145/351180432:1(1-37)Online publication date: 13-Feb-2023
    • (2023)If You’re Scanning This, It’s Too Late! A QR Code-Based Fuzzing Methodology to Identify Input Vulnerabilities in Mobile AppsApplied Cryptography and Network Security Workshops10.1007/978-3-031-41181-6_30(553-570)Online publication date: 4-Oct-2023
    • (2022)PermDroid: automatically testing permission-related behaviour of Android applicationsProceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3533767.3534221(593-604)Online publication date: 18-Jul-2022
    • (2022)Fuzzing: A Survey for RoadmapACM Computing Surveys10.1145/3512345Online publication date: 28-Jan-2022
    • (2022)Enabling Mutant Generation for Open- and Closed-Source Android AppsIEEE Transactions on Software Engineering10.1109/TSE.2020.298263848:1(186-208)Online publication date: 1-Jan-2022
    • (2022)Detecting Connectivity Issues in Android Apps2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)10.1109/SANER53432.2022.00087(697-708)Online publication date: Mar-2022
    • (2022)AppSPIN: reconfiguration-based responsiveness testing and diagnosing for Android AppsAutomated Software Engineering10.1007/s10515-022-00347-929:2Online publication date: 1-Nov-2022
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    EPUB

    View this article in ePub.

    ePub

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media