skip to main content
10.1145/3460319.3464806acmconferencesArticle/Chapter ViewAbstractPublication PagesisstaConference Proceedingsconference-collections
research-article

Understanding and finding system setting-related defects in Android apps

Published: 11 July 2021 Publication History

Abstract

Android, the most popular mobile system, offers a number of user-configurable system settings (e.g., network, location, and permission) for controlling devices and apps. Even popular, well-tested apps may fail to properly adapt their behaviors to diverse setting changes, thus frustrating their users. However, there exists no effort to systematically investigate such defects. To this end, we conduct the first empirical study to understand the characteristics of these setting-related defects (in short as "setting defects"), which reside in apps and are triggered by system setting changes. We devote substantial manual effort (over three person-months) to analyze 1,074 setting defects from 180 popular apps on GitHub. We investigate their impact, root causes, and consequences. We find that setting defects have a wide, diverse impact on apps' correctness, and the majority of these defects (≈70.7%) cause non-crash (logic) failures, and thus could not be automatically detected by existing app testing techniques due to the lack of strong test oracles. Motivated and guided by our study, we propose setting-wise metamorphic fuzzing, the first automated testing approach to effectively detect setting defects without explicit oracles. Our key insight is that an app's behavior should, in most cases, remain consistent if a given setting is changed and later properly restored, or exhibit expected differences if not restored. We realize our approach in SetDroid, an automated, end-to-end GUI testing tool, for detecting both crash and non-crash setting defects. SetDroid has been evaluated on 26 popular, open-source apps and detected 42 unique, previously unknown setting defects in 24 apps. To date, 33 have been confirmed and 21 fixed. We also apply SetDroid on five highly popular industrial apps, namely WeChat, QQMail, TikTok, CapCut, and AlipayHK, all of which each have billions of monthly active users. SetDroid successfully detects 17 previously unknown setting defects in these apps' latest releases, and all defects have been confirmed and fixed by the app vendors. The majority of SetDroid-detected defects (49 out of 59) cause non-crash failures, which could not be detected by existing testing tools (as our evaluation confirms). These results demonstrate SetDroid's strong effectiveness and practicality.

References

[1]
Christoffer Quist Adamsen, Gianluca Mezzetti, and Anders Møller. 2015. Systematic execution of android test suites in adverse conditions. In Proceedings of the 2015 International Symposium on Software Testing and Analysis (ISSTA). 83–93. https://doi.org/10.1145/2771783.2771786
[2]
Abdulaziz Alshayban, Iftekhar Ahmed, and Sam Malek. 2020. Accessibility issues in Android apps: state of affairs, sentiments, and ways forward. In 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE). 1323–1334. https://doi.org/10.1145/3377811.3380392
[3]
Domenico Amalfitano, Vincenzo Riccio, Ana CR Paiva, and Anna Rita Fasolino. 2018. Why does the orientation change mess up my Android application? From GUI failures to code faults. In Software Testing, Verification and Reliability (STVR). e1654. https://doi.org/10.1002/stvr.1654
[4]
Tanzirul Azim and Iulian Neamtiu. 2013. Targeted and depth-first exploration for systematic testing of android apps. In Proceedings of the 2013 ACM SIGPLAN International Conference on Object Oriented Programming Systems Languages & Applications (OOPSLA). 641–660. https://doi.org/10.1145/2509136.2509549
[5]
Tsong Y. Chen, Shing C. Cheung, and Shiu Ming Yiu. 1998. Metamorphic testing: a new approach for generating next test cases. HKUST-CS98-01, Hong Kong University of Science and Technology. arxiv:2002.12543
[6]
Shauvik Roy Choudhary, Alessandra Gorla, and Alessandro Orso. 2015. Automated test input generation for Android: are we there yet? (E). In 30th IEEE/ACM International Conference on Automated Software Engineering (ASE). 429–440. https://doi.org/10.1109/ASE.2015.89
[7]
Zhen Dong, Marcel Böhme, Lucia Cojocaru, and Abhik Roychoudhury. 2020. Time-travel testing of Android apps. In Proceedings of the 42nd International Conference on Software Engineering (ICSE). 1–12. https://doi.org/10.1145/3377811.3380402
[8]
Lingling Fan, Ting Su, Sen Chen, Guozhu Meng, Yang Liu, Lihua Xu, and Geguang Pu. 2018. Efficiently manifesting asynchronous programming errors in Android apps. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering (ASE). 486–497. https://doi.org/10.1145/3238147.3238170
[9]
Lingling Fan, Ting Su, Sen Chen, Guozhu Meng, Yang Liu, Lihua Xu, Geguang Pu, and Zhendong Su. 2018. Large-scale analysis of framework-specific exceptions in Android apps. In 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE). 408–419. https://doi.org/10.1145/3180155.3180222
[10]
Mattia Fazzini and Alessandro Orso. 2017. Automated cross-platform inconsistency detection for mobile apps. In 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE). 308–318. https://doi.org/10.1109/ASE.2017.8115644
[11]
fpernice518. 2018. NextCloud issue #2979. Retrieved 2021-1 from https://github.com/nextcloud/android/issues/2979
[12]
Tianxiao Gu, Chengnian Sun, Xiaoxing Ma, Chun Cao, Chang Xu, Yuan Yao, Qirun Zhang, Jian Lu, and Zhendong Su. 2019. Practical GUI testing of Android applications via model abstraction and refinement. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). 269–280. https://doi.org/10.1109/ICSE.2019.00042
[13]
Wunan Guo, Liwei Shen, Ting Su, Xin Peng, and Weiyang Xie. 2020. Improving Automated GUI Exploration of Android Apps via Static Dependency Analysis. In 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME). 557–568. https://doi.org/10.1109/ICSME46990.2020.00059
[14]
Axel Halin, Alexandre Nuttinck, Mathieu Acher, Xavier Devroey, Gilles Perrouin, and Benoit Baudry. 2019. Test them all, is it worth it? Assessing configuration sampling on the JHipster Web development stack. In Empirical Software Engineering (EMSE). 674–717. https://doi.org/10.1145/3382025.3414985
[15]
Jiajun Hu, Lili Wei, Yepang Liu, Shing-Chi Cheung, and Huaxun Huang. 2018. A tale of two cities: how WebView induces bugs to Android applications. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering (ASE). 702–713. https://doi.org/10.1145/3238147.3238180
[16]
Ajay Kumar Jha, Sunghee Lee, and Woo Jin Lee. 2017. Developer mistakes in writing Android manifests: an empirical study of configuration error. In IEEE/ACM 14th International Conference on Mining Software Repositories (MSR). 25–36. https://doi.org/10.1109/MSR.2017.41
[17]
Pingfan Kong, Li Li, Jun Gao, Kui Liu, Tegawendé F Bissyandé, and Jacques Klein. 2018. Automated testing of android apps: a systematic literature review. In IEEE Transactions on Reliability. 45–66. https://doi.org/10.1109/TR.2018.2865733
[18]
Emily Kowalczyk, Myra B. Cohen, and Atif M. Memon. 2018. Configurations in Android testing: they matter. Proceedings of the 1st International Workshop on Advances in Mobile App Analysis (A-Mobile), 1–6. https://doi.org/10.1145/3243218.3243219
[19]
langid Team. 2021. langid. Retrieved 2021-1 from https://github.com/saffsd/langid.py
[20]
Yuanchun Li, Ziyue Yang, Yao Guo, and Xiangqun Chen. 2017. DroidBot: a lightweight UI-guided test input generator for Android. In 2017 IEEE/ACM 39th International Conference on Software Engineering Companion (ICSE-C). 23–26. https://doi.org/10.1109/ICSE-C.2017.8
[21]
Max Lillack, Christian Kästner, and Eric Bodden. 2014. Tracking load-time configuration options. In Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering (ASE). 445–456. https://doi.org/10.1145/2642937.2643001
[22]
Yifei Lu, Minxue Pan, Juan Zhai, Tian Zhang, and Xuandong Li. 2019. Preference-wise testing for Android applications. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (FSE). 268–278. https://doi.org/10.1145/3338906.3338980
[23]
Riyadh Mahmood, Nariman Mirzaei, and Sam Malek. 2014. Evodroid: segmented evolutionary testing of android apps. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE). 599–609. https://doi.org/10.1145/2635868.2635896
[24]
malinajirka. 2019. WordPress issue #10096. Retrieved 2021-1 from https://github.com/wordpress-mobile/WordPress-Android/issues/10096
[25]
Ke Mao, Mark Harman, and Yue Jia. 2016. Sapienz: multi-objective automated testing for Android applications. In Proceedings of the 25th International Symposium on Software Testing and Analysis (ISSTA). 94–105. https://doi.org/10.1145/2931037.2931054
[26]
Flávio Medeiros, Christian Kästner, Márcio Ribeiro, Rohit Gheyi, and Sven Apel. 2016. A comparison of 10 sampling algorithms for configurable systems. In Proceedings of the 38th International Conference on Software Engineering (ICSE). 643–654. https://doi.org/10.1145/2884781.2884793
[27]
Nariman Mirzaei, Hamid Bagheri, Riyadh Mahmood, and Sam Malek. 2015. Sig-droid: automated system input generation for android applications. In 2015 IEEE 26th International Symposium on Software Reliability Engineering (ISSRE). 461–471. https://doi.org/10.1145/2509136.2509549
[28]
mzorz. 2017. WordPress issue #6026. Retrieved 2021-1 from https://github.com/wordpress-mobile/WordPress-Android/issues/6026
[29]
Minxue Pan, An Huang, Guoxin Wang, Tian Zhang, and Xuandong Li. 2020. Reinforcement learning based curiosity-driven testing of Android applications (ISSTA). In Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis. 153–164. https://doi.org/10.1145/3395363.3397354
[30]
pcqpcq. 2021. opensource-android-apps. Retrieved 2021-1 from https://github.com/pcqpcq/open-source-android-apps/
[31]
Oliviero Riganelli, Simone Paolo Mottadelli, Claudio Rota, Daniela Micucci, and Leonardo Mariani. 2020. Data loss detector: automatically revealing data loss bugs in Android apps. In Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA). 141–152. https://doi.org/10.1145/3395363.3397379
[32]
Alireza Sadeghi, Reyhaneh Jabbarvand, and Sam Malek. 2017. PATDroid: permission-aware gui testing of android. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering (FSE). 220–232. https://doi.org/10.1145/3106237.3106250
[33]
setting-defect fuzzing. 2021. Dataset. Retrieved 2021-1 from https://github.com/setting-defect-fuzzing/home
[34]
Sabrina Souto, Marcelo d’Amorim, and Rohit Gheyi. 2017. Balancing soundness and efficiency for practical testing of configurable systems. In Proceedings of the 39th International Conference on Software Engineering (ICSE). 632–642. https://doi.org/10.1109/ICSE.2017.64
[35]
Ting Su. 2016. FSMdroid: guided GUI testing of android apps. In Proceedings of the 38th International Conference on Software Engineering (ICSE). 689–691. https://doi.org/10.1145/2889160.2891043
[36]
Ting Su, Lingling Fan, Sen Chen, Yang Liu, Lihua Xu, Geguang Pu, and Zhendong Su. 2020. Why my app crashes? Understanding and benchmarking framework-specific exceptions of Android apps. IEEE Transactions on Software Engineering (TSE), https://doi.org/10.1109/TSE.2020.3013438
[37]
Ting Su, Guozhu Meng, Yuting Chen, Ke Wu, Weiming Yang, Yao Yao, Geguang Pu, Yang Liu, and Zhendong Su. 2017. Guided, stochastic model-based GUI testing of Android apps. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering (FSE). 245–256. https://doi.org/10.1145/3106237.3106298
[38]
AlipayHK Team. 2021. AlipayHK. Retrieved 2021-1 from https://www.alipayhk.com
[39]
Android Team. 2021. Android Developers Documentation. Retrieved 2021-1 from https://developer.android.com
[40]
Android Team. 2021. Android Help. Retrieved 2021-1 from https://support.google.com/android
[41]
AnkiDroid Team. 2021. AnkiDroid. Retrieved 2021-1 from https://github.com/ankidroid/Anki-Android
[42]
Android Team. 2021. Request App Permissions. Retrieved 2021-1 from https://developer.android.com/training/permissions/requesting#perm-check
[43]
CapCut Team. 2021. CapCut. Retrieved 2021-1 from https://lv.faceueditor.com
[44]
Clover Team. 2021. Clover. Retrieved 2021-1 from https://github.com/chandevel/Clover
[45]
Commons Team. 2021. Commons. Retrieved 2021-1 from https://github.com/commons-app/apps-android-commons
[46]
Frost Team. 2021. Frost. Retrieved 2021-1 from https://github.com/AllanWang/Frost-for-Facebook
[47]
GitHub Team. 2021. GitHub REST API. Retrieved 2021-1 from https://docs.github.com/en/rest/
[48]
K-9 Team. 2021. K-9. Retrieved 2021-1 from https://github.com/k9mail/k-9
[49]
Monkey Team. 2021. Android Monkey. Retrieved 2021-1 from https://developer.android.com/studio/test/monkey
[50]
NextCloud Team. 2021. NextCloud. Retrieved 2021-1 from https://github.com/nextcloud/android
[51]
OpenFoodFacts Team. 2021. OpenFoodFacts. Retrieved 2021-1 from https://github.com/openfoodfacts/openfoodfacts-androidapp
[52]
Openlauncher Team. 2021. Openlauncher. Retrieved 2021-1 from https://github.com/OpenLauncherTeam/openlauncher
[53]
QQMail Team. 2021. QQMail. Retrieved 2021-1 from https://en.mail.qq.com
[54]
Signal Team. 2021. Signal. Retrieved 2021-1 from https://github.com/signalapp/Signal-Android
[55]
Status Team. 2021. Status. Retrieved 2021-1 from https://github.com/status-im/status-react
[56]
Syncthing Team. 2021. Syncthing. Retrieved 2021-1 from https://github.com/syncthing/syncthing-android
[57]
TikTok Team. 2021. TikTok. Retrieved 2021-1 from https://www.tiktok.com
[58]
WeChat Team. 2021. WeChat. Retrieved 2021-1 from https://www.wechat.com
[59]
WordPress Team. 2021. WordPress. Retrieved 2021-1 from https://github.com/wordpress-mobile/WordPress-Android
[60]
Porfirio Tramontana, Domenico Amalfitano, Nicola Amatucci, and Anna Rita Fasolino. 2019. Automated functional testing of mobile applications: a systematic mapping study. In Software Quality Journal (SQJ). 149–201. https://doi.org/10.1007/s11219-018-9418-6
[61]
uiautomator2 Team. 2021. uiautomator2. Retrieved 2021-1 from https://github.com/openatx/uiautomator2
[62]
Mario Linares Vásquez, Kevin Moran, and Denys Poshyvanyk. 2017. Continuous, evolutionary and large-scale: a new perspective for automated mobile app testing. In International Conference on Software Maintenance and Evolution (ICSME). https://doi.org/10.1109/ICSME.2017.27
[63]
Wenyu Wang, Dengfeng Li, Wei Yang, Yurui Cao, Zhenwen Zhang, Yuetang Deng, and Tao Xie. 2018. An empirical study of Android test generation tools in industrial cases. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering (ASE). https://doi.org/10.1145/3238147.3240465
[64]
Tianyin Xu, Xinxin Jin, Peng Huang, Yuanyuan Zhou, Shan Lu, Long Jin, and Shankar Pasupathy. 2016. Early detection of configuration errors to reduce failure damage. In 12th $USENIX$ Symposium on Operating Systems Design and Implementation (OSDI). 619–634. https://dl.acm.org/doi/10.5555/3026877.3026925
[65]
Zuoning Yin, Xiao Ma, Jing Zheng, Yuanyuan Zhou, Lakshmi N Bairavasundaram, and Shankar Pasupathy. 2011. An empirical study on configuration errors in commercial and open source systems. In Proceedings of the Twenty-Third ACM Symposium on Operating Systems Principles (SOSP). 159–172. https://doi.org/10.1145/2043556.2043572
[66]
Sai Zhang and Michael D Ernst. 2014. Which configuration option should I change? In Proceedings of the 36th International Conference on Software Engineering (ICSE). 152–163. https://doi.org/10.1145/2568225.2568251

Cited By

View all
  • (2024)General and Practical Property-based Testing for Android AppsProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3694986(53-64)Online publication date: 27-Oct-2024
  • (2024)When Your Infrastructure Is a Buggy Program: Understanding Faults in Infrastructure as Code EcosystemsProceedings of the ACM on Programming Languages10.1145/36897998:OOPSLA2(2490-2520)Online publication date: 8-Oct-2024
  • (2024)Testing Updated Apps by Adapting Learned ModelsACM Transactions on Software Engineering and Methodology10.1145/366460133:6(1-40)Online publication date: 29-Jun-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ISSTA 2021: Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis
July 2021
685 pages
ISBN:9781450384599
DOI:10.1145/3460319
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 July 2021

Permissions

Request permissions for this article.

Check for updates

Badges

Author Tags

  1. Android
  2. Empirical study
  3. Setting
  4. Testing

Qualifiers

  • Research-article

Conference

ISSTA '21
Sponsor:

Acceptance Rates

Overall Acceptance Rate 58 of 213 submissions, 27%

Upcoming Conference

ISSTA '25

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)84
  • Downloads (Last 6 weeks)7
Reflects downloads up to 17 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)General and Practical Property-based Testing for Android AppsProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3694986(53-64)Online publication date: 27-Oct-2024
  • (2024)When Your Infrastructure Is a Buggy Program: Understanding Faults in Infrastructure as Code EcosystemsProceedings of the ACM on Programming Languages10.1145/36897998:OOPSLA2(2490-2520)Online publication date: 8-Oct-2024
  • (2024)Testing Updated Apps by Adapting Learned ModelsACM Transactions on Software Engineering and Methodology10.1145/366460133:6(1-40)Online publication date: 29-Jun-2024
  • (2024)Property-Based Testing for Validating User Privacy-Related Functionalities in Social Media AppsCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering10.1145/3663529.3663863(440-451)Online publication date: 10-Jul-2024
  • (2024)App review driven collaborative bug findingEmpirical Software Engineering10.1007/s10664-024-10489-x29:5Online publication date: 26-Jul-2024
  • (2023)An Empirical Study of Functional Bugs in Android AppsProceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3597926.3598138(1319-1331)Online publication date: 12-Jul-2023
  • (2023)ωTest: WebView-Oriented Testing for Android ApplicationsProceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3597926.3598112(992-1004)Online publication date: 12-Jul-2023
  • (2023)DDLDroid: Efficiently Detecting Data Loss Issues in Android AppsProceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3597926.3598089(703-714)Online publication date: 12-Jul-2023
  • (2023)Characterizing and Finding System Setting-Related Defects in Android AppsIEEE Transactions on Software Engineering10.1109/TSE.2023.323644949:4(2941-2963)Online publication date: 1-Apr-2023
  • (2023)Effectively Finding ICC-related Bugs in Android Apps via Reinforcement Learning2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE)10.1109/ISSRE59848.2023.00032(403-414)Online publication date: 9-Oct-2023
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media