Skip to main content
SpringerLink
Log in

Multi-device coverage testing of mobile applications

  • Published:
Software Quality Journal Aims and scope Submit manuscript

Abstract

This paper evaluates the effectiveness of coverage approaches for selecting mobile devices (i.e., smartphones and tablets) to test mobile software applications. Due to the large number of such devices on the market and the variations in their characteristics, it is hard to guarantee that an application will work as intended on all devices. For this reason, multi-device testing is necessary. The goal of this research was to determine how many devices must be tested and which methods for device selection are best for revealing device-specific faults. We experimentally investigated a simple coverage of all values of each device’s features separately and the each-choice coverage (i.e., the coverage of all device characteristics at the same time). To collect the experimental data, 15 Android applications were tested on 30 mobile devices and 24 device-specific faults were detected. Our research shows that a random selection of 13 devices achieved 100% effectiveness. However, coverage of device characteristics in the selection process yielded an acceptable 90% level of effectiveness with a set of only five devices. The most successful approaches were the coverage of different types of Android operating systems and the each-choice coverage. Our results include recommendations for increasing the effectiveness while decreasing the costs of mobile testing.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  • Adobe Edge Inspect (2016). https://creative.adobe.com/products/inspect. Accessed 27 December 2016

  • Barksdale M.: Mobile platform fragmentation—what it is and how we deal with it at Engage Mobile (2014). http://www.engagemobile.com/mobile-platform-fragmentation-deal-engage-mobile/. Accessed 27 December 2016

  • Cutler K.: How do top Android developers QA test their apps? (2012). http://techcrunch.com/2012/06/02/android-qa-testing-quality-assurance/. Accessed 27 December 2016

  • Duggan M.: Cell phone activities 2013 (2013). http://www.pewinternet.org/2013/09/19/cell-phone-activities-2013/. Accessed 27 December 2016

  • Gao, J., Bai, X., Tsai, W. T., & Uehara, T. (2014). Mobile application testing: a tutorial. Computer, 2, 46–55.

    Article  Google Scholar 

  • Ghostlab (2016). http://vanamco.com/ghostlab/. Accessed 27 December 2016

  • Google Play Services (2016). https://developer.android.com/google/play-services/index.html. Accessed 27 December 2016

  • Grindal, M., Offutt, J., & Andler, S. F. (2005). Combination testing strategies: a survey. Software Testing, Verification and Reliability, 15(3), 167–199.

    Article  Google Scholar 

  • Han, D., Zhang, C., Fan, X., Hindle, A., Wong, K., Stroulia, E.: Understanding Android fragmentation with topic analysis of vendor-specific bugs. Proceedings of the 19th Working Conference on Reverse Engineering (WCRE 2012), Kingston, ON, Canada, October 15–18, pp. 83–92 (2012)

  • Huang, J.: AppACTS: mobile app automated compatibility testing service. Proceedings of the 2nd IEEE Int. Conference on Mobile Cloud Computing, Services, and Engineering (MobileCloud), Oxford, UK, 8–11 April, pp. 85–90 (2014)

  • Keynote Mobile Testing. DeviceAnywhere (2016). http://www.keynote.com/solutions/testing/mobile-testing. Accessed 27 December 2016

  • Kuhn, D. R., Kacker, R. N., Lei, Y.: Introduction to combinatorial testing, Chapman and Hall/CRC, 341 pages (2013)

  • Kuhn, D. R., Lei, Y., Kacker, R. N.: Practical combinatorial testing—beyond pairwise. IEEE IT Professional, June, pp. 19–23 (2008)

  • Moore, M. F., Vilkomir, S.: Practical combinatorial testing approaches: a case study of a university portal application. Proceedings of the 28th International Conference on Software Engineering & Knowledge Engineering (SEKE 2016), Redwood City, CA, USA, July 1–3 (2016)

  • Muccini, H., Di Francesco, A., Esposito, P.: Software testing of mobile applications: challenges and future research directions. Proceedings of the 7th IEEE/ACM International Workshop on Automation of Software Test (AST 2012), Zurich, Switzerland, 2–3 June (2012)

  • OpenSignal. Android Fragmentation Visualized (2015). https://opensignal.com/reports/2015/08/android-fragmentation/. Accessed 27 December 2016

  • Park, J.H., Park, Y.B., Ham, H.K.: Fragmentation problem in Android. Proceedings of the International Conference on Information Science and Applications (ICISA 2013), 24–26 June, Pattaya, Thailand (2013)

  • pCloudy (2016). http://www.pcloudy.com. Accessed 27 December 2016

  • Perez, S.: Consumers spend 85% of time on smartphones in apps, but only 5 apps see heavy use (2015). http://techcrunch.com/2015/06/22/consumers-spend-85-of-time-on-smartphones-in-apps-but-only-5-apps-see-heavy-use/. Accessed 27 December 2016.

  • Perfecto Mobile (2016). www.perfectomobile.com. Accessed 27 December 2016

  • Silk Mobile (2016). http://www.borland.com/Products/Software-Testing/Automated-Testing/Silk-Mobile. Accessed 27 December 2016

  • Starov, O., Vilkomir S., Kharchenko, V.: Cloud testing for mobile software systems: concept and prototyping. Proceedings of the 8th International Conference on Software Engineering and Applications (ICSOFT-EA 2013), July 29–31, Reykjavik, Iceland, pp. 124–131 (2013)

  • Starov, O., Vilkomir, S.: Integrated TaaS platform for mobile development: architecture solutions. Proceedings of the Eighth International Workshop on Automation of Software Test (AST’13), San Francisco, USA, May 18–19 (2013)

  • Tao, C., Gao, J.: Modeling mobile application test platform and environment: testing criteria and complexity analysis. Proceedings of 2014 Workshop on Joining AcadeMiA and Industry Contributions to Test Automation and Model-Based Testing (JAMAICA 2014), San Jose, CA, USA, July 21, pp. 28–33 (2014)

  • TestObject (2016). www.testobject.com.Accessed 27 December 2016

  • Vilkomir, S., Amstutz, B.: Using combinatorial approaches for testing mobile applications. Proceedings of the IEEE Seventh International Conference on Software Testing, Verification and Validation Workshops (ICSTW 2014), March 31–April 4, Cleveland, Ohio, USA, pp. 78–83 (2014)

  • Vilkomir, S., Marszalkowski, K., Perry C., Mahendrakar, S.: Effectiveness of multi-device testing mobile applications. Proceedings of the 2nd ACM International Conference on Mobile Software Engineering and Systems (MobileSoft 2015), May 16–17, Florence, Italy, pp. 44–47 (2015)

  • Zhang, T., Gao, J., Cheng, J., Uehara, T.: Compatibility testing service for mobile applications. IEEE Symposium on Service-Oriented System Engineering (SOSE), Mar 30, San Francisco Bay, USA, pp. 179–186 (2015)

Download references

Acknowledgments

The author wishes to thank his former students Brandi Amstutz, Katherine Marszalkowski, Swetha Mahendrakar, and Chauncey Perry for their contribution at the early stages of this research.

Author information

Authors and Affiliations

  1. Department of Computer Science, East Carolina University, Greenville, NC, 27858, USA

    Sergiy Vilkomir

Authors
  1. Sergiy Vilkomir
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sergiy Vilkomir.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Vilkomir, S. Multi-device coverage testing of mobile applications. Software Qual J 26, 197–215 (2018). https://doi.org/10.1007/s11219-017-9357-7

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11219-017-9357-7

Keywords

Search

Navigation