Skip to main content
SpringerLink
Log in

Exploring output-based coverage for testing PHP web applications

  • Published:
Automated Software Engineering Aims and scope Submit manuscript

Abstract

In software testing, different testers focus on different aspects of the software such as functionality, performance, design, and other attributes. While many tools and coverage metrics exist to support testers at the code level, not much support is targeted for testers who want to inspect the output of a program such as a dynamic web application. To support this category of testers, we propose a family of output-coverage metrics (similar to statement, branch, and path coverage metrics on code) that measure how much of the possible output has been produced by a test suite and what parts of the output are still uncovered. To do that, we first approximate the output universe using our existing symbolic execution technique. Then, given a set of test cases, we map the produced outputs onto the output universe to identify the covered and uncovered parts and compute output-coverage metrics. In our empirical evaluation on seven real-world PHP web applications, we show that selecting test cases by output coverage is more effective at identifying presentation faults such as HTML validation errors and spelling errors than selecting test cases by traditional code coverage. In addition, to help testers understand output coverage and augment test cases, we also develop a tool called WebTest that displays the output universe in one single web page and allows testers to visually explore covered and uncovered parts of the output.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Notes

  1. http://quercus.caucho.com/.

  2. http://crawljax.com/.

  3. http://jtidy.sourceforge.net/.

  4. http://sourceforge.net/projects/jazzy/.

  5. http://php.net/manual/en/language.oop5.php.

References

  • Ali, S., Briand, L.C., Hemmati, H., Panesar-Walawege, R.K.: A systematic review of the application and empirical investigation of search-based test case generation. IEEE Trans. Softw. Eng. 36(6), 742–762 (2010). https://doi.org/10.1109/TSE.2009.52. ISSN 0098-5589

    Article  Google Scholar 

  • Alshahwan, N., Harman, M.: Automated web application testing using search based software engineering. In: Proceedings of the 2011 26th IEEE/ACM International Conference on Automated Software Engineering, ASE ’11, Washington, DC, USA, pp. 3–12. IEEE Computer Society (2011). https://doi.org/10.1109/ASE.2011.6100082. ISBN 978-1-4577-1638-6

  • Alshahwan, N., Harman, M.: Augmenting test suites effectiveness by increasing output diversity. In: Proceedings of the 34th International Conference on Software Engineering, ICSE ’12, pp. 1345–1348. IEEE Press (2012)

  • Alshahwan, N., Harman, M.: Coverage and fault detection of the output-uniqueness test selection criteria. In: Proceedings of the 2014 International Symposium on Software Testing and Analysis, New York, NY, USA, pp. 181–192. ACM (2014)

  • Ammann, P., Offutt, J.: Introduction to Software Testing, 1st edn. Cambridge University Press, New York (2008)

    Book  Google Scholar 

  • Andrews, A.A., Offutt, J., Alexander, R.T.: Testing web applications by modeling with FSMs. Softw. Syst. Model. 4, 326–345 (2005)

    Article  Google Scholar 

  • Artzi, S., Kiezun, A., Dolby, J., Tip, F., Dig, D., Paradkar, A., Ernst, M.D.: Finding bugs in web applications using dynamic test generation and explicit-state model checking. IEEE Trans. Softw. Eng. 36(4), 474–494 (2010). https://doi.org/10.1109/TSE.2010.31. ISSN 0098-5589

    Article  Google Scholar 

  • Artzi, S., Dolby, J., Jensen, S.H., Møller, A., Tip, F.: A framework for automated testing of JavaScript web applications. In: Proceedings of the 33rd International Conference on Software Engineering, ICSE ’11, pp. 571–580, New York, NY, USA. ACM (2011). https://doi.org/10.1145/1985793.1985871. ISBN 978-1-4503-0445-0

  • Brady, P.: Mutation testing framework for PHP (2016). https://github.com/padraic/humbug. Accessed 30 June 2017

  • Doğan, S., Betin-Can, A., Garousi, V.: Web application testing: a systematic literature review. J. Syst. Softw. 91, 174–201 (2014). https://doi.org/10.1016/j.jss.2014.01.010. ISSN 0164-1212

    Article  Google Scholar 

  • Elbaum, S., Karre, S., Rothermel, G.: Improving web application testing with user session data. In: Proceedings of the 25th International Conference on Software Engineering, ICSE ’03, Washington, DC, USA, pp. 49–59. IEEE Computer Society (2003). http://dl.acm.org/citation.cfm?id=776816.776823. ISBN 0-7695-1877-X. Accessed 30 June 2017

  • Elbaum, S., Chilakamarri, K.-R., Fisher II, M., Rothermel, G.: Web application characterization through directed requests. In: Proceedings of the 2006 International Workshop on Dynamic Systems Analysis, WODA ’06, New York, NY, USA, pp. 49–56. ACM (2006). https://doi.org/10.1145/1138912.1138923. ISBN 1-59593-400-6

  • Frantzen, L., Huerta, M.L.N., Kiss, Z.G., Wallet, T.: Chapter on-the-fly model-based testing of web services with Jambition. In: Bruni, R., Wolf, K. (eds.) Web Services and Formal Methods, pp. 143–157. Springer, Berlin (2009). https://doi.org/10.1007/978-3-642-01364-5_9. ISBN 978-3-642-01363-8

    Chapter  Google Scholar 

  • Girardi, C., Ricca, F., Tonella, P.: Web crawlers compared. Int. J. Web Inf. Syst. 2(2), 85–94 (2006)

    Article  Google Scholar 

  • Goodenough, J.B., Gerhart, S.L.: Toward a theory of test data selection. In: Proceedings of the International Conference on Reliable Software, pp. 493–510. ACM (1975)

  • Heidegger, P., Thiemann, P.: Contract-driven testing of JavaScript code. In: Proceedings of the 48th International Conference on Objects, Models, Components, Patterns, TOOLS ’10, pp. 154–172. Springer, Berlin (2010). http://dl.acm.org/citation.cfm?id=1894386.1894395. ISBN 3-642-13952-3, 978-3-642-13952-9. Accessed 30 June 2017

  • Kästner, C., Giarrusso, P.G., Rendel, T., Erdweg, S., Ostermann, K., Berger, T.: Variability-aware parsing in the presence of lexical macros and conditional compilation. In: Proceedings of the 2011 ACM International Conference on Object Oriented Programming Systems Languages and Applications, OOPSLA ’11, New York, NY, USA, pp. 805–824. ACM (2011). https://doi.org/10.1145/2048066.2048128. ISBN 978-1-4503-0940-0

  • Li, Y.-F., Das, P.K., Dowe, D.L.: Two decades of web application testing-a survey of recent advances. Inf. Syst. 43(3), 20–54 (2014). https://doi.org/10.1016/j.is.2014.02.001. ISSN 0306-4379

    Article  Google Scholar 

  • McMinn, P.: Search-based software test data generation: a survey: research articles. Softw. Test. Verif. Reliab. 14(2), 105–156 (2004). https://doi.org/10.1002/stvr.v14:2. ISSN 0960-0833

    Article  Google Scholar 

  • Mesbah, A., van Deursen, A.: Invariant-based automatic testing of ajax user interfaces. In: Proceedings of the 31st International Conference on Software Engineering, ICSE ’09, Washington, DC, USA, pp. 210–220. IEEE Computer Society (2009). https://doi.org/10.1109/ICSE.2009.5070522. ISBN 978-1-4244-3453-4

  • Mesbah, Ali, van Deursen, Arie, Lenselink, Stefan: Crawling ajax-based web applications through dynamic analysis of user interface state changes. ACM Trans. Web 6(1), 3:1–3:30 (2012). https://doi.org/10.1145/2109205.2109208. ISSN 1559-1131

    Article  Google Scholar 

  • Milani Fard, A., Mirzaaghaei, M., Mesbah, A.: Leveraging existing tests in automated test generation for web applications. In: Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering, ASE ’14, New York, NY, USA, pp. 67–78. ACM (2014). https://doi.org/10.1145/2642937.2642991. ISBN 978-1-4503-3013-8

  • Miller, J.C., Maloney, C.J.: Systematic mistake analysis of digital computer programs. Commun. ACM 6(2), 58–63 (1963)

    Article  MATH  Google Scholar 

  • Minamide, Y.: Static approximation of dynamically generated web pages. In: Proceedings of the 14th International Conference on World Wide Web, WWW ’05, New York, NY, USA, pp. 432–441. ACM (2005). https://doi.org/10.1145/1060745.1060809. ISBN 1-59593-046-9

  • Mirzaaghaei, M., Mesbah, A.: Dom-based test adequacy criteria for web applications. In: Proceedings of the 2014 International Symposium on Software Testing and Analysis, ISSTA 2014, New York, NY, USA, pp. 71–81. ACM (2014). https://doi.org/10.1145/2610384.2610406. ISBN 978-1-4503-2645-2

  • Nguyen, H.V., Nguyen, H.A., Nguyen, T.T., Nguyen, T.N.: Auto-locating and fix-propagating for HTML validation errors to PHP server-side code. In: Proceedings of the 2011 26th IEEE/ACM International Conference on Automated Software Engineering, Washington, DC, USA, pp. 13–22. IEEE Computer Society (2011)

  • Nguyen, H.V., Kästner, C., Nguyen, T.N.: Building call graphs for embedded client-side code in dynamic web applications. In: Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, New York, NY, USA, pp. 518–529. ACM (2014)

  • Nguyen, H.V., Kästner, C., Nguyen, T.N.: Cross-language program slicing for dynamic web applications. In: Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, New York, NY, USA, pp. 369–380. ACM (2015a)

  • Nguyen, H.V., Kästner, C., Nguyen, T.N.: Varis: IDE support for embedded client code in PHP web applications. In: Proceedings of the 37th International Conference on Software Engineering, vol. 2, pp. 693–696, Piscataway, NJ, USA. IEEE Press (2015b)

  • Ostrand, T.J., Balcer, M.J.: The category-partition method for specifying and generating fuctional tests. Commun. ACM 31(6), 676–686 (1988). https://doi.org/10.1145/62959.62964. ISSN 0001-0782

    Article  Google Scholar 

  • Praphamontripong, U, Offutt, J.: Applying mutation testing to web applications. In: Proceedings of the 2010 Third International Conference on Software Testing, Verification, and Validation Workshops, ICSTW ’10, Washington, DC, USA, pp. 132–141. IEEE Computer Society (2010). https://doi.org/10.1109/ICSTW.2010.38. ISBN 978-0-7695-4050-4

  • Raghavan, S., Garcia-Molina, H.: Crawling the hidden web. In: Proceedings of the 27th International Conference on Very Large Data Bases, VLDB ’01, San Francisco, CA, USA, pp. 129–138. Morgan Kaufmann Publishers Inc (2001). http://dl.acm.org/citation.cfm?id=645927.672025. ISBN 1-55860-804-4. Accessed 30 June 2017

  • Ricca, F., Tonella, P.: Analysis and testing of web applications. In: Proceedings of the 23rd International Conference on Software Engineering, ICSE ’01, Washington, DC, USA, pp. 25–34. IEEE Computer Society (2001). http://dl.acm.org/citation.cfm?id=381473.381476. ISBN 0-7695-1050-7. Accessed 30 June 2017

  • Richardson, D.J., Clarke, L.A.: A partition analysis method to increase program reliability. In: Proceedings of the 5th International Conference on Software Engineering, ICSE ’81, Piscataway, NJ, USA, pp. 244–253. IEEE Press (1981). http://dl.acm.org/citation.cfm?id=800078.802537. ISBN 0-89791-146-6. Accessed 30 June 2017

  • Samimi, H., Schäfer, M., Artzi, S., Millstein, T., Tip, F., Hendren, L.: Automated repair of HTML generation errors in PHP applications using string constraint solving. In: Proceedings of the 34th International Conference on Software Engineering, Piscataway, NJ, USA, pp. 277–287. IEEE Press (2012a)

  • Samimi, H., Schäfer, M., Artzi, S., Millstein, T., Tip, F., Hendren, L.: Automated repair of html generation errors in PHP applications using string constraint solving. In: Proceedings of the 34th International Conference on Software Engineering, ICSE ’12, Piscataway, NJ, USA, pp. 277–287. IEEE Press (2012b). http://dl.acm.org/citation.cfm?id=2337223.2337257. ISBN 978-1-4673-1067-3. Accessed 30 June 2017

  • Saxena, P., Akhawe, D., Hanna, S., Mao, F., McCamant, S., Song, D.: A symbolic execution framework for JavaScript. In: Proceedings of the 2010 IEEE Symposium on Security and Privacy, SP ’10, Washington, DC, USA, pp. 513–528. IEEE Computer Society (2010). https://doi.org/10.1109/SP.2010.38. ISBN 978-0-7695-4035-1

  • Wang, X., Zhang, L., Xie, T., Xiong, Y., Mei, H.: Automating presentation changes in dynamic web applications via collaborative hybrid analysis. In: Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering, FSE ’12, New York, NY, USA, pp. 16:1–16:11. ACM (2012). https://doi.org/10.1145/2393596.2393614. ISBN 978-1-4503-1614-9

  • Wassermann, G., Su, Z.: Static detection of cross-site scripting vulnerabilities. In: Proceedings of the 30th International Conference on Software Engineering, ICSE ’08, New York, NY, USA, pp. 171–180. ACM (2008). https://doi.org/10.1145/1368088.1368112. ISBN 978-1-60558-079-1

  • Wassermann, G., Yu, D., Chander, A., Dhurjati, D., Inamura, H., Su, Z.: Dynamic test input generation for web applications. In: Proceedings of the 2008 International Symposium on Software Testing and Analysis, ISSTA ’08, New York, NY, USA, pp. 249–260. ACM (2008). https://doi.org/10.1145/1390630.1390661. ISBN 978-1-60558-050-0

  • Weyuker, E.J., Ostrand, T.J.: Theories of program testing and the application of revealing subdomains. IEEE Trans. Softw. Eng. 6(3), 236–246 (1980). https://doi.org/10.1109/TSE.1980.234485. ISSN 0098-5589

    Article  MathSciNet  MATH  Google Scholar 

  • Xie, Y., Aiken, A.: Static detection of security vulnerabilities in scripting languages. In: Proceedings of the 15th Conference on USENIX Security Symposium, USENIX-SS ’06, Berkeley, CA, USA, vol. 15. USENIX Association (2006). http://dl.acm.org/citation.cfm?id=1267336.1267349. Accessed 30 June 2017

  • Yu, F., Alkhalaf, M., Bultan, T.: Patching vulnerabilities with sanitization synthesis. In: Proceedings of the 33rd International Conference on Software Engineering, ICSE ’11, New York, NY, USA, pp. 251–260. ACM (2011). https://doi.org/10.1145/1985793.1985828. ISBN 978-1-4503-0445-0

  • Zou, Y., Chen, Z., Zheng, Y., Zhang, X., Gao, Z.: Virtual dom coverage for effective testing of dynamic web applications. In: Proceedings of the 2014 International Symposium on Software Testing and Analysis, ISSTA 2014, New York, NY, USA, pp. 60–70. ACM (2014). https://doi.org/10.1145/2610384.2610399. ISBN 978-1-4503-2645-2

Download references

Acknowledgements

Kästner’s work has been supported in part by the National Science Foundation (Awards 1318808, 1552944, and 1717022) and AFRL and DARPA (FA8750-16-2-0042). Nguyen’s work has been supported in part by CCF-1349153, CCF-1320578, and CCF-1413927.

Author information

Authors and Affiliations

  1. Google LLC, Mountain View, USA

    Hung Viet Nguyen

  2. ECpE Department, Iowa State University, Ames, USA

    Hung Dang Phan

  3. School of Computer Science, Carnegie Mellon University, Pittsburgh, USA

    Christian Kästner

  4. School of Engineering and Computer Science, University of Texas at Dallas, Richardson, USA

    Tien N. Nguyen

Authors
  1. Hung Viet Nguyen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hung Dang Phan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Christian Kästner
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Tien N. Nguyen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Christian Kästner.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Nguyen, H.V., Phan, H.D., Kästner, C. et al. Exploring output-based coverage for testing PHP web applications. Autom Softw Eng 26, 59–85 (2019). https://doi.org/10.1007/s10515-018-0246-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10515-018-0246-5

Keywords

Search

Navigation