Skip to main content
SpringerLink
Log in

Reproducing failures based on semiformal failure scenario descriptions

  • Published:
Software Quality Journal Aims and scope Submit manuscript

Abstract

Due to the increasing size and complexity of software systems, it becomes hard to test these systems exhaustively. As a result, some faults can be left undetected. Undetected faults can lead to failures in deployed systems. Such failures are usually reported by the users from the field or test engineers back to developers. It requires considerable time and effort to analyze and reproduce the reported failures because their descriptions are not always complete, structured and formal. In this paper, we introduce a novel approach for automatically reproducing failures to aid their debugging. Our approach relies on semi-structured failure scenario descriptions that employ a set of keywords. These descriptions are preprocessed and mapped to a set of predefined test case templates with valid input sets. Then, test cases are generated and executed to reproduce the reported failure scenarios. The approach is evaluated with an industrial case study performed in a company from the telecommunications domain. Several failures were successfully reproduced. The approach is also adopted in the quality assurance process of the company. After one-time preparation of reusable test case templates and training of test engineers, 24.9 % of the reported failures (and 40 % of those that were manually reproducible) could be reproduced without any manual effort.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Notes

  1. We refer to both software test engineers and end users of the system as users.

  2. http://www.fitnesse.org.

  3. http://robotframework.org.

  4. http://www.piworks.net.

  5. http://www.specflow.org/.

  6. http://www.seleniumhq.org.

  7. http://srl.ozyegin.edu.tr/projects/piatt.

  8. The number of compatible issues were decreased due to a major version upgrade.

  9. The average time to reproduce a failure is basically expert opinion. We collected estimations of all the involved engineers in the company.

References

  • Andrica, S., & Candea, G. (2011). Warr: A tool for high-fidelity web application record and replay. In Proceedings of the 41st annual IEEE/IFIP international conference on dependable systems and networks (pp. 403–410).

  • Arya, K., & Verma, H. (2014). Keyword driven automated testing framework for web application. In Proceedings of the 9th international conference on industrial and information systems (pp. 1–6).

  • Avizienis, A., Laprie, J. C., Randell, B., & Landwehr, C. (2004). Basic concepts and taxonomy of dependable and secure computing. IEEE Transactions on Dependable and Secure Computing, 1, 11–33.

    Article  Google Scholar 

  • Bajpai, N., et al. (2012). A keyword driven framework for testing web applications. International Journal of Advanced Computer Science and Applications, 3(3), 8–14.

    Google Scholar 

  • Briand, L., Labiche, Y., & He, S. (2009). Automating regression test selection based on UML designs. Information and Software Technology, 51(1), 16–30.

    Article  Google Scholar 

  • Friedl, J. (2006). Mastering regular expressions. Sebastopol: O’Reilly Media Inc.

    MATH  Google Scholar 

  • Grottke, M., Nikora, A., & Trivedi, K. (2010). An empirical investigation of fault types in space mission system software. In Proceedings of the 40th IEEE/IFIP international conference on dependable systems and networks (pp. 447–456).

  • Hui, J., Yuqing, L., Pei, L., Shuhang, G., & Jing, G. (2008). LKDT: A keyword-driven based distributed test framework. In Proceedings of the international conference on computer science and software engineering (Vol. 2, pp. 719–722).

  • Jin, W., & Orso, A. (2012). Bugredux: Reproducing field failures for in-house debugging. In Proceedings of the 34th international conference on software engineering (pp. 474–484).

  • Lerman, J. (2013). Behavior-driven design with specflow. MSDN Magazine, 28(7), 12–22.

    Google Scholar 

  • Mickens, J., Elson, J., & Howell, J. (2010). Mugshot: Deterministic capture and replay for javascript applications. In Proceedings of the 7th USENIX symposium on networked systems design and implementation (pp. 159–174).

  • Naveen, K., & Hunter, P. (2012). Cost effective agile test practices and test automation using open source tools specflow and white. In Proceedings of the PNSQC (pp. 165–178).

  • North, D. (2006). Behavior modification (pp. 27–31). TechWell Corporation.

  • Pajunen, T., Takala, T., & Katara, M. (2011). Model-based testing with a general purpose keyword-driven test automation framework. In Proceedings of the IEEE fourth international conference on software testing, verification and validation workshops (pp. 242–251).

  • Sarma, M., & Mall, R. (2007). Automatic test case generation from UML models. In Proceedings of the 10th international conference on information technology (pp. 196–201).

  • Soeken, M., Wille, R., & Drechsler, R. (2012). Assisted behavior driven development using natural language processing. Proceedings of the 50th international conference on objects, models: Components, patterns (pp. 269–287). Springer.

  • Utting, M., Pretschner, A., & Legeard, B. (2012). A taxonomy of model-based testing approaches. Software Testing Verification and Reliability, 22(5), 297–312.

    Article  Google Scholar 

  • Vikram, K., Prateek, A., & Livshits, B. (2009) Ripley: Automatically securing web 2.0 applications through replicated execution. In Proceedings of the 16th ACM conference on computer and communications security (pp. 173–186).

  • Wang, C., Pastore, F., Goknil, A., Briand, L., & Iqbal, Z. (2015). Automatic generation of system test cases from use case specifications. In Proceedings of the international symposium on software testing and analysis (pp. 385–396).

  • Wynne, M., & Hellesoy, A. (2012). The cucumber book: Behaviour-driven development for testers and developers. Pragmatic Bookshelf.

  • Yue, T., Ali, S., & Briand, L. (2011). Automated transition from use cases to UML state machines to support state-based testing. In R. France, J. Kuester, B. Bordbar, & R. Paige (Eds.), Modelling foundations and applications, lecture notes in computer science (Vol. 6698, pp. 115–131). Berlin: Springer.

    Chapter  Google Scholar 

Download references

Acknowledgments

This work is partially supported by P.I.Works. We would like to thank software developers and software test engineers at P.I.Works for sharing their code base with us and supporting our case study.

Author information

Authors and Affiliations

  1. P.I.Works HQ, Teknopark İstanbul Sanayi Mah. Teknopark Bul. No: 1/3A 103, 34906, Pendik, Istanbul, Turkey

    Gün Karagöz

  2. School of Engineering, Ozyegin University, Nişantepe Mah. Orman Sk. No: 34-36, Alemdağ, Çekmeköy, 34794, Istanbul, Turkey

    Hasan Sözer

Authors
  1. Gün Karagöz
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hasan Sözer
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Gün Karagöz.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Karagöz, G., Sözer, H. Reproducing failures based on semiformal failure scenario descriptions. Software Qual J 25, 111–129 (2017). https://doi.org/10.1007/s11219-016-9310-1

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11219-016-9310-1

Keywords

Search

Navigation