Skip to main content
SpringerLink
Log in

Model-based testing of digital TVs: an industry-as-laboratory approach

  • Published:
Software Quality Journal Aims and scope Submit manuscript

Abstract

Model-based testing is a promising approach for increasing the efficiency of the testing process and for improving software quality. It has been employed in the industry for more than a decade. Nevertheless, there are still challenges regarding its application in different domains. Some of these challenges are general, while some others are domain-specific. In this paper, we explain our experiences in enhancing model-based testing for its adoption in the consumer electronics domain, in particular for Digital TV systems. We applied the so-called industry-as-laboratory approach to define/refine research problems and evaluate our research results. We summarize these results and provide an evaluation of relevant research problems for our context. We observed that the industry-as-laboratory approach is highly effective for industry-academia collaboration and technology transfer in the scope of model-based software testing.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Notes

  1. www.vestel.com.tr.

  2. Vestel currently employs MaTeLo (http://www.all4tec.net) for developing test models and generating test cases from these models.

  3. In each of the case studies, we used the same version of the system to compare the results before and after model refinement.

  4. http://www.klocwork.com.

References

  • Aksit, M., Tekinerdogan, B., Sozer, H., Safi, H., Ayas, M. (2015). The DESARC method: An effective approach for university-industry cooperation. In Proceedings of the international conference on advances in computing, control and networking (pp. 51–53).

  • Apfelbaum, L., Doyle, J. (1997). Model-based testing. In Software quality week conference (pp. 296–300).

  • Arias, T. (2011). Execution architecture views for evolving software-intensive systems. Ph.D. Thesis, University of Groningen.

  • Barr, E., Harman, M., McMinn, P., Shahbaz, M., & Yoo, S. (2015). The oracle problem in software testing: A survey. IEEE Transactions on Software Engineering, 41(5), 507–525.

    Article  Google Scholar 

  • Belli, F. (2001). Finite state testing and analysis of graphical user interfaces. In Proceedings of 12th international symposium on software reliability engineering, ISSRE2001 (pp. 34–43).

  • Belli, F., Beyazit, M., Guler, N. (2011). Event-based GUI testing and reliability assessment techniques—An experimental insight and preliminary results. In Proceedings of the 4th IEEE international conference on software testing, verification and validation workshops (pp. 212–221).

  • Belli, F., Endo, A., Linschulte, M., & Simao, A. (2014). A holistic approach to model-based testing of web service compositions. Software: Practice and Experience, 44(2), 201–234.

    Google Scholar 

  • Boberg, J. (2008). Early fault detection with model-based testing. In Proceedings of the 7th ACM SIGPLAN workshop on ERLANG (pp. 9–20).

  • Boogerd, C., Moonen, L. (2006). Prioritizing software inspection results using static profiling. In Proceedings of the 6th international workshop on source code analysis and manipulation (pp. 149–160).

  • Brinksma E, Hooman J (2008) Dependability for high-tech systems: an industry-as-laboratory approach. In Proceedings of the conference on design, automation and test in Europe (pp. 1226–1231).

  • Dalal, S., Jain, A., Karunanithi, N., Leaton, J., Lott, C., Patton, G., Horowitz, B. (1999). Model-based testing in practice. In Proceedings of the international conference on software engineering (pp. 285–294).

  • Damm, L., Lundberg, L., & Wohlin, C. (2008). A model for software rework reduction through a combination of anomaly metrics. Journal of Systems and Software, 81(11), 1968–1982.

    Article  Google Scholar 

  • de Visser, I. (2008). Analyzing user perceived failure severity in consumer electronics products : Incorporating the user perspective into the development process. Ph.D. thesis, Eindhoven University of Technology.

  • Dustin, E., Rashka, J., & Paul, J. (1999). Automated software testing: Introduction, management, and performance. Boston, MA: Addison-Wesley Longman Publishing Co. Inc.

    Google Scholar 

  • Engstrm, E., & Runeson, P. (2011). Software product line testing a systematic mapping study. Information and Software Technology, 53(1), 2–13.

    Article  Google Scholar 

  • Eriksson, M. (2007). Engineering families of softwareintensive systems using features, goals and scenarios. Ph.D. Thesis, Umea University.

  • Gebizli, C., Metin, D., Sozer, H. (2015). Combining model-based and risk-based testing for effective test case generation. In Proceedings of the 8th IEEE conference on software testing, verification and validation workshops (pp. 1–4).

  • Gebizli, C., Sozer, H. (2014). Improving models for model-based testing based on exploratory testing. In Proceedings of the 6th IEEE workshop on software test automation (pp. 656–661).

  • Gebizli, C, Sozer, H. (2015). Automated refinement of models for model-based testing using exploratory testing. Software Quality Journal (under review).

  • Grottke, M., Matias, R., Trivedi, K.S. (2008). The fundamentals of software aging. In Proceedings of the 19th international symposium on software reliability engineering workshops (pp. 1–6).

  • Guen, H.L., Marie, R., Thelin, T. (2004). Reliability estimation for statistical usage testing using markov chains. In Proceedings of the 15th international symposium on software reliability engineering (pp. 54–65).

  • Joye, C. (2014). Matelo test case generation algorithms: Explanation on available algorithms for test case generation. http://www.all4tec.net/MaTeLo-How-To/understanding-of-test-cases-generation-algorithms.html.

  • Keranen, J., & Raty, T. (2011). Model-based testing of embedded systems in hardware in the loop environment. IET Software, 6(4), 364–376.

    Article  Google Scholar 

  • Klockwork, (2016). C and C++ checker reference, [available online]. https://developer.klocwork.com/documentation/en/insight/10-1/c-and-c-checker-reference.

  • Lackner, H., Thomas, M., Wartenberg, F., Weißleder, S. (2014). Model-based test design of product lines: Raising test design to the product line level. In Proceedings of the 7th IEEE international conference on software testing, verification and validation (pp. 51–60).

  • Lindvall, M., Ganesan, D., Ardal, R., Wiegand, R. (2015). Metamorphic model-based testing applied on NASA DAT—An experience report. In Proceedings of the 37th international conference on software engineering (pp. 129–138).

  • Malik, Q.A., Jskelinen, A., Virtanen, H., Katara, M., Abbors, F., Truscan, D., Lilius, J. (2010). Model-based testing using system vs. test models—What is the difference? In Proceedings of the 17th IEEE international conference and workshops on engineering of computer based systems (pp. 291–299).

  • Muller, G. (2012). Industry-as-laboratory applied in practice: The boderc project. http://www.gaudisite.nl/IndustryAsLaboratoryAppliedPaper.pdf. Accessed 2015.

  • Neto, A., Subramanyan, R., Vieira, M., Travassos, G. (2007). A survey on model-based testing approaches: A systematic review. In Proceedings of the 1st ACM international workshop on Empirical assessment of software engineering languages and technologies (pp. 31–36).

  • Nguyen, B., & Memon, A. (2014). An observe-model-exercise* paradigm to test event-driven systems with undetermined input spaces. IEEE Transactions on Software Engineering, 40(3), 216–234.

    Article  Google Scholar 

  • Nguyen, B., Robbins, B., Banerjee, I., & Memon, A. (2014). GUITAR: an innovative tool for automated testing of gui-driven software. Automated Software Engineering, 21(1), 65–105.

    Article  Google Scholar 

  • Potts, C. (1993). Software-engineering research revisited. IEEE Software, 10(5), 19–28.

    Article  Google Scholar 

  • Samih, H., Bogusch, R. (2014). MPLM—Matelo product line manager: [Relating variability modelling and model-based testing]. In Proceedings of the 18th international software product line conference: Companion volume for workshops, demonstrations and tools, (Vol. 2, pp. 138–142).

  • Schieferdecker, I. (2012). Model-based testing. IEEE Software, 29(1), 14–18.

    Article  Google Scholar 

  • Sozer, H., Hofmann, C., Tekinerdogan, B., Aksit, M. (2011). Runtime verification of component-based embedded software. In Proceedings of the 26th international symposium on computer and information sciences (pp. 471–477).

  • Sozer, H., Tekinerdogan, B., & Aksit, M. (2009). FLORA: A framework for decomposing software architecture to introduce local recovery. Software: Practice and Experience, 39(10), 869–889.

    Google Scholar 

  • Sozer, H., Tekinerdogan, B., & Aksit, M. (2013). Optimizing decomposition of software architecture for local recovery. Software Quality Journal, 21(2), 203–240.

    Article  Google Scholar 

  • Tekinerdogan, B., Sozer, H., & Aksit, M. (2008). Software architecture reliability analysis using failure scenarios. Journal of Systems and Software, 81(4), 558–575.

    Article  Google Scholar 

  • Utting, M., & Legeard, B. (2007). Practical model-based testing: A tools approach. San Francisco, CA: Morgan Kaufmann Publishers Inc.

    Google Scholar 

  • Utting, M., Pretschner, A., & Legeard, B. (2012). A taxonomy of model-based testing approaches. Software Testing Verification and Reliability, 22(5), 297–312.

    Article  Google Scholar 

  • van de Laar, P. (2010). Observations from the industry-as-laboratory research project darwin. In Proceedings of the 8th conference on systems engineering research (pp. 658–667).

  • Weißleder, S. (2010). Test models and coverage criteria for automatic model-based test generation with uml state machines. Ph.D. Thesis, Humboldt-Universitt zu Berlin.

  • Weißleder, S., Lackner, H. (2010). System models vs. test models -distinguishing the undistinguishable?. In Informatik 2010: Service Science - Neue Perspektiven für die Informatik, Beiträge der 40. Jahrestagung der Gesellschaft für Informatik e.V. (GI), Band 2, 27.09. - 1.10.2010, Leipzig (pp. 321–326).

  • Whittaker, J. (2009). Exploratory software testing: Tips, tricks, tours, and techniques to guide test design (1st ed.). Boston: Addison-Wesley Professional.

    Google Scholar 

  • Zoeteweij, P., Abreu, R., Golsteijn, R., van Gemund, A. (2007). Diagnosis of embedded software using program spectra. In Proceedings of the 14th annual IEEE international conference and workshop on engineering of computer based systems (pp. 213–220).

Download references

Acknowledgments

This work is supported by the joint Grant of Vestel Electronics and the Turkish Ministry of Science, Industry and Technology (909.STZ.2015). The contents of this article reflect the ideas and positions of the authors and do not necessarily reflect the ideas or positions of Vestel Electronics and the Turkish Ministry of Science, Industry and Technology. We would like to thank software developers and software test engineers at Vestel Electronics for sharing their code base with us and supporting our case studies.

Author information

Authors and Affiliations

  1. School of Engineering, Ozyegin University, Nişantepe Mah. Orman Sk. No: 34-36, Alemdağ-Çekmeköy, 34794, Istanbul, Turkey

    Hasan Sözer

  2. Vestel Electronics, Manisa, Turkey

    Ceren Şahin Gebizli

Authors
  1. Hasan Sözer
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ceren Şahin Gebizli
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hasan Sözer.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sözer, H., Gebizli, C.Ş. Model-based testing of digital TVs: an industry-as-laboratory approach. Software Qual J 25, 1185–1202 (2017). https://doi.org/10.1007/s11219-016-9321-y

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11219-016-9321-y

Keywords

Search

Navigation