Analyzing an automotive testing process with evidence-based software engineering

https://doi.org/10.1016/j.infsof.2013.01.005Get rights and content

Abstract

Context

Evidence-based software engineering (EBSE) provides a process for solving practical problems based on a rigorous research approach. The primary focus so far was on mapping and aggregating evidence through systematic reviews.

Objectives

We extend existing work on evidence-based software engineering by using the EBSE process in an industrial case to help an organization to improve its automotive testing process. With this we contribute in (1) providing experiences on using evidence based processes to analyze a real world automotive test process and (2) provide evidence of challenges and related solutions for automotive software testing processes.

Methods

In this study we perform an in-depth investigation of an automotive test process using an extended EBSE process including case study research (gain an understanding of practical questions to define a research scope), systematic literature review (identify solutions through systematic literature), and value stream mapping (map out an improved automotive test process based on the current situation and improvement suggestions identified). These are followed by reflections on the EBSE process used.

Results

In the first step of the EBSE process we identified 10 challenge areas with a total of 26 individual challenges. For 15 out of those 26 challenges our domain specific systematic literature review identified solutions. Based on the input from the challenges and the solutions, we created a value stream map of the current and future process.

Conclusions

Overall, we found that the evidence-based process as presented in this study helps in technology transfer of research results to industry, but at the same time some challenges lie ahead (e.g. scoping systematic reviews to focus more on concrete industry problems, and understanding strategies of conducting EBSE with respect to effort and quality of the evidence).

Introduction

Evidence-based software engineering consists of the steps (1) identify the need for information (evidence) and formulate a question, (2) track down the best evidence to answer the question and critically appraise the evidence, and (3) critically reflect on the evidence provided with respect to the problem and context that the evidence should help to solve [1]. Overall, the aim is that practitioners who face an issue in their work shall be supported to make good decisions of how to solve the problems by relying on evidence.

In software engineering two approaches for identifying and aggregating evidence (systematic map [2] and systematic review [3]) have received much attention, which is visible in a high number of systematic reviews and maps published that cover a variety of areas. This ranges from very specific and scoped questions (e.g. within-company vs. cross-company cost estimation [4]) to very generic questions (e.g. what we know about software productivity measurement [5] or pair programming [6]). So far no studies exist that used the evidence based process coming from a concrete problem raised in an industrial case and then attempting to provide a solution for that concrete problem/case through the evidence based process. We extend existing work on evidence based software engineering by using the evidence based process for analyzing an industrial automotive software testing process.

In particular, the first contribution of this paper is to demonstrate the use of the evidence-based process in an industrial case providing reflections of researchers using the process to help the company in their improvement efforts. The process is a staged process where subsequent stages use the input of the previous ones, allowing to provide a traceable and holistic picture from practical challenges to recommendations for improvements. The steps are: identify need for information/problems to be solved (through case study); identify solutions and critical appraisal of those (through systematic literature review); critically reflect on the solutions with respect to the problem and mapping them to solve the problem (through value stream mapping); reflection on the EBSE process.

The second contribution is an in-depth understanding of the automotive software testing process with respect to the current situation (strengths and weaknesses) as well as defining a target process based on evidence presented in literature. In other words, it is shown how an improved automotive software testing process based on evidence from literature would look like. There is a need to better understand and address the challenges related to software testing in the automotive domain (see e.g. [7], [8]), and to identify what solutions are available to automotive companies to deal with these challenges, which is of particular importance due to the specific domain profile of automotive software engineering (as presented in Pretchner et al. [9]).

The remainder of the paper is structured as follows: Section 2 presents the related work, elaborating on the characteristics of automotive software engineering. Section 3 presents the staged evidence-based process using a mixed method design, as well as the research questions asked in each step. Sections 4 EBSE Step 1: Case study on strengths and challenges, 2 Related work, 3 Evidence-based software engineering process used in the case study, 4 EBSE Step 1: Case study on strengths and challenges, 5 EBSE Step 2: Identifying improvements through systematic literature review, 6 Step 3: Critically reflect on the evidence and how to use it to solve problems in the current situation, 7 EBSE Step 4: Evaluate and reflect on the EBSE process present the different stages of the EBSE process proposed in this study, and their outcomes. Section 8 presents the validity threats. Thereafter, Section 9 discusses the results, followed by the conclusions in Section 10.

Section snippets

Related work

The related work focuses on characterizing the automotive software engineering domain and provides a motivation for the study. Details on solutions for automotive software testing in response to the challenges identified are provided through the systematic literature review (see Section 5).

Evidence-based software engineering process used in the case study

In evidence based software engineering (EBSE) the “best” solution for a practical problem should be selected based on evidence. EBSE consists of the following steps: (1) Identify the need for information (evidence) and formulate a question, (2) track down the “best” evidence to answer the question and critically appraise the evidence, and (3) critically reflect on the evidence provided with respect to the problem and context that the evidence should help to solve. In the end (Step 4), the

EBSE Step 1: Case study on strengths and challenges

We conducted an industrial case study [15] to investigate problems and challenges in test process in automotive software domain and identify improvement potentials, answering RQ1 and RQ2.

EBSE Step 2: Identifying improvements through systematic literature review

This section describes the purpose and design of our method in systematic literature review and value stream mapping to improve the testing process. The research question for this part is RQ3: What improvements for the automotive testing process based on practical experiences were suggested in the literature?

In Section 4 we identified the challenges related to software testing in the industry. In this section, we present EBSE Step 2. We perform a domain specific systematic literature review to

Step 3: Critically reflect on the evidence and how to use it to solve problems in the current situation

Value stream mapping (VSM) as a process analysis tool is used to evaluate the findings of strengths and weaknesses. This tool is used for uncovering and eliminating waste [23], [14]. A value stream captures all activities (both value added and non-value added) currently required to bring a product through the main process steps to the customer (end-to-end flow of the process). Value adding activities are those that add value to a product (e.g. by assuring the quality of a feature), while

EBSE Step 4: Evaluate and reflect on the EBSE process

We presented a staged EBSE process, which incorporates systematic literature review and value stream mapping, and used it for software process improvement applied to an automotive test process. In particular, our EBSE process included four steps. First, we performed a case study to investigate the challenges. Second, we performed a domain specific systematic literature. Based on that we formulated solutions proposals and linked it to our literature study findings (see Table 9). Third, we

Validity threats

A validity threat is a specific way in which you might be wrong [64] Research based on empirical studies does have threats to consider. Potential threats relevant to this case study are: Construct validity, external validity and reliability or conclusion validity.

RQ1: What are the practices in testing that can be considered as strengths in automotive domain?

The strengths of the testing were first listed in Section 4.5.2 and further elaborated in Section 6 where they were mapped general value producing activities (see Table 11). Working in small agile teams was considered as a benefit as it reduces the need for documentation and bureaucracy. Small teams were also perceived to lead to more iterative development, easier continuous integration and allowing a better alignment of testing with software requirements and design. Furthermore, team size and

Conclusions

We used a staged evidence-based process that incorporates case study, systematic literature review, and value stream analysis to study the challenges and to create a map of solution proposals for our case company. These techniques have been widely applied, but to our knowledge this is the first time they have been used in combination for solving a problem in a concrete case study. We see that combining these approaches is a good way to do industry academia collaboration as it allows studying

Acknowledgements

We would like to thank all the participants in the study who provided valuable input in interviews. Furthermore, we thank the anonymous reviewers for valuable comments that helped in improving the paper. This work has been supported by ELLIIT, the Strategic Area for ICT research, funded by the Swedish Government.

References (89)

  • K. Petersen et al.

    A comparison of issues and advantages in agile and incremental development between state of the art and an industrial case

    Journal of Systems and Software

    (2009)
  • H. Saiedian et al.

    Requirements engineering: making the connection between the software developer and customer

    Information and Software Technology

    (2000)
  • L. Layman et al.

    Essential communication practices for extreme programming in a global software development team

    Information and Software Technology

    (2006)
  • H. Zhang et al.

    Identifying relevant studies in software engineering

    Information & Software Technology

    (2011)
  • D. Cruzes et al.

    Research synthesis in software engineering: a tertiary study

    Information & Software Technology

    (2011)
  • B. Kitchenham et al.

    Evidence-based software engineering

  • K. Petersen et al.

    Systematic mapping studies in software engineering

  • B. Kitchenham, Procedures for Performing Systematic Reviews, Tech. Rep. TR/SE-0401, Department of Computer Science,...
  • B.A. Kitchenham et al.

    Cross versus within-company cost estimation studies: a systematic review

    IEEE Transaction on Software Engineering

    (2007)
  • K. Grimm, Software technology in an automotive company – major challenges, in: Proceedings of the 25th International...
  • D. Sundmark, K. Petersen, S. Larsson, An exploratory case study of testing in an automotive electrical system release...
  • A. Pretschner, M. Broy, I.H. Krüger, T. Stauner, Software engineering for automotive systems: A roadmap, in:...
  • E. Bringmann, A. Krämer, Model-based testing of automotive systems, in: First International Conference on Software...
  • R. Feldt, R. Torkar, E. Ahmad, B. Raza, Challenges with software verification and validation activities in the space...
  • B. Regnell et al.

    Supporting roadmapping of quality requirements

    Software, IEEE

    (2008)
  • S. Mujtaba, R. Feldt, K. Petersen, Waste and lead time reduction in a software product customization process with value...
  • P. Runeson et al.

    Guidelines for conducting and reporting case study research in software engineering

    Empirical Software Engineering

    (2009)
  • H.L. McManus, Product Development Value Stream Mapping (PDVSM) Manual, Tech. rep., Center for Technology, Policy, and...
  • Y. Cai, J. You, Research on value stream analysis and optimization methods, in: WiCOM’08, 4th International Conference...
  • R.K. Yin

    Case Study Research: Design and Methods

    (2009)
  • C. Robson

    Real World Research: A Resource for Social Scientists and Practitioner–Researchers

    (2002)
  • K. Petersen et al.

    The effect of moving from a plan-driven to an incremental software development approach with agile practices – an industrial case study

    Empirical Software Engineering

    (2010)
  • M. Khurum, T. Gorschek, M. Wilson, The software value map – an exhaustive collection of value aspects for the...
  • J.M. Morgan et al.

    The Toyota product development system: integrating people, process, and technology

    (2006)
  • M. Poppendieck et al.

    Lean Software Development: An Agile Toolkit

    (2003)
  • B. Kitchenham, S. Charters, Guidelines for Performing Systematic Literature Reviews in Software Engineering, Tech. Rep....
  • G. Park, D. Ku, S. Lee, W. Won, W. Jung, Test methods of the AUTOSAR application software components, in: ICCAS-SICE,...
  • G. Mueller, J. Borzuchowski, Extreme embedded a report from the front line, in: OOPSLA 2002 Practitioners Reports, ACM,...
  • S. Islam, H. Omasreiter, Systematic use case interviews for specification of automotive systems, in: 12th Asia-Pacific...
  • S. Bühne, G. Halmans, K. Pohl, M. Weber, H. Kleinwechter, T. Wierczoch, Defining requirements at different levels of...
  • X. Liu et al.

    Modeling requirements of automotive software with an extended EAST-ADL2 architecture description language

  • A. Puschnig, R.T. Kolagari, Requirements engineering in the development of innovative automotive embedded software...
  • H. Post, C. Sinz, F. Merz, T. Gorges, T. Kropf, Linking functional requirements and software verification, in: RE 2009,...
  • B. Hwong, X. Song, Tailoring the process for automotive software requirements engineering, in: AuRE’06, International...
  • Cited by (54)

    • Assessing attitudes towards evidence-based software engineering in a government agency

      2023, Information and Software Technology
      Citation Excerpt :

      However, the study of non-academic stakeholders has not been extensively explored in EBSE research. We are only aware of three reports of non-academic use of EBSE [10–12], all three with industry companies and only one includes a report of the attitudes of the industry stakeholders [11]. Thus, when the School of Engineering at the Universidad de la República was invited to give a presentation on its research to the Uruguayan E-Government agency (AGESIC), we took the opportunity to introduce EBSE to AGESIC staff and assess their attitudes to the concept.

    • Assessing test artifact quality—A tertiary study

      2021, Information and Software Technology
    • Emerging Software Testing Technologies

      2018, Advances in Computers
    View all citing articles on Scopus
    View full text