Analyzing an automotive testing process with evidence-based software engineering
Introduction
Evidence-based software engineering consists of the steps (1) identify the need for information (evidence) and formulate a question, (2) track down the best evidence to answer the question and critically appraise the evidence, and (3) critically reflect on the evidence provided with respect to the problem and context that the evidence should help to solve [1]. Overall, the aim is that practitioners who face an issue in their work shall be supported to make good decisions of how to solve the problems by relying on evidence.
In software engineering two approaches for identifying and aggregating evidence (systematic map [2] and systematic review [3]) have received much attention, which is visible in a high number of systematic reviews and maps published that cover a variety of areas. This ranges from very specific and scoped questions (e.g. within-company vs. cross-company cost estimation [4]) to very generic questions (e.g. what we know about software productivity measurement [5] or pair programming [6]). So far no studies exist that used the evidence based process coming from a concrete problem raised in an industrial case and then attempting to provide a solution for that concrete problem/case through the evidence based process. We extend existing work on evidence based software engineering by using the evidence based process for analyzing an industrial automotive software testing process.
In particular, the first contribution of this paper is to demonstrate the use of the evidence-based process in an industrial case providing reflections of researchers using the process to help the company in their improvement efforts. The process is a staged process where subsequent stages use the input of the previous ones, allowing to provide a traceable and holistic picture from practical challenges to recommendations for improvements. The steps are: identify need for information/problems to be solved (through case study); identify solutions and critical appraisal of those (through systematic literature review); critically reflect on the solutions with respect to the problem and mapping them to solve the problem (through value stream mapping); reflection on the EBSE process.
The second contribution is an in-depth understanding of the automotive software testing process with respect to the current situation (strengths and weaknesses) as well as defining a target process based on evidence presented in literature. In other words, it is shown how an improved automotive software testing process based on evidence from literature would look like. There is a need to better understand and address the challenges related to software testing in the automotive domain (see e.g. [7], [8]), and to identify what solutions are available to automotive companies to deal with these challenges, which is of particular importance due to the specific domain profile of automotive software engineering (as presented in Pretchner et al. [9]).
The remainder of the paper is structured as follows: Section 2 presents the related work, elaborating on the characteristics of automotive software engineering. Section 3 presents the staged evidence-based process using a mixed method design, as well as the research questions asked in each step. Sections 4 EBSE Step 1: Case study on strengths and challenges, 2 Related work, 3 Evidence-based software engineering process used in the case study, 4 EBSE Step 1: Case study on strengths and challenges, 5 EBSE Step 2: Identifying improvements through systematic literature review, 6 Step 3: Critically reflect on the evidence and how to use it to solve problems in the current situation, 7 EBSE Step 4: Evaluate and reflect on the EBSE process present the different stages of the EBSE process proposed in this study, and their outcomes. Section 8 presents the validity threats. Thereafter, Section 9 discusses the results, followed by the conclusions in Section 10.
Section snippets
Related work
The related work focuses on characterizing the automotive software engineering domain and provides a motivation for the study. Details on solutions for automotive software testing in response to the challenges identified are provided through the systematic literature review (see Section 5).
Evidence-based software engineering process used in the case study
In evidence based software engineering (EBSE) the “best” solution for a practical problem should be selected based on evidence. EBSE consists of the following steps: (1) Identify the need for information (evidence) and formulate a question, (2) track down the “best” evidence to answer the question and critically appraise the evidence, and (3) critically reflect on the evidence provided with respect to the problem and context that the evidence should help to solve. In the end (Step 4), the
EBSE Step 1: Case study on strengths and challenges
We conducted an industrial case study [15] to investigate problems and challenges in test process in automotive software domain and identify improvement potentials, answering RQ1 and RQ2.
EBSE Step 2: Identifying improvements through systematic literature review
This section describes the purpose and design of our method in systematic literature review and value stream mapping to improve the testing process. The research question for this part is RQ3: What improvements for the automotive testing process based on practical experiences were suggested in the literature?
In Section 4 we identified the challenges related to software testing in the industry. In this section, we present EBSE Step 2. We perform a domain specific systematic literature review to
Step 3: Critically reflect on the evidence and how to use it to solve problems in the current situation
Value stream mapping (VSM) as a process analysis tool is used to evaluate the findings of strengths and weaknesses. This tool is used for uncovering and eliminating waste [23], [14]. A value stream captures all activities (both value added and non-value added) currently required to bring a product through the main process steps to the customer (end-to-end flow of the process). Value adding activities are those that add value to a product (e.g. by assuring the quality of a feature), while
EBSE Step 4: Evaluate and reflect on the EBSE process
We presented a staged EBSE process, which incorporates systematic literature review and value stream mapping, and used it for software process improvement applied to an automotive test process. In particular, our EBSE process included four steps. First, we performed a case study to investigate the challenges. Second, we performed a domain specific systematic literature. Based on that we formulated solutions proposals and linked it to our literature study findings (see Table 9). Third, we
Validity threats
A validity threat is a specific way in which you might be wrong [64] Research based on empirical studies does have threats to consider. Potential threats relevant to this case study are: Construct validity, external validity and reliability or conclusion validity.
RQ1: What are the practices in testing that can be considered as strengths in automotive domain?
The strengths of the testing were first listed in Section 4.5.2 and further elaborated in Section 6 where they were mapped general value producing activities (see Table 11). Working in small agile teams was considered as a benefit as it reduces the need for documentation and bureaucracy. Small teams were also perceived to lead to more iterative development, easier continuous integration and allowing a better alignment of testing with software requirements and design. Furthermore, team size and
Conclusions
We used a staged evidence-based process that incorporates case study, systematic literature review, and value stream analysis to study the challenges and to create a map of solution proposals for our case company. These techniques have been widely applied, but to our knowledge this is the first time they have been used in combination for solving a problem in a concrete case study. We see that combining these approaches is a good way to do industry academia collaboration as it allows studying
Acknowledgements
We would like to thank all the participants in the study who provided valuable input in interviews. Furthermore, we thank the anonymous reviewers for valuable comments that helped in improving the paper. This work has been supported by ELLIIT, the Strategic Area for ICT research, funded by the Swedish Government.
References (89)
Measuring and predicting software productivity: a systematic map and review
Information and Software Technology
(2011)- et al.
The effectiveness of pair programming: a meta-analysis
Information and Software Technology
(2009) - et al.
Requirements engineering challenges in market-driven software development – an interview study with practitioners
Information & Software Technology
(2007) - et al.
Empirical studies of agile software development: a systematic review
Information & Software Technology
(2008) - et al.
Requirements engineering in automotive development: experiences and challenges
IEEE Software
(2003) - et al.
Systematic model-based testing of embedded automotive software
Electronic Notes in Theoretical Computer Science
(2005) - et al.
Feature interaction aware test case generation for embedded control systems
Electronic Notes in Theoretical Computer Science
(2010) - et al.
Evolutionary functional testing
Computers & OR
(2008) - et al.
Lightweight embedded software performance analysis method by kernel hack and its industrial field study
Journal of Systems and Software
(2012) - et al.
Evolving car designs using model-based automated safety analysis and optimisation techniques
Journal of Systems and Software
(2005)
A comparison of issues and advantages in agile and incremental development between state of the art and an industrial case
Journal of Systems and Software
Requirements engineering: making the connection between the software developer and customer
Information and Software Technology
Essential communication practices for extreme programming in a global software development team
Information and Software Technology
Identifying relevant studies in software engineering
Information & Software Technology
Research synthesis in software engineering: a tertiary study
Information & Software Technology
Evidence-based software engineering
Systematic mapping studies in software engineering
Cross versus within-company cost estimation studies: a systematic review
IEEE Transaction on Software Engineering
Supporting roadmapping of quality requirements
Software, IEEE
Guidelines for conducting and reporting case study research in software engineering
Empirical Software Engineering
Case Study Research: Design and Methods
Real World Research: A Resource for Social Scientists and Practitioner–Researchers
The effect of moving from a plan-driven to an incremental software development approach with agile practices – an industrial case study
Empirical Software Engineering
The Toyota product development system: integrating people, process, and technology
Lean Software Development: An Agile Toolkit
Modeling requirements of automotive software with an extended EAST-ADL2 architecture description language
Cited by (54)
Checklists to support decision-making in regression testing
2023, Journal of Systems and SoftwareAssessing attitudes towards evidence-based software engineering in a government agency
2023, Information and Software TechnologyCitation Excerpt :However, the study of non-academic stakeholders has not been extensively explored in EBSE research. We are only aware of three reports of non-academic use of EBSE [10–12], all three with industry companies and only one includes a report of the attitudes of the industry stakeholders [11]. Thus, when the School of Engineering at the Universidad de la República was invited to give a presentation on its research to the Uruguayan E-Government agency (AGESIC), we took the opportunity to introduce EBSE to AGESIC staff and assess their attitudes to the concept.
Exploring the challenges in software testing of the 5G system at Nokia: A survey
2023, Information and Software TechnologyAssessing test artifact quality—A tertiary study
2021, Information and Software TechnologyEmerging Software Testing Technologies
2018, Advances in Computers