Skip to main content

Advertisement

SpringerLink
Log in

Experience report on the application of genetic algorithms to reduce costs of the software validation process in the automotive sector during an engine control unit project

  • Published:
Software Quality Journal Aims and scope Submit manuscript

Abstract

The number of electronic control units (ECU) installed in vehicles is increasingly high. Manufacturers must improve the software quality and reduce cost by proposing innovative techniques. This research proposes a technique being able to generate not only test-cases in real time but to decide the best means to run them (hardware-in-the-loop simulations or prototype vehicles) to reduce the cost and software testing time. It is focused on the engine ECU software which is one of the most complex software installed in vehicles. This software is coded by using Simulink® models. Two genetic algorithms (GAs) were coded. The first one is in charge of choosing which parts of the Simulink® models should be validated by using hardware-in-the-loop (HIL) simulations and by using prototype vehicles. The second one tunes the inputs of the software module (SM) under validation to cover these parts of the Simulink® models. The usage of dynamic-linked libraries (dlls) is described to deal with the issues linked to SM interactions when running HIL simulations. GAs found at least 7 more bugs than traditional techniques and improved the functional and code coverage by between 3 and 11% for functional coverage and by between 1.4 and 7% for code coverage depending on the SM complexity. The validation time is reduced by 11.9% compared to traditional techniques. GAs perform better than traditional techniques improving software quality and reducing costs and validation time. The usage of dlls allows testing the software in real time as described in this study.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20

Similar content being viewed by others

Notes

  1. “Orthogonal array testing is a black box testing technique that is a systematic, statistical way of software testing. It is used when the number of inputs to the system is relatively small, but too large to allow for exhaustive testing of every possible input to the systems. It is particularly effective in finding errors associated with faulty logic within computer software systems” (Delius, 2004).

  2. Wang and Winner present a method, in which the automated driving function is executed passively, in series production vehicles, which is sometimes known as the shadow mode (Riedmaier et al., 2020; Wang & Winner, 2019).

  3. Low system states are functional states at low level. Consequently, the functional state cannot be detected by the driver.

  4. The optimal path is a part of the model (Fig. 2) that is composed of functional states. These states meet the conditions to change states and the path has the lowest cost.

  5. The reader can find a beta version of the code used in this research. See Appendix.

  6. The number of vectors that will be mutated depends on the need. In this research, the authors have chosen 30% of the vectors to be mutated with good empirical results.

  7. Feedback from other projects means bugs found in a project which could impact another project.

  8. Considering the complexity of this case-study, the number of variables used as predictors must be limited. Otherwise, it would be extremely complex to draw conclusions.

References

  • Abadeh, M. N. (2020). Performance-driven software development: An incremental refinement approach for high-quality requirement engineering. Requirements Engineering, 25, 95–113.

    Article  Google Scholar 

  • Ågren, S. M., Knauss, E., Heldal, R., Pelliccione, P., Malmqvist, G., & Bodén, J. (2019). The impact of requirements on systems development speed: A multiple-case study in automotive. Requirements Engineering, 24, 315–340.

    Article  Google Scholar 

  • ASPICE. (2020) ISO - ISO/IEC 33001:2015 - Information technology — Process assessment — Concepts and terminology. Accessed 30 January 2020.

  • Banish, G. (2007). Engine management: Advanced tuning. Minnesota: Cartech.

  • Chunduri, A. (2016). http://www.diva-portal.org/smash/get/diva2:945731/FULLTEXT02. Accessed 3 February 2020.

  • Conrad, M., Fey, I., & Sadeghipour, S. (2005). systematic model-based testing of embedded automotive software. Electronic Notes in Theoretical Computer Science, 1111, 13–26.

    Article  Google Scholar 

  • Delius, G. W. (2004). Orthogonal Arrays (Taguchi Designs). University of York. https://www.york.ac.uk/depts/maths/tables/orthogonal.htm. Accessed 29 December 2021.

  • Dos Santos, J., Martins, L. E. G., de Santiago Junior, V. A., Povoa, L. V., & dos Santos, L. B. R. (2019). Software requirements testing approaches: A systematic literature review. Requirements Engineering. https://doi.org/10.1007/s00766-019-00325-w

    Article  Google Scholar 

  • dSpace. (2018). https://www.dspace.com/en/inc/home.cfm. Accessed 29 December 2021.

  • dSpace Supplier. (2019a). https://www.dSpace.com/en/inc/home/products/hw/simulator_hardware/dSpace_simulator_full_size.cfm. Accessed 10 December 2019.

  • dSpace Supplier. (2019b). https://www.dSpace.com/en/inc/home/products/sw/experimentandvisualization/controldesk.cfm. Accessed 10 December 2019.

  • El-Rewini, Z., Sadatsharan, K., Flor, D., Siby, S., Plathottam, J., & Ranganathana, P. (2019) Cybersecurity challenges in vehicular communications. Vehicular Communications, 23, 100214

  • Esfandyari, S., & Rafe, V. (2018). A tuned version of genetic algorithm for efficient test suite generation in interactive t-way testing strategy. Information and Software Technology, 94, 165–185.

    Article  Google Scholar 

  • ETAS supplier. (2019). https://www.etas.com/en/products/inca_software_products.php. Accessed 9 March 2020.

  • Feldhütter, A., Segler, C., & Bengler, K. (2018). Does shifting between conditionally and partially automated driving lead to a loss of mode awareness? In N. Stanton (Ed.), Advances in human aspects of transportation. AHFE 2017. Advances in Intelligent Systems and Computing, 597, 730–741.

  • Gajjar, M. J. (2017). Mobile sensors and context-aware computing. Morgan Kaufmann Publishers.

    Google Scholar 

  • Garousi, V., Felderer, M., & Kilicaslan, F. N. (2018). A survey on software testability. Cornell University. https://arxiv.org/abs/1801.02201. Accessed 17 January 2020.

  • Garousi, V., & Mäntylä, M. V. (2016). A systematic literature review of literature reviews in software testing. Information and Software Technology, 80, 195–216.

  • Haghighatkhah, A., Banijamali, A., Pekka Pakanen, O., Oivo, M., & Kuvaja, P. (2017). Automotive software engineering: A systematic mapping study. Journal of Systems and Software, 128, 25–55.

    Article  Google Scholar 

  • Hooshyar, H., Mahmood, F., Vanfretti, L., & Baudette, M. (2015). Specification, implementation, and hardware-in-the-loop real-time simulation of an active distribution grid. Sustainable Energy, Grids and Networks, 3, 36–51.

    Article  Google Scholar 

  • Huang, W.L., Wang, K. Ly, Y., & Zhu, F. (2016). Autonomous vehicles testing methods review. In IEEE 19th international conference on intelligent transportation systems (ITSC) (pp. 163–168).

  • ISO. (2019). Cybersecurity standard. https://www.iso.org/standard/70939.html. Accessed 20 September 2020.

  • ISO. (2020). Autonomous driving safety standard. https://www.iso.org/standard/70918.html. Accessed 20 September 2020.

  • Kasoju, A., Petersen, K., & Mäntylä, M. V. (2013). Analyzing an automotive testing process with evidence-based software engineering. Information and Software Technology, 55(7), 1237–1259.

    Article  Google Scholar 

  • Kim, Y., Lee, D., Baek, J., & Kim, M. (2020). MAESTRO: Automated test generation framework for high test coverage and reduced human effort in automotive industry. Information and Software Technology, 123, 106221.

  • Koegel, M., & Wolf, M. (2018). Auto update – Safe and secure over-the-air (SOTA) software update for advanced driving assistance systems. Springer.

    Book  Google Scholar 

  • Köhl, S., Lemp, D., & Plöger, M. (2003). ECU network testing by hardware-in-the-loop simulation. ATZ Worldwide, 105(10), 10–12.

    Article  Google Scholar 

  • Krûguer, M., Straube, S., Middendorf, A., Hahn, D., Dobs, T., & Lang, K. D. (2016). Requirements for the application of ECUs in e-mobility originally qualified for gasoline cars. Microelectronics Reliability, 64, 140–144.

    Article  Google Scholar 

  • Linderman, U., Maurer, M., & Braun, T. (2009). Structural complexity management. Springer.

    Book  Google Scholar 

  • Lockledge, J. C., & Salustri, F. A. (2010). Defining the engine design process. Journal of Engineering Design, 10, 109–124. https://doi.org/10.1080/095448299261344

  • Martin, H., Ma, Z. , Schmittner, C., Winkler, B., & Kreiner, C. (2020). Combined automotive safety and security pattern engineering approach. Reliability Engineering & System Safety, 198, Article 106773.

  • Matelo® Software. (2018). https://www.all4tec.com/. Accessed 7 February 2020.

  • McAfee. (2016). https://www.mcafee.com/enterprise/en-us/assets/white-papers/wp-automotive-security.pdf. Accessed 7 September 2020.

  • Meloa, S. M., Carver, J. C., Souza, P. S. L., & Souza, S. R. S. (2019). Empirical research on concurrent software testing: A systematic mapping study. Information and Software Technology, 105, 226–251.

    Article  Google Scholar 

  • Möller, D., & Haas, R. (2019). Guide to automotive connectivity and cybersecurity. Wiesbaden: Springer

  • Morris, D., Madzudzo, G., & Garcia-Pereza, A. (2020). Cybersecurity threats in the auto industry: Tensions in the knowledge environment. Technical Forcasting and Social Change, 157, 120102.

  • National Instrument. (2019). https://www.ni.com/fr-fr/innovations/white-papers/17/what-is-hardware-in-the-loop-.html. Accessed 3 March 2020.

  • Ortega-Cabezas, P. M., Colmenar-Santos, A., Borge-Diez, D., & Blanes-Peiró, J. J. (2019a). Application of rule-based expert systems and dynamic-link libraries to enhance hardware-in-the-loop simulation results. The Journal of Software, 14(6), 265–292.

    Article  Google Scholar 

  • Ortega‐Cabezas, P. M., Colmenar‐Santos, A., Borge‐Diez, D., & Blanes‐Peiró, J. J. (2019b). Application of rule‐based expert systems in hardware‐in‐the‐loop simulation case study: Software and performance validation of an engine electronic control unit. Journal of Software: Evolution and Process. https://doi.org/10.1002/smr.2223

  • Petrenko, A., Nguena-Timo, T., & Ramesh, S. (2015). Model-based testing of automotive software: Some challenges and solutions. 52nd Congress ACM/IEEE Design Automation Conference.

  • Placho, T., Schmittner, C., Bonitz, A., & Wana, O. (2020). Management of automotive software updates. Microprocessors and Microsystems, 78, 103257.

  • Plummer, A. R. (2006). Model-in-the-loop testing, proceedings of the institution of mechanical engineers part I. Journal of Systems and Control Engineering, 220(3), 183–199.

    Google Scholar 

  • Raffaëlli, L., Vallée, F., Fayolle, G., Armines, A., de Souza, P., Rouah, X., Pfeiffer, M., Géronimi, S., Pétrot, F., & Ahiad, S. (2016). Embedded Real Time Software and Systems Conference.

  • Raikwar, S., Jijyabhau, L. W., Arun Kumar, S., & Sreenivasulu Rao, M. (2019). Hardware-in-the-loop test automation of embedded systems for agricultural tractors. Measurement, 133, 271–280.

    Article  Google Scholar 

  • Rajan, A., & Wahl, T. (2013). CESAR - Cost-efficient methods and processes for safety-relevant embedded systems. Springer.

    Book  Google Scholar 

  • Riedmaier, S., Ponn, T., Ludwig, B., Shick, F., & Diermeyer, F. (2020). Survey on scenario-based safety assessment of automated vehicles. IEEE Access, 8, 87456–87477.

    Article  Google Scholar 

  • Roychoudhury, A. (2009). Embedded systems and software validation. Morgan Kaufmann Publishers.

    Google Scholar 

  • Sharma, C., Sabharwal, S., & Sibal, R. (2013). A survey on software testing techniques using genetic algorithm. IJCSI International Journal of Computer Science Issues, 20(1), 381–387.

    Google Scholar 

  • Sharma, A., Patani, R., & Aggarwal, A. (2016). Software testing using genetic algorithms. International Journal of Computer Science & Engineering Survey (IJCSES), 7(2), 21–33.

    Article  Google Scholar 

  • Sopan-Barhate, S. (2015). Effective test strategy for testing automotive software. International Congress of Electronic Instrumentation and Control.

  • Sun, W., Cai, X., & Meng, Q. (2016). Testing flight software on the ground: Introducing the hardware-in-the-loop simulation method to the alpha magnetic spectrometer on the International Space Station. Nuclear Instruments and Methods in Physics Research Section a: Accelerators, Spectrometers, Detectors and Associated Equipment, 815, 83–90.

    Article  Google Scholar 

  • Tatar, M., & Mauss, J. (2014). Systematic test and validation of complex embedded systems. Embedded Real Time Software and Systems.

  • Utesch, F., Brandies, A., Pekezou, P., Schiessl, F., & Schiessl, F. (2020). Towards behaviour based testing to understand the black box of autonomous cars. European Transport Research Review, 12, 48.

    Article  Google Scholar 

  • Vandi, G., Nicolò, C., Corti, E., Mancini, G., Moro, D., Ponti, F., & Ravaglioli, V. (2014). Development of a software in the loop environment for automotive powertrain system. Energy Procedia, 45, 789–798.

    Article  Google Scholar 

  • Vector. (2019). https://www.vector.com/int/en/know-how/technologies/safety-security/automotive-cybersecurity/#c2941. Accessed 10 September 2020.

  • Vivas, J. L., Agudo, I., & Lopez, J. (2011). A methodology for security assurance-driven system development. Requirements Engineering, 16, 55–73.

    Article  Google Scholar 

  • Walia, G. S., & Carver, J. C. (2009). A systematic literature review to identify and classify software requirement errors. Information and Software Technology, 51(7), 1087–1109.

    Article  Google Scholar 

  • Wang, C., & Winner, H. (2019). Overcoming challenges of validation automated driving and identification of critical scenarios. Proceeding IEEE Intelligent Transportation Systems Conference (ITSC), 2639–2644.

  • Yi, L., He, H., & Peng, J. (2016). Hardware-in-loop simulation for the energy management system development of a plug-in hybrid electric bus. Energy Procedia, 88, 950–956.

    Article  Google Scholar 

  • Zhan, Y., & Clark, J. A. (2008). A search-based framework for automatic testing of MATLAB/Simulink models. Journal of Systems and Software, 81(2), 262–285.

    Article  Google Scholar 

  • Zhou, J., Zhang, Z., Xie, P., & Wang, J. (2015). A test data generation approach for automotive software. IEEE International Conference on Software Quality, Reliability and Security.

Download references

Author information

Authors and Affiliations

  1. Department of Electric, Electronic and Control Engineering, UNED, Juan del Rosal, 12 – Ciudad Universitaria, 28040, Madrid, Spain

    Pedro Miguel Ortega-Cabezas, Antonio Colmenar-Santos, Jorge Higuera-Pérez & Eric Alcaide

  2. Department of Electrical and Control Engineering, Universidad de León, León, Spain

    David Borge-Diez & Jorge Juan Blanes-Peiró

Authors
  1. Pedro Miguel Ortega-Cabezas
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Antonio Colmenar-Santos
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. David Borge-Diez
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jorge Juan Blanes-Peiró
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jorge Higuera-Pérez
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Eric Alcaide
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Pedro Miguel Ortega-Cabezas.

Additional information

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix

Appendix

For confidentiality reasons, only a beta version can be provided. It can be downloaded in the following link:

https://github.com/pedroai1980/ga.git

The version provided by the authors tries to solve the problem shown in Fig. 21. The reader could reuse it with some changes such as adding automation scripts to each transition between states. Adding calls to Simulink® models to assess conditions to go from one state to another one.

Fig. 21
figure 21

Problem to be solved by using the code provided by the authors

Full size image

The code is composed of the following files:

  1. i.

    utils.py. This file defines the functions necessary to assess conditions for going from one state to another one. The reader can replace and add the functions they want or they can even add calls to Simulink® models.

  2. ii.

    main_v2.py. This file runs the code to solve the problem.

  3. iii.

    g2_func.py and genetic_funcs.py contain the code of the two genetic algorithms necessary to solve the problem.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ortega-Cabezas, P.M., Colmenar-Santos, A., Borge-Diez, D. et al. Experience report on the application of genetic algorithms to reduce costs of the software validation process in the automotive sector during an engine control unit project. Software Qual J 30, 687–728 (2022). https://doi.org/10.1007/s11219-021-09582-x

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11219-021-09582-x

Keywords

Search

Navigation