Article

Automated testing of industrial automation software: practical receipts and lessons learned

Authors:

Werner Putschögl,

Dietmar WinklerAuthors Info & Claims

MoSEMInA 2014: Proceedings of the 1st International Workshop on Modern Software Engineering Methods for Industrial Automation

Pages 7 - 16

https://doi.org/10.1145/2593783.2593788

Published: 31 May 2014 Publication History

Abstract

The share of software in industrial automation systems is steadily increasing. Thus, software quality issues become a critical concern for many automation projects, which require effective software quality assurance measures. In this paper we describe an architecture for automated testing of software applications part of industrial automation systems. We focus on testing programmable logic controller (PLC) software for machineries, which can be achieved by using test automation frameworks derived from software development. The paper provides a collection of practical receipts describing how different approaches from software engineering best-practices can be applied in the context of industrial automation systems. A combination of these receipts has been used for automating software tests in an industrial automation project. In this project, more than 200 tests have been developed to assure the quality of critical PLC components. The tests are an integral part of the project’s automated build and continuous integration system. Every software change triggers an automated build and test process to ensure that no defects or unintended side effects have been introduced. The feedback from the project confirmed the applicability and usefulness of the outlined testing approach in context of industry automation. The paper concludes with providing recommendations and sharing additional lessons we have learned from automated testing of PLC software.

References

[1]

Achatz, R., and Paulisch, F. 2003. Industrial Strength Software and Quality: Software and Engineering at Siemens. In Proc. of the Third Int. Conf. on Quality Software (November, 2003). QSIC'03. IEEE.

Digital Library

[2]

Angerer, F., Prahofer, H., Ramler, R., and Grillenberger, F. 2013. Points-to Analysis of IEC 61131-3 Programs: Implementation and Application. In Proc. of the 18th Conf. on Emerging Technologies Factory Automation (Cagliari, Italy, Sept. 10-13, 2013). ETFA'13. IEEE.

[3]

Barth, M., and Fay, A. 2013. Automated Generation of Simulation Models for Control Code Tests. Control Engineering Practice. 21, 2 (February 2013), 218-230.

[4]

Broy, M. 2006. The ‘Grand Challenge’ in Informatics: Engineering Software-Intensive Systems. IEEE Computer. 39, 10 (October 2006), 72–80.

Digital Library

[5]

Cervantes, A. 2009. Exploring the Use of a Test Automation Framework. In Proc. of the Aerospace Conference (Big Sky, MT, March 7-14, 2009). IEEE.

[6]

Doganay, K., Bohlin, M., and Sellin, O. 2013. Search Based Testing of Embedded Systems Implemented in IEC 61131-3: An Industrial Case Study. In Proc. of the Sixth Int. Conf. on Software Testing, Verification and Validation Workshops (Luxembourg, March 18-22, 2013). ICSTW'13. IEEE.

Digital Library

[7]

Dubey, A. 2011. Evaluating Software Engineering Methods in the Context of Automation Applications. In Proc. of the 9th Int. Conf. on Industrial Informatics (Caparica, Lisbon, July 26-29, 2011). INDIN’11. IEEE.

[8]

Freiseisen, W., Keber, R., Medetz, W., Pau, P., and Stelzmueller, D. 2002. Using Modelica for Testing Embedded Systems. In Proc. of the 2nd International Modelica Conference (Oberpfaffenhofen, Germany, March, 2002).

[9]

Gu, F., Harrison, W.S., Tilbury, D.M., and Yuan, C. 2007. Hardware-In-The-Loop for Manufacturing Automation Control: Current Status and Identified Needs. In Proc. of the Int. Conf. on Automation Science and Engineering (Scottsdale, AZ, Sept. 22-25, 2007). CASE’07. IEEE.

[10]

Hametner, R., Winkler, D., streicher, T., Biffl, S., and Zoitl, A. 2010. The Adaptation of Test-Driven Software Processes to Industrial Automation Engineering. In Proc. of the 8th Int. Conf. on Industrial Informatics (Osaka, Japan, July 13-16, 2010).INDIN’10. IEEE.

[11]

Hametner, R., Winkler, D., and Zoitl, A. 2012. Agile Testing Concepts Based on Keyword-Driven Testing for Industrial Automation Systems. In Proc. of the 38th Annual Conference on Industrial Electronics (Montreal, QC, Oct. 25-28, 2012). IECON’12. IEEE.

[12]

Hametner, R., Kormann, B., Vogel-Heuser, B., Winkler, D., and Zoitl, A. 2013. Automated Test Case Generation for Industrial Control Applications. In Recent Advances in Robotics and Automation, Studies in Computational Intelligence. Vol. 480, 263-273.

[13]

Hamill, P. 2004. Unit Test Frameworks. O’Reilly.

Digital Library

[14]

Huang, L., and Boehm, B. 2006. How Much Software Quality Investment Is Enough: A Value-Based Approach. IEEE Software. 23, 5 (Sept.-Oct. 2006). 88-95.

Digital Library

[15]

Humble, J., and Farley, D. 2011. Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation. Addison-Wesley, Upper Saddle River, NJ.

Digital Library

[16]

Jones, C., and Bonsignour, O. 2012. The Economics of Software Quality. Addison-Wesley, Upper Saddle River, NJ.

Digital Library

[17]

Jolly, S.A., Garousi, V., and Eskandar, M.M. 2012. Automated Unit Testing of a SCADA Control Software: An Industrial Case Study Based on Action Research. In Proc. of the Fifth Int. Conf. on Software Testing, Verification and Validation (Montreal, April 17-21, 2012). ICST'12. IEEE.

Digital Library

[18]

Krapfenbauer, H., Ertl, D., Zoitl, A., and Kupzog, F. 2009. Improving Component Testing of Industrial Automation Software. In Proc. of the 4th International Multi-Conf. on Computing in the Global Information Technology (Cannes, La Bocca, Aug. 23-29, 2009). ICCGI’09. IEEE.

Digital Library

[19]

Kristiansen, D., and Lindegaard, K.-P. 2006. A Framework for Automatic Testing of Industrial Controller Code. In Proc. of the Int. Conf. on Software Engineering Research and Practice (Las Vegas, NV, June 26-29, 2006). SERP’06. CSREA Press.

[20]

Kuhn, D.R., Kacker, R.N., and Lei, Y. 2013. Introduction to Combinatorial Testing. Chapman & Hall/CRC.

Digital Library

[21]

Larndorfer, S., Ramler, R., Federspiel, C., and Lehner, K. 2007. Testing High-Reliability Software for Continuous Casting Steel Plants - Experiences and Lessons Learned from Siemens VAI. In Proc. of the 33rd Euromicro Conf. on Software Engineering and Advanced Applications (Lübeck, Germany, Aug. 28-31, 2007), SEAA’07. IEEE.

Digital Library

[22]

Mahnke, W., Leitner, S.-H., and Damm, M. 2009. OPC Unified Architecture. Springer Publishing Company.

Digital Library

[23]

Mathur, A.P. 2013. Foundations of Software Testing, 2. Ed. Pearson Education India.

[24]

Pingping, W., Xiaoyao, Q., and Jiang, Z. 2012. Testing Process Model and Classification of Test Methods for Embedded Software of Electrical Products. In Proc. of the 3rd Int. Conf. on Software Engineering and Service Science (Beijing, June 22-24, 2012). ICSESS’12. IEEE.

[25]

Prahofer, H., Schatz, R., Wirth, C., and Mossenböck, H. 2010. Deterministic Replay Debugging of IEC 61131-3 SoftPLC Programs. In Proc. of the 8th Int. Conf. on Industrial Informatics (Osaka, July 13-16, 2010).INDIN’10. IEEE.

[26]

Ramler, R., and Putschögl, W. 2013. A Retrospection on Building a Custom Tool for Automated System Testing. In Proc. of the 37th Annual Computer Software and Applications Conference (Kyoto, Japan, July 22-26, 2013). COMPSAC’13. IEEE.

Digital Library

[27]

Ramler, R, and Wolfmaier, K. 2006. Economic Perspectives in Test Automation: Balancing Automated and Manual Testing with Opportunity Cost. In Proc. of the International Workshop on Automation of Software Test (Shanghai, China, May, 2006). AST’06. ACM.

Digital Library

[28]

Sünder, C., Zoitl, A., Christensen, J.H., Steininger, H., and Rritsche, J. 2008. Considering IEC 61131-3 and IEC 61499 in the Context of Component Frameworks. In Proc. of the 6th Int. Conf. on Industrial Informatics (Daejeon, July 13-16, 2008).INDIN’08. IEEE.

[29]

Stoeppler, G., Menzel, T., and Douglas, S. 2005. Hardwarein-the-Loop Simulation of Machine Tools and Manufacturing Systems. Computing Control Engineering Journal, 16, 1, (Feb.-March 2005). 10-15.

[30]

Thomas, D., and Hunt, A. 2002. Mock Objects. IEEE Software. 19, 3, (May/Jun 2002). 22-24.

Digital Library

[31]

Tillmann, N., De Halleux, J., and Xie, T. 2010. Parameterized Unit Testing: Theory and Practice. In Proc. of the 32nd Int. Conf. on Software Engineering (Cape Town, May 2-8, 2010). ICSE’10. ACM/IEEE.

Digital Library

[32]

Tsai, W.-T., Yu, L., Zhu, F., and Paul, R. 2005. Rapid Embedded System Testing Using Verification Patterns. IEEE Software. 22, 4, (July-Aug. 2005). 68-75.

Digital Library

[33]

Vyatkin, V. 2013. Software Engineering in Industrial Automation: State-of-the-Art Review. IEEE Transactions on Industrial Informatics. 9, 3, (Aug. 2013). 1234-1249.

[34]

Winkler, D., Hametner, R., and Biffl, S. 2009. Automation Component Aspects for Efficient Unit Testing. In Proc. of the Conf. on Emerging Technologies and Factory Automation (Mallorca, Spain, Sept. 22-25, 2009).ETFA’09. IEEE.

Digital Library

[35]

Winkler, D., Hametner, R., streicher, T., and Biffl, S. 2010. A Framework for Automated Testing of Automation Systems. In Proc. of the Conf. on Emerging Technologies and Factory Automation (Bilbao, Spain, Sept. 13-16, 2010).ETFA’10. IEEE.

Cited By

Walker MKlingel LOechsle SNeubauer MLechler AVerl A(2023)Safeguarded Continuous Deployment of Control Containers through Real-Time Simulation2023 IEEE 28th International Conference on Emerging Technologies and Factory Automation (ETFA)10.1109/ETFA54631.2023.10275659(1-8)Online publication date: 12-Sep-2023
https://doi.org/10.1109/ETFA54631.2023.10275659
Fischer SKlammer CFernández ARabiser RRamler R(2023)Testing of highly configurable cyber–physical systems — Results from a two-phase multiple case studyJournal of Systems and Software10.1016/j.jss.2023.111624199(111624)Online publication date: May-2023
https://doi.org/10.1016/j.jss.2023.111624
Buchgeher GDorninger BKlammer CWalchshofer AKern A(2022)Migrating Cyber-Physical Systems to OPC UAProcedia Computer Science10.1016/j.procs.2022.01.226200(276-283)Online publication date: 2022
https://doi.org/10.1016/j.procs.2022.01.226
Show More Cited By

Index Terms

Automated testing of industrial automation software: practical receipts and lessons learned
1. Applied computing
  1. Computers in other domains
2. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Software test automation practices in agile development environment: an industry experience report
AST '12: Proceedings of the 7th International Workshop on Automation of Software Test

The increased importance of Test Automation in software engineering is very evident considering the number of companies investing in automated testing tools nowadays, with the main aim of preventing defects during the development process. Test ...
An orchestrated survey of methodologies for automated software test case generation

Test case generation is among the most labour-intensive tasks in software testing. It also has a strong impact on the effectiveness and efficiency of software testing. For these reasons, it has been one of the most active research topics in software ...
Automated System Testing for FPGA-based ADC Reference Design Boards
ICCIP '20: Proceedings of the 6th International Conference on Communication and Information Processing

System testing is one of the test stages of a software test, making sure that the software intended for the system is working as it should be. This study has aimed to automate the manual system testing of the Field Programmable Gate Array (FPGA) -based ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MoSEMInA 2014: Proceedings of the 1st International Workshop on Modern Software Engineering Methods for Industrial Automation

May 2014

71 pages

ISBN:9781450328517

DOI:10.1145/2593783

General Chairs:
Anil R. Nair,
Herbert Prähofer,
Alois Zoitl,
Program Chairs:
Raoul Jetley,
Alpana Dubey,
Atul Kumar

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

In-Cooperation

TCSE: IEEE Computer Society's Tech. Council on Software Engin.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 31 May 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

ICSE '14

Sponsor:

SIGSOFT

ICSE '14: 36th International Conference on Software Engineering

May 31, 2014

Hyderabad, India

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
614
Total Downloads

Downloads (Last 12 months)35
Downloads (Last 6 weeks)3

Reflects downloads up to 25 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Walker MKlingel LOechsle SNeubauer MLechler AVerl A(2023)Safeguarded Continuous Deployment of Control Containers through Real-Time Simulation2023 IEEE 28th International Conference on Emerging Technologies and Factory Automation (ETFA)10.1109/ETFA54631.2023.10275659(1-8)Online publication date: 12-Sep-2023
https://doi.org/10.1109/ETFA54631.2023.10275659
Fischer SKlammer CFernández ARabiser RRamler R(2023)Testing of highly configurable cyber–physical systems — Results from a two-phase multiple case studyJournal of Systems and Software10.1016/j.jss.2023.111624199(111624)Online publication date: May-2023
https://doi.org/10.1016/j.jss.2023.111624
Buchgeher GDorninger BKlammer CWalchshofer AKern A(2022)Migrating Cyber-Physical Systems to OPC UAProcedia Computer Science10.1016/j.procs.2022.01.226200(276-283)Online publication date: 2022
https://doi.org/10.1016/j.procs.2022.01.226
Wang YMäntylä MLiu ZMarkkula J(2022)Test automation maturity improves product quality—Quantitative study of open source projects using continuous integrationJournal of Systems and Software10.1016/j.jss.2022.111259188(111259)Online publication date: Jun-2022
https://doi.org/10.1016/j.jss.2022.111259
Klammer CWetzlmaier TPfeiffer MSteiner TKonnerth M(2021)Testing Autogenerated OPC UA NodeSet Models for Product Variants in IndustrySoftware Quality: Future Perspectives on Software Engineering Quality10.1007/978-3-030-65854-0_2(15-29)Online publication date: 6-Jan-2021
https://doi.org/10.1007/978-3-030-65854-0_2
Meixner KKathrein LWinkler DLuder ABiffl S(2020)Efficient Test Case Generation from Product and Process Model Properties and Preconditions2020 25th IEEE International Conference on Emerging Technologies and Factory Automation (ETFA)10.1109/ETFA46521.2020.9212003(859-866)Online publication date: Sep-2020
https://doi.org/10.1109/ETFA46521.2020.9212003
Schlie ABougouffa SFischer JSchaefer IVogel-Heuser B(2018)Change analysis on evolving PLC software in automated production systemsat - Automatisierungstechnik10.1515/auto-2018-003766:10(806-818)Online publication date: 17-Oct-2018
https://doi.org/10.1515/auto-2018-0037
Winkler DMeixner KBiffl S(2018)Towards Flexible and Automated Testing in Production Systems Engineering Projects2018 IEEE 23rd International Conference on Emerging Technologies and Factory Automation (ETFA)10.1109/ETFA.2018.8502650(169-176)Online publication date: Sep-2018
https://doi.org/10.1109/ETFA.2018.8502650
Ramler RZiebermayr T(2017)What You See Is What You Test - Augmenting Software Testing with Computer Vision2017 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)10.1109/ICSTW.2017.76(398-400)Online publication date: Mar-2017
https://doi.org/10.1109/ICSTW.2017.76
Barabanov AMarkov ATsirlov V(2016)Procedure for substantiated development of measures to design secure software for automated process control systems2016 International Siberian Conference on Control and Communications (SIBCON)10.1109/SIBCON.2016.7491660(1-4)Online publication date: May-2016
https://doi.org/10.1109/SIBCON.2016.7491660

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten