Abstract
Conventional approaches for the validation of robotic systems include simulations and functional software testing. Although these approaches are interesting, they do not offer coverage information to guide the testing activity. In this sense, the introduction of coverage testing to the validation of mobile robotic systems seems to promise. This paper proposes a integration testing approach for robotic systems in a simulation environment. The approach is composed of test scenarios, constructed based on functional testing, and structural testing towards revealing defects and providing a better systematization of the testing activity in this application domain. The proposed approach focuses on systems developed in ROS, the robot operating system, in which communication can be established through a publish/subscribe interaction schema. The paper provides an example of the way the integration testing approach can be applied to robotic systems. An experimental study conducted with developers of mobile robotic systems evaluated the gains and proved the applicability of the approach to the industry. The results confirmed its advantages for the integration testing of mobile robotic systems. It can check functional behavior (test scenarios), or reach a high structural coverage (coverage criteria), and reveal common defects in mobile robotic systems.
Similar content being viewed by others
References
Abdelgawad, M., McLeod, S., Andrews, A., & Xiao, J. (2017). Model-based testing of real-time adaptive motion planning (RAMP). In: IEEE International Conference on Simulation, Modeling, and Programming for Autonomous Robots, SIMPAR, 162–169.
Ammann, P., & Offutt, J. (2008). Introduction to Software Testing. Cambridge University Press.
Baresi, L., & Pezzé, M. (2006). An introduction to software testing. Electronic Notes in Theoretical Computer Science, proceedings of the School of SegraVis Research Training Network on Foundations of Visual Modelling Techniques (FoVMT 2004), 148(1), 89–111.
Belli, F., Hollmann, A., & Padberg, S. (2009). Communication sequence graphs for mutation-oriented integration testing. In: Secure Software Integration and Reliability Improvement - SSIRI, Third IEEE International Conference on, 387–392.
Blondeau, V., Etien, A., Anquetil, N., Cresson, S., Croisy, P., & Ducasse, S. (2017). Test case selection in industry: an analysis of issues related to static approaches. Software Quality Journal, 25(4), 1203–1237.
Brito, MAS. (2017). Study and definition of integration software testing for mobile robotic systems. PhD thesis, ICMC/USP, Sao Carlos - Sao Paulo - Brasil, portuguese.
Brito, MAS., Santos, M., Souza, PSL., & Souza, SRS. (2015). Integration testing criteria for mobile robotic systems. In: International Conference on Software Engineering & Knowledge Engineering.
Brito, P., De Lemos, R., Rubira, C., & Martins, E. (2009). Architecting fault tolerance with exception handling: Verification and validation. Journal of Computer Science and Technology, 24(2), 212–237.
Carlson, J., Murphy, R. R., & Nelson, A. (2004). Follow-up analysis of mobile robot failures. International Conference on Robotics and Automation, ICRA, 5, 4987–4994.
Chen, Q., Wang, L., Yang, Z., & Stoller, S. (2009). HAVE: Detecting atomicity violations via integrated dynamic and static analysis. Fundamental Approaches to Software Engineering, Springer, Berlin Heidelberg, 5503, 425–439.
Damm, W., Hungar, H., Josko, B., Peikenkamp, T., & Stierand, I. (2011). Using contract-based component specifications for virtual integration testing and architecture design. In: Design, Automation Test in Europe Conference Exhibition, 1023–1028.
Eugster, P., Felber, P. A., Guerraoui, R., & Kermarrec, A. (2003). The many faces of publish/subscribe. ACM Comput Surv, 35(2), 114–131.
Fernandes, LC., Souza, JR., Pessim, G., Shinzato, PY., Sales, DO., Grassi, Jr., V., ... Wolf, DF. (2013). CaRINA intelligent robotic car: Architectural design and implementations. Journal of Systems Architecture.
Frankl, F., & Weyuker, E. (1986). Data ow testing in the presence of unexecutable paths. In: Proceedings of the Workshop on Software Testing, 4–13.
Garlan, D., Khersonsky, S., & Kim, J. S. (2003). Model checking publish-subscribe systems. In T. Ball & S. K. Rajamani (Eds.), Model Checking Software Heidelberg: Springer (pp. 166–180). Berlin: Berlin Heidelberg.
Kaner, C. (2003). An introduction to scenario testing. Tech. rep.: University Florida Tech.
Kang, JS., & Park, HS. (2011). RPIST: Required and provided interface specification-based test case generation and execution methodology for robot software component. In: Int. Conference on Ubiquitous Robots and Ambient Intelligence, 647–651.
Kang, J. S., & Park, H. S. (2012). Web-based automated black-box testing framework for component based robot software. Proceedings of the ACM Conference on Ubiquitous Computing, UbiComp, 852–859. https://doi.org/10.1145/2370216.2370410, http://doi.acm.org/10.1145/2370216.2370410.
Kang, SS., Maeng, SW., Kim, SW., & Park, HS. (2010). SITAT: Simulation-based interface testing automation tool for robot software component. In: Int. Conference on Control Automation and Systems, 1781–1784.
Kim, D., Hong, J. E., & Chung, L. (2018). Investigating relationships between functional coupling and the energy efficiency of embedded software. Software Quality Journal, 26(2), 491–519.
Kramer, J., & Scheutz, M. (2007). Development environments for autonomous mobile robots: A survey. Auton Robots, 22(2), 101–132.
Lee, N., Kim, T., & Cha, S. (2002). Construction of global finite state machine for testing task interactions written in message sequence charts. International conference on Software engineering and knowledge engineering, 27, 369–376.
Leung, HKN., & White, L. (1990). A study of integration testing and software regression at the integration level. In: Conference on Software Maintenance, 290–301.
Lim, JH., Song, SH., Kuc, TY., Park, HS., & Kim, HS. (2009). A hierarchical test model and automated test framework for RTC. In: Int. Conf. on Future Generation Information Technology, 198–207.
Lima, LP., & Cavalli, AR. (1997). A pragmatic approach to generating test sequences for embedded systems. In: International Federation for Information Processing, 288–307.
Lu, XL., Dong, YW., Sun, B., & Zhao, HB. (2011). Research of embedded software testing method based on AADL modes. In: 3rd International Conference on Communication Software and Networks, 89–92.
Lucas, C., Elbaum, S., & Rosenblum, DS. (2012). Detecting problematic message sequences and frequencies in distributed systems. In: Proceedings of the ACM International Conference on Object Oriented Programming Systems Languages and Applications, ACM, New York, NY, USA, OOPSLA ’12, 915–926.
Lussier, B., Chatila, R., Ingrand, F., Killijian, M., & Powell, D. (2004). On fault tolerance and robustness in autonomous systems. In: 3rdI ARPIEEE/RASEURON Joint Workshop on Technical Challenges for Dependable Robots in Human Environments.
Matias, LPN., Santos, TC., Wolf, DF., & Souza, JR. (2015). Path planning and autonomous navigation using amcl and AD*. In: 12th Latin American Robotics Symposium and 3rd Brazilian Symposium on Robotics (LARS-SBR), 320–324.
Mattiello-Francisco, F., Martins, E., Cavalli, A., & Yano, E. (2012). Inrob: An approach for testing interoperability and robustness of real-time embedded software. Journal of Systems and Software, 85(1), 3–15.
Melo, S., Souza, S. R. S., Sarmanho, F. S., & Souza, P. S. L. (2017). Contributions for the structural testing of multithreaded programs: coverage criteria, testing tool, and experimental evaluation. Software Quality Journal, 26(1), 921–959.
Myers, G. J., Sandler, C., Badgett, T., & Thomas, T. M. (2004). The Art of Software Testing. Hoboken, New Jersey: John Wiley & Sons Inc.
Neves, VO., Delamaro, ME., Masiero, P., Mendes, CCT., & Wolf, DF. (2013). Structural testing of autonomous vehicles. In: 25th International Conference on Software Engineering & Knowledge Engineering, 200–205.
Pakala, H., Khan, I., & Raju, K. (2010). Sensors integration in embedded systems. In: International Conference on Power, Control and Embedded Systems.
Pakala, H. G. M., Raju, K. V. S. V. N., & Khan, I. (2011). Integration testing of multiple embedded processing components. Advanced Computing, 133, 200–209.
Park, H., & Seok Kang, J. (2012). SITAF: Simulation-based interface testing automation framework for robot software component (pp. 453–470). InTech: Automation.
Planthaber, S., Vogelgesang, J., & Nieben, E. (2011). CoHoN: A fault-tolerant publish/subscribe tree-based middleware for robots with heterogeneous communication hardware. In: International Conference on Robotics and Biomimetics. IEEE International Conference on Robotics and Biomimetics (IEEE-ROBIO), 232–235.
Prado, RR., Souza, PSL., Souza, SRS., Dourado, GGM., & Batista, RN. (2016). Valipar service: structural testing of concurrent programs a web service composition. In: 13th International Conference on Information Technology: New Generations (ITNG), 581–590.
Pressman, R. S. (2005). Software Engineering: Practitioner’s Approach (6th ed.). McGraw-Hill.
Rapps, S., & Weyuker, E. J. (1985). Selecting software test data using data flow information. IEEE Transactions on Software Engineering, SE, 11(4), 367–375.
Siegwart, R., & Nourbakhsh, I. R. (2004). Introduction to Autonomous Mobile Robots. Scituate, MA, USA: Bradford Company.
Son, J., Kuc, T., Park, J., & Kim, H., (2011). Simulation based functional and per- formance evaluation of robot components and modules. In: International Conference on Information Science and Applications, 1–7.
Sotiropoulos, T., Waeselynck, H., Guiochet, J., & Ingrand, F. (2017). Can robot navigation bugs be found in simulation? an exploratory study. In: IEEE Inter- national Conference on Software Quality, Reliability and Security (QRS), 150–159.
Souza, P. S. L., Souza, S. R. S., & Zaluska, E. (2014). Structural testing for message-passing concurrent programs: an extended test model. Concurrency and Computation: Practice and Experience, 26(1), 21–50.
Souza, S. R. S., Vergilio, S. R., Souza, P. S. L., Simão, A. S., & Hausen, A. C. (2008). Structural testing criteria for message-passing parallel programs. Concurrency and Computation: Practice and Experience, 20(16), 1893–1916.
Söylemez, M., & Tarhan, A. (2018). Challenges of software process and product quality improvement: catalyzing defect root-cause investigation by process enactment data analysis. Software Quality Journal, 26(2), 779–807.
Steinbauer, G. (2013). A survey about faults of robots used in robocup. In X. Chen, P. Stone, L. E. Sucar, & T. van der Zant (Eds.), RoboCup 2012: Robot Soccer World Cup XVI Heidelberg: Springer (pp. 344–355). Berlin: Berlin Heidelberg.
Szyperski, C. (2002). Component Software: Beyond Object-Oriented Program- ming. Boston, MA: Addison-Wesley.
Tzafestas, S. (2014). Introduction to Mobile Robot Control. Elsevier.
Weisshardt, F., Kett, J., Araujo, De Freitas Oliveira, T., Bubeck, A., & Verl, A., (2014). Enhancing software portability with a testing and evaluation platform. Proceedings for the Joint Conference of ISR 2014–45th International Sym- posium on Robotics and Robotik - 8th German Conference on Robotics Munich, Germany: ISR/ROBOTIK, 219–224.
Acknowledgements
The authors acknowledge Brazilian funding agency FAPESP, under processes 2013/03459-4 and 2019/06937-0 for the financial support provided for this research.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Brito, M.A.S., Souza, S.R.S. & Souza, P.S.L. Integration testing for robotic systems. Software Qual J 30, 3–35 (2022). https://doi.org/10.1007/s11219-020-09535-w
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11219-020-09535-w