Nauman bin Ali
ABSTRACT
Context: Systems of systems (SoS) are highly complex and are integrated on multiple levels (unit, component, system, system of systems). Many of the characteristics of SoS (such as operational and managerial independence, integration of system into system of systems, SoS comprised of complex systems) make their development and testing challenging.
Contribution: This paper provides an understanding of SoS testing in large-scale industry settings with respect to challenges and how to address them.
Method: The research method used is case study research. As data collection methods we used interviews, documentation, and fault slippage data.
Results: We identified challenges related to SoS with respect to fault slippage, test turn-around time, and test maintainability. We also classified the testing challenges to general testing challenges, challenges amplified by SoS, and challenges that are SoS specific. Interestingly, the interviewees agreed on the challenges, even though we sampled them with diversity in mind, which meant that the number of interviews conducted was sufficient to answer our research questions. We also identified solution proposals to the challenges that were categorized under four classes of developer quality assurance, function test, testing in all levels, and requirements engineering and communication.
Conclusion: We conclude that although over half of the challenges we identified can be categorized as general testing challenges still SoS systems have their unique and amplified challenges stemming from SoS characteristics. Furthermore, it was found that interviews and fault slippage data indicated that different areas in the software process should be improved, which indicates that using only one of these methods would have led to an incomplete picture of the challenges in the case company.
- JUnit: A programmer-oriented testing framework for Java. http://www.junit.org/. {Acc. Mar. 2012}.Google Scholar
- Sonar - an open platform to manage code quality. www.sonarsource.org. {Accessed Mar. 10, 2012}.Google Scholar
- Testing and Test Control Notation Version 3 (TTCN-3). http://www.ttcn-3.org/. {Acc. Mar. 2012}.Google Scholar
- TestNG - a testing framework. http://testng.org/doc/index.html. {Acc. Mar. 2012}.Google Scholar
- M. C. B. Alves, D. Drusinsky, J. B. Michael, and M. T. Shing. Formal validation and verification of space flight software using statechart-assertions and runtime execution monitoring. In Proceedings of the 6th International Conference on System of Systems Engineering, pages 155--160. IEEE, 2011.Google Scholar
- J. Christie. The seductive and Dangerous V-model. Testing Experience, pages 73--77, 2008.Google Scholar
- J. Colombi, B. C. Cohee, and C. W. Turner. Interoperability test and evaluation: A system of systems field study. The Journal of Defense Software Engineering, 21(11):10--14, 2008.Google Scholar
- J. Dahmann and K. Baldwin. Understanding the current state of us defense systems of systems and the implications for systems engineering. In Proceedings of the 2nd Annual IEEE Systems Conference, pages 1--7. IEEE, 2008.Google ScholarCross Ref
- L. O. Damm and L. Lundberg. Identification of test process improvements by combining fault trigger classification and faults-slip-through measurement. In 2005 International Symposium on Empirical Software Engineering, 2005. IEEE, Nov. 2005.Google ScholarCross Ref
- L.-O. Damm, L. Lundberg, and C. Wohlin. Faults-slip-through - a concept for measuring the efficiency of the test process. Software Process: Improvement and Practice, 11(1):47--59, 2006.Google ScholarCross Ref
- DoD. Systems and software engineering. systems engineering guide for systems of systems, version 1.0. Technical Report ODUSD(A&T)SSE, Office of the Deputy Under Secretary of Defense for Acquisition and Technology, Washington, DC, USA, 2008.Google Scholar
- M. Fewster and D. Graham. Software Test Automation. Addison-Wesley Professional, Sept. 1999.Google Scholar
- R. A. Gougal and A. Monti. The virtual test bed as a tool for rapid system engineering. In Proceedings of the 1st Annual IEEE Systems Conference, pages 1--6. IEEE, 2007.Google Scholar
- J. E. Hannay and H. C. Benestad. Perceived productivity threats in large agile development projects. In Proceedings of the International Symposium on Empirical Software Engineering and Measurement (ESEM 2010), 2010. Google ScholarDigital Library
- J. A. Lane and R. Valerdi. Synthesizing sos concepts for use in cost estimation. Systems Engineering, 10(4):297--307, 2007. Google ScholarDigital Library
- G. Lewis, E. Morris, P. Place, S. Simanta, D. Smith, and L. Wrage. Engineering systems of systems. In Proceedings of the IEEE International Systems Conference (SysCon 2008), 2008.Google ScholarCross Ref
- G. A. Lewis, E. J. Morris, S. Simanta, and D. B. Smith. Service orientation and systems of systems. IEEE Software, 28(1):58--63, 2011. Google ScholarDigital Library
- R. O. Lewis. Independent verification and validation {Elektronisk resurs} : a life cycle engineering process for quality software. Wiley, New York, 1992. Google ScholarDigital Library
- M. W. Maier. Architecting principles for systems-of-systems. Systems Engineering, 1(4):267--284, 1998.Google ScholarCross Ref
- K. Petersen and C. Wohlin. Context in industrial software engineering research. In Proceedings of the Third International Symposium on Empirical Software Engineering and Measurement (ESEM 2009), pages 401--404, 2009. Google ScholarDigital Library
- G. Rothermel, R. H. Untch, C. Chu, and M. J. Harrold. Test case prioritization: An empirical study. In Proceedings of the International Conference on Software Maintenance (ICSM 99), pages 179--188, 1999. Google ScholarDigital Library
- P. Runeson and M. Host. Guidelines for conducting and reporting case study research in software engineering. Empirical Software Engineering, 14(2):131--164, 2009. Google ScholarDigital Library
- C. Solis and X. Wang. A study of the characteristics of behaviour driven development. In Proceedings of the 37th EUROMICRO Conference on Software Engineering and Advanced Applications (SEAA 2011), pages 383--387, 2011. Google ScholarDigital Library
- R. K. Yin. Case study research: design and methods. Sage Publications, Thousand Oaks, 3 ed. edition, 2003.Google Scholar
- S. Yoo and M. Harman. Regression testing minimization, selection and prioritization: a survey. Software Testing, Verification and Reliability, 2010. Google ScholarDigital Library
- B. Zeiss, H. Neukirchen, J. Grabowski, D. Evans, and P. Baker. Refactoring and metrics for ttcn-3 test suites. In Proceedings of the 5th International Workshop on System Analysis and Modeling: Language Profiles (SAM 2006), pages 148--165, 2006. Google ScholarDigital Library
Index Terms
- Testing highly complex system of systems: an industrial case study
Recommendations
A Systematic Mapping Study on Software Testing for Systems-of-Systems
SAST '20: Proceedings of the 5th Brazilian Symposium on Systematic and Automated Software TestingContext: Smart Cities and other Systems-of-Systems (SoS) have attracted attention due to their potential for innovation in the services provided to the society. SoS are composed of several independent systems and often support critical domains. As such, ...
Mutation analysis for system of systems policy testing
JSOS '17: Proceedings of the Joint 5th International Workshop on Software Engineering for Systems-of-Systems and 11th Workshop on Distributed Software Development, Software Ecosystems and Systems-of-SystemsA System of Systems (SoS) is a set of the constituent systems (CS) which has managerial and operational independence. To address an SoS-level goal that cannot be satisfied by each CS, an SoS policy guides or forces the CSs to collaborate with each ...
Application of robust design in engineering software testing
ICNSER2020: Proceedings of the 2nd International Conference on Industrial Control Network And System Engineering ResearchAccording to the characteristics of engineering software testing including development process self-testing, tripartite testing and identification /type testing, a engineering software testing method based on robust design is constructed, and the ...
Comments