Testing highly complex system of systems: An industrial case study

Document type: Conference Papers
Peer reviewed: Yes
Author(s): Nauman Bin Ali, Kai Petersen, Mika Mäntylä
Title: Testing highly complex system of systems: An industrial case study
Conference name: 6th ACM-IEEE International Symposium on Empirical Software Engineering and Measurement, ESEM
Year: 2012
Pagination: 211-220
ISSN: 1949-3770
Publisher: ACM/IEEE
City: Lund
URI/DOI: 10.1145/2372251.2372290
Organization: Blekinge Institute of Technology
Department: School of Computing (Sektionen för datavetenskap och kommunikation)
School of Computing S-371 79 Karlskrona
+46 455 38 50 00
Language: English
Abstract: Systems of systems (SoS) are highly complex and are integrated on multiple levels (unit, component, system, system of systems). Many of the characteristics of SoS (such as operational and managerial independence, integration of system into system of systems, SoS comprised of complex systems) make their development and testing challenging. Contribution: This paper provides an understanding of SoS testing in large-scale industry settings with respect to challenges and how to address them. Method: The research method used is case study research. As data collection methods we used interviews, documentation, and fault slippage data. Results: We identified challenges related to SoS with respect to fault slippage, test turn-around time, and test maintainability. We also classified the testing challenges to general testing challenges, challenges amplified by SoS, and challenges that are SoS specific. Interestingly, the interviewees agreed on the challenges, even though we sampled them with diversity in mind, which meant that the number of interviews conducted was sufficient to answer our research questions. We also identified solution proposals to the challenges that were categorized under four classes of developer quality assurance, function test, testing in all levels, and requirements engineering and communication. Conclusion: We conclude that although over half of the challenges we identified can be categorized as general testing challenges still SoS systems have their unique and amplified challenges stemming from SoS characteristics. Furthermore, it was found that interviews and fault slippage data indicated that different areas in the software process should be improved, which indicates that using only one of these methods would have led to an incomplete picture of the challenges in the case company.
Subject: Software Engineering\General
Keywords: Case study; Software test; System of systems