A Method for Evaluating Rigor and Industrial Relevance of Technology Evaluations

Document type: Journal Articles
Article type: Original article
Peer reviewed: Yes
Author(s): Martin Ivarsson, Tony Gorschek
Title: A Method for Evaluating Rigor and Industrial Relevance of Technology Evaluations
Journal: Empirical Software Engineering
Year: 2011
Volume: 16
Issue: 3
Pagination: 365-395
ISSN: 1382-3256
Publisher: Springer verlag
URI/DOI: 10.1007/s10664-010-9146-4
ISI number: 000289254600003
Organization: Blekinge Institute of Technology
Department: School of Computing (Sektionen för datavetenskap och kommunikation)
School of Computing S-371 79 Karlskrona
+46 455 38 50 00
Language: English
Abstract: One of the main goals of an applied research field such as software engineering is the transfer and widespread use of research results in industry. To impact industry, researchers developing technologies in academia need to provide tangible evidence of the advantages of using them. This can be done trough step-wise validation, enabling researchers to gradually test and evaluate technologies to finally try them in real settings with real users and applications. The evidence obtained, together with detailed information
on how the validation was conducted, offers rich decision support material for industry practitioners seeking to adopt new technologies and researchers looking for an empirical
basis on which to build new or refined technologies. This paper presents model for evaluating the rigor and industrial relevance of technology evaluations in software engineering. The model is applied and validated in a comprehensive systematic literature review of evaluations of requirements engineering technologies published in software engineering journals. The aim is to show the applicability of the model and to characterize how evaluations are carried out and reported to evaluate the state-of-research. The review shows that the model can be applied to characterize evaluations in requirements engineering. The findings from applying the model also show that the majority of technology evaluations in requirements engineering lack both industrial relevance and rigor.
In addition, the research field does not show any improvements in terms of industrial relevance over time.
Subject: Software Engineering\General
Keywords: research methodology, empirical, case study, requirements engineering