research-article

From craft to science: the road ahead for empirical software engineering research

Authors:

Mehdi MirakhorliAuthors Info & Claims

ICSE-NIER '18: Proceedings of the 40th International Conference on Software Engineering: New Ideas and Emerging Results

Pages 77 - 80

https://doi.org/10.1145/3183399.3183421

Published: 27 May 2018 Publication History

Get Access

Abstract

Empirical software engineering (SE) research is often criticized for poorly designed and reported studies, a lack of replications to build up bodies of knowledge, and little practical relevance. In this paper, we discuss issues in empirical software architecture research as an illustration of these issues in one subfield of SE and as a step towards better understanding empirical research in SE in general. Based on feedback from software architecture researchers and practitioners, we explore why, despite persistent discussions in the SE research community, there are still disagreements about why and how to conduct empirical research. Then, we explore how empirical SE research can progress beyond "one-off studies and endless "new and exciting" results toward SE research as a mature science. This would allow us to establish foundations for evaluating existing and future empirical research and help researchers design and publish better studies.

References

[1]

M. Baker. 2016. Is there a Reproducability Crisis? Nature, 33: 452--454.

Crossref

Google Scholar

[2]

L. Briand, D. Bianculli, S. Nejati, F. Pastore and M. Sabetzadeh. 2017. The Case for Context-driven Software Engineering Research. IEEE Software, 34, 5: 72--75.

Crossref

Google Scholar

[3]

D. Falessi, M. A. Babar, G. Cantone and P. Kruchten. 2010. Applying Empirical Software Engineering to Software Architecture: Challenges and Lessons Learned. Empirical Software Engineering, 15, 3: 250--276.

Digital Library

Google Scholar

[4]

M. Galster and D. Weyns. 2016. Empirical Research in Software Architecture - How far have we come? In Proceedings of the 13th Working IEEE/IFIP Conference on Software Architecture (WICSA). Venice, Italy, 2016, 11--20.

Google Scholar

[5]

J. Ioannidis. 2005. Why Most Published Research Findings are False. PLoS Medicine, 2, 8: 696--701.

Crossref

Google Scholar

[6]

V. Ivanov, A. Rogers, G. Succi, J. Yi and V. Zorin. 2017. What Do Software Engineers Care About? Gaps Between Research and Practice. In Proceedings of the 11th Joint Meeting on Foundations of Software Engineering (FSE). Paderborn, Germany, 2017, 890--895.

Digital Library

Google Scholar

[7]

D. Lo, N. Nagappan and T. Zimmermann. 2015. How Practitioners Perceive the Relevance of Software Engineering Research. In Proceedings of the 10th Joint Meeting on Foundations of Software Engineering (FSE). Bergamo, Italy, 2015, 415--425.

Digital Library

Google Scholar

[8]

S. T. Redwine and W. E. Riddle. 1985. Software Technology Maturation. In Proceedings of the 8th International Conference on Software Engineering (ICSE). London, England, 1985, 189--200.

Digital Library

Google Scholar

[9]

J. Siegmund, N. Siegmund and S. Apel. 2015. Views on Internal and External Validity in Empirical Software Engineering. In Proceedings of the 37th International Conference on Software Engineering (ICSE). Florence, Italy, 2015, 9--19.

Digital Library

Google Scholar

[10]

A. Tang and R. Kazman. 2017. On the Worthiness of Software Engineering Research.

Google Scholar

[11]

C. Wohlin and A. Aurum. 2015. Towards a Decision-making Structure for Selecting a Research Design in Empirical Software Engineering. Empirical Software Engineering, 20, 6: 1427--1455.

Digital Library

Google Scholar

Cited By

View all

Galster MWeyns D(2023)Empirical research in software architecture — Perceptions of the communityJournal of Systems and Software10.1016/j.jss.2023.111684202(111684)Online publication date: Aug-2023
https://doi.org/10.1016/j.jss.2023.111684
Konersmann MKaplan AKuhn THeinrich RKoziolek AReussner RJurjens Jal-Doori MBoltz NEhl MFuchs DGroser KHahner SKeim JLohr MSaglam TSchulz SToberg J(2022)Evaluation Methods and Replicability of Software Architecture Research Objects2022 IEEE 19th International Conference on Software Architecture (ICSA)10.1109/ICSA53651.2022.00023(157-168)Online publication date: Mar-2022
https://doi.org/10.1109/ICSA53651.2022.00023
Anchundia CC E(2020)Resources for reproducibility of experiments in empirical software engineering: Topics derived from a secondary studyIEEE Access10.1109/ACCESS.2020.2964587(1-1)Online publication date: 2020
https://doi.org/10.1109/ACCESS.2020.2964587

Index Terms

From craft to science: the road ahead for empirical software engineering research
1. General and reference
  1. Cross-computing tools and techniques
    1. Empirical studies
2. Software and its engineering
  1. Software organization and properties
    1. Software system structures
      1. Software architectures

Recommendations

Ethical Issues in Software Engineering Research: A Survey of Current Practice

In this paper we explore how the software engineering research community is currently dealing with the ethical issues that some empirical research presents. We discuss how the immaturity of the software engineering discipline is reflected in an approach ...
Point/Counterpoint

Is software engineering suffering from a lack of sound empirical research? "In Empirical Research in Software Engineering: A Critical View," David Lorge Parnas (Middle Road Software) debates this issue with Bill Curtis (Cast Software) who writes "Are ...
A Holistic Overview of Software Engineering Research Strategies
CESI '15: Proceedings of the 2015 IEEE/ACM 3rd International Workshop on Conducting Empirical Studies in Industry

Empirical research studies are the principal mechanism through which the software engineering research community studies and learns from software engineering practice. The focus on empirical studies has increased significantly in the past decade, more ...

Comments

Information & Contributors

Information

Published In

ICSE-NIER '18: Proceedings of the 40th International Conference on Software Engineering: New Ideas and Emerging Results

May 2018

130 pages

ISBN:9781450356626

DOI:10.1145/3183399

Conference Chairs:
Andrea Zisman
The Open University
,
Sven Apel
University of Passau, Germany

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 May 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICSE '18

Sponsor:

SIGSOFT
IEEE-CS

ICSE '18: 40th International Conference on Software Engineering

May 27 - June 3, 2018

Gothenburg, Sweden

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
152
Total Downloads

Downloads (Last 12 months)1
Downloads (Last 6 weeks)1

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Galster MWeyns D(2023)Empirical research in software architecture — Perceptions of the communityJournal of Systems and Software10.1016/j.jss.2023.111684202(111684)Online publication date: Aug-2023
https://doi.org/10.1016/j.jss.2023.111684
Konersmann MKaplan AKuhn THeinrich RKoziolek AReussner RJurjens Jal-Doori MBoltz NEhl MFuchs DGroser KHahner SKeim JLohr MSaglam TSchulz SToberg J(2022)Evaluation Methods and Replicability of Software Architecture Research Objects2022 IEEE 19th International Conference on Software Architecture (ICSA)10.1109/ICSA53651.2022.00023(157-168)Online publication date: Mar-2022
https://doi.org/10.1109/ICSA53651.2022.00023
Anchundia CC E(2020)Resources for reproducibility of experiments in empirical software engineering: Topics derived from a secondary studyIEEE Access10.1109/ACCESS.2020.2964587(1-1)Online publication date: 2020
https://doi.org/10.1109/ACCESS.2020.2964587

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Abstract

References

Cited By

Index Terms

Recommendations

Ethical Issues in Software Engineering Research: A Survey of Current Practice

Point/Counterpoint

A Holistic Overview of Software Engineering Research Strategies

Comments

Information

Published In

Sponsors

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Conference

Upcoming Conference

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

Share

Share this Publication link

Share on social media

Affiliations