Skip to main content
SpringerLink
Log in

Guidelines for Conducting Software Engineering Research

  • Chapter
  • First Online:
Contemporary Empirical Methods in Software Engineering

Abstract

This chapter presents a holistic overview of software engineering research strategies. It identifies the two main modes of research within the software engineering research field, namely knowledge-seeking and solution-seeking research—the Design Science model corresponding well with the latter. We present the ABC framework for research strategies as a model to structure knowledge-seeking research. The ABC represents three desirable aspects of research—generalizability over actors (A), precise control of behavior (B), and realism of context (C). Unfortunately, as our framework illustrates, these three aspects cannot be simultaneously maximized. We describe the two dimensions that provide the foundation of the ABC framework—generalizability and control, explain the four different types of settings in which software engineering research is conducted, and position eight archetypal research strategies within the ABC framework. We illustrate each strategy with examples, identify appropriate metaphors, and present an example of how the ABC framework can be used to design a research program.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.00
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 169.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  • Australian Electoral Commission (2016) Australian Electoral Commission image library, 2016 federal election. Opening the house of representatives ballot papers (election night). https://upload.wikimedia.org/wikipedia/commons/9/93/AEC-Senate-election-night-opening.jpg, distributed under Creative Commons CC BY 3.0, https://creativecommons.org/licenses/by/3.0

  • Barcomb A (2019) Retaining and managing episodic contributors in free/libre/open source software communities. PhD thesis, University of Limerick

    Google Scholar 

  • Barcomb A, Kaufmann A, Riehle D, Stol KJ, Fitzgerald B (2019a) Uncovering the periphery: a qualitative survey of episodic volunteering in free/libre and open source software communities. IEEE Trans Softw Eng (in press)

    Google Scholar 

  • Barcomb A, Stol KJ, Riehle D, Fitzgerald B (2019b) Why do episodic volunteers stay in FLOSS communities? In: Proceedings of the 41st international conference on software engineering. ACM, New York, pp 948–954

    Google Scholar 

  • Barcomb A, Stol KJ, Fitzgerald B, Riehle D (2020) Managing episodic volunteers in free/libre/open source software communities. IEEE Trans Softw Eng (in press)

    Google Scholar 

  • Bos N, Sadat Shami N, Olson J, Cheshin A, Nan N (2004) In-group/out-group effects in distributed teams: an experimental simulation. In: Proceedings of the international conference on computer-supported cooperative work and social computing, CSCW’04. ACM, New York, pp 429–436

    Google Scholar 

  • Bourgeois L (1979) Toward a method of middle-range theorizing. Acad Manag Rev 4(3):443–447

    Google Scholar 

  • Dalkey N, Helmer O (1963) An experimental application of the Delphi method to the use of experts. Manag Sci 9(3):458–467

    Google Scholar 

  • Damschen E, Baker D, Bohrer G, Nathan R, Orrock J, Turner JR, Brudvig L, Haddad N, Levey D, Tewksbury J (2014) How fragmentation and corridors affect wind dynamics and seed dispersal in open habitats. Proc Natl Acad Sci USA 111(9):3484–3489. https://doi.org/10.1073/pnas.1308968111

    Google Scholar 

  • Easterbrook S, Singer J, Storey MA, Damian D (2008) Selecting empirical methods for software engineering research. In: Shull F, Singer J, Sjøberg DI (eds) Guide to advanced software engineering. Springer, Berlin

    Google Scholar 

  • Ebert C, Parro C, Suttels R, Kolarczyk H (2001) Better validation in a world-wide development environment. In: Proceedings of the 7th international software metrics symposium (METRICS)

    Google Scholar 

  • Edwards H, McDonald S, Young M (2009) The repertory grid technique: its place in empirical software engineering research. Inform Softw Tech 51(4):785–798

    Google Scholar 

  • Fayerollinson (2010) The Victorian Civil Courtroom at the National Justice Museum. https://commons.wikimedia.org/wiki/File:Victorian_Civil_Courtroom_National_Justice_Mus eum_June_2010.jpg, distributed under Creative Commons BY-SA 3.0, https://creativecommons.org/licenses/by-sa/3.0)

  • Fitzgerald B, Stol KJ, O’Sullivan R, O’Brien D (2013) Scaling agile methods to regulated environments: an industry case study. In: Proceedings of the 2013 international conference on software engineering. IEEE Press, New York, pp 863–872

    Google Scholar 

  • Glaser B (1978) Theoretical sensitivity. The Sociology Press, Mill Valley

    Google Scholar 

  • Glaser B, Strauss A (1967) The discovery of grounded theory. AldineTransaction, Piscataway

    Google Scholar 

  • Glass RL, Vessey I, Ramesh V (2002) Research in software engineering: an analysis of the literature. Inform Softw Tech 44(8):491–506

    Google Scholar 

  • Hassan NR (2015) Seeking middle-range theories in information systems research. In: Proceedings of the 36th international conference on information systems, Fort Worth

    Google Scholar 

  • Hevner A, March S, Park J, Ram S (2004) Design science in information systems research. MIS Q 28(1):75–105

    Google Scholar 

  • Hoda R, Noble J, Marshall S (2013) Self-organizing roles on agile software development teams. IEEE Trans Softw Eng 39(3):422–444

    Google Scholar 

  • Jiang S, McMillan C, Santelices R (2017) Do programmers do change impact analysis in debugging? Empir Softw Eng 22(2):631–669

    Google Scholar 

  • Juristo N, Moreno A (2001) Basics of software engineering experimentation. Springer, Berlin

    MATH  Google Scholar 

  • Kalliamvakou E, Gousios G, Blincoe K, Singer L, German D, Damian D (2016) An in-depth study of the promises and perils of mining GitHub. Empir Softw Eng 21(5):2035–2071

    Google Scholar 

  • Kitchenham B, Pfleeger S (2002) Principles of survey research part 2: designing a survey. ACM Softw Eng Notes 27(1):18–20

    Google Scholar 

  • Kitchenham B, Pfleeger S, Pickard L, Jones P, Hoaglin D, Emam KE, Rosenberg J (2002) Preliminary guidelines for empirical research in software engineering. IEEE Trans Softw Eng 28(8):721–734

    Google Scholar 

  • Kitchenham B, Dybå T, Jørgensen M (2004) Evidence-based software engineering. In: Proceedings of the 26th international conference on software engineering. IEEE, Piscataway, pp 273–281

    Google Scholar 

  • Kontio J, Bragge J, Lehtola L (2008) The focus group method as an empirical tool in software engineering. In: Guide to advanced empirical software engineering. Springer, Berlin

    Google Scholar 

  • Krafft M, Stol K, Fitgerald B (2016) How do free/open source developers pick their tools? A Delphi study of the Debian project. In: Proceedings of the 38th ACM/IEEE international conference on software engineering (SEIP), pp 232–241

    Google Scholar 

  • Lee A, Baskerville R (2003) Generalizing generalizability in information systems research. Inform Syst Res 14:221–243

    Google Scholar 

  • Lethbridge T, Sim S, Singer J (2005) Studying software engineers: data collection techniques for software field studies. Empir Softw Eng 10:311–341

    Google Scholar 

  • Li Y, Yue T, Ali S, Zhang L (2017) Zen-ReqOptimizer: a search-based approach for requirements assignment optimization. Empir Softw Eng 22(1):175–234

    Google Scholar 

  • Linstone H, Turoff M (eds) (2002) The Delphi method techniques and applications. Addison-Wesley, Reading

    Google Scholar 

  • March ST, Smith G (1995) Design and natural science research on information technology. Decis Support Syst 15(4):251–266

    Google Scholar 

  • maxpixel.net (no date) Creative Commons CC0 1.0 Universal. https://www.maxpixel.net/Nature-Green-Jungle-Animals-Fauna-Forest-3828424

  • McGrath JE (1981) Dilemmatics: the study of research choices and dilemmas. Am Behav Sci 25(2):179–210

    Google Scholar 

  • McGrath JE (1984) Groups: interaction and performance. Prentice Hall, Englewood

    Google Scholar 

  • McGrath JE (1995) Methodology matters: doing research in the behavioral sciences. In: Baecker R, Grudin J, Buxton W, Greenberg S (eds) Readings in human computer interaction: toward the year 2000, 2nd edn. Morgan Kaufmann, Los Altos, pp 152–169

    Google Scholar 

  • Méndez Fernández D, Passoth JH (2019) Empirical software engineering: from discipline to interdiscipline. J Syst Softw 148:170–179

    Google Scholar 

  • Merton RK (1968) Social theory and social structure. Free Press

    Google Scholar 

  • Mockus A, Fielding R, Herbsleb J (2000) A case study of open source software development: the Apache server. In: Proceedings of the international conference on software engineering. IEEE, Piscataway

    Google Scholar 

  • Müller M, Pfahl D (2008) Simulation methods. In: Shull F, Singer J, Sjøberg DI (eds) Guide to advanced software engineering. Springer, Berlin

    Google Scholar 

  • Niknafs A, Berry D (2017) The impact of domain knowledge on the effectiveness of requirements engineering activities. Empir Softw Eng 22(1):80–133

    Google Scholar 

  • Nisbett R (2005) The geography of thought: how Asians and Westerners think differently and why. Nicholas Brealey Publishing, Boston

    Google Scholar 

  • Ralph P (2015) The sensemaking-coevolution-implementation theory of software design. Sci Comput Program 101:21–41

    Google Scholar 

  • Ralph P (2018) Toward methodological guidelines for process theories and taxonomies in software engineering. IEEE Trans Softw Eng 45(7):712–735

    Google Scholar 

  • Runeson P, Höst M, Rainer A, Regnell B (2012) Case study research in software engineering: guidelines and examples. Wiley, London

    Google Scholar 

  • Runkel PJ, McGrath JE (1972) Research on human behavior: a systematic guide to method. Holt, Rinehart and Winston, New York

    Google Scholar 

  • Russo D, Stol K (2019) Soft theory: a pragmatic alternative to conduct quantitative empirical studies. In: Proceedings of the joint 7th international workshop on conducting empirical studies in industry and 6th international workshop on software engineering research and industrial practice, CESSER-IP@ICSE 2019, Montreal, pp 30–33

    Google Scholar 

  • Seaman CB (1999) Qualitative methods in empirical studies of software engineering. IEEE Trans Softw Eng 24(4):557–572

    Google Scholar 

  • Setamanit SO (2007) A software process simulation model of global software development (GSD) projects. PhD thesis, Portland State University

    Google Scholar 

  • Setamanit SO, Wakeland W, Raffo D (2007) Using simulation to evaluate global software development task allocation strategies. Softw Process Improve Pract 12:491–503

    Google Scholar 

  • Sharma G, Stol KJ (2019) Exploring onboarding success, organizational fit, and turnover intention of software professionals. J Syst Softw 159:110442

    Google Scholar 

  • Sharp H, Dittrich Y, de Souza C (2016) The role of ethnographic studies in empirical software engineering. IEEE Trans Softw Eng 42(8):786–804

    Google Scholar 

  • Shaw M (2003) Writing good software engineering research papers. In: Proceedings of the 25th international conference on software engineering, pp 726–736

    Google Scholar 

  • Sim S, Easterbrook S, Holt R (2003) Using benchmarking to advance research: a challenge to software engineering. In: Proceedings of the 25th international conference on software engineering. IEEE Computer Society, Silver Spring

    Google Scholar 

  • Simon H (1996) The sciences of the artificial, 3rd edn. MIT Press, Cambridge,

    Google Scholar 

  • Singer J, Storey MA, Sim SE (2000) Beg, borrow, or steal: using multidisciplinary approaches in empirical software engineering research. In: Proceedings of the international conference on software engineering

    Google Scholar 

  • Spinellis D, Avgeriou P (2019) Evolution of the Unix system architecture: an exploratory case study. IEEE Trans Softw Eng (in press)

    Google Scholar 

  • Stol K, Fitzgerald B (2015) Theory-oriented software engineering. Sci Comput Program 101:79–98

    Google Scholar 

  • Stol K, Fitzgerald B (2018) The ABC of software engineering research. ACM Trans Softw Eng Methodol 27(3):51

    Google Scholar 

  • Stol K, Goedicke M, Jacobson I (2016a) Introduction to the special section—general theories of software engineering: new advances and implications for research. Inform Softw Tech 70:176–180

    Google Scholar 

  • Stol K, Ralph P, Fitzgerald B (2016b) Grounded theory in software engineering research: a critical review and guidelines. In: Proceedings of the 38th International Conference on Software Engineering. ACM, New York, pp 120–131

    Google Scholar 

  • Stol K, Caglayan B, Fitzgerald B (2017) Competition-based crowdsourcing software development: a multi-method study from a customer perspective. IEEE Trans Softw Eng 45(3):237–260

    Google Scholar 

  • Storey MD, Zagalsky A, Filho FMF, Singer L, Germán DM (2017) How social and communication channels shape and challenge a participatory culture in software development. IEEE Trans Softw Eng 43(2):185–204

    Google Scholar 

  • SuperJet International (2011) Full flight simulator. https://www.flickr.com/photos/superjetinternational/5573438825, distributed under Creative Commons CC BY 2.0, https://creativecommons.org/licenses/by-sa/2.0/legalcode

  • Tyndall J (1896) Fragment of science, volume one. Taken from an electronic copy of the book at Archive.Org (1896 edition of the book) and subsequently annotated in colored typeface. Public Domain, https://commons.wikimedia.org/w/index.php?curid=57653822

  • Wieringa R (2009) Design science as nested problem solving. In: Proceedings of the 4th international conference on design science research in information systems and technology, DESRIST ’09. ACM, New York

    Google Scholar 

  • Wieringa R, Heerkens M (2006) The methodological soundness of requirements engineering papers: a conceptual framework and two case studies. Requir Eng 11:295–307

    Google Scholar 

  • Wieringa R, Maiden N, Mead N, Rolland C (2006) Requirements engineering paper classification and evaluation criteria: a proposal and a discussion. Requir Eng 11:102–107

    Google Scholar 

  • Wohlin C, Runeson P, Höst M, Ohlsson M, Regnell B, Wesslén A (2012) Experimentation in software engineering, 2nd edn. Springer, Berlin

    MATH  Google Scholar 

  • Wohlin C, Smite D, Moe NB (2015) A general theory of software engineering: balancing human, social and organizational capitals. J Syst Soft 109:229–242

    Google Scholar 

  • Yin R (2014) Case study research design and methods, 5th edn. Sage Publications, Thousand Oaks

    Google Scholar 

Download references

Acknowledgements

This work was supported, in part, by Science Foundation Ireland grant 15/SIRG/3293 and 13/RC/2094 and co-funded under the European Regional Development Fund through the Southern & Eastern Regional Operational Programme to Lero—the Irish Software Research Centre (http://www.lero.ie).

Author information

Authors and Affiliations

  1. Lero—The Irish Software Research Centre and School of Computer Science and Information Technology, University College Cork, Cork, Ireland

    Klaas-Jan Stol

  2. Lero—The Irish Software Research Centre and Department of Computer Science and Information Systems, University of Limerick, Limerick, Ireland

    Brian Fitzgerald

Authors
  1. Klaas-Jan Stol
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Brian Fitzgerald
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Klaas-Jan Stol .

Editor information

Editors and Affiliations

  1. Institute of Computer Science, University of Innsbruck, Innsbruck, Austria

    Michael Felderer

  2. Systems Engineering and Computer Science, Federal University of Rio de Janeiro, Rio de Janeiro, Rio de Janeiro, Brazil

    Guilherme Horta Travassos

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Stol, KJ., Fitzgerald, B. (2020). Guidelines for Conducting Software Engineering Research. In: Felderer, M., Travassos, G. (eds) Contemporary Empirical Methods in Software Engineering. Springer, Cham. https://doi.org/10.1007/978-3-030-32489-6_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-32489-6_2

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-32488-9

  • Online ISBN: 978-3-030-32489-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation