skip to main content
10.1145/3530019.3533680acmotherconferencesArticle/Chapter ViewAbstractPublication PageseaseConference Proceedingsconference-collections
abstract

Tutorial 3: Pitfalls in the Measurement Methods Applied in Experimental Software Engineering - Assessment and Suggestions for Improvement

Published: 13 June 2022 Publication History

Abstract

Measurement is an essential issue in empirical software engineering. It is subject to different sources of error that must be kept as small as possible. Measuring instruments is one of these sources of error. In this tutorial, we provide awareness of potential pitfalls in the measurement methods—specifically measuring instruments—applied in empirical software engineering, describing statistical techniques that can be used for measures assessment, and making recommendations to improve the measurement practice in software engineering. Test suites are used as measuring instruments in many software engineering experiments. We will use the case of test suites when measuring the external quality of the code developed by participants of TDD-related experiments.

References

[1]
M.J. Bland and D.G. Altman. 1990. A note on the use of the intraclass correlation coefficient in the evaluation of agreement between two methods of measurement. Computers in Biology and Medicine 20, 5 (1990), 337–340.
[2]
Oscar Dieste, Ayse Tosun, Sira Vegas, Adrian Santos, Fernando Uyaguari, Jarno Kyykkä, and Natalia Juristo. 2021. The role of slicing in test-driven development. ACM Transactions on Software Engineering and Methodology (2021). In preparation.
[3]
Oscar Dieste, Fernando Uyaguari, and Natalia Juristo. 2021. Test cases as a measurement instrument in experimentation. https://arxiv.org/abs/2111.05287v2(2021).
[4]
Norman Fenton. 1992. When a software measure is not a measure. Software Engineering Journal 7, 5 (1992), 357–362.
[5]
N. Fenton and J. Bieman. 2014. Software Metrics – A Rigorous and Practical Approach. CRC Press.
[6]
International Standards Organization. 1994. Accuracy (trueness and precision) of measurement methods and results – Parts 1 to 6. Number ISO 5725.
[7]
Joint Committee for Guides in Metrology. 2008. Evaluation of measurement data – Guide to the expression of uncertainty in measurement. Number JCGM 100:2008.
[8]
Joint Committee for Guides in Metrology. 2012. International vocabulary of metrology – Basic and general concepts and associated terms. Number JCGM 200:2012.
[9]
James Lee, David Koh, and CN Ong. 1989. Statistical evaluation of agreement between two methods for measuring a quantitative variable. Computers in biology and medicine 19, 1 (1989), 61–70.
[10]
Bertil Magnusson. 2003. Handbook for calculation of measurement uncertainty in environmental laboratories.
[11]
Ayse Tosun, Oscar Dieste, Sira Vegas, Dietmar Pfahl, Kerli Rungi, and Natalia Juristo. 2019. Investigating the impact of development task on external quality in test-driven development: An industry experiment. IEEE Transactions on Software Engineering(2019).
[12]
Eric W. Weisstein. Last visited on 22/5/2019. Measure Theory. http://mathworld.wolfram.com/MeasureTheory.html
[13]
Claes Wohlin, Per Runeson, Martin Höst, Magnus C Ohlsson, Björn Regnell, and Anders Wesslén. 2012. Experimentation in software engineering. Springer Science & Business Media.

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
EASE '22: Proceedings of the 26th International Conference on Evaluation and Assessment in Software Engineering
June 2022
466 pages
ISBN:9781450396134
DOI:10.1145/3530019
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 June 2022

Check for updates

Author Tags

  1. Experimentation
  2. accuracy
  3. agreement
  4. measurement
  5. measuring instrument
  6. test suite.

Qualifiers

  • Abstract
  • Research
  • Refereed limited

Funding Sources

  • Ministerio de Ciencia e Innovación
  • FEDER

Conference

EASE 2022

Acceptance Rates

Overall Acceptance Rate 71 of 232 submissions, 31%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • 0
    Total Citations
  • 34
    Total Downloads
  • Downloads (Last 12 months)7
  • Downloads (Last 6 weeks)1
Reflects downloads up to 17 Feb 2025

Other Metrics

Citations

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media