skip to main content
10.1145/3463274.3463362acmotherconferencesArticle/Chapter ViewAbstractPublication PageseaseConference Proceedingsconference-collections
research-article

About the Assessment of Grey Literature in Software Engineering

Published: 21 June 2021 Publication History

Abstract

There is an ongoing interest in the Software Engineering field for multivocal literature reviews including grey literature. However, at the same time, the role of the grey literature is still controversial, and the benefits of its inclusion in systematic reviews are object of discussion. Some of these arguments concern the quality assessment methods for grey literature entries, which is often considered a challenging and critical task. On the one hand, apart from a few proposals, there is a lack of an acknowledged methodological support for the inclusion of Software Engineering grey literature in systematic surveys. On the other hand, the unstructured shape of the grey literature contents could lead to bias in the evaluation process impacting on the quality of the surveys. This work leverages an approach on fuzzy Likert scales, and it proposes a methodology for managing the explicit uncertainties emerging during the assessment of entries from the grey literature. The methodology also strengthens the adoption of consensus policies that take into account the individual confidence level expressed for each of the collected scores.

References

[1]
Jean Adams, Frances C Hillier-Brown, Helen J Moore, Amelia A Lake, Vera Araujo-Soares, Martin White, and Carolyn Summerbell. 2016. Searching and synthesising ‘grey literature’and ‘grey information’in public health: critical reflections on three case studies. Systematic reviews 5, 1 (2016), 1–11.
[2]
Antonia Bertolino, Antonello Calabrò, Francesca Lonetti, Eda Marchetti, and Breno Miranda. 2018. A categorization scheme for software engineering conference papers and its application. J. Syst. Softw. 137(2018), 114–129.
[3]
Francisco Javier Cabrerizo, Francisco Chiclana, Rami Al-Hmouz, Ali Morfeq, Abdullah Saeed Balamash, and Enrique Herrera-Viedma. 2015. Fuzzy decision making and consensus: challenges. Journal of Intelligent & Fuzzy Systems 29, 3 (2015), 1109–1118.
[4]
Deborah J Cook, Gordon H Guyatt, Gerard Ryan, Joanne Clifton, Lisa Buckingham, Andrew Willan, William Mcllroy, and Andrew D Oxman. 1993. Should unpublished data be included in meta-analyses?: Current convictions and controversies. Jama 269, 21 (1993), 2749–2753.
[5]
Fabio QB Da Silva, André LM Santos, Sérgio Soares, A César C França, Cleviton VF Monteiro, and Felipe Farias Maciel. 2011. Six years of systematic literature reviews in software engineering: An updated tertiary study. Information and Software Technology 53, 9 (2011), 899–913.
[6]
Sara de la Rosa de Sáa, María Ángeles Gil, Gil González-Rodríguez, María Teresa López, and María Asunción Lubiano. 2014. Fuzzy rating scale-based questionnaires and their statistical analysis. IEEE Transactions on Fuzzy Systems 23, 1 (2014), 111–126.
[7]
Pierpaolo D’Urso. 2017. Exploratory multivariate analysis for empirical information affected by uncertainty and modeled in a fuzzy manner: a review. Granular Computing 2, 4 (2017), 225–247.
[8]
Vahid Garousi, Michael Felderer, and Mika V. Mäntylä. 2016. The Need for Multivocal Literature Reviews in Software Engineering: Complementing Systematic Literature Reviews with Grey Literature. In Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering(EASE ’16). Article 26, 6 pages.
[9]
Vahid Garousi, Michael Felderer, and Mika V. Mäntylä. 2019. Guidelines for including grey literature and conducting multivocal literature reviews in software engineering. Information & Software Technology 106 (2019), 101–121.
[10]
S Gopalakrishnan and P Ganeshkumar. 2013. Systematic reviews and meta-analysis: understanding the best evidence in primary healthcare. Journal of family medicine and primary care 2, 1 (2013), 9.
[11]
James Hartley. 2014. Some thoughts on Likert-type scales. International journal of clinical and health psychology 14, 1(2014), 83–86.
[12]
Tamás Jónás, Zsuzsanna Eszter Tóth, and Gábor Árva. 2018. Applying a fuzzy questionnaire in a peer review process. Total Quality Management & Business Excellence 29, 9-10(2018), 1228–1245.
[13]
Barbara Kitchenham and Stuart Charters. 2007. Guidelines for performing systematic literature reviews in software engineering. (2007).
[14]
Qing Li. 2013. A novel Likert scale based on fuzzy sets theory. Expert Systems with Applications 40, 5 (2013), 1609 – 1618.
[15]
Pierre Loslever, Taisa Guidini Gonçalves, Káthia Marçal de Oliveira, and Christophe Kolski. 2019. Using fuzzy coding with qualitative data: example with subjective data in human-computer interaction. Theoretical Issues in Ergonomics Science 20, 4 (2019), 459–488.
[16]
Quenby Mahood, Dwayne Van Eerd, and Emma Irvin. 2014. Searching for grey literature for systematic reviews: challenges and benefits. Research synthesis methods 5, 3 (2014), 221–234.
[17]
Sebastia Massanet, Juan Vicente Riera, Joan Torrens, and Enrique Herrera-Viedma. 2014. A new linguistic computational model based on discrete fuzzy numbers for computing with words. Information Sciences 258(2014), 277–290.
[18]
Laura McAuley, Peter Tugwell, David Moher, 2000. Does the inclusion of grey literature influence estimates of intervention effectiveness reported in meta-analyses?The Lancet 356, 9237 (2000), 1228–1231.
[19]
Ralf W Schlosser. 2006. The role of systematic reviews in evidence-based practice, research, and development. Focus 15(2006), 1–4.
[20]
Joachim Schöpfel and Dominic J Farace. 2010. Grey literature. In Encyclopedia of library and information sciences. 2029–2039.
[21]
M. Shaw. 2003. Writing good software engineering research papers. In Proceedings of 25th International Conference on Software Engineering. 726–736.
[22]
William J. Tastle and Mark J. Wierman. 2007. Consensus and dissention: A measure of ordinal dispersion. International Journal of Approximate Reasoning 45, 3 (2007), 531 – 545.
[23]
Jennifer Tetzlaff, David Moher, Ba Pham, and Douglas Altman. 2006. Survey of views on including grey literature in systematic reviews. In 16th Cochrane Colloquium, Dublin, Ireland.
[24]
J Tyndall and J Tyndall. 2010. AACODS checklist. Flinders University (2010).
[25]
H. Zhang, X. Zhou, X. Huang, H. Huang, and M. A. Babar. 2020. An Evidence-Based Inquiry into the Use of Grey Literature in Software Engineering. In 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE). 1422–1434.
[26]
Xin Zhou. 2020. How to Treat the Use of Grey Literature in Software Engineering. In Proceedings of the International Conference on Software and System Processes(ICSSP ’20). 189–192.

Cited By

View all
  • (2022)RETRACTED ARTICLE: A Novel Framework in Software Engineering for Deep LearningSN Computer Science10.1007/s42979-022-01173-23:4Online publication date: 8-Jun-2022
  • (2022)A dive in white and grey shades of ML and non-ML literature: a multivocal analysis of mathematical expressionsArtificial Intelligence Review10.1007/s10462-022-10330-156:7(7047-7135)Online publication date: 6-Dec-2022

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
EASE '21: Proceedings of the 25th International Conference on Evaluation and Assessment in Software Engineering
June 2021
417 pages
ISBN:9781450390538
DOI:10.1145/3463274
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 June 2021

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Fuzzy rating scale
  2. Grey Literature
  3. Likert scale
  4. Quality Assessment

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Funding Sources

  • Italian Research Group: INdAM-GNCS
  • Italian MIUR PRIN 2017

Conference

EASE 2021

Acceptance Rates

Overall Acceptance Rate 71 of 232 submissions, 31%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)18
  • Downloads (Last 6 weeks)3
Reflects downloads up to 15 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2022)RETRACTED ARTICLE: A Novel Framework in Software Engineering for Deep LearningSN Computer Science10.1007/s42979-022-01173-23:4Online publication date: 8-Jun-2022
  • (2022)A dive in white and grey shades of ML and non-ML literature: a multivocal analysis of mathematical expressionsArtificial Intelligence Review10.1007/s10462-022-10330-156:7(7047-7135)Online publication date: 6-Dec-2022

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media