Skip to main content
SpringerLink
Log in

SeSG: a search string generator for Secondary Studies with hybrid search strategies using text mining

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

A Secondary Study (SS) is an important research method used in several areas. A crucial step in the Conduction phase of a SS is the search of studies. This step is time-consuming and error-prone, mainly due to the refinement of the search string. The objective of this study is to validate the effectiveness of an automatic formulation of search strings for SS. Our approach, termed Search String Generator (SeSG), takes as input a small set of studies (as a Quasi-Gold Standard) and processes them using text mining. After that, SeSG generates search strings that deliver a high F1-Score on the start set of a hybrid search strategy. To achieve this objective, we (1) generate a structured textual representation of the initial set of input studies as a bag-of-words using Term Frequency and Document Frequency; (2) perform automatic topic modeling using LDA (Latent Dirichlet Allocation) and enrichment of terms with a pre-trained dense language representation (embedding) called BERT (Bidirectional Encoder Representations from Transformers); (3) formulate and evaluate the search string using the obtained terms; and (4) use the developed search strings in a digital library. For the validation of our approach, we conduct an experiment—using some SS as objects—comparing the effectiveness of automatically formulated search strings by SeSG with manual search strings reported in these studies. SeSG generates search strings that achieve a better final F1-Score on the start set than the searches reported by these SS. Our study shows that SeSG can effectively supersede the formulation of search strings, in hybrid search strategies, since it dismisses the manual string refinements.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Notes

  1. https://dev.elsevier.com/

  2. https://github.com/zhiyzuo/python-SCOPUS

  3. http://cermine.ceon.pl/index.html

  4. https://scikit-learn.org

  5. https://github.com/sesg-creator/SeSG

References

  • Aggarwal CC, Zhai CX (2012) Mining text data. Springer Science+Business Media

  • Agrawal R, Srikant R (1994) Fast algorithms for mining association rules. In: Proceedings of the 20th international conference on very large data bases (VLDB), vol 1215, pp 487–499

  • Ali NB, Usman M (2018) Reliability of search in systematic reviews: towards a quality assessment framework for the automated-search strategy. Inf Softw Technol 99:133–147

    Article  Google Scholar 

  • Ampatzoglou A, Bibi S, Avgeriou P, Verbeek M, Chatzigeorgiou A (2019) Identifying, categorizing and mitigating threats to validity in software engineering secondary studies. Inf Softw Technol 106:201–230

    Article  Google Scholar 

  • Arampatzis A, Van Der Weide TP, van Bommel P, Koster CH (1999) Linguistically-motivated information retrieval. Encycl Library Inf Sci 69:201–222

    Google Scholar 

  • Azeem MI, Palomba F, Shi L, Wang Q (2019) Machine learning techniques for code smell detection: a systematic literature review and meta-analysis. Inf Softw Technol 108:115–138

    Article  Google Scholar 

  • Babar MA, Zhang H (2009) Systematic literature reviews in software engineering: preliminary results from interviews with researchers. In: Proceedings of the 2009 3rd international symposium on empirical software engineering and measurement. IEEE Computer Society, pp 346–355

  • Badampudi D, Wohlin C, Petersen K (2015) Experiences from using snowballing and database searches in systematic literature studies. In: Proceedings of the 19th international conference on evaluation and assessment in software engineering, pp 1–10

  • Bengio Y, Ducharme R, Vincent P, Jauvin C (2003) A neural probabilistic language model. J Mach Learn Res 3:1137–1155

    MATH  Google Scholar 

  • Blei DM, Ng AY, Jordan MI (2003) Latent dirichlet allocation. J Mach Learn Res 3:993–1022

    MATH  Google Scholar 

  • Booth A (2016) Searching for qualitative research for inclusion in systematic reviews: a structured methodological review. Syst Rev 5(1):74

    Article  Google Scholar 

  • Bramer WM, Rethlefsen ML, Kleijnen J, Franco OH (2017) Optimal database combinations for literature searches in systematic reviews: a prospective exploratory study. Syst Rev 6(1):1–12

    Article  Google Scholar 

  • Briscoe S, Bethel A, Rogers M (2020) Conduct and reporting of citation searching in cochrane systematic reviews: a cross-sectional study. Res Synthesis Methods 11(2):169–180

    Article  Google Scholar 

  • Campbell DT, Cook TD (1979) Quasi-experimentation: design and analysis issues for field settings. Houghton Mifflin Company, Dallas

    Google Scholar 

  • Chang AA, Heskett KM, Davidson TM (2006) Searching the literature using medical subject headings versus text word with pubmed. Laryngoscope 116(2):336–340

    Article  Google Scholar 

  • Collobert R, Weston J (2008) A unified architecture for natural language processing: deep neural networks with multitask learning. In: Proceedings of the 25th international conference on machine learning, pp 160–167

  • Collobert R, Weston J, Bottou L, Karlen M, Kavukcuoglu K, Kuksa P (2011) Natural language processing (almost) from scratch. J Mach Learn Res 12:2493–2537

    MATH  Google Scholar 

  • Cooper C, Booth A, Britten N, Garside R (2017) A comparison of results of empirical studies of supplementary search techniques and recommendations in review methodology handbooks: a methodological review. Syst Rev 6(1):1–16

    Article  Google Scholar 

  • Cooper C, Booth A, Varley-Campbell J, Britten N, Garside R (2018) Defining the process to literature searching in systematic reviews: a literature review of guidance and supporting studies. BMC Med Res Methodol 18(1):85

    Article  Google Scholar 

  • Devlin J, Chang MW, Lee K, Toutanova K (2019) Bert: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 conference of the north American chapter of the association for computational linguistics: human language technologies, pp 4171–4186

  • Dickersin K, Scherer R, Lefebvre C (1994) Identifying relevant studies for systematic reviews. BMJ (Clinical research ed) 309:1286–91

    Article  Google Scholar 

  • Dieste O, Padua AG (2007) Developing search strategies for detecting relevant experiments for systematic reviews. In: First international symposium on empirical software engineering and measurement (ESEM 2007), pp 215–224

  • Dieste O, Grimán A, Juristo N (2009) Developing search strategies for detecting relevant experiments. Empir Softw Eng 14(5):513–539. https://doi.org/10.1007/s10664-008-9091-7

    Article  Google Scholar 

  • Feng L, Chiam YK, Lo SK (2017) Text-mining techniques and tools for systematic literature reviews: a systematic literature review. In: 2017 24th Asia-Pacific software engineering conference (APSEC), pp 41–50

  • Ghafari M, Saleh M, Ebrahimi T (2012) A federated search approach to facilitate systematic literature review in software engineering. Int J Softw Eng Appl (IJSEA) 3(2):13–24

    Google Scholar 

  • Gonzalez MAI, de Lima VLS, de Lima JV (2006) Tools for nominalization: an alternative for lexical normalization. In: Computational processing of the Portuguese language, pp 100–109

  • Grames EM, Stillman AN, Tingley MW, Elphick CS (2019) An automated approach to identifying search terms for systematic reviews using keyword co-occurrence networks. Methods Ecol Evol 10(10):1645–1654

    Article  Google Scholar 

  • Haynes RB, Kastner M, Wilczynski NL (2005) Developing optimal search strategies for detecting clinically sound and relevant causation studies in embase. BMC Med Inform Decis Making 5(1):1–7

    Article  Google Scholar 

  • Horsley T, Dingwall O, Sampson M (2011) Checking reference lists to find additional studies for systematic reviews. Cochrane Database of Systematic Reviews (8)

  • Hosseini S, Turhan B, Gunarathna D (2019) A systematic literature review and meta-analysis on cross project defect prediction. IEEE Trans Softw Eng 45(2):111–147

    Article  Google Scholar 

  • Imtiaz S, Bano M, Ikram N, Niazi M (2013) A tertiary study: experiences of conducting systematic literature reviews in software engineering. In: Proceedings of the 17th international conference on evaluation and assessment in software engineering, pp 177–182

  • Jones KS, Willett P (1997) Readings in information retrieval. Morgan Kaufmann Publishers Inc., San Francisco

    Google Scholar 

  • Juristo N, Moreno AM (2001) Basics of software engineering experimentation. Springer Science & Business Media

  • Kitchenham B (2004) Procedures for performing systematic reviews, vol 33. Keele University, Keele, pp 1–26

    Google Scholar 

  • Kitchenham B, Charters S (2007) Guidelines for performing systematic literature reviews in software engineering. Technical Report—Department of Computer Science, University of Durham

  • Kitchenham BA, Li Z, Burn AJ (2011) Validating search processes in systematic literature reviews. In: EAST, pp 3–9

  • Kitchenham BA, Budgen D, Brereton P (2015) Evidence-based software engineering and systematic reviews. Chapman & Hall/CRC

  • Krüger J, Lausberger C, von Nostitz-Wallwitz I, Saake G, Leich T (2020) Search. Review. Repeat? An empirical study of threats to replicating slr searches. Empir Softw Eng 25(1):627–677

    Article  Google Scholar 

  • Krovetz R (1993) Viewing morphology as an inference process. In: Proceedings of the 16th annual international ACM SIGIR conference on research and development in information retrieval, pp 191–202

  • Kuhrmann M, Fernández DM, Daneva M (2017) On the pragmatic design of literature studies in software engineering: an experience-based guideline. Empir Softw Eng 22(6):2852–2891

    Article  Google Scholar 

  • Kuper H, Nicholson A, Hemingway H (2006) Searching for observational studies: what does citation tracking add to pubmed? A case study in depression and coronary heart disease. BMC Med Res Methodol 6(1):1–4

    Article  Google Scholar 

  • Laguna MdSC, Pardo TAS, Rezende SO (2014) Extração automática de termos simples baseada em aprendizado de máquina. Doctoral thesis in ciências de computação e matemática computacional, Instituto de Ciências Matemáticas e de Computação, University of São Paulo, São Carlos, SP

  • Lee J, Yoon W, Kim S, Kim D, Kim S, So CH, Kang J (2019) Biobert: a pre-trained biomedical language representation model for biomedical text mining. arXiv:190108746

  • Luhn HP (1958) The automatic creation of literature abstracts. IBM J Res Dev 2(2):159–165

    Article  MathSciNet  Google Scholar 

  • Manning C, Raghavan P, Schütze H (2010) Introduction to information retrieval. Nat Lang Eng 16(1):100–103

    MATH  Google Scholar 

  • Marcos-Pablos S, García-Peñalvo FJ (2018) Information retrieval methodology for aiding scientific database search. Soft Comput 1–10

  • Marshall C, Brereton P (2013) Tools to support systematic literature reviews in software engineering: a mapping study. In: 2013 ACM/IEEE international symposium on empirical software engineering and measurement, pp 296–299

  • Marshall C, Brereton P, Kitchenham B (2014) Tools to support systematic reviews in software engineering: a feature analysis. In: Proceedings of the 18th international conference on evaluation and assessment in software engineering, association for computing machinery

  • McCann B, Bradbury J, Xiong C, Socher R (2017) Learned in translation: contextualized word vectors. In: Proceedings of the 31st international conference on neural information processing systems, pp 6297–6308

  • Mergel GD, Silveira MS, da Silva TS (2015) A method to support search string building in systematic literature reviews through visual text mining. In: Proceedings of the 30th annual ACM symposium on applied computing. Association for Computing Machinery, pp 1594–1601

  • Mikolov T, Sutskever I, Chen K, Corrado GS, Dean J (2013) Distributed representations of words and phrases and their compositionality. In: Proceedings of the 26th international conference on neural information processing systems, vol 2, pp 3111–3119

  • Mikolov T, Grave E, Bojanowski P, Puhrsch C, Joulin A (2018) Advances in pre-training distributed word representations. In: Proceedings of the eleventh international conference on language resources and evaluation. European Language Resources Association (ELRA)

  • Montgomery DC (2017) Design and analysis of experiments. Wiley, New York

    Google Scholar 

  • Montgomery DC, Runger GC (2018) Applied statistics and probability for engineers. Wiley, New York

    MATH  Google Scholar 

  • Mourão E, Kalinowski M, Murta L, Mendes E, Wohlin C (2017) Investigating the use of a hybrid search strategy for systematic reviews. In: 2017 ACM/IEEE international symposium on empirical software engineering and measurement, pp 193–198

  • Mourão E, Pimentel JF, Murta L, Kalinowski M, Mendes E, Wohlin C (2020) On the performance of hybrid search strategies for systematic literature reviews in software engineering. Inf Softw Technol 106–294

  • Münch J, Armbrust O, Kowalczyk M, Soto M (2012) Software process definition and management. Springer Publishing Company, Incorporated

  • Nogueira BM (2009) Avaliação de métodos não-supervisionados de seleção de atributos para mineração de textos. Doctoral thesis in ciências de computação e matemática computacional, Instituto de ciências matemáticas e de computação University of São Paulo, São Carlos

  • Nogueira BM (2013) Hierarchical semi-supervised confidence-based active clustering and its application to the extraction of topic hierarchies from document collections. Masters dissertation in ciências de computação e matemática computacional, Instituto de ciências matemáticas e de computação University of São Paulo São Carlos

  • O’Mara-Eves A, Thomas J, McNaught J, Miwa M, Ananiadou S (2015) Using text mining for study identification in systematic reviews: a systematic review of current approaches. Syst Rev 4(1):5

    Article  Google Scholar 

  • Pennington J, Socher R, Manning C (2014) Glove: global vectors for word representation. In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp 1532–1543

  • Peters ME, Neumann M, Iyyer M, Gardner M, Clark C, Lee K, Zettlemoyer L (2018) Deep contextualized word representations. In: Proceedings of the 2018 conference of the North American chapter of the association for computational linguistics: human language technologies, vol 1, pp 2227–2237

  • Petitti DB, et al. (2000) Meta-analysis, decision analysis, and cost-effectiveness analysis: methods for quantitative synthesis in medicine. 31, OUP USA

  • Pravin A, Srinivasan S (2012) Detecting of software bugs in source code using data mining approach. Natl J Syst Inf Technol 6(1):1–8

    Google Scholar 

  • Relevo R (2012) Effective search strategies for systematic reviews of medical tests. J Gen Internal Med 27(1):28–32

    Article  Google Scholar 

  • Rezende SO (2003) Sistemas inteligentes: fundamentos e aplicações. Editora Manole Ltda, Barueri

    Google Scholar 

  • Rogers A, Kovaleva O, Rumshisky A (2020) A primer in bertology: what we know about how bert works. Trans Assoc Comput Linguist 8:842–866. https://doi.org/10.1162/tacl_a_00349

    Article  Google Scholar 

  • Ros R, Bjarnason E, Runeson P (2017) A machine learning approach for semi-automated search and selection in literature studies. In: Proceedings of the 21st international conference on evaluation and assessment in software engineering. Association for Computing Machinery, pp 118–127

  • Salton G, Wong A, Yang CS (1975) A vector space model for automatic indexing. Commun ACM 18(11):613–620

    Article  MATH  Google Scholar 

  • Shapiro SS, Wilk MB (1965) An analysis of variance test for normality (complete samples). Biometrika 52(3/4):591–611

    Article  MathSciNet  MATH  Google Scholar 

  • Shearer C (2000) The crisp-dm model: the new blueprint for data mining. J Data Warehous 5(4):13–22

    Google Scholar 

  • Smalheiser NR, Lin C, Jia L, Jiang Y, Cohen AM, Yu C, Davis JM, Adams CE, McDonagh MS, Meng W (2014) Design and implementation of metta, a metasearch engine for biomedical literature retrieval intended for systematic reviewers. Health Inf Sci Syst 2(1):1

    Article  Google Scholar 

  • Stansfield C, O’Mara-Eves A, Thomas J (2017) Text mining for search term development in systematic reviewing: a discussion of some methods and challenges. Res Synth Methods 8(3):355–365

    Article  Google Scholar 

  • Sullivan GM, Feinn R (2012) Using effect size—or why the p value is not enough. J Grad Med Educ 4(3):279–282

    Article  Google Scholar 

  • Tomassetti F, Rizzo G, Vetro A, Ardito L, Torchiano M, Morisio M (2011) Linked data approach for selection process automation in systematic reviews. In: 15th Annual conference on evaluation & assessment in software engineering (EASE 2011), pp 31–35

  • Trochim WM, Donnelly JP (2020) Research methods knowledge base. https://conjointly.com/kb/ (version current as of 27 April 2020)

  • Tsafnat G, Glasziou P, Choong MK, Dunn A, Galgani F, Coiera E (2014) Systematic review automation technologies. Syst Rev 3(1):74

    Article  Google Scholar 

  • Turian J, Ratinov L, Bengio Y (2010) Word representations: a simple and general method for semi-supervised learning. In: Proceedings of the 48th annual meeting of the association for computational linguistics. Association for Computational Linguistics, pp 384–394

  • van Rijsbergen C (1979) Information retrieval. http://www.dcs.gla.ac.uk/Keith/Preface.html. Accessed 7 July 2020

  • Vasconcellos FJ, Landre GB, Cunha JAO, Oliveira JL, Ferreira RA, Vincenzi AM (2017) Approaches to strategic alignment of software process improvement: a systematic literature review. J Syst Softw 123:45–63

    Article  Google Scholar 

  • Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser U, Polosukhin I (2017) Attention is all you need. In: Proceedings of the 31st international conference on neural information processing systems, , NIPS’17. Curran Associates Inc., Red Hook, pp 6000–6010

  • Wang J, Wang Q (2016) Analyzing and predicting software integration bugs using network analysis on requirements dependency network. Requir Eng 21(2):161–184

    Article  MathSciNet  Google Scholar 

  • Weiss SM, Indurkhya N (1998) Predictive data mining: a practical guide. Morgan Kaufmann Publishers Inc., San Francisco

    MATH  Google Scholar 

  • Weiss SM, Indurkhya N, Zhang T, Damerau F (2010) Text mining: predictive methods for analyzing unstructured information, 1st edn. Springer Publishing Company, Incorporated

  • White VJ, Glanville JM, Lefebvre C, Sheldon TA (2001) A statistical approach to designing search filters to find systematic reviews: objectivity enhances accuracy. J Inf Sci 27(6):357–370

    Article  Google Scholar 

  • Wieringa RJ (2014) Design science methodology for information systems and software engineering. Springer, Berlin

    Book  Google Scholar 

  • Wieringa R, Daneva M (2015) Six strategies for generalizing software engineering theories. Sci Comput Program 101:136–152

    Article  Google Scholar 

  • Wilczynski NL, Haynes RB (2005) Embase search strategies for identifying methodologically sound diagnostic studies for use by clinicians and researchers. BMC Med 3(1):1–6

    Article  Google Scholar 

  • Wohlin C (2014) Guidelines for snowballing in systematic literature studies and a replication in software engineering. In: Proceedings of the 18th international conference on evaluation and assessment in software engineering. Association for Computing Machinery

  • Wohlin C, Runeson P, Höst M, Ohlsson MC, Regnell B, Wesslén A (2012) Experimentation in software engineering. Springer Publishing Company, Incorporated

  • Wolf T, Debut L, Sanh V, Chaumond J, Delangue C, Moi A, Cistac P, Rault T, Louf R, Funtowicz M, Davison J, Shleifer S, von Platen P, Ma C, Jernite Y, Plu J, Xu C, Le Scao T, Gugger S, Drame M, Lhoest Q, Rush A (2020) Transformers: state-of-the-art natural language processing. In: Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations. Association for Computational Linguistics, pp 38–45

  • Yang L, Zhang H, Shen H, Huang X, Zhou X, Rong G, Shao D (2021) Quality assessment in systematic literature reviews: a software engineering perspective. Inf Softw Technol 130:106397. https://doi.org/10.1016/j.infsof.2020.106397. https://www.sciencedirect.com/science/article/pii/S0950584920301610

    Article  Google Scholar 

  • Zhang H, Babar MA (2011) An empirical investigation of systematic reviews in software engineering. In: 2011 International symposium on empirical software engineering and measurement. https://doi.org/10.1109/ESEM.2011.17, pp 87–96

  • Zhang H, Babar MA, Tell P (2011) Identifying relevant studies in software engineering. Inf Softw Technol 53(6):625–637

    Article  Google Scholar 

  • Zwakman M, Verberne LM, Kars MC, Hooft L, van Delden JJ, Spijker R (2018) Introducing palette: an iterative method for conducting a literature search for a review in palliative care. BMC Palliative Care 17(1):1–9

    Article  Google Scholar 

Download references

Acknowledgements

The authors would like to thank the Coordination for the Improvement of Higher Education Personnel (CAPES) for financially assisting this study.

Author information

Authors and Affiliations

  1. Faculty of Computing, Federal University of Mato Grosso do Sul, Campo Grande, MS, Brazil

    Leonardo Fuchs Alves, Francisco J. S. Vasconcellos & Bruno Magalhães Nogueira

Authors
  1. Leonardo Fuchs Alves
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Francisco J. S. Vasconcellos
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bruno Magalhães Nogueira
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Leonardo Fuchs Alves.

Additional information

Communicated by: Tim Menzies

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This work was supported by the Coordination for the Improvement of Higher Education Personnel (CAPES) Grant No. 88882.466597/2019-01.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Alves, L.F., Vasconcellos, F.J.S. & Nogueira, B.M. SeSG: a search string generator for Secondary Studies with hybrid search strategies using text mining. Empir Software Eng 27, 105 (2022). https://doi.org/10.1007/s10664-021-10084-4

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10664-021-10084-4

Keywords

Search

Navigation