Skip to main content
SpringerLink
Log in

A Semi-automatic Document Screening System for Computer Science Systematic Reviews

  • Conference paper
  • First Online:
Pattern Recognition and Artificial Intelligence (MedPRAI 2021)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1543))

  • 645 Accesses

Abstract

The elaboration of systematic reviews has become a common practice in computer science after being exclusively related to healthcare and medical sciences. The process incorporates several steps to collect and analyze relevant papers to answer a set of well-formulated research questions. The search process starts by exploring different sources and digital libraries. This often results in a huge number of documents. After deduplication, the metadata of all the retrieved documents are checked for relevance before being approved for inclusion in the review. This task is known to be long and tiresome. In this paper, we propose a semi-automatic system that helps in reducing the efforts required for screening papers. The proposed system combines unsupervised and semi-supervised machine learning models and makes use of the domain ontology. Several features are extracted from metadata and used for classification. With the adoption of semi-supervised learning, researchers are only asked to manually label a subset of retrieved papers. Those papers are used to train a semi-supervised model which can then automatically classify the remaining papers. The proposed system is experimented with seven datasets built from pre-elaborated systematic reviews in computer science. We found that the system can save 50% of the efforts reaching up to 89% in terms of macro F1-score and up to 97% in terms of accuracy.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Semantic Scholar home page: https://www.semanticscholar.org/.

  2. 2.

    The systematic review toolbox: http://systematicreviewtools.com.

  3. 3.

    Springer Nature: https://www.springer.com.

  4. 4.

    Elsevier Science Direct: https://www.elsevier.com/.

  5. 5.

    PubMed - National Library of Medicine: https://pubmed.ncbi.nlm.nih.gov.

  6. 6.

    Semantic Scholar API: https://pypi.org/project/semanticscholar/.

  7. 7.

    LDAMulticore: https://radimrehurek.com/gensim_3.8.3/models/ldamulticore.html.

  8. 8.

    Scopus abstract and citation database: https://www.scopus.com.

  9. 9.

    Datasets for automatic screening of papers: https://github.com/hannousse/Semantic-Scholar-Evaluation.

References

  1. Al-Zubidy, A., Carver, J.C., Hale, D.P., Hassler, E.E.: Vision for SLR tooling infrastructure: prioritizing value-added requirements. Inf. Softw. Technol. 91, 72–81 (2017). https://doi.org/10.1016/j.infsof.2017.06.007

    Article  Google Scholar 

  2. Alhammad, M.M., Moreno, A.M.: Gamification in software engineering education: a systematic mapping. J. Syst. Softw. 141, 131–150 (2018). https://doi.org/10.1016/j.jss.2018.03.065

    Article  Google Scholar 

  3. Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent Dirichlet allocation. J. Mach. Learn. Res. 3, 993–1022 (2003). https://doi.org/10.5555/944919.944937

  4. Cairo, L.S., de Figueiredo Carneiro, G., da Silva, B.C.: Adoption of machine learning techniques to perform secondary studies: a systematic mapping study for the computer science field. In: Filipe, J., Smialek, M., Brodsky, A., Hammoudi, S. (eds.) Proceedings of the 21st International Conference on Enterprise Information Systems, ICEIS 2019, Heraklion, Crete, Greece, 3–5 May 2019, pp. 351–356. SciTePress (2019). https://doi.org/10.5220/0007780603510356

  5. Dieste, O., Padua, A.G.: Developing search strategies for detecting relevant experiments for systematic reviews. In: 2013 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, Madrid, Spain, 20–21 September 2007, pp. 215–224. IEEE Computer Society (2007). https://doi.org/10.1109/ESEM.2007.19

  6. Dybå, T., Kitchenham, B.A., Jørgensen, M.: Evidence-based software engineering for practitioners. IEEE Softw. 22(1), 58–65 (2005). https://doi.org/10.1109/MS.2005.6

    Article  Google Scholar 

  7. Felizardo, K.R., Andery, G.F., Paulovich, F.V., Minghim, R., Maldonado, J.C.: A visual analysis approach to validate the selection review of primary studies in systematic reviews. Inf. Softw. Technol. 54(10), 1079–1091 (2012). https://doi.org/10.1016/j.infsof.2012.04.003

    Article  Google Scholar 

  8. Ghafari, M., Saleh, M., Ebrahimi, T.: A federated search approach to facilitate systematic literature review in software engineering. Int. J. Softw. Eng. Appl. 3(2), 13–24 (2012). https://doi.org/10.5121/ijsea.2012.3202

    Article  Google Scholar 

  9. Ghasemi, M., Amyot, D.: From event logs to goals: a systematic literature review of goal-oriented process mining. Requirements Eng. 25(1), 67–93 (2019). https://doi.org/10.1007/s00766-018-00308-3

    Article  Google Scholar 

  10. Ghawi, R., Pfeffer, J.: Efficient hyperparameter tuning with grid search for text categorization using KNN approach with BM25 similarity. Open Comput. Sci. 9(1), 160–180 (2019). https://doi.org/10.1515/comp-2019-0011

    Article  Google Scholar 

  11. González-Toral, S., Freire, R., Gualán, R., Saquicela, V.: A ranking-based approach for supporting the initial selection of primary studies in a systematic literature review. In: XLV Latin American Computing Conference, CLEI 2019, Panama, Panama, 30 September–4 October 2019, pp. 1–10. IEEE Computer Society (2019). https://doi.org/10.1109/CLEI47609.2019.235079

  12. Goulão, M., Amaral, V., Mernik, M.: Quality in model-driven engineering: a tertiary study. Softw. Qual. J. 24(3), 601–633 (2016). https://doi.org/10.1007/s11219-016-9324-8

    Article  Google Scholar 

  13. Guinea, A.S., Nain, G., Traon, Y.L.: A systematic review on the engineering of software for ubiquitous systems. J. Syst. Softw. 118, 251–276 (2016). https://doi.org/10.1016/j.jss.2016.05.024

    Article  Google Scholar 

  14. Hannousse, A.: Searching relevant papers for software engineering secondary studies: semantic scholar coverage and identification role. IET Softw. 15(1), 126–146 (2021). https://doi.org/10.1049/sfw2.12011

    Article  Google Scholar 

  15. Kitchenham, B.A., Budgen, D., Brereton, P.: Evidence-Based Software Engineering and Systematic Reviews. Chapman & Hall/CRC, Boca Raton (2015)

    Google Scholar 

  16. Marshall, C., Kitchenham, B.A., Brereton, P.: Tool features to support systematic reviews in software engineering - a cross domain study. e-Informatica Softw. Eng. J. 12(1), 79–115 (2018). https://doi.org/10.5277/e-Inf180104

  17. Michelson, M., Reuter, K.: The significant cost of systematic reviews and meta-analyses: a call for greater involvement of machine learning to assess the promise of clinical trials. Contemp. Clin. Trials Commun. 16, 100443 (2019). https://doi.org/10.1016/j.conctc.2019.100443

  18. Molléri, J.S., Petersen, K., Mendes, E.: Towards understanding the relation between citations and research quality in software engineering studies. Scientometrics 117(3), 1453–1478 (2018). https://doi.org/10.1007/s11192-018-2907-3

  19. Oghbaie, M., Mohammadi Zanjireh, M.: Pairwise document similarity measure based on present term set. J. Big Data 5(1), 1–23 (2018). https://doi.org/10.1186/s40537-018-0163-2

  20. Portenoy, J., West, J.D.: Constructing and evaluating automated literature review systems. Scientometrics 125(3), 3233–3251 (2020). https://doi.org/10.1007/s11192-020-03490-w

  21. Salatino, A.A., Thanapalasingam, T., Mannocci, A., Osborne, F., Motta, E.: The computer science ontology: a large-scale taxonomy of research areas. In: Vrandečić, D., et al. (eds.) ISWC 2018. LNCS, vol. 11137, pp. 187–205. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00668-6_12

  22. Santos, J.A.M., Rocha-Junior, J.B., Prates, L.C.L., do Nascimento, R.S., Freitas, M.F., de Mendonça, M.G.: A systematic review on the code smell effect. J. Syst. Softw. 144, 450–477 (2018). https://doi.org/10.1016/j.jss.2018.07.035

  23. Shahin, M., Babar, M.A., Zhu, L.: Continuous integration, delivery and deployment: a systematic review on approaches, tools, challenges and practices. IEEE Access 5, 3909–3943 (2017). https://doi.org/10.1109/ACCESS.2017.2685629

  24. Silva, G., Neto, P.S., Moura, R.S., Araujo, A.C., da Costa Castro, O.C., Ibiapina, I.: An approach to support the selection of relevant studies in systematic review and systematic mappings. In: 8th Brazilian Conference on Intelligent Systems BRACIS 2019, Salvador, Brazil, 15–18 October 2019, pp. 824–829. IEEE Computer Society (2019). https://doi.org/10.1109/BRACIS.2019.00147

  25. Van Dinter, R., Catal, C., Tekinerdogan, B.: A decision support system for automating document retrieval and citation screening. Expert Syst. Appl. 182, 115261 (2021). https://doi.org/10.1016/j.eswa.2021.115261

  26. Vassilvitskii, S., Arthur, D.: k-means++: the advantages of careful seeding. In: Proceedings of the Eighteenth Annual ACM-SIAM Symposium on Discrete Algorithms, New Orleans Louisiana, USA, 7–9 January 2006, pp. 1027–1035. ACM (2006). https://dl.acm.org/doi/10.5555/1283383.1283494

  27. Yang, C., Liang, P., Avgeriou, P.: A systematic mapping study on the combination of software architecture and agile development. J. Syst. Softw. 111, 157–184 (2016). https://doi.org/10.1016/j.jss.2015.09.028

  28. Yu, Z., Kraft, N.A., Menzies, T.: Finding better active learners for faster literature reviews. Empir. Softw. Eng. 23(6), 3161–3186 (2018). https://doi.org/10.1007/s10664-017-9587-0

  29. Yu, Z., Menzies, T.: Fast2: an intelligent assistant for finding relevant papers. Expert Syst. Appl. 120, 57–71 (2019). https://doi.org/10.1016/j.eswa.2018.11.021

  30. Zhu, X.: Semi-supervised learning literature survey. Technical report TR 1530, University of Wisconsin, July 2008

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of 8 May 1945 Gulema, BP 401, 24000, Guelma, Algeria

    Abdelhakim Hannousse

  2. LRS Laboratory, Badji Mokhtar University, BP 12, 23000, Annaba, Algeria

    Salima Yahiouche

Authors
  1. Abdelhakim Hannousse
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Salima Yahiouche
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Abdelhakim Hannousse .

Editor information

Editors and Affiliations

  1. Larbi Tebessi University, Tebessa, Algeria

    Chawki Djeddi

  2. Bahria University, Islamabad, Pakistan

    Imran Siddiqi

  3. FAST National University, Islamabad, Pakistan

    Akhtar Jamil

  4. Istanbul Sabahattin Zaim University, Istanbul, Turkey

    Alaa Ali Hameed

  5. Istanbul Sabahattin Zaim University, Istanbul, Turkey

    İsmail Kucuk

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Hannousse, A., Yahiouche, S. (2022). A Semi-automatic Document Screening System for Computer Science Systematic Reviews. In: Djeddi, C., Siddiqi, I., Jamil, A., Ali Hameed, A., Kucuk, İ. (eds) Pattern Recognition and Artificial Intelligence. MedPRAI 2021. Communications in Computer and Information Science, vol 1543. Springer, Cham. https://doi.org/10.1007/978-3-031-04112-9_15

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-04112-9_15

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-04111-2

  • Online ISBN: 978-3-031-04112-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation