skip to main content
10.1145/3613905.3637110acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
extended-abstract

Accelerating Scoping Reviews: A Case Study in the User-Centered Design of an AI-Enabled Interdisciplinary Research Tool

Published: 11 May 2024 Publication History

Abstract

This case study presents the user-centered design process our multidisciplinary team of computer scientists, librarians and social scientists engaged in to develop a research tool utilized as part of a scoping review to explore the vast literature on virtual humans. Our process was guided by the Information Systems Research (ISR) Framework to define the parameters of this AI-enabled accessible tool: a semantically organized, interactive evidence map, clustered by salient topics and linked to Google Scholar to expedite the discovery of relevant resources. Specifically, we aimed to achieve several desiderata: (1) replicability, (2) objectivity, (3) automation & scalability, and (4) ease of discovery. Lessons learned include 1) how to apply the ISR Framework to benefit multidisciplinary collaboration, 2) user-centered design benefits from in-house, cross-discipline training, 3) interdisciplinary challenges benefit from multidisciplinary teams, and 4) intentional feedback loops remove the pressure from brainstorming and provide equal opportunity for multidisciplinary input throughout the development life cycle. To enable replicability, we provide full access to the dataset, the AI-enabled interactive evidence map, and source code.

Supplemental Material

MP4 File
Talk Video
Transcript for: Talk Video

References

[1]
Hilary Arksey and Lisa O’Malley. 2005. Scoping studies: towards a methodological framework. International journal of social research methodology 8, 1 (2005), 19–32.
[2]
Gordon Bell, Tony Hey, and Alex Szalay. 2009. Beyond the data deluge. Science 323, 5919 (2009), 1297–1298.
[3]
Elaine Beller, Justin Clark, Guy Tsafnat, Clive Adams, Heinz Diehl, Hans Lund, Mourad Ouzzani, Kristina Thayer, James Thomas, Tari Turner, 2018. Making progress with the automation of systematic reviews: principles of the International Collaboration for the Automation of Systematic Reviews (ICASR). Systematic reviews 7, 1 (2018), 1–7.
[4]
Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150 (2020).
[5]
David Burden and Maggi Savin-Baden. 2019. Virtual humans: Today and tomorrow. CRC Press.
[6]
Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Céspedes, Steve Yuan, Chris Tar, 2018. Universal sentence encoder. arXiv preprint arXiv:1803.11175 (2018).
[7]
Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel S. Weld. 2020. SPECTER: Document-level Representation Learning using Citation-informed Transformers. In ACL.
[8]
Corinna Cortes and Vladimir Vapnik. 1995. Support-vector networks. Machine learning 20, 3 (1995), 273–297.
[9]
Marc Damashek. 1995. Gauging similarity with n-grams: Language-independent categorization of text. Science 267, 5199 (1995), 843–848.
[10]
Sayan Ghosh, Mathieu Chollet, Eugene Laksana, Louis-Philippe Morency, and Stefan Scherer. 2017. Affect-lm: A neural language model for customizable affective text generation. arXiv preprint arXiv:1704.06851 (2017).
[11]
Michael Gusenbauer and Neal R Haddaway. 2020. Which academic search systems are suitable for systematic reviews or meta-analyses? Evaluating retrieval qualities of Google Scholar., and 26 other resources. Research synthesis methods 11, 2 (2020), 181–217.
[12]
Greg Hamerly and Charles Elkan. 2003. Learning the k in k-means. Advances in neural information processing systems 16 (2003), 281–288.
[13]
Florian Heimerl, Steffen Lohmann, Simon Lange, and Thomas Ertl. 2014. Word cloud explorer: Text analytics based on word clouds. In 2014 47th Hawaii International Conference on System Sciences. IEEE, 1833–1842.
[14]
Alan R Hevner. 2007. A three cycle view of design science research. Scandinavian journal of information systems 19, 2 (2007), 4.
[15]
Anthony JG Hey and Anne E Trefethen. 2003. The data deluge: An e-science perspective. (2003).
[16]
Siddhartha R Jonnalagadda, Pawan Goyal, and Mark D Huffman. 2015. Automating data extraction in systematic reviews: a systematic review. Systematic reviews 4, 1 (2015), 1–16.
[17]
Kyo Kageura and Bin Umino. 1996. Methods of automatic term recognition: A review. Terminology. International Journal of Theoretical and Applied Issues in Specialized Communication 3, 2 (1996), 259–289.
[18]
Judith Logan, Jenaya Webb, Nalini Singh, Ben Walsh, Nailisa Tanner, Margaret Wall, and Ana Patricia Ayala. 2021. Scoping review search practices in the social sciences: A scoping review protocol. (2021).
[19]
Sharon Mozgai, Cari Kaurloto, Jade Winn, Andrew Leeds, Dirk Heylen, Arno Hartholt, and Stefan Scherer. 2023. Machine Learning for Semi-Automated Scoping Reviews. Intelligent Systems with Applications (2023), 200249.
[20]
Zachary Munn, Micah DJ Peters, Cindy Stern, Catalin Tufanaru, Alexa McArthur, and Edoardo Aromataris. 2018. Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC medical research methodology 18, 1 (2018), 1–7.
[21]
Matthew J Page, Joanne E McKenzie, Patrick M Bossuyt, Isabelle Boutron, Tammy C Hoffmann, Cynthia D Mulrow, Larissa Shamseer, Jennifer M Tetzlaff, Elie A Akl, Sue E Brennan, 2021. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. Bmj 372 (2021).
[22]
Christopher Peters, Catherine Pelachaud, Elisabetta Bevacqua, Maurizio Mancini, Isabella Poggi, and Universita Roma Tre. 2005. Engagement capabilities for ecas. In AAMAS’05 workshop Creating Bonds with ECAs.
[23]
Micah DJ Peters, Christina M Godfrey, Hanan Khalil, Patricia McInerney, Deborah Parker, and Cassia Baldini Soares. 2015. Guidance for conducting systematic scoping reviews. JBI Evidence Implementation 13, 3 (2015), 141–146.
[24]
Jessica Peterson, Patricia F Pearce, Laurie Anne Ferguson, and Cynthia A Langford. 2017. Understanding scoping reviews: Definition, purpose, and process. Journal of the American Association of Nurse Practitioners 29, 1 (2017), 12–16.
[25]
Lilia Raitskaya and Elena Tikhonova. 2019. Scoping reviews: What is in a name?Journal of Language and Education 5, 2 (2019), 4–9.
[26]
Ville Satopaa, Jeannie Albrecht, David Irwin, and Barath Raghavan. 2011. Finding a" kneedle" in a haystack: Detecting knee points in system behavior. In 2011 31st international conference on distributed computing systems workshops. IEEE, 166–171.
[27]
James Thomas, John McNaught, and Sophia Ananiadou. 2011. Applications of text mining within systematic reviews. Research synthesis methods 2, 1 (2011), 1–14.
[28]
Mercedes Torres Torres and Clive E Adams. 2017. RevManHAL: towards automatic text generation in systematic reviews. Systematic reviews 6, 1 (2017), 1–7.
[29]
Andrea C Tricco, Erin Lillie, Wasifa Zarin, Kelly O’Brien, Heather Colquhoun, Monika Kastner, Danielle Levac, Carmen Ng, Jane Pearson Sharpe, Katherine Wilson, 2016. A scoping review on the conduct and reporting of scoping reviews. BMC medical research methodology 16, 1 (2016), 1–10.
[30]
Guy Tsafnat, Paul Glasziou, Miew Keen Choong, Adam Dunn, Filippo Galgani, and Enrico Coiera. 2014. Systematic review automation technologies. Systematic reviews 3, 1 (2014), 1–15.
[31]
Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE.Journal of machine learning research 9, 11 (2008).
[32]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems. 5998–6008.
[33]
Byron C Wallace, Kevin Small, Carla E Brodley, Joseph Lau, and Thomas A Trikalinos. 2012. Deploying an interactive machine learning system in an evidence-based practice center: abstrackr. In Proceedings of the 2nd ACM SIGHIT international health informatics symposium. 819–824.
[34]
Byron C Wallace, Thomas A Trikalinos, Joseph Lau, Carla Brodley, and Christopher H Schmid. 2010. Semi-automated screening of biomedical citations for systematic reviews. BMC bioinformatics 11, 1 (2010), 1–11.
[35]
Hanna M Wallach. 2006. Topic modeling: beyond bag-of-words. In Proceedings of the 23rd international conference on Machine learning. 977–984.

Index Terms

  1. Accelerating Scoping Reviews: A Case Study in the User-Centered Design of an AI-Enabled Interdisciplinary Research Tool

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Conferences
      CHI EA '24: Extended Abstracts of the CHI Conference on Human Factors in Computing Systems
      May 2024
      4761 pages
      ISBN:9798400703317
      DOI:10.1145/3613905
      Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

      Sponsors

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 11 May 2024

      Check for updates

      Author Tags

      1. Scoping reviews
      2. representation learning
      3. user-centered design
      4. virtual human

      Qualifiers

      • Extended-abstract
      • Research
      • Refereed limited

      Funding Sources

      • US Army
      • NIH NIAAA

      Conference

      CHI '24

      Acceptance Rates

      Overall Acceptance Rate 6,164 of 23,696 submissions, 26%

      Upcoming Conference

      CHI 2025
      ACM CHI Conference on Human Factors in Computing Systems
      April 26 - May 1, 2025
      Yokohama , Japan

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • 0
        Total Citations
      • 262
        Total Downloads
      • Downloads (Last 12 months)262
      • Downloads (Last 6 weeks)31
      Reflects downloads up to 08 Feb 2025

      Other Metrics

      Citations

      View Options

      Login options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Full Text

      View this article in Full Text.

      Full Text

      HTML Format

      View this article in HTML Format.

      HTML Format

      Figures

      Tables

      Media

      Share

      Share

      Share this Publication link

      Share on social media