Integrating measures of replicability into scholarly search: Challenges and opportunities
Article No.: 18, Pages 1 - 18
Abstract
Challenges to reproducibility and replicability have gained widespread attention, driven by large replication projects with lukewarm success rates. A nascent work has emerged developing algorithms to estimate the replicability of published findings. The current study explores ways in which AI-enabled signals of confidence in research might be integrated into the literature search. We interview 17 PhD researchers about their current processes for literature search and ask them to provide feedback on a replicability estimation tool. Our findings suggest that participants tend to confuse replicability with generalizability and related concepts. Information about replicability can support researchers throughout the research design processes. However, the use of AI estimation is debatable due to the lack of explainability and transparency. The ethical implications of AI-enabled confidence assessment must be further studied before such tools could be widely accepted. We discuss implications for the design of technological tools to support scholarly activities and advance replicability.
Supplemental Material
MP4 File - Video Preview
Video Preview
- Download
- 4.01 MB
- Download
- 46.15 MB
- Transcript
MP4 File - Video Demonstration
Video demonstration of the tool used in the study.
- Download
- 80.57 MB
References
[1]
Ahmed Al-Zubidy and Jeffrey C Carver. 2019. Identification and prioritization of SLR search tool requirements: an SLR and a survey. Empirical Software Engineering 24 (2019), 139–169.
[2]
Sajid Ali, Tamer Abuhmed, Shaker El-Sappagh, Khan Muhammad, Jose M Alonso-Moral, Roberto Confalonieri, Riccardo Guidotti, Javier Del Ser, Natalia Díaz-Rodríguez, and Francisco Herrera. 2023. Explainable Artificial Intelligence (XAI): What we know and what is left to attain Trustworthy Artificial Intelligence. Information Fusion 99 (2023), 101805.
[3]
Adam Altmejd, Anna Dreber, Eskil Forsell, Juergen Huber, Taisuke Imai, Magnus Johannesson, Michael Kirchler, Gideon Nave, and Colin Camerer. 2019. Predicting the replicability of social science lab experiments. PloS one 14, 12 (2019), e0225826.
[4]
Tawfiq Ammari and Sarita Schoenebeck. 2015. Understanding and supporting fathers and fatherhood on social media sites. In Proceedings of the 33rd annual ACM conference on human factors in computing systems. 1905–1914.
[5]
Alia Arshad and Kanwal Ameen. 2019. Scholarly information seeking of academic engineers and technologists. International Information & Library Review 51, 1 (2019), 1–8.
[6]
Kumaripaba Athukorala, Dorota Głowacka, Giulio Jacucci, Antti Oulasvirta, and Jilles Vreeken. 2016. Is exploratory search different? A comparison of information search behavior for exploratory and lookup tasks. Journal of the Association for Information Science and Technology 67, 11 (2016), 2635–2651.
[7]
Kumaripaba Athukorala, Eve Hoggan, Anu Lehtiö, Tuukka Ruotsalo, and Giulio Jacucci. 2013. Information-seeking behaviors of computer scientists: Challenges for electronic literature search tools. Proceedings of the American Society for Information Science and Technology 50, 1 (2013), 1–11.
[8]
Lauren Z. Atkinson and Andrea Cipriani. 2018. How to carry out a literature search for a systematic review: a practical guide. BJPsych Advances 24, 2 (March 2018), 74–82. https://doi.org/10.1192/bja.2017.3
[9]
Monya Baker. 2016. 1,500 scientists lift the lid on reproducibility. Nature 533, 7604 (May 2016), 452–454. https://doi.org/10.1038/533452a Number: 7604 Publisher: Nature Publishing Group.
[10]
Monya Baker. 2016. Reproducibility crisis. Nature 533, 26 (2016), 353–66.
[11]
Michael J. Baker. 2000. Writing a literature review. The marketing review 1, 2 (2000), 219–247. ISBN: 1469-347X Publisher: Westburn Publishers Ltd.
[12]
Robert E Bartholomew. 2014. Science for sale: the rise of predatory journals., 384–385 pages.
[13]
James Berry, Lucas C Coffman, Douglas Hanley, Rania Gihleb, and Alistair J Wilson. 2017. Assessing the rate of replication in economics. American Economic Review 107, 5 (2017), 27–31.
[14]
Steven Bethard and Dan Jurafsky. 2010. Who should I cite: learning literature search models from citation behavior. In Proceedings of the 19th ACM international conference on Information and knowledge management. ACM, Toronto ON Canada, 609–618. https://doi.org/10.1145/1871437.1871517
[15]
Abeba Birhane, Atoosa Kasirzadeh, David Leslie, and Sandra Wachter. 2023. Science in the age of large language models. Nature Reviews Physics (2023), 1–4. ISBN: 2522-5820 Publisher: Nature Publishing Group UK London.
[16]
Martin Boeker, Werner Vach, and Edith Motschall. 2013. Google Scholar as replacement for systematic literature searches: good relative recall and precision are not enough. BMC medical research methodology 13, 1 (2013), 1–12. ISBN: 1471-2288 Publisher: BioMed Central.
[17]
Lutz Bornmann. 2014. Do altmetrics point to the broader impact of research? An overview of benefits and disadvantages of altmetrics. Journal of informetrics 8, 4 (2014), 895–903. ISBN: 1751-1577 Publisher: Elsevier.
[18]
Rotem Botvinik-Nezer, Felix Holzmeister, Colin F Camerer, Anna Dreber, Juergen Huber, Magnus Johannesson, Michael Kirchler, Roni Iwanir, Jeanette A Mumford, R Alison Adcock, 2020. Variability in the analysis of a single neuroimaging dataset by many teams. Nature 582, 7810 (2020), 84–88.
[19]
Samuel R Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, Scott Heiner, Kamilė Lukošiūtė, Amanda Askell, Andy Jones, Anna Chen, 2022. Measuring progress on scalable oversight for large language models. arXiv preprint arXiv:2211.03540 (2022).
[20]
Wichor M. Bramer, Gerdien B. de Jonge, Melissa L. Rethlefsen, Frans Mast, and Jos Kleijnen. 2018. A systematic approach to searching: an efficient and complete method to develop literature searches. Journal of the Medical Library Association : JMLA 106, 4 (Oct. 2018), 531–541. https://doi.org/10.5195/jmla.2018.283
[21]
Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative research in psychology 3, 2 (2006), 77–101.
[22]
Jan vom Brocke, Alexander Simons, Bjoern Niehaves, Bjorn Niehaves, Kai Reimer, Ralf Plattfaut, and Anne Cleven. 2009. Reconstructing the giant: On the importance of rigour in documenting the literature search process. (2009).
[23]
Colin F. Camerer, Anna Dreber, Eskil Forsell, Teck-Hua Ho, Jürgen Huber, Magnus Johannesson, Michael Kirchler, Johan Almenberg, Adam Altmejd, and Taizan Chan. 2016. Evaluating replicability of laboratory experiments in economics. Science 351, 6280 (2016), 1433–1436. ISBN: 0036-8075 Publisher: American Association for the Advancement of Science.
[24]
Colin F Camerer, Anna Dreber, Eskil Forsell, Teck-Hua Ho, Jürgen Huber, Magnus Johannesson, Michael Kirchler, Johan Almenberg, Adam Altmejd, Taizan Chan, 2016. Evaluating replicability of laboratory experiments in economics. Science 351, 6280 (2016), 1433–1436.
[25]
Colin F Camerer, Anna Dreber, Felix Holzmeister, Teck-Hua Ho, Jürgen Huber, Magnus Johannesson, Michael Kirchler, Gideon Nave, Brian A Nosek, Thomas Pfeiffer, 2018. Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nature human behaviour 2, 9 (2018), 637–644.
[26]
Daniel N Cassenti and Lance M Kaplan. 2021. Robust uncertainty representation in human-AI collaboration. In Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications III, Vol. 11746. SPIE, 249–262.
[27]
Julen Cestero, David Velásquez, Elizabeth Suescún, Mikel Maiza, and Marco Quartulli. 2022. Pysurveillance: A Novel Tool for Supporting Researchers in the Systematic Literature Review Process. Advanced Intelligent Technologies for Industry (2022), 239–248.
[28]
Joseph Chee Chang, Amy X. Zhang, Jonathan Bragg, Andrew Head, Kyle Lo, Doug Downey, and Daniel S. Weld. 2023. CiteSee: Augmenting Citations in Scientific Papers with Persistent and Personalized Historical Context. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. ACM, Hamburg Germany, 1–15. https://doi.org/10.1145/3544548.3580847
[29]
Catherine Chen and Carsten Eickhoff. 2022. Evaluating Search Explainability with Psychometrics and Crowdsourcing. arXiv preprint arXiv:2210.09430 (2022).
[30]
Kiroong Choe, Seokweon Jung, Seokhyeon Park, Hwajung Hong, and Jinwook Seo. 2021. Papers101: Supporting the Discovery Process in the Literature Review Workflow for Novice Researchers. In 2021 IEEE 14th Pacific Visualization Symposium (PacificVis). 176–180. https://doi.org/10.1109/PacificVis52677.2021.00037 ISSN: 2165-8773.
[31]
Yu-Liang Chou, Catarina Moreira, Peter Bruza, Chun Ouyang, and Joaquim Jorge. 2022. Counterfactuals and causability in explainable artificial intelligence: Theory, algorithms, and applications. Information Fusion 81 (May 2022), 59–83. https://doi.org/10.1016/j.inffus.2021.11.003
[32]
Morris F Cohen. 2011. An introduction to logic and scientific method. Read Books Ltd.
[33]
Nicki Lisa Cole, Sven Ulpts, Tony Ross-Hellauer, Agata Bochynska, and Thomas Klebel. 2023. Integrative review of conceptions and facilitators of and barriers to reproducibility of qualitative research. (July 2023). https://doi.org/10.17605/OSF.IO/Q4XWK Publisher: OSF.
[34]
Open Science Collaboration. 2015. Estimating the reproducibility of psychological science. Science 349, 6251 (2015), aac4716. ISBN: 0036-8075 Publisher: American Association for the Advancement of Science.
[35]
Open Science Collaboration. 2015. Estimating the reproducibility of psychological science. Science 349, 6251 (2015), aac4716.
[36]
Christian Collberg, Todd Proebsting, Gina Moraila, Akash Shankaran, Zuoming Shi, and Alex M Warren. 2014. Measuring reproducibility in computer systems research. Department of Computer Science, University of Arizona, Tech. Rep 37 (2014).
[37]
David Colquhoun. 2017. The reproducibility of research and the misinterpretation of p-values. Royal society open science 4, 12 (2017), 171085.
[38]
Chris Cooper, Andrew Booth, Jo Varley-Campbell, Nicky Britten, and Ruth Garside. 2018. Defining the process to literature searching in systematic reviews: a literature review of guidance and supporting studies. BMC Medical Research Methodology 18, 1 (Aug. 2018), 85. https://doi.org/10.1186/s12874-018-0545-3
[39]
Florian Cova, Brent Strickland, Angela Abatista, Aurélien Allard, James Andow, Mario Attie, James Beebe, Renatas Berniūnas, Jordane Boudesseul, Matteo Colombo, 2021. Estimating the reproducibility of experimental philosophy. Review of Philosophy and Psychology 12 (2021), 9–44.
[40]
John W Creswell and Dana L Miller. 2000. Determining validity in qualitative inquiry. Theory into practice 39, 3 (2000), 124–130.
[41]
Harriet Dashnow, Andrew Lonsdale, and Philip E. Bourne. 2014. Ten Simple Rules for Writing a PLOS Ten Simple Rules Article. PLOS Computational Biology 10, 10 (Oct. 2014), e1003858. https://doi.org/10.1371/journal.pcbi.1003858 Publisher: Public Library of Science.
[42]
Fabrice De Chaumont, Stéphane Dallongeville, Nicolas Chenouard, Nicolas Hervé, Sorin Pop, Thomas Provoost, Vannary Meas-Yedid, Praveen Pankajakshan, Timothée Lecomte, and Yoann Le Montagner. 2012. Icy: an open bioimage informatics platform for extended reproducible research. Nature methods 9, 7 (2012), 690–696. ISBN: 1548-7091 Publisher: Nature Publishing Group US New York.
[43]
Nurullah Demir, Matteo Große-Kampmann, Tobias Urban, Christian Wressnegger, Thorsten Holz, and Norbert Pohlmann. 2022. Reproducibility and Replicability of Web Measurement Studies. In Proceedings of the ACM Web Conference 2022. ACM, Virtual Event, Lyon France, 533–544. https://doi.org/10.1145/3485447.3512214
[44]
Berna Devezer, Luis G Nardin, Bert Baumgaertner, and Erkan Ozge Buzbas. 2019. Scientific discovery in a model-centric framework: Reproducibility, innovation, and epistemic diversity. PloS one 14, 5 (2019), e0216125.
[45]
Xianghua Ding, Yubo Kou, Yiwen Xu, and Peng Zhang. 2022. “As Uploaders, We Have the Responsibility”: Individualized Professionalization of Bilibili Uploaders. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems. 1–14.
[46]
Cody Dunne, Ben Shneiderman, Robert Gove, Judith Klavans, and Bonnie Dorr. 2012. Rapid understanding of scientific paper collections: Integrating statistics, text analytics, and visualization. Journal of the American Society for Information Science and Technology 63, 12 (2012), 2351–2369. https://doi.org/10.1002/asi.22652 _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/asi.22652.
[47]
Charles G. Durbin. 2009. How to read a scientific research paper. Respiratory care 54, 10 (2009), 1366–1371. ISBN: 0020-1324 Publisher: Respiratory Care.
[48]
Upol Ehsan, Q Vera Liao, Michael Muller, Mark O Riedl, and Justin D Weisz. 2021. Expanding explainability: Towards social transparency in ai systems. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–19.
[49]
David Eriksson. 2023. The art and science of scholarly writing: framing symmetry of specificity beyond IMRAD. European Business Reviewahead-of-print (2023).
[50]
Timothy M Errington, Elizabeth Iorns, William Gunn, Fraser Elisabeth Tan, Joelle Lomax, and Brian A Nosek. 2014. An open investigation of the reproducibility of cancer biology research. Elife 3 (2014), e04333.
[51]
Andres Fortino, Qitong Zhong, Luke Yeh, and Sijia Fang. 2020. Using Text Data Mining to Enhance the Literature Search Process for Novice STEM Researchers. In 2020 IEEE Integrated STEM Education Conference (ISEC). IEEE, 1–6.
[52]
Erin D. Foster and Ariel Deardorff. 2017. Open science framework (OSF). Journal of the Medical Library Association: JMLA 105, 2 (2017), 203. Publisher: Medical Library Association.
[53]
Jeremy Freese, Tamkinat Rauf, and Jan Gerrit Voelkel. 2022. Advances in transparency and reproducibility in the social sciences. Social Science Research 107 (2022), 102770.
[54]
Darcy E Furlong and Jessica Nina Lester. 2023. Toward a Practice of Qualitative Methodological Literature Reviewing. Qualitative Inquiry 29, 6 (2023), 669–677.
[55]
Niloy Ganguly, Dren Fazlija, Maryam Badar, Marco Fisichella, Sandipan Sikdar, Johanna Schrader, Jonas Wallat, Koustav Rudra, Manolis Koubarakis, Gourab K Patro, 2023. A review of the role of causality in developing trustworthy ai systems. arXiv preprint arXiv:2302.06975 (2023).
[56]
Eugene Garfield. 1977. Proposal for a new profession-scientific reviewer. Current contents14 (1977), 5–8. Publisher: INST SCI INFORM INC 3501 MARKET ST, PHILADELPHIA, PA 19104.
[57]
Trisha Greenhalgh. 1997. How to read a paper: Assessing the methodological quality of published papers. Bmj 315, 7103 (1997), 305–308. ISBN: 0959-8138 Publisher: British Medical Journal Publishing Group.
[58]
Odd Erik Gundersen and Sigbjørn Kjensmo. 2018. State of the art: Reproducibility in artificial intelligence. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 32. Issue: 1.
[59]
David Gunning, Mark Stefik, Jaesik Choi, Timothy Miller, Simone Stumpf, and Guang-Zhong Yang. 2019. XAI—Explainable artificial intelligence. Science Robotics 4, 37 (Dec. 2019), eaay7120. https://doi.org/10.1126/scirobotics.aay7120
[60]
Michael B. Harari, Heather R. Parola, Christopher J. Hartwell, and Amy Riegelman. 2020. Literature searches in systematic reviews and meta-analyses: A review, evaluation, and recommendations. Journal of Vocational Behavior 118 (April 2020), 103377. https://doi.org/10.1016/j.jvb.2020.103377
[61]
Andrew Head, Kyle Lo, Dongyeop Kang, Raymond Fok, Sam Skjonsberg, Daniel S. Weld, and Marti A. Hearst. 2021. Augmenting Scientific Papers with Just-in-Time, Position-Sensitive Definitions of Terms and Symbols. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems(CHI ’21). Association for Computing Machinery, New York, NY, USA, 1–18. https://doi.org/10.1145/3411764.3445648
[62]
Megan L. Head, Luke Holman, Rob Lanfear, Andrew T. Kahn, and Michael D. Jennions. 2015. The extent and consequences of p-hacking in science. PLoS biology 13, 3 (2015), e1002106. ISBN: 1545-7885 Publisher: Public Library of Science.
[63]
Elisa L Hill-Yardin, Mark R Hutchinson, Robin Laycock, and Sarah J Spencer. 2023. A Chat (GPT) about the future of scientific publishing. Brain Behav Immun 110 (2023), 152–154.
[64]
Andreas Hinderks, Francisco José Domínguez Mayo, Jörg Thomaschewski, and María José Escalona. 2020. An SLR-tool: Search process in practice: A tool to conduct and manage systematic literature review (SLR). In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering: Companion Proceedings. 81–84.
[65]
Xiaoyun Huang, Jessica Vitak, and Yla Tausczik. 2020. " You Don’t Have To Know My Past": How WeChat Moments Users Manage Their Evolving Self-Presentation. In Proceedings of the 2020 CHI Conference on Human Factors in Computing systems. 1–13.
[66]
Martin E. Héroux, Annie A. Butler, Aidan G. Cashin, Euan J. McCaughey, Andrew J. Affleck, Michael A. Green, Andrew Cartwright, Matthew Jones, Kim M. Kiely, Kimberley S. van Schooten, Jasmine C. Menant, Michael Wewege, and Simon C. Gandevia. 2022. Quality Output Checklist and Content Assessment (QuOCCA): a new tool for assessing research quality and reproducibility. BMJ Open 12, 9 (Sept. 2022), e060976. https://doi.org/10.1136/bmjopen-2022-060976 Publisher: British Medical Journal Publishing Group Section: Communication.
[67]
Sharon Favaro Ince, Christopher Hoadley, and Paul A Kirschner. 2018. A study of search practices in doctoral student scholarly workflows. In Proceedings of the 2018 Conference on Human Information Interaction & Retrieval. 245–248.
[68]
Srinivasan Keshav. 2007. How to read a paper. ACM SIGCOMM Computer Communication Review 37, 3 (2007), 83–84. ISBN: 0146-4833 Publisher: ACM New York, NY, USA.
[69]
Mallory C. Kidwell, Ljiljana B. Lazarević, Erica Baranski, Tom E. Hardwicke, Sarah Piechowski, Lina-Sophia Falkenberg, Curtis Kennett, Agnieszka Slowik, Carina Sonnleitner, and Chelsey Hess-Holden. 2016. Badges to acknowledge open practices: A simple, low-cost, effective method for increasing transparency. PLoS biology 14, 5 (2016), e1002456. ISBN: 1544-9173 Publisher: Public Library of Science San Francisco, CA USA.
[70]
Sunnie SY Kim, Elizabeth Anne Watkins, Olga Russakovsky, Ruth Fong, and Andrés Monroy-Hernández. 2023. " Help Me Help the AI": Understanding How Explainability Can Support Human-AI Interaction. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–17.
[71]
Seth A. King, Douglas Kostewicz, Olivia Enders, Taneal Burch, Argnue Chitiyo, Johanna Taylor, Sarah DeMaria, and Milsha Reid. 2020. Search and Selection Procedures of Literature Reviews in Behavior Analysis. Perspectives on Behavior Science 43, 4 (Dec. 2020), 725–760. https://doi.org/10.1007/s40614-020-00265-9
[72]
Sai Koneru, Jian Wu, and Sarah Rajtmajer. 2023. Can Large Language Models Discern Evidence for Scientific Hypotheses? Case Studies in the Social Sciences. arXiv preprint arXiv:2309.06578 (2023).
[73]
Janice Y. Kung. 2023. Elicit. The Journal of the Canadian Health Libraries Association 44, 1 (April 2023), 15–18. https://doi.org/10.29173/jchla29657
[74]
Tamarinde L. Haven and Dr. Leonie Van Grootel. 2019. Preregistering qualitative research. Accountability in Research 26, 3 (April 2019), 229–244. https://doi.org/10.1080/08989621.2019.1580147 Publisher: Taylor & Francis _eprint: https://doi.org/10.1080/08989621.2019.1580147.
[75]
Manoj Mathew Lalu, Larissa Shamseer, Kelly D Cobey, and David Moher. 2017. How stakeholders can respond to the rise of predatory journals. Nature Human Behaviour 1, 12 (2017), 852–855.
[76]
Sabina Leonelli. 2018. Rethinking Reproducibility as a Criterion for Research Quality. In Including a Symposium on Mary Morgan: Curiosity, Imagination, and Surprise. Research in the History of Economic Thought and Methodology, Vol. 36B. Emerald Publishing Limited, 129–146. https://doi.org/10.1108/S0743-41542018000036B009
[77]
Pantelis Linardatos, Vasilis Papastefanopoulos, and Sotiris Kotsiantis. 2021. Explainable AI: A Review of Machine Learning Interpretability Methods. Entropy 23, 1 (Jan. 2021), 18. https://doi.org/10.3390/e23010018 Number: 1 Publisher: Multidisciplinary Digital Publishing Institute.
[78]
Nelson F Liu, Tianyi Zhang, and Percy Liang. 2023. Evaluating verifiability in generative search engines. arXiv preprint arXiv:2304.09848 (2023).
[79]
Shengbo Liu, Chaomei Chen, Kun Ding, Bo Wang, Kan Xu, and Yuan Lin. 2014. Literature retrieval based on citation context. Scientometrics 101, 2 (Nov. 2014), 1293–1307. https://doi.org/10.1007/s11192-014-1233-7
[80]
Matthew C. Makel, Jonathan A. Plucker, and Boyd Hegarty. 2012. Replications in psychology research: How often do they really occur?Perspectives on Psychological Science 7, 6 (2012), 537–542. ISBN: 1745-6916 Publisher: Sage Publications Sage CA: Los Angeles, CA.
[81]
Alan Maloney and Lettie Y Conrad. 2016. Expecting the unexpected: Serendipity, discovery, and the scholarly research process. White Paper (2016).
[82]
Jiaxin Mao, Yiqun Liu, Noriko Kando, Min Zhang, and Shaoping Ma. 2018. How does domain expertise affect users’ search interaction and outcome in exploratory search?ACM Transactions on Information Systems (TOIS) 36, 4 (2018), 1–30.
[83]
Alberto Martín-Martín, Rodrigo Costas, Thed Van Leeuwen, and Emilio Delgado López-Cózar. 2018. Evidence of open access of scientific publications in Google Scholar: A large-scale analysis. Journal of informetrics 12, 3 (2018), 819–841. ISBN: 1751-1577 Publisher: Elsevier.
[84]
Erin C. McKiernan, Philip E. Bourne, C. Titus Brown, Stuart Buck, Amye Kenall, Jennifer Lin, Damon McDougall, Brian A. Nosek, Karthik Ram, and Courtney K. Soderberg. 2016. How open science helps researchers succeed. elife 5 (2016), e16800. ISBN: 2050-084X Publisher: eLife Sciences Publications, Ltd.
[85]
Blakeley B McShane, Jennifer L Tackett, Ulf Böckenholt, and Andrew Gelman. 2019. Large-scale replication projects in contemporary psychological research. The American Statistician 73, sup1 (2019), 99–105.
[86]
Wondimagegn Mengist, Teshome Soromessa, and Gudina Legese. 2020. Method for conducting systematic literature review and meta-analysis for environmental science research. MethodsX 7 (2020), 100777.
[87]
Marcin Miłkowski, Witold M. Hensel, and Mateusz Hohol. 2018. Replicability or reproducibility? On the replication crisis in computational neuroscience and sharing only relevant detail. Journal of Computational Neuroscience 45, 3 (Dec. 2018), 163–172. https://doi.org/10.1007/s10827-018-0702-z
[88]
Marcus R. Munafò, Brian A. Nosek, Dorothy VM Bishop, Katherine S. Button, Christopher D. Chambers, Nathalie Percie du Sert, Uri Simonsohn, Eric-Jan Wagenmakers, Jennifer J. Ware, and John Ioannidis. 2017. A manifesto for reproducible science. Nature human behaviour 1, 1 (2017), 1–9. ISBN: 2397-3374 Publisher: Nature Publishing Group.
[89]
Nishanth Nakshatri, Arjun Menon, C Lee Giles, Sarah Rajtmajer, and Christopher Griffin. 2021. Design and Analysis of a Synthetic Prediction Market using Dynamic Convex Sets. arXiv preprint arXiv:2101.01787 (2021).
[90]
National Academies of Sciences, Engineering, and Medicine 2019. Reproducibility and replicability in science. National Academies Press.
[91]
Brian A Nosek, Tom E Hardwicke, Hannah Moshontz, Aurélien Allard, Katherine S Corker, Anna Dreber Almenberg, Fiona Fidler, Joseph Hilgard, Melissa Kline, Michèle B Nuijten, 2021. Replicability, robustness, and reproducibility in psychological science. (2021).
[92]
Matthew J Page, Joanne E McKenzie, Patrick M Bossuyt, Isabelle Boutron, Tammy C Hoffmann, Cynthia D Mulrow, Larissa Shamseer, Jennifer M Tetzlaff, Elie A Akl, Sue E Brennan, 2021. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. International journal of surgery 88 (2021), 105906.
[93]
Srishti Palani, Aakanksha Naik, Doug Downey, Amy X. Zhang, Jonathan Bragg, and Joseph Chee Chang. 2023. Relatedly: Scaffolding Literature Reviews with Existing Related Work Sections. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. ACM, Hamburg Germany, 1–20. https://doi.org/10.1145/3544548.3580841
[94]
Samuel Pawel and Leonhard Held. 2020. Probabilistic forecasting of replication studies. PloS one 15, 4 (2020), e0231416.
[95]
Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Larivière, Alina Beygelzimer, Florence d’Alché Buc, Emily Fox, and Hugo Larochelle. 2021. Improving Reproducibility in Machine Learning Research. Journal of Machine Learning Research 22 (2021).
[96]
Hans E. Plesser. 2018. Reproducibility vs. Replicability: A Brief History of a Confused Terminology. Frontiers in Neuroinformatics 11 (2018). https://www.frontiersin.org/articles/10.3389/fninf.2017.00076
[97]
Diego Ponte and Judith Simon. 2011. Scholarly Communication 2.0: Exploring Researchers’ Opinions on Web 2.0 for Scientific Knowledge Creation, Evaluation and Dissemination. Serials Review 37, 3 (Sept. 2011), 149–156. https://doi.org/10.1080/00987913.2011.10765376 Publisher: Routledge _eprint: https://www.tandfonline.com/doi/pdf/10.1080/00987913.2011.10765376.
[98]
Snehal Prabhudesai, Leyao Yang, Sumit Asthana, Xun Huan, Q Vera Liao, and Nikola Banovic. 2023. Understanding Uncertainty: How Lay Decision-makers Perceive and Interpret Uncertainty in Human-AI Decision Making. In Proceedings of the 28th International Conference on Intelligent User Interfaces. 379–396.
[99]
Edward Raff. 2019. A step toward quantifying independently reproducible machine learning research. Advances in Neural Information Processing Systems 32 (2019).
[100]
Sarah Rajtmajer, Christopher Griffin, Jian Wu, Robert Fraleigh, Laxmaan Balaji, Anna Squicciarini, Anthony Kwasnica, David Pennock, Michael McLaughlin, Timothy Fritton, 2021. A Synthetic Prediction Market for Estimating Confidence in Published Work. arXiv preprint arXiv:2201.06924 (2021).
[101]
Sarah Rajtmajer, Christopher Griffin, Jian Wu, Robert Fraleigh, Laxmaan Balaji, Anna Squicciarini, Anthony Kwasnica, David Pennock, Michael McLaughlin, Timothy Fritton, 2022. A synthetic prediction market for estimating confidence in published work. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 36. 13218–13220.
[102]
Melissa L Rethlefsen, Shona Kirtley, Siw Waffenschmidt, Ana Patricia Ayala, David Moher, Matthew J Page, and Jonathan B Koffel. 2021. PRISMA-S: an extension to the PRISMA statement for reporting literature searches in systematic reviews. Systematic reviews 10 (2021), 1–19.
[103]
Wojciech Samek and Klaus-Robert Müller. 2019. Towards Explainable Artificial Intelligence. In Explainable AI: Interpreting, Explaining and Visualizing Deep Learning, Wojciech Samek, Grégoire Montavon, Andrea Vedaldi, Lars Kai Hansen, and Klaus-Robert Müller (Eds.). Springer International Publishing, Cham, 5–22. https://doi.org/10.1007/978-3-030-28954-6_1
[104]
Frances Shiely, Kerrie Gallagher, and Seán R Millar. 2024. How, and why, science and health researchers read scientific (IMRAD) papers. Plos one 19, 1 (2024), e0297034.
[105]
Luciana B. Sollaci and Mauricio G. Pereira. 2004. The introduction, methods, results, and discussion (IMRAD) structure: a fifty-year survey. Journal of the medical library association 92, 3 (2004), 364. Publisher: Medical Library Association.
[106]
Piotr Sorokowski, Emanuel Kulczycki, Agnieszka Sorokowska, and Katarzyna Pisanski. 2017. Predatory journals recruit fake editor. Nature 543, 7646 (2017), 481–483.
[107]
Marek Sośnicki and Lech Madeyski. 2021. ASH: A New Tool for Automated and Full-Text Search in Systematic Literature Reviews. In International Conference on Computational Science. Springer, 362–369.
[108]
Ayah Soufan, Ian Ruthven, and Leif Azzopardi. 2022. Searching the literature: an analysis of an exploratory search task. In Proceedings of the 2022 Conference on Human Information Interaction and Retrieval. 146–157.
[109]
Nicole Sultanum, Christine Murad, and Daniel Wigdor. 2020. Understanding and Supporting Academic Literature Review Workflows with LitSense. In Proceedings of the International Conference on Advanced Visual Interfaces. ACM, Salerno Italy, 1–5. https://doi.org/10.1145/3399715.3399830
[110]
Katherine A. Tamminen and Zoë A. Poucher. 2018. Open science in sport and exercise psychology: Review of current approaches and considerations for qualitative inquiry. Psychology of Sport and Exercise 36 (2018), 17–28. https://doi.org/10.1016/j.psychsport.2017.12.010 Place: Netherlands Publisher: Elsevier Science.
[111]
Carol Tenopir, Lisa Christian, and Jordan Kaufman. 2019. Seeking, Reading, and Use of Scholarly Articles: An International Study of Perceptions and Behavior of Researchers. Publications 7, 1 (March 2019), 18. https://doi.org/10.3390/publications7010018 Number: 1 Publisher: Multidisciplinary Digital Publishing Institute.
[112]
Xuanhui Wang and ChengXiang Zhai. 2007. Learn from web search logs to organize search results. In Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval. 87–94.
[113]
Ronald L Wasserstein and Nicole A Lazar. 2016. The ASA statement on p-values: context, process, and purpose., 129–133 pages.
[114]
Ryen W White. 2018. Opportunities and challenges in search interaction. Commun. ACM 61, 12 (2018), 36–38.
[115]
Ryen W White, Susan T Dumais, and Jaime Teevan. 2009. Characterizing the influence of domain expertise on web search behavior. In Proceedings of the second ACM international conference on web search and data mining. 132–141.
[116]
Jian Wu, Rajal Nivargi, Sree Sai Teja Lanka, Arjun Manoj Menon, Sai Ajay Modukuri, Nishanth Nakshatri, Xin Wei, Zhuoer Wang, James Caverlee, Sarah M Rajtmajer, 2021. Predicting the Reproducibility of Social and Behavioral Science Papers Using Supervised Learning Models. arXiv preprint arXiv:2104.04580 (2021).
[117]
Erjia Yan and Ying Ding. 2010. Weighted citation: An indicator of an article’s prestige. Journal of the American Society for Information Science and Technology 61, 8 (2010), 1635–1643. ISBN: 1532-2882 Publisher: Wiley Online Library.
[118]
Yang Yang, Wu Youyou, and Brian Uzzi. 2020. Estimating the deep replicability of scientific findings using human and artificial intelligence. Proceedings of the National Academy of Sciences 117, 20 (2020), 10762–10768.
[119]
Yifan Zhu, Qika Lin, Hao Lu, Kaize Shi, Ping Qiu, and Zhendong Niu. 2021. Recommending scientific paper via heterogeneous knowledge embedding based attentive recurrent neural networks. Knowledge-Based Systems 215 (March 2021), 106744. https://doi.org/10.1016/j.knosys.2021.106744
Index Terms
- Integrating measures of replicability into scholarly search: Challenges and opportunities
Index terms have been assigned to the content through auto-classification.
Recommendations
Reproducibility, Replicability and Repeatability: A survey of reproducible research with a focus on high performance computing
AbstractReproducibility is widely acknowledged as a fundamental principle in scientific research. Currently, the scientific community grapples with numerous challenges associated with reproducibility, often referred to as the “reproducibility crisis”. ...
A Study on Reproducibility and Replicability of Table Structure Recognition Methods
Document Analysis and Recognition - ICDAR 2023AbstractConcerns about reproducibility in artificial intelligence (AI) have emerged, as researchers have reported unsuccessful attempts to directly reproduce published findings in the field. Replicability, the ability to affirm a finding using the same ...
KheOps: Cost-effective Repeatability, Reproducibility, and Replicability of Edge-to-Cloud Experiments
ACM REP '23: Proceedings of the 2023 ACM Conference on Reproducibility and ReplicabilityDistributed infrastructures for computation and analytics are now evolving towards an interconnected ecosystem allowing complex scientific workflows to be executed across hybrid systems spanning from IoT Edge devices to Clouds, and sometimes to ...
Comments
Information & Contributors
Information
Published In
May 2024
18961 pages
ISBN:9798400703300
DOI:10.1145/3613904
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 11 May 2024
Check for updates
Badges
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Funding Sources
- DARPA
Conference
Acceptance Rates
Overall Acceptance Rate 6,199 of 26,314 submissions, 24%
Upcoming Conference
CHI 2025
- Sponsor:
- sigchi
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 802Total Downloads
- Downloads (Last 12 months)802
- Downloads (Last 6 weeks)201
Reflects downloads up to 01 Mar 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderFull Text
View this article in Full Text.
Full TextHTML Format
View this article in HTML Format.
HTML Format