Foldit Drug Design Game Usability Study: Comparison of Citizen and Expert Scientists
Article No.: 20, Pages 1 - 19
Abstract
In building a new drug design mode for the popular citizen scientist game Foldit, we focus on creating an easy-to-use and intuitive interface to confer complex scientific concepts to citizen scientist players. We hypothesize that to be efficient in the hands of citizen scientists such an interface will look different from well-established drug-design software used by experts. We used the relaxed think-aloud method to compare citizen and expert scientists working with our prototype interface for Foldit Drug Design Mode (FDDM). First, we tested if the two groups are providing different feedback when it comes to the usability of the prototype interface. Second, we investigated how the difference between the two groups might inform a new game design. As expected, the results confirm that experienced scientists differ from citizen scientists in engaging their background knowledge when interacting with the game. We then provided a prioritization list of background knowledge employed by the expert scientists to derive design suggestions for FDDM.
Supplementary Material
Supplement to "Foldit Drug Design Game Usability Study: Comparison of Citizen and Expert Scientists" by Liu et al., Motion, Interaction and Games (MIG '20).
- Download
- 1.64 MB
References
[1]
Rick Bonney, Jennifer L. Shirk, Tina B. Phillips, Andrea Wiggins, Heidi L. Ballard, Abraham J. Miller-Rushing, and Julia K. Parrish. 2014. Next Steps for Citizen Science. Science 343, 6178 (2014), 1436–1437. https://doi.org/10.1126/science.1251554 arXiv:https://science.sciencemag.org/content/343/6178/1436.full.pdf
[2]
Anne Bowser, Derek Hansen, Yurong He, Carol Boston, Matthew Reid, Logan Gunnell, and Jennifer Preece. 2013. Using Gamification to Inspire New Citizen Science Volunteers. In Proceedings of the First International Conference on Gameful Design, Research, and Applications(Toronto, Ontario, Canada) (Gamification ’13). Association for Computing Machinery, New York, NY, USA, 18–25. https://doi.org/10.1145/2583008.2583011
[3]
Seth Cooper, Firas Khatib, Adrien Treuille, Janos Barbero, Jeehyung Lee, Michael Beenen, Andrew Leaver-Fay, David Baker, Zoran Popović, 2010. Predicting protein structures with a multiplayer online game. Nature 466, 7307 (2010), 756–760.
[4]
K. Crowston and N. R. Prestopnik. 2013. Motivation and Data Quality in a Citizen Science Game: A Design Science Evaluation. In 2013 46th Hawaii International Conference on System Sciences. 450–459. https://doi.org/10.1109/HICSS.2013.413
[5]
J A DiMasi, L Feldman, A Seckler, and A Wilson. 2010. Trends in Risks Associated With New Drug Development: Success Rates for Investigational Drugs. Clinical Pharmacology & Therapeutics 87, 3 (2010), 272–277. https://doi.org/10.1038/clpt.2009.295 arXiv:https://ascpt.onlinelibrary.wiley.com/doi/pdf/10.1038/clpt.2009.295
[6]
Christopher B Eiben, Justin B Siegel, Jacob B Bale, Seth Cooper, Firas Khatib, Betty W Shen, Barry L Stoddard, Zoran Popovic, and David Baker. 2012. Increased Diels-Alderase activity through backbone remodeling guided by Foldit players. Nature biotechnology 30, 2 (2012), 190–192.
[7]
K Anders Ericsson and Herbert A Simon. 1984. Protocol analysis: Verbal reports as data.the MIT Press.
[8]
Robert Heck, Oana Vuculescu, Jens Jakob Sørensen, Jonathan Zoller, Morten G. Andreasen, Mark G. Bason, Poul Ejlertsen, Ottó Elíasson, Pinja Haikka, Jens S. Laustsen, Lærke L. Nielsen, Andrew Mao, Romain Müller, Mario Napolitano, Mads K. Pedersen, Aske R. Thorsen, Carsten Bergenholtz, Tommaso Calarco, Simone Montangero, and Jacob F. Sherson. 2018. Remote optimization of an ultracold atoms experiment by experts and citizen scientists. Proceedings of the National Academy of Sciences 115, 48(2018), E11231–E11237. https://doi.org/10.1073/pnas.1716869115 arXiv:https://www.pnas.org/content/115/48/E11231.full.pdf
[9]
Scott Horowitz, Brian Koepnick, Raoul Martin, Agnes Tymieniecki, Amanda A Winburn, Seth Cooper, Jeff Flatten, David S Rogawski, Nicole M Koropatkin, Tsinatkeab T Hailu, 2016. Determining crystal structures through crowdsourcing and coursework. Nature communications 7, 1 (2016), 1–11.
[10]
Ioanna Iacovides, Charlene Jennett, Cassandra Cornish-Trestrail, and Anna L. Cox. 2013. Do Games Attract or Sustain Engagement in Citizen Science? A Study of Volunteer Motivations. In CHI ’13 Extended Abstracts on Human Factors in Computing Systems (Paris, France) (CHI EA ’13). Association for Computing Machinery, New York, NY, USA, 1101–1106. https://doi.org/10.1145/2468356.2468553
[11]
Monique W.M. Jaspers. 2009. A comparison of usability methods for testing interactive health technologies: Methodological aspects and empirical evidence. International Journal of Medical Informatics 78, 5(2009), 340 – 353. https://doi.org/10.1016/j.ijmedinf.2008.10.002
[12]
Monique W.M. Jaspers, Thiemo Steen, Cor van den Bos, and Maud Geenen. 2004. The think aloud method: a guide to user interface design. International Journal of Medical Informatics 73, 11(2004), 781 – 795. https://doi.org/10.1016/j.ijmedinf.2004.08.003
[13]
Victor Jupp. 2006. The Sage dictionary of social research methods. Sage.
[14]
Firas Khatib, Seth Cooper, Michael D. Tyka, Kefan Xu, Ilya Makedon, Zoran Popović, David Baker, and Foldit Players. 2011a. Algorithm discovery by protein folding game players. Proceedings of the National Academy of Sciences 108, 47(2011), 18949–18953. https://doi.org/10.1073/pnas.1115898108
[15]
Firas Khatib, Frank DiMaio, Seth Cooper, Maciej Kazmierczyk, Miroslaw Gilski, Szymon Krzywda, Helena Zabranska, Iva Pichova, James Thompson, Zoran Popović, 2011b. Crystal structure of a monomeric retroviral protease solved by protein folding game players. Nature structural & molecular biology 18, 10 (2011), 1175–1177.
[16]
Brian Koepnick, Jeff Flatten, Tamir Husain, Alex Ford, Daniel-Adriano Silva, Matthew J Bick, Aaron Bauer, Gaohua Liu, Yojiro Ishida, Alexander Boykov, 2019. De novo protein design by citizen scientists. Nature 570, 7761 (2019), 390–394.
[17]
E. Korpela, D. Werthimer, D. Anderson, J. Cobb, and M. Leboisky. 2001. SETI@home-massively distributed computing for SETI. Computing in Science Engineering 3, 1 (Jan 2001), 78–83. https://doi.org/10.1109/5992.895191
[18]
Andrew Leaver-Fay, Michael Tyka, Steven M. Lewis, Oliver F. Lange, James Thompson, Ron Jacak, Kristian W. Kaufman, P. Douglas Renfrew, Colin A. Smith, Will Sheffler, Ian W. Davis, Seth Cooper, Adrien Treuille, Daniel J. Mandell, Florian Richter, Yih-En Andrew Ban, Sarel J. Fleishman, Jacob E. Corn, David E. Kim, Sergey Lyskov, Monica Berrondo, Stuart Mentzer, Zoran Popović, James J. Havranek, John Karanicolas, Rhiju Das, Jens Meiler, Tanja Kortemme, Jeffrey J. Gray, Brian Kuhlman, David Baker, and Philip Bradley. 2011. Chapter nineteen - Rosetta3: An Object-Oriented Software Suite for the Simulation and Design of Macromolecules. In Computer Methods, Part C, Michael L. Johnson and Ludwig Brand (Eds.). Methods in Enzymology, Vol. 487. Academic Press, 545 – 574. https://doi.org/10.1016/B978-0-12-381270-4.00019-6
[19]
Jens Meiler and David Baker. 2006. ROSETTALIGAND: Protein–small molecule docking with full side-chain flexibility. Proteins: Structure, Function, and Bioinformatics 65, 3(2006), 538–548. https://doi.org/10.1002/prot.21086
[20]
Josh Aaron Miller, Vivian Lee, Seth Cooper, and Magy Seif El-Nasr. 2019. Large-Scale Analysis of Visualization Options in a Citizen Science Game. In Extended Abstracts of the Annual Symposium on Computer-Human Interaction in Play Companion Extended Abstracts (Barcelona, Spain) (CHI PLAY ’19 Extended Abstracts). Association for Computing Machinery, New York, NY, USA, 535–542. https://doi.org/10.1145/3341215.3356274
[21]
Garrett M. Morris, Ruth Huey, William Lindstrom, Michel F. Sanner, Richard K. Belew, David S. Goodsell, and Arthur J. Olson. 2009. AutoDock4 and AutoDockTools4: Automated docking with selective receptor flexibility. Journal of Computational Chemistry 30, 16 (2009), 2785–2791. https://doi.org/10.1002/jcc.21256
[22]
Michael Shirts and Vijay S. Pande. 2000. Screen Savers of the World Unite!Science 290, 5498 (2000), 1903–1904. https://doi.org/10.1126/science.290.5498.1903
[23]
Jonathan Silvertown. 2009. A new dawn for citizen science. Trends in Ecology & Evolution 24, 9 (2009), 467 – 471. https://doi.org/10.1016/j.tree.2009.03.017
[24]
Kristin Siu, Alexander Zook, and Mark O. Riedl. 2017. A Framework for Exploring and Evaluating Mechanics in Human Computation Games. In Proceedings of the 12th International Conference on the Foundations of Digital Games (Hyannis, Massachusetts) (FDG ’17). Association for Computing Machinery, New Yorkk NYk USA, Article 38, 4 pages. https://doi.org/10.1145/3102071.3106344
[25]
Joanna Stanczyk, Caroline Ospelt, and Steffen Gay. 2008. Is there a future for small molecule drugs in the treatment of rheumatic diseases?Current opinion in rheumatology 20, 3 (2008), 257–262.
[26]
Brian L. Sullivan, Christopher L. Wood, Marshall J. Iliff, Rick E. Bonney, Daniel Fink, and Steve Kelling. 2009. eBird: A citizen-based bird observation network in the biological sciences. Biological Conservation 142, 10 (2009), 2282 – 2292. https://doi.org/10.1016/j.biocon.2009.05.006
[27]
Marcel L. Verdonk, Jason C. Cole, Michael J. Hartshorn, Christopher W. Murray, and Richard D. Taylor. 2003. Improved protein–ligand docking using GOLD. Proteins: Structure, Function, and Bioinformatics 52, 4(2003), 609–623. https://doi.org/10.1002/prot.10465 arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1002/prot.10465
[28]
L. von Ahn. 2006. Games with a purpose. Computer 39, 6 (June 2006), 92–94. https://doi.org/10.1109/MC.2006.196
[29]
Luis Von Ahn and Laura Dabbish. 2008. Designing games with a purpose. Commun. ACM 51, 8 (2008), 58–67.
Recommendations
Is Virtual Citizen Science A Game?
The use of game elements within virtual citizen science is increasingly common, promising to bring increased user activity, motivation, and engagement to large-scale scientific projects. However, there is an ongoing debate about whether or not gamifying ...
A new framework of usability evaluation for massively multi-player online game: case study of "World of warcraft" game
HCI'07: Proceedings of the 12th international conference on Human-computer interaction: applications and servicesAccording to traditional marketing wisdom, customers' preferences are a core issue in designing successful products, and the design process in game is crucial for guaranteeing gamers' satisfaction. This research aimed to explore critical factors for the ...
Gamers, citizen scientists, and data
Two key problems for crowd-sourcing systems are motivating contributions from participants and ensuring the quality of these contributions. Games have been suggested as a motivational approach to encourage contribution, but attracting participation ...
Comments
Information & Contributors
Information
Published In
October 2020
190 pages
ISBN:9781450381710
DOI:10.1145/3424636
Copyright © 2020 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 22 November 2020
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
MIG '20
Sponsor:
Acceptance Rates
Overall Acceptance Rate -9 of -9 submissions, 100%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 118Total Downloads
- Downloads (Last 12 months)14
- Downloads (Last 6 weeks)1
Reflects downloads up to 02 Mar 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format