Rajan Vaish
Snehalkumar (Neil) S. Gaikwad
Geza Kovacs
Ranjay Krishna
Michael S. Bernstein
ABSTRACT
Research experiences today are limited to a privileged few at select universities. Providing open access to research experiences would enable global upward mobility and increased diversity in the scientific workforce. How can we coordinate a crowd of diverse volunteers on open-ended research? How could a PI have enough visibility into each person's contributions to recommend them for further study? We present Crowd Research, a crowdsourcing technique that coordinates open-ended research through an iterative cycle of open contribution, synchronous collaboration, and peer assessment. To aid upward mobility and recognize contributions in publications, we introduce a decentralized credit system: participants allocate credits to each other, which a graph centrality algorithm translates into a collectively-created author order. Over 1,500 people from 62 countries have participated, 74% from institutions with low access to research. Over two years and three projects, this crowd has produced articles at top-tier Computer Science venues, and participants have gone on to leading graduate programs.
Supplemental Material
Available for Download
- 2017. The Times Higher Education World University Rankings. (2017). https://www.timeshighereducation. com/world-university-rankingsGoogle Scholar
- Ricardo A Baeza-Yates, Carlos Castillo, Vicente López, Martin Shubik, John Hopcroft, and Daniel Sheldon. 2007. Pagerank Increase under Different Collusion Topologies. In AIRWeb, Vol. 5. Springer, Sage Publications, 68--81.Google Scholar
- Julie A Bianchini. 2011. Expanding underrepresented minority participation: America's science and technology talent at the crossroads. National Academies Press, Washington, District of Columbia.Google Scholar
- Rick Bonney, Heidi Ballard, Rebecca Jordan, Ellen McCallie, Tina Phillips, Jennifer Shirk, and Candie C Wilderman. 2009. Public Participation in Scientific Research: Defining the field and Assessing Its Potential for Informal Science Education. (2009).Google Scholar
- William G Bowen and Derek Bok. 2016. The shape of the river: Long-term consequences of considering race in college and university admissions. Princeton University Press.Google Scholar
- Nama Budhathoki. 2016. Who are the mappers and why do they map in OpenStreetMap. (2016). https://www.youtube.com/watch?v=LvakiUOsDrMGoogle Scholar
- Yan Chen, Steve Oney, and Walter S Lasecki. 2016. Towards providing on-demand expert support for software developers. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 3192--3203. Google ScholarDigital Library
- CIA. 2016. The world factbook. (2016). https://www.cia.gov/library/publications/the-world-factbook/Google Scholar
- Open Science Collaboration. 2015. Estimating the reproducibility of psychological science. Science 349, 6251 (2015).Google Scholar
- Seth Cooper, firas Khatib, Adrien Treuille, Janos Barbero, Jeehyung Lee, Michael Beenen, Andrew Leaver-Fay, David Baker, Zoran Popovi´c, and Foldit players. 2010. Predicting protein structures with a multiplayer online game. Nature 466, 7307 (2010), 756--760.Google ScholarCross Ref
- Joe Cox, Eun Young Oh, Brooke Simmons, Chris Lintott, Karen Masters, Anita Greenhill, Gary Graham, and Kate Holmes. 2015. Defining and measuring success in online citizen science: A case study of Zooniverse projects. Computing in Science & Engineering 17, 4 (2015), 28--41.Google ScholarDigital Library
- Catherine Durnell Cramton and Pamela J Hinds. 2004. Subgroup dynamics in internationally distributed teams: Ethnocentrism or cross-national learning? Research in organizational behavior 26 (2004), 231--263.Google Scholar
- Justin Cranshaw and Aniket Kittur. 2011. The polymath project: lessons from a successful online collaboration in mathematics. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1865--1874. Google ScholarDigital Library
- Ruth Cronje, Spencer Rohlinger, Alycia Crall, and Greg Newman. 2011. Does Participation in Citizen Science Improve Scientific Literacy? A Study to Compare Assessment Methods. Applied Environmental Education & Communication 10, 3 (7 2011), 135--145.Google Scholar
- Tawanna R Dillahunt, Bingxin Chen, and Stephanie Teasley. 2014. Model thinking: demographics and performance of MOOC students unable to afford a formal education. In Proceedings of the first ACM conference on Learning@ scale conference. ACM, 145--146. Google ScholarDigital Library
- Tawanna R Dillahunt, Sandy Ng, Michelle fiesta, and Zengguang Wang. 2016. Do Massive Open Online Course Platforms Support Employability?. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing. ACM Press, NY, NY, USA, 232--243. Google ScholarDigital Library
- Ying Ding, Erjia Yan, Arthur Frazho, and James Caverlee. 2009. PageRank for ranking authors in co-citation networks. Journal of the American Society for Information Science and Technology 60, 11 (11 2009), 2229--2243. Google ScholarDigital Library
- Steven Dow, Anand Kulkarni, Scott Klemmer, and Björn Hartmann. 2012. Shepherding the crowd yields better work. In Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work. ACM Press, NY, NY, USA, 1013. Google ScholarDigital Library
- M Kevin Eagan, Sylvia Hurtado, Mitchell J Chang, Gina A Garcia, Felisha A Herrera, Juan C Garibay, and Juan C. Garibay. 2013. Making a Difference in Science Education: The Impact of Undergraduate Research Programs. American Educational Research Journal 50, 4 (8 2013), 683--713.Google ScholarCross Ref
- Snehalkumar (Neil) Gaikwad, Durim Morina, Rohit Nistala, Megha Agarwal, Alison Cossette, Radhika Bhanu, Saiph Savage, Vishwajeet Narwal, Karan Rajpal, Jeff Regino, Aditi Mithal, Adam Ginzberg, Aditi Nath, Karolina R. Ziulkoski, Trygve Cossette, Dilrukshi Gamage, Angela Richmond-Fuller, Ryo Suzuki, Jeerel Herrejón, Kevin Le, Claudia flores-Saviaga, Haritha Thilakarathne, Kajal Gupta, William Dai, Ankita Sastry, Shirish Goyal, Thejan Rajapakshe, Niki Abolhassani, Angela Xie, Abigail Reyes, Surabhi Ingle, Verónica Jaramillo, Martin Godinez, Walter Angel, Carlos Toxtli, Juan flores, Asmita Gupta, Vineet Sethia, Diana Padilla, Kristy Milland, Kristiono Setyadi, Nuwan Wajirasena, Muthitha Batagoda, Rolando Cruz, James Damon, Divya Nekkanti, Tejas Sarma, Mohamed Saleh, Gabriela Gongora-Svartzman, Soroosh Bateni, Gema Toledo Barrera, Alex Pena, Ryan Compton, Deen Aariff, Luis Palacios, Manuela Paula Ritter, Nisha K.K., Alan Kay, Jana Uhrmeister, Srivalli Nistala, Milad Esfahani, Elsa Bakiu, Christopher Diemert, Luca Matsumoto, Manik Singh, Krupa Patel, Ranjay Krishna, Geza Kovacs, Rajan Vaish, and Michael Bernstein. 2015. Daemo: A Self-Governed Crowdsourcing Marketplace. In Adjunct Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology. ACM, 101--102. Google ScholarDigital Library
- Snehalkumar (Neil) S. Gaikwad, Mark E. Whiting, Dilrukshi Gamage, Catherine A. Mullings, Dinesh Majeti, Shirish Goyal, Aaron Gilbee, Nalin Chhibber, Adam Ginzberg, Angela Richmond-Fuller, Sekandar Matin, Vibhor Sehgal, Tejas Seshadri Sarma, Ahmed Nasser, Alipta Ballav, Jeff Regino, Sharon Zhou, Kamila Mananova, Preethi Srinivas, Karolina Ziulkoski, Dinesh Dhakal, Alexander Stolzoff, Senadhipathige S. Niranga, Mohamed Hashim Salih, Akshansh Sinha, Rajan Vaish, and Michael S. Bernstein. 2017. The Daemo Crowdsourcing Marketplace. In Companion of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing. ACM, 1--4. Google ScholarDigital Library
- S S Gaikwad, D Morina, A Ginzberg, C Mullings, S Goyal, D Gamage, C Diemert, M Burton, S Zhou, M Whiting, K Ziulkoski, A Ballav, A Gilbee, S S Niranga, V Sehgal, J Lin, L Kristianto, J Regino, N Chhibber, D Majeti, S Sharma, K Mananova, D Dhakal, W Dai, V Purynova, S Sandeep, V Chandrakanthan, T Sarma, S Matin, A Nassar, R Nistala, A Stolzoff, K Milland, V Mathur, R Vaish, and M S Bernstein. 2016. Boomerang: Rebounding the Consequences of Reputation Feedback on Crowdsourcing Platforms. In Proceedings of the 29th Annual ACM Symposium on User Interface Software and Technology (UIST '16). ACM, NY, NY, USA. Google ScholarDigital Library
- C R Gerstner and D V Day. 1997. Meta-analytic review of leader-member exchange theory: Correlates and construct issues. Journal of Applied Psychology 82, 6 (1997), 827--844.Google ScholarCross Ref
- Eric Gilbert. 2013. Widespread underprovision on Reddit. In Proceedings of the 2013 conference on Computer supported cooperative work. ACM, 803--808. Google ScholarDigital Library
- Timothy Gowers and Michael Nielsen. 2009. Massively collaborative mathematics. Nature 461, 7266 (2009), 879--881.Google Scholar
- Patricia Gurin, Eric Dey, Sylvia Hurtado, and Gerald Gurin. 2002. Diversity and higher education: Theory and impact on educational outcomes. Harvard Educational Review 72, 3 (2002), 330--367.Google ScholarCross Ref
- Russel S Hathaway, Biren A Nagda, and Sandra R Gregerman. 2002. The relationship of undergraduate research participation to graduate and professional education pursuit: an empirical study. Journal of College Student Development 43, 5 (2002), 614.Google Scholar
- Pamela J Hinds and Diane E Bailey. 2003. Out of sight, out of sync: Understanding conflict in distributed teams. Organization science 14, 6 (2003), 615--632. Google ScholarDigital Library
- Pamela J Hinds and Mark Mortensen. 2005. Understanding conflict in geographically distributed teams: The moderating effects of shared identity, shared context, and spontaneous communication. Organization science 16, 3 (2005), 290--307. Google ScholarDigital Library
- Lu Hong and Scott E Page. 2004. Groups of diverse problem solvers can outperform groups of high-ability problem solvers. Proceedings of the National Academy of Sciences 101, 46 (2004), 16385--16389.Google ScholarCross Ref
- Anne-Barrie Hunter, Sandra L. Laursen, and Elaine Seymour. 2007. Becoming a scientist: The role of undergraduate research in students' cognitive, personal, and professional development. Science Education 91, 1 (1 2007), 36--74.Google Scholar
- Lilly C Irani and M Silberman. 2013. Turkopticon: Interrupting worker invisibility in amazon mechanical turk. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 611--620. Google ScholarDigital Library
- Firas Khatib, Frank DiMaio, Seth Cooper, Maciej Kazmierczyk, Miroslaw Gilski, Szymon Krzywda, Helena Zabranska, Iva Pichova, James Thompson, Zoran Popović, Mariusz Jaskolski, and David Baker. 2011. Crystal structure of a monomeric retroviral protease solved by protein folding game players. Nature Structural & Molecular Biology 18, 10 (9 2011), 1175--1177.Google Scholar
- Sara Kiesler, Robert Kraut, Paul Resnick, and Aniket Kittur. 2012. Regulating behavior in online communities. Building Successful Online Communities: Evidence-Based Social Design. MIT Press, Cambridge, MA (2012).Google ScholarDigital Library
- Aniket Kittur, Jeffrey V. Nickerson, Michael Bernstein, Elizabeth Gerber, Aaron Shaw, John Zimmerman, Matt Lease, and John Horton. 2013. The future of crowd work. In Proceedings of the 2013 conference on Computer supported cooperative work. ACM Press, NY, NY, USA, 1301. Google ScholarDigital Library
- Aniket Kittur, Boris Smus, Susheel Khamkar, and Robert E Kraut. 2011. Crowdforge: Crowdsourcing complex work. In Proceedings of the 24th annual ACM symposium on User interface software and technology. ACM, 43--52. Google ScholarDigital Library
- René F. Kizilcec and Sherif Halawa. 2015. Attrition and Achievement Gaps in Online Learning. In Proceedings of the Second ACM Conference on Learning @ Scale. ACM Press, NY, NY, USA, 57--66. Google ScholarDigital Library
- Jon M Kleinberg. 1999. Authoritative sources in a hyperlinked environment. Journal of the ACM (JACM) 46, 5 (1999), 604--632. Google ScholarDigital Library
- Chinmay Kulkarni, Koh Pang Wei, Huy Le, Daniel Chia, Kathryn Papadopoulos, Justin Cheng, Daphne Koller, and Scott R Klemmer. 2013. Peer and self assessment in massive online classes. ACM Transactions on Computer-Human Interaction (TOCHI) 20, 6 (2013), 33. Google ScholarDigital Library
- Chinmay E Kulkarni, Michael S Bernstein, and Scott R Klemmer. 2015. PeerStudio: rapid peer feedback emphasizes revision and improves performance. In Proceedings of the Second (2015) ACM Conference on Learning@ Scale. ACM, 75--84. Google ScholarDigital Library
- Karim R Lakhani, Hila Lifshitz-Assaf, and Michael Tushman. 2013. Open innovation and organizational boundaries: task decomposition, knowledge distribution and the locus of innovation. Handbook of economic organization: Integrating economic and organizational theory (2013), 355--382.Google Scholar
- Kate Land, Anže Slosar, Chris Lintott, Dan Andreescu, Steven Bamford, Phil Murray, Robert Nichol, M Jordan Raddick, Kevin Schawinski, Alex Szalay, Daniel Thomas, and Jan Vandenberg. 2008. Galaxy Zoo: the large-scale spin statistics of spiral galaxies in the Sloan Digital Sky Survey. Monthly Notices of the Royal Astronomical Society 388, 4 (2008), 1686--1692.Google ScholarCross Ref
- R. Eric Landrum and Lisa R. Nelsen. 2002. The Undergraduate Research Assistantship: An Analysis of the Benefits. Teaching of Psychology 29, 1 (2 2002), 15--19.Google Scholar
- Adam Lashinsky. 2012. Inside Apple: How America's Most Admired--and Secretive--Company Really Works. John Murray Publishers. Google ScholarDigital Library
- Thomas D LaToza, W Ben Towne, Christian M Adriano, and André Van Der Hoek. 2014. Microtask programming: Building software with a crowd. In Proceedings of the 27th annual ACM symposium on User interface software and technology. ACM, 43--54. Google ScholarDigital Library
- Edith Law, Andrea Wiggins, Mary L. Gray, and Alex Williams. 2017. Crowdsourcing as a Tool for Research: Implications of Uncertainty. In Proceedings of the 20th ACM Conference on Computer-Supported Cooperative Work & Social Computing (CSCW '17). ACM, NY, NY, USA. Google ScholarDigital Library
- Jeehyung Lee, Wipapat Kladwang, Minjae Lee, Daniel Cantu, Martin Azizyan, Hanjoo Kim, Alex Limpaecher, Snehal Gaikwad, Sungroh Yoon, Adrien Treuille, and others. 2014. RNA design rules from a massive open laboratory. Proceedings of the National Academy of Sciences 111, 6 (2014), 2122--2127.Google ScholarCross Ref
- Jin Li. 2003. US and Chinese cultural beliefs about learning. Journal of educational psychology 95, 2 (2003), 258.Google ScholarCross Ref
- David Lopatto. 2004. Survey of Undergraduate Research Experiences (SURE): first findings. Cell biology education 3, 4 (2004), 270--7.Google Scholar
- Andrew Mao, Winter Mason, Siddharth Suri, Duncan J. Watts, and TW Malone. 2016. An Experimental Study of Team Size and Performance on a Complex Task. PLOS ONE 11, 4 (4 2016), e0153048.Google Scholar
- David Martin, Benjamin V. Hanrahan, Jacki O'Neill, and Neha Gupta. 2014. Being a turker. In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing. ACM Press, NY, NY, USA, 224--235. Google ScholarDigital Library
- Robert C Miller, Haoqi Zhang, Eric Gilbert, and Elizabeth Gerber. 2014. Pair research: matching people for collaboration, learning, and productivity. In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing. ACM, 1043--1048. Google ScholarDigital Library
- Michael Nebeling, Alexandra To, Anhong Guo, Adrian A de Freitas, Jaime Teevan, Steven P Dow, and Jeffrey P Bigham. 2016. WearWrite: Crowd-assisted writing from smartwatches. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 3834--3846. Google ScholarDigital Library
- Nigini Oliveira, Eunice Jun, and Katharina Reinecke. 2017. Citizen Science Opportunities in Volunteer-Based Online Experiments. (2017).Google Scholar
- Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry Winograd. 1999. The PageRank citation ranking: bringing order to the web. (1999).Google Scholar
- Scott E Page. 2008. The difference: How the power of diversity creates better groups, firms, schools, and societies. Princeton University Press. Google ScholarDigital Library
- Vineet Pandey, Amnon Amir, Justine Debelius, Embriette R Hyde, Tomasz Kosciolek, Rob Knight, and Scott Klemmer. 2017. Gut Instinct: Creating scientific theories with online learners. (2017).Google ScholarDigital Library
- David A Patterson. 1994. How to have a bad career in research/academia. In Keynote, 1994 USENIX Symposium on Operating System Design and Implementation.Google Scholar
- M Jordan Raddick, Georgia Bracey, Pamela L Gay, Chris J Lintott, Phil Murray, Kevin Schawinski, Alexander S Szalay, and Jan Vandenberg. 2009. Galaxy zoo: Exploring the motivations of citizen science volunteers. arXiv preprint arXiv:0909.2925 (2009).Google Scholar
- Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing.Google ScholarCross Ref
- Katharina Reinecke and Krzysztof Z Gajos. 2015. LabintheWild: Conducting large-scale online experiments with uncompensated samples. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing. ACM, 1364--1378. Google ScholarDigital Library
- Daniela Retelny, Sébastien Robaszkiewicz, Alexandra To, Walter S Lasecki, Jay Patel, Negar Rahmati, Tulsee Doshi, Melissa Valentine, and Michael S Bernstein. 2014. Expert crowdsourcing with flash teams. In Proceedings of the 27th annual ACM symposium on User interface software and technology. ACM, 75--85. Google ScholarDigital Library
- Olga Russakovsky, Li-Jia Li, and Li Fei-Fei. 2015. Best of both worlds: Human-machine collaboration for object annotation. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2121--2131.Google ScholarCross Ref
- Susan H Russell, Mary P Hancock, and James McCullough. 2007. Benefits of undergraduate research experiences. Science 316, 5824 (2007), 548--549.Google Scholar
- Heather Sarsons. 2015. Gender differences in recognition for group work. Working Paper, Harvard University (2015).Google Scholar
- Alok Shankar Mysore, Vikas S Yaligar, Imanol Arrieta Ibarra, Camelia Simoiu, Sharad Goel, Ramesh Arvind, Chiraag Sumanth, Arvind Srikantan, Bhargav HS, Mayank Pahadia, Tushar Dobha, Atif Ahmed, Mani Shankar, Himani Agarwal, Rajat Agarwal, Sai Anirudh-Kondaveeti, Shashank Arun-Gokhale, Aayush Attri, Arpita Chandra, Yogitha Chilukur, Sharath Dharmaji, Deepak Garg, Naman Gupta, Paras Gupta, Glincy Mary Jacob, Siddharth Jain, Shashank Joshi, Tarun Khajuria, Sameeksha Khillan, Sandeep Konam, Praveen Kumar-Kolla, Sahil Loomba, Rachit Madan, Akshansh Maharaja, Vidit Mathur, Bharat Munshi, Mohammed Nawazish, Venkata Neehar-Kurukunda, Venkat Nirmal-Gavarraju, Sonali Parashar, Harsh Parikh, Avinash Paritala, Amit Patil, Rahul Phatak, Mandar Pradhan, Abhilasha Ravichander, Krishna Sangeeth, Sreecharan Sankaranarayanan, Vibhor Sehgal, Ashrith Sheshan, Suprajha Shibiraj, Aditya Singh, Anjali Singh, Prashant Sinha, Pushkin Soni, Bipin Thomas, Kasyap Varma-Dattada, Sukanya Venkataraman, Pulkit Verma, and Ishan Yelurwar. 2015. Investigating the "Wisdom of Crowds" at Scale. In Adjunct Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology (UIST '15 Adjunct). ACM, NY, NY, USA, 75--76. Google ScholarDigital Library
- Hua-Wei Shen and Albert-László Barabási. 2014. Collective credit allocation in science. Proceedings of the National Academy of Sciences 111, 34 (2014), 12325--12330.Google ScholarCross Ref
- Jonathan Silvertown. 2009. A new dawn for citizen science. Trends in Ecology & Evolution 24, 9 (9 2009), 467--471.Google Scholar
- Günter K Stahl, Martha L Maznevski, Andreas Voigt, and Karsten Jonsen. 2010. Unraveling the effects of cultural diversity in teams: A meta-analysis of research on multicultural work groups. Journal of international business studies 41, 4 (2010), 690--709.Google ScholarCross Ref
- Matthias Stevens, Michalis Vitos, Julia Altenbuchner, Gillian Conquest, Jerome Lewis, and Muki Haklay. 2014. Taking participatory citizen science to extremes. IEEE Pervasive Computing 13, 2 (2014), 20--29.Google ScholarCross Ref
- Brian L. Sullivan, Christopher L. Wood, Marshall J. Iliff, Rick E. Bonney, Daniel fink, and Steve Kelling. 2009. eBird: A citizen-based bird observation network in the biological sciences. Biological Conservation 142, 10 (10 2009), 2282--2292.Google Scholar
- Yizhou Sun, Yintao Yu, and Jiawei Han. 2009. Ranking-based clustering of heterogeneous information networks with star network schema. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 797--806. Google ScholarDigital Library
- Ryo Suzuki, Niloufar Salehi, Michelle S Lam, Juan C Marroquin, and Michael S Bernstein. 2016. Atelier: Repurposing Expert Crowdsourcing Tasks as Micro-internships. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 2645--2656. Google ScholarDigital Library
- Jaime Teevan, Shamsi T Iqbal, and Curtis von Veh. 2016. Supporting collaborative writing with microtasks. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 2657--2668. Google ScholarDigital Library
- Ginka Toegel and Jay A Conger. 2003. 360-degree assessment: Time for reinvention. Academy of Management Learning & Education 2, 3 (2003), 297--311.Google ScholarCross Ref
- Bill Tomlinson, Joel Ross, Paul Andre, Eric Baumer, Donald Patterson, Joseph Corneli, Martin Mahaux, Syavash Nobarany, Marco Lazzari, Birgit Penzenstadler, and others. 2012. Massively distributed authorship of academic papers. In Extended Abstracts of the 2012 ACM annual conference on Human Factors in Computing Systems. ACM, 11--20. Google ScholarDigital Library
- Jeanne L Tsai. 2007. Ideal affect: Cultural causes and behavioral consequences. Perspectives on Psychological Science 2, 3 (2007), 242--259.Google ScholarCross Ref
- Melissa A Valentine, Daniela Retelny, Alexandra To, Negar Rahmati, Tulsee Doshi, and Michael S Bernstein. 2017. flash Organizations: Crowdsourcing Complex Work by Structuring Crowds As Organizations. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, 3523--3537. Google ScholarDigital Library
- Andreas Veit, Michael J Wilber, Rajan Vaish, Serge J Belongie, James Davis, Vishal Anand, Anshu Aviral, Prithvijit Chakrabarty, Yash Chandak, Sidharth Chaturvedi, Chinmaya Devaraj, Ankit Dhall, Utkarsh Dwivedi, Sanket Gupte, Sharath N Sridhar, Karthik Paga, Anuj Pahuja, Aditya Raisinghani, Ayush Sharma, Shweta Sharma, Darpana Sinha, Nisarg Thakkar, K Bala Vignesh, Utkarsh Verma, Kanniganti Abhishek, Amod Agrawal, Arya Aishwarya, Aurgho Bhattacharjee, Sarveshwaran Dhanasekar, Venkata Karthik Gullapalli, Shuchita Gupta, Chandana G, Kinjal Jain, Simran Kapur, Meghana Kasula, Shashi Kumar, Parth Kundaliya, Utkarsh Mathur, Alankrit Mishra, Aayush Mudgal, Aditya Nadimpalli, Munakala Sree Nihit, Akanksha Periwal, Ayush Sagar, Ayush Shah, Vikas Sharma, Yashovardhan Sharma, Faizal Siddiqui, Virender Singh, Abhinav S., Pradyumna Tambwekar, Rashida Taskin, Ankit Tripathi, and Anurag D Yadav. 2015. On Optimizing Human-Machine Task Assignments. HCOMP 2015 Extended Abstracts (2015).Google Scholar
- Mark E Whiting, Dilrukshi Gamage, Snehalkumar (Neil) S Gaikwad, Aaron Gilbee, Shirish Goyal, Alipta Ballav, Dinesh Majeti, Nalin Chhibber, Angela Richmond-Fuller, Freddie Vargus, Tejas Seshadri Sarma, Varshine Chandrakanthan, Teogenes Moura, Mohamed Hashim Salih, Gabriel Bayomi Tinoco Kalejaiye, Adam Ginzberg, Catherine A Mullings, Yoni Dayan, Kristy Milland, Henrique Orefice, Jeff Regino, Sayna Parsi, Kunz Mainali, Vibhor Sehgal, Sekandar Matin, Akshansh Sinha, Rajan Vaish, and Michael S Bernstein. 2017. Crowd Guilds: Worker-led Reputation and Feedback on Crowdsourcing Platforms. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (CSCW '17). ACM, NY, NY, USA, 1902--1913. Google ScholarDigital Library
- Haizi Yu, Biplab Deka, Jerry O Talton, and Ranjitha Kumar. 2016. Accounting for taste: ranking curators and content in social networks. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 2383--2389. Google ScholarDigital Library
- Lixiu Yu, Aniket Kittur, and Robert E Kraut. 2014. Distributed analogical idea generation: inventing with crowds. In Proceedings of the 32nd annual ACM conference on Human factors in computing systems. ACM, 1245--1254. Google ScholarDigital Library
- Haoqi Zhang, Matthew W. Easterday, Elizabeth M. Gerber, Daniel Rees Lewis, and Leesha Maliakal. 2017. Agile Research Studios. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing - CSCW '17. ACM Press, NY, NY, USA, 220--232. Google ScholarDigital Library
- Andrew L. Zydney, Joan S. Bennett, Abdus Shahid, and Karen W. Bauer. 2002. Impact of Undergraduate Research Experience in Engineering. Journal of Engineering Education 91, April (4 2002), 151--157.Google Scholar
Index Terms
- Crowd Research: Open and Scalable University Laboratories
Recommendations
Modus Operandi of Crowd Workers: The Invisible Role of Microtask Work Environments
The ubiquity of the Internet and the widespread proliferation of electronic devices has resulted in flourishing microtask crowdsourcing marketplaces, such as Amazon MTurk. An aspect that has remained largely invisible in microtask crowdsourcing is that ...
The Contours of Crowd Capability
HICSS '14: Proceedings of the 2014 47th Hawaii International Conference on System SciencesIn this work we use the theory of Crowd Capital as a lens to compare and contrast a number of IS tools currently in use by organizations for crowd-engagement purposes. In doing so, we contribute to both the practitioner and research domains. For the ...
Crowd Dynamics: Exploring Conflicts and Contradictions in Crowdsourcing
CHI EA '16: Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing SystemsUnfair reputation systems, slow payments, lack of transparency, and socio-spatial inequalities are only some of the many reasons for conflicts in crowdsourcing. The divisive logic of the system and the sharing processes in the peer-community create ...
Comments