skip to main content
10.1145/3025453.3026025acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article
Public Access

VoxPL: Programming with the Wisdom of the Crowd

Published: 02 May 2017 Publication History

Abstract

Having a crowd estimate a numeric value is the original inspiration for the notion of "the wisdom of the crowd." Quality control for such estimated values is challenging because prior, consensus-based approaches for quality control in labeling tasks are not applicable in estimation tasks. We present VoxPL, a high-level programming framework that automatically obtains high-quality crowdsourced estimates of values. The VoxPL domain-specific language lets programmers concisely specify complex estimation tasks with a desired level of confidence and budget. VoxPL's runtime system implements a novel quality control algorithm that automatically computes sample sizes and obtains high quality estimates from the crowd at low cost. To evaluate VoxPL, we implement four estimation applications, ranging from facial feature recognition to calorie counting. The resulting programs are concise---under 200 lines of code---and obtain high quality estimates from the crowd quickly and inexpensively.

Supplementary Material

TGZ File (pn4706-file4.tgz)
suppl.mov (pn4706p.mp4)
Supplemental video

References

[1]
2016. Scala Language Specification. Technical Report Version 2.11. École Polytechnique Fédérale de Lausanne. https://www.scala-lang.org/files/archive/spec/2.11/
[2]
Greg Aloupis. 2001. On Computing Geometric Estimators of Location (M.Sc Thesis). McGill University School of Computer Science.
[3]
Yukino Baba and Hisashi Kashima. 2013. Statistical Quality Estimation for General Crowdsourcing Tasks. In Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD '13). ACM, New York, NY, USA, 554--562.
[4]
Daniel W. Barowy, Charlie Curtsinger, Emery D. Berger, and Andrew McGregor. 2012. AutoMan: A Platform for Integrating Human-based and Digital Computation. In OOPSLA 2012. 639--654.
[5]
Daniel W. Barowy, Charlie Curtsinger, Emery D. Berger, and Andrew McGregor. 2016. AutoMan: A Platform for Integrating Human-based and Digital Computation. Commun. ACM 59, 6 (May 2016), 102--109.
[6]
Gerald Van Belle. 2008. Statistical Rules of Thumb (second edition ed.). John Wiley & Sons, Inc., Hoboken, NJ, USA.
[7]
Peter J. Bickel and David A. Freedman. 1981. Some Asymptotic Theory for the Bootstrap. The Annals of Statistics 9, 6 (1981), 1196--1217.
[8]
Jonathan Bragg, Mausam, and Daniel S. Weld. 2013. Crowdsourcing Multi-Label Classification for Taxonomy Creation. In Proceedings of the First AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2013, November 7-9, 2013, Palm Springs, CA, USA. http://www. aaai.org/ocs/index.php/HCOMP/HCOMP13/paper/view/7560
[9]
Peng Dai, Christopher H. Lin, Mausam, and Daniel S. Weld. 2013. POMDP-based control of workflows for crowdsourcing. Artificial Intelligence 202 (Sept. 2013), 52--85.
[10]
Jia Deng, Olga Russakovsky, Jonathan Krause, Michael S. Bernstein, Alex Berg, and Li Fei-Fei. 2014. Scalable Multi-label Annotation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '14). ACM, New York, NY, USA, 3099--3102.
[11]
Olive Jean Dunn. 1961. Multiple Comparisons Among Means. J. Amer. Statist. Assoc. 56, 293 (1961), 52--64.
[12]
B. Efron. 1979. Bootstrap Methods: Another Look at the Jackknife. The Annals of Statistics 7, 1 (1979), 1--26.
[13]
Bradley Efron. 1981. Nonparametric Standard Errors and Confidence Intervals. Canadian Journal of Statistics 9, 2 (1981), 139--158.
[14]
Bradley Efron. 1987. Better Bootstrap Confidence Intervals. J. Amer. Statist. Assoc. 82, 397 (1987), 171--185.
[15]
Michael J. Franklin, Donald Kossmann, Tim Kraska, Sukriti Ramesh, and Reynold Xin. 2011. CrowdDB: Answering Queries with Crowdsourcing. In Proceedings of the 2011 ACM SIGMOD International Conference on Management of Data (SIGMOD '11). ACM, New York, NY, USA, 61--72.
[16]
Francis Galton. 1907. Vox Populi. Nature 75, 1949 (March 1907), 450--451. http://www.nature.com/doifinder/10.1038/075450a0
[17]
Björn Hartman and Eric Horvitz (Eds.). 2013. Proceedings of the First AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2013, November 7--9, 2013, Palm Springs, CA, USA. AAAI. http://www.aaai.org/Library/HCOMP/hcomp13contents.php
[18]
Sture Holm. 1979. A Simple Sequentially Rejective Multiple Test Procedure. Scandinavian Journal of Statistics 6, 2 (1979), 65--70.
[19]
Gary B. Huang, Manjunath Narayana, and Erik G. Learned-Miller. 2008. Towards unconstrained face recognition. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR Workshops 2008, Anchorage, AK, USA, 23-28 June, 2008. IEEE Computer Society, 1--8.
[20]
John P. A. Ioannidis. 2005. Why Most Published Research Findings Are False. PLoS Med 2, 8 (08 2005), e124.
[21]
Panagiotis G. Ipeirotis, Foster Provost, and Jing Wang. 2010. Quality Management on Amazon Mechanical Turk. In Proceedings of the ACM SIGKDD Workshop on Human Computation (HCOMP '10). ACM, New York, NY, USA, 64--67.
[22]
Lilly C. Irani and M. Six Silberman. 2013. Turkopticon: Interrupting Worker Invisibility in Amazon Mechanical Turk. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, New York, NY, USA, 611--620.
[23]
Adam Kapelner. 2014. Statistical Analysis and Design of Crowdsourcing Applications. Ph.D. Dissertation. University of Pennsylvania. http://repository.upenn.edu/dissertations/AAI3622073
[24]
David Kestenbaum. 2015. Weighty Issue: Cow Guessing Game Helps To Explain The Stock Market. (15 Aug. 2015). http://www.npr.org/2015/08/20/432978431
[25]
Aniket Kittur, Jeffrey V. Nickerson, Michael Bernstein, Elizabeth Gerber, Aaron Shaw, John Zimmerman, Matt Lease, and John Horton. 2013. The Future of Crowd Work. In Proceedings of the 2013 Conference on Computer Supported Cooperative Work (CSCW '13). ACM, New York, NY, USA, 1301--1318.
[26]
Aniket Kittur, Boris Smus, Susheel Khamkar, and Robert E. Kraut. 2011. CrowdForge: Crowdsourcing Complex Work. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (UIST '11). ACM, New York, NY, USA, 43--52.
[27]
Ranjay A. Krishna, Kenji Hata, Stephanie Chen, Joshua Kravitz, David A. Shamma, Li Fei-Fei, and Michael S. Bernstein. 2016. Embracing Error to Enable Rapid Crowdsourcing. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 3167--3179.
[28]
Daan Leijen and Erik Meijer. 1999. Domain Specific Embedded Compilers. In Proceedings of the 2Nd Conference on Domain-specific Languages (DSL '99). ACM, New York, NY, USA, 109--122.
[29]
Christopher H. Lin, Mausam, and Daniel S. Weld. 2012. Crowdsourcing Control: Moving Beyond Multiple Choice. In Proceedings of the Twenty-Eighth Conference on Uncertainty in Artificial Intelligence, Catalina Island, CA, USA, August 14-18, 2012, Nando de Freitas and Kevin P. Murphy (Eds.). AUAI Press, 491--500. https://dslpitt.org/papers/12/p491-lin.pdf
[30]
Greg Little, Lydia B. Chilton, Max Goldman, and Robert C. Miller. 2010. TurKit: Human Computation Algorithms on Mechanical Turk. In UIST 2010. 57--66.
[31]
Benjamin Livshits and Todd Mytkowicz. 2014. Saving Money While Polling with InterPoll Using Power Analysis. In Proceedings of the Seconf AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2014, November 2-4, 2014, Pittsburgh, Pennsylvania, USA, Jeffrey P. Bigham and David C. Parkes (Eds.). AAAI. http://www.aaai.org/ocs/index.php/HCOMP/ HCOMP14/paper/view/8979
[32]
Adam Marcus, David Karger, Samuel Madden, Robert Miller, and Sewoong Oh. 2012. Counting with the Crowd. Proc. VLDB Endow. 6, 2 (Dec. 2012), 109--120.
[33]
Adam Marcus, Eugene Wu, Samuel Madden, and Robert C. Miller. 2011. Crowdsourced Databases: Query Processing with People. In CIDR 2011, Fifth Biennial Conference on Innovative Data Systems Research, Asilomar, CA, USA, January 9-12, 2011, Online Proceedings. www.cidrdb.org, 211--214. http: //www.cidrdb.org/cidr2011/Papers/CIDR11_Paper29.pdf
[34]
Winter Mason and Siddharth Suri. 2012. Conducting behavioral research on Amazon's Mechanical Turk. Behavior Research Methods 44, 1 (2012), 1--23.
[35]
Luyi Mo, Reynold Cheng, Ben Kao, Xuan S. Yang, Chenghui Ren, Siyu Lei, David W. Cheung, and Eric Lo. 2013. Optimizing plurality for human intelligence tasks. In Proceedings of the 22nd ACM International Conference on Information & Knowledge Management (CIKM '13). ACM, New York, NY, USA, 1929--1938.
[36]
Austin Myers, Nick Johnston, Vivek Rathod, Anoop Korattikara, Alexander N. Gorban, Nathan Silberman, Sergio Guadarrama, George Papandreou, Jonathan Huang, and Kevin Murphy. 2015. Im2Calories: Towards an Automated Mobile Vision Food Diary. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015. IEEE Computer Society, 1233--1241.
[37]
National Institute of Standards and Technology / SEMATECH. 2013. NIST/SEMATECH e-Handbook of Statistical Methods. http://www.itl.nist.gov/div898/handbook
[38]
Jon Noronha, Eric Hysen, Haoqi Zhang, and Krzysztof Z. Gajos. 2011. Platemate: Crowdsourcing Nutritional Analysis from Food Photographs. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (UIST '11). ACM, New York, NY, USA, 1--12.
[39]
M. Puri, Zhiwei Zhu, Q. Yu, A. Divakaran, and H. Sawhney. 2009. Recognition and Volume Estimation of Food Intake Using a Mobile Device. In Applications of Computer Vision (WACV), 2009 Workshop on. 1--8.
[40]
Alexander J. Quinn and Benjamin B. Bederson. 2011. Human-Machine Hybrid Computation. In CHI 2011 Workshop On Crowdsourcing And Human Computation. http: //crowdresearch.org/chi2011-workshop/papers/quinn.pdf
[41]
Norman Ramsey and Avi Pfeffer. 2002. Stochastic Lambda Calculus and Monads of Probability Distributions. In Proceedings of the 29th ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL '02). ACM, New York, NY, USA, 154--165.
[42]
Pawan Sinha, Benjamin Balas, Yuri Ostrovsky, and Richard Russell. 2007. Face Recognition by Humans: Nineteen Results All Computer Vision Researchers Should Know About. Proc. IEEE 94, 11 (2007), 1948--1962.
[43]
Rion Snow, Brendan O'Connor, Daniel Jurafsky, and Andrew Y. Ng. 2008. Cheap and Fast - But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks. In 2008 Conference on Empirical Methods in Natural Language Processing, EMNLP 2008, Proceedings of the Conference, 25-27 October 2008, Honolulu, Hawaii, USA, A meeting of SIGDAT, a Special Interest Group of the ACL. ACL, 254--263. http://www.aclweb.org/anthology/D08-1027
[44]
James Surowiecki. 2005. The Wisdom of Crowds. Anchor.
[45]
Yaniv Taigman, Ming Yang, Marc-Aurelio Ranzato, and Lior Wolf. 2014. DeepFace: Closing the Gap to Human-Level Performance in Face Verification. In Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR '14). IEEE Computer Society, Washington, DC, USA, 1701--1708.
[46]
Bradley Efron Thomas J. DiCiccio. 1996. Bootstrap Confidence Intervals. Statist. Sci. 11, 3 (1996), 189--212.
[47]
Matthew A. Turk and Alex Pentland. 1991. Face Recognition Using Eigenfaces. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 1991, 3-6 June, 1991, Lahaina, Maui, Hawaii, USA. IEEE, 586--591.
[48]
Luis von Ahn and Laura Dabbish. 2004. Labeling Images with a Computer Game. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '04). ACM, New York, NY, USA, 319--326.
[49]
L. Wasserman. 2006. All of Nonparametric Statistics. Springer New York.

Cited By

View all
  • (2022)The Practice of CrowdsourcingundefinedOnline publication date: 10-Mar-2022
  • (2021)A Safe Collaborative Chatbot for Smart Home AssistantsSensors10.3390/s2119664121:19(6641)Online publication date: 6-Oct-2021
  • (2019)The Practice of CrowdsourcingSynthesis Lectures on Information Concepts, Retrieval, and Services10.2200/S00904ED1V01Y201903ICR06611:1(1-149)Online publication date: 28-May-2019
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
CHI '17: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems
May 2017
7138 pages
ISBN:9781450346559
DOI:10.1145/3025453
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 May 2017

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. crowdprogramming
  2. crowdsourcing
  3. domain-specific languages
  4. quality control
  5. scalability
  6. wisdom of the crowd

Qualifiers

  • Research-article

Funding Sources

Conference

CHI '17
Sponsor:

Acceptance Rates

CHI '17 Paper Acceptance Rate 600 of 2,400 submissions, 25%;
Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI 2025
ACM CHI Conference on Human Factors in Computing Systems
April 26 - May 1, 2025
Yokohama , Japan

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)126
  • Downloads (Last 6 weeks)17
Reflects downloads up to 13 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2022)The Practice of CrowdsourcingundefinedOnline publication date: 10-Mar-2022
  • (2021)A Safe Collaborative Chatbot for Smart Home AssistantsSensors10.3390/s2119664121:19(6641)Online publication date: 6-Oct-2021
  • (2019)The Practice of CrowdsourcingSynthesis Lectures on Information Concepts, Retrieval, and Services10.2200/S00904ED1V01Y201903ICR06611:1(1-149)Online publication date: 28-May-2019
  • (2019)PlanAlyzer: assessing threats to the validity of online experimentsProceedings of the ACM on Programming Languages10.1145/33606083:OOPSLA(1-30)Online publication date: 10-Oct-2019
  • (2019)Computational thinking with the web crowd using CodeMapperProceedings of the 34th ACM/SIGAPP Symposium on Applied Computing10.1145/3297280.3298913(2532-2534)Online publication date: 8-Apr-2019
  • (2018)Integrated Analytics for IIoT Predictive Maintenance Using IoT Big Data Cloud Systems2018 IEEE International Conference on Industrial Internet (ICII)10.1109/ICII.2018.00020(109-118)Online publication date: Oct-2018
  • (2018)A Free-Choice Social Learning Network for Computational Thinking2018 IEEE 18th International Conference on Advanced Learning Technologies (ICALT)10.1109/ICALT.2018.00023(69-71)Online publication date: Jul-2018

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Login options

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media