Skip to main content
SpringerLink
Log in
Book cover

International Symposium on Intelligent Data Analysis

IDA 2017: Advances in Intelligent Data Analysis XVI pp 260–271Cite as

The Combination of Decision in Crowds When the Number of Reliable Annotator Is Scarce

  • Conference paper
  • First Online:

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 10584))

Abstract

Crowdsourcing appears as one of cheap and fast solutions of distributed labor networks. Since the workers have various expertise levels, several approaches to measure annotators reliability have been addressed. There is a condition when annotators who give random answer are abundance and few number of expert is available Therefore, we proposed an iterative algorithm in crowds problem when it is hard to find expert annotators by selecting expert annotator based on EM-Bayesian algorithm, Entropy Measure, and Condorcet Jury’s Theorem. Experimental results using eight datasets show the best performance of our proposed algorithm compared to previous approaches.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Notes

  1. 1.

    https://www.innocentive.com.

  2. 2.

    https://www.mturk.com.

  3. 3.

    www.r-project.org.

  4. 4.

    https://github.com/agusbudi/CRA.

References

  1. Estellés-Arolas, E., González-Ladrón-De-Guevara, F.: Towards an integrated crowdsourcing definition. J. Inf. Sci. 38(2), 189–200 (2012)

    Article  Google Scholar 

  2. Howe, J.: Crowdsourcing: How the Power of the Crowd is Driving the Future of Business. Business books. Random House Business (2008)

    Google Scholar 

  3. Tarasov, A., Delany, S.J., Namee, B.M.: Dynamic estimation of worker reliability in crowdsourcing for regression tasks: making it work. Expert Syst. Appl. 41(14), 6190–6210 (2014)

    Article  Google Scholar 

  4. Heer, J., Bostock, M.: Crowdsourcing graphical perception: using mechanical turk to assess visualization design. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 2010, pp. 203–212. ACM, New York (2010)

    Google Scholar 

  5. Ho, C.J., Jabbari, S., Vaughan, J.W.: Adaptive task assignment for crowdsourced classification. In: Proceedings of the 30th International Conference on International Conference on Machine Learning, ICML2013, vol. 28, pp. I-534-I-542, JMLR.org (2013)

    Google Scholar 

  6. Boutsis, I., Kalogeraki, V.: On task assignment for real-time reliable crowdsourcing. In: 2014 IEEE 34th International Conference on Distributed Computing Systems, pp. 1–10, June 2014

    Google Scholar 

  7. Moayedikia, A., Ong, K.L., Boo, Y.L., Yeoh, W.: Bee colony based worker reliability estimation algorithm in microtask crowdsourcing. In: 2016 15th IEEE International Conference on Machine Learning and Applications (ICMLA), pp. 713–717, December 2016

    Google Scholar 

  8. Dekel, O., Gentile, C., Sridharan, K.: Selective sampling and active learning from single and multiple teachers. J. Mach. Learn. Res. 13(Sep), 2655–2697 (2012)

    MathSciNet  MATH  Google Scholar 

  9. Downs, J.S., Holbrook, M.B., Sheng, S., Cranor, L.F.: Are your participants gaming the system? Screening mechanical turk workers. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 2010, pp. 2399–2402. ACM, New York (2010)

    Google Scholar 

  10. Raykar, V.C., Yu, S.: Eliminating spammers and ranking annotators for crowdsourced labeling tasks. J. Mach. Learn. Res. 13(1), 491–518 (2012)

    MathSciNet  MATH  Google Scholar 

  11. Hernández-González, J., Inza, I., Lozano, J.A.: Multidimensional learning from crowds: usefulness and application of expertise detection. Int. J. Intell. Syst. 30(3), 326–354 (2015)

    Article  Google Scholar 

  12. Zhang, J., Sheng, V.S., Li, Q., Wu, J., Wu, X.: Consensus algorithms for biased labeling in crowdsourcing. Inf. Sci. 382–383, 254–273 (2017)

    Article  Google Scholar 

  13. Dempster, A.P., Laird, N.M., Rubin, D.B.: Maximum likelihood from incomplete data via the em algorithm. J. Roy. Stat. Soc. B 39(1), 1–38 (1977)

    MathSciNet  MATH  Google Scholar 

  14. Condorcet, M.d.: Essai sur l’application de l’analyse à la probabilité des décisions rendues à la pluralité des voix (1785)

    Google Scholar 

  15. Lichman, M.: UCI machine learning repository (2013)

    Google Scholar 

  16. Whitehill, J., Ruvolo, P., Wu, T., Bergsma, J., Movellan, J.: Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In: Proceedings of the 22nd International Conference on Neural Information Processing Systems, NIPS 2009, USA, Curran Associates Inc., pp. 2035–2043 (2009)

    Google Scholar 

  17. Welinder, P., Perona, P.: Online crowdsourcing: Rating annotators and obtaining cost-effective labels. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops, pp. 25–32, June 2010

    Google Scholar 

  18. Raykar, V.C., Yu, S., Zhao, L.H., Valadez, G.H., Florin, C., Bogoni, L., Moy, L.: Learning from crowds. J. Mach. Learn. Res. 11, 1297–1322 (2010)

    MathSciNet  Google Scholar 

  19. Yan, Y., Fung, G., Schmidt, M., Hermosillo, G., Bogoni, L., Moy, L., Dy, J.G.: Modeling annotator expertise: learning when everyone knows a bit of something. In: In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics (AISTATS, 2010), pp. 932–939 (2010)

    Google Scholar 

  20. Wolley, C., Quafafou, M.: Scalable experts selection when learning from noisy labelers. In: 12th International Conference on Machine Learning and Applications (ICMLA) Poster Session (2013)

    Google Scholar 

  21. Zighed, D.A., Ritschard, G., Marcellin, S.: Asymmetric and sample size sensitive entropy measures for supervised learning. In: Ras Z.W., Tsay L.S. (eds.) Advances in Intelligent Information Systems. Studies in Computational Intelligence, vol 265, pp. 27–42. Springer, Heidelberg (2010)

    Google Scholar 

  22. Peleg, B., Zamir, S.: Extending the condorcet jury theorem to a general dependent jury. Soc. Choice Welfare 39(1), 91–125 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  23. Gottlieb, K., Hussain, F.: Voting for image scoring and assessment (visa) - theory and application of a 2 + 1 reader algorithm to improve accuracy of imaging endpoints in clinical trials. BMC Med. Imaging 15(1), 6 (2015)

    Article  Google Scholar 

  24. Xia, L.: Quantitative extensions of the condorcet jury theorem with strategic agents. In: Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12–17, 2016, Phoenix, Arizona, USA, pp. 644–650 (2016)

    Google Scholar 

  25. Jain, B.J.: Condorcet’s jury theorem for consensus clustering. CoRR abs/1604.07711 (2016)

    Google Scholar 

  26. Gehrlein, W.V.: Condorcet’s paradox and the likelihood of its occurrence: different perspectives on balanced preferences*. Theor. Decis. 52(2), 171–199 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  27. Peyton, H.: Group choice and individual judgements, pp. 181–200. Cambridge University Press, Cambridge (1997)

    Google Scholar 

  28. Dawid, A.P.: A.M.S.: Maximum likelihood estimation of observer error-rates using the em algorithm. J. Roy. Stat. Soc.: Ser. C (Appl. Stat.) 28(1), 20–28 (1979)

    Google Scholar 

  29. Yeh, I.C., Yang, K.J., Ting, T.M.: Knowledge discovery on RFM model using Bernoulli sequence. Expert Syst. Appl. 36(3(Part 2)), 5866–5871 (2009)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Aix-Marseille University, CNRS, LSIS UMR 7296, 13397, Marseille, France

    Agus Budi Raharjo & Mohamed Quafafou

Authors
  1. Agus Budi Raharjo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mohamed Quafafou
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Agus Budi Raharjo .

Editor information

Editors and Affiliations

  1. Imperial College London, London, United Kingdom

    Niall Adams

  2. Brunel University London, Uxbridge, United Kingdom

    Allan Tucker

  3. Birkbeck, University of London, London, United Kingdom

    David Weston

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this paper

Cite this paper

Raharjo, A.B., Quafafou, M. (2017). The Combination of Decision in Crowds When the Number of Reliable Annotator Is Scarce. In: Adams, N., Tucker, A., Weston, D. (eds) Advances in Intelligent Data Analysis XVI. IDA 2017. Lecture Notes in Computer Science(), vol 10584. Springer, Cham. https://doi.org/10.1007/978-3-319-68765-0_22

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-68765-0_22

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-68764-3

  • Online ISBN: 978-3-319-68765-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation