Abstract
Majority vote is one of the most common methods for crowdsourced label aggregation to get higher-quality labels. In this paper, we extend the work of Donmez et al. that estimates majority labels with a small subset of crowdsourcing workers in order to reduce financial and time costs. Our proposed method estimates the majority labels more accurately by completing missing labels to approximate the whole crowds even if some workers do not answer labels. Experimental results show that the proposed method approximates crowds more accurately than the method without label completion.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Auer, P., Cesa-Bianchi, N., Fischer, P.: Finite-time analysis of the multiarmed bandit problem. Machine Learning 47(2-3), 235–256 (2002)
Donmez, P., Carbonell, J.G., Schneider, J.: Efficiently learning the accuracy of labeling sources for selective sampling. In: Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 259–268 (2009)
Ertekin, S., Hirsh, H., Rudin, C.: Learning to predict the wisdom of crowds. In: Proceedings of Collective Intelligence (2012)
Jung, H.J., Lease, M.: Improving quality of crowdsourced labels via probabilistic matrix factorization. In: Proceedings of the 4th Human Computation Workshop at AAAI, pp. 101–106 (2012)
Lewis, D.D., Gale, W.A.: A sequential algorithm for training text classifiers. In: Proceedings of the 17th Annual International ACM SIGIR Conference on Reserch and Development in Information Retrieval, pp. 3–12 (1994)
Resnick, P., Iacovou, N., Suchak, M., Bergstrom, P., Riedl, J.: Grouplens: an open architecture for collaborative filering of netnews. In: Proceedings of the 1994 ACM Conference on Computer Supported Cooperative Work, pp. 175–186 (1994)
Salakhutdinov, R., Mnih, A.: Probabilistic matrix factorization. In: Advances in Neural Information Processing Systems (NIPS), vol. 20, pp. 1257–1264 (2007)
Snow, R., O’Connor, B., Jurafsky, D., Ng, A.Y.: Cheap and fast - but is it good? evaluating non-expert annotations for natural language tasks. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 254–263 (2008)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer International Publishing Switzerland
About this paper
Cite this paper
Watanabe, T., Kashima, H. (2014). A Label Completion Approach to Crowd Approximation. In: Loo, C.K., Yap, K.S., Wong, K.W., Teoh, A., Huang, K. (eds) Neural Information Processing. ICONIP 2014. Lecture Notes in Computer Science, vol 8835. Springer, Cham. https://doi.org/10.1007/978-3-319-12640-1_46
Download citation
DOI: https://doi.org/10.1007/978-3-319-12640-1_46
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-12639-5
Online ISBN: 978-3-319-12640-1
eBook Packages: Computer ScienceComputer Science (R0)