Abstract
Subjective judgment is an important manner of collecting labels which are used for training and evaluating models. Researchers always use a large enough and fixed number of workers to label an instance and then aggregate the labels of an instance from multiple workers into a single label. However, some easy instances only need a small number of workers and some difficult instances need more workers to reach stable aggregated labels. Using a fixed number of workers cannot efficiently use the limited budget. We thus propose an approach for reducing the cost of label collection by assigning a dynamic number of workers to each instance. We propose an Exploration-Focused Upper-Confidence Bound (EFUCB) approach which tends to explore the stable aggregated labels for all instances in the entire dataset. It iteratively selects an instance to ask the workers for one more label. To select the instances for labeling, in contrast to using the collected labels as the reward directly, it utilizes the disagreement of workers to measure the reward of collecting labels for an instance. The experiments based on real datasets verify that our approach can reduce the same budget cost with fewer influences on the aggregated labels so that the utility of the collected labels is preserved.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Abraham, I., Alonso, O., Kandylas, V., Patel, R., Shelford, S., Slivkins, A.: How many workers to ask? Adaptive exploration for collecting high quality labels. In: Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2016, pp. 473–482 (2016)
Auer, P., Cesa-Bianchi, N., Fischer, P.: Finite-time analysis of the multiarmed bandit problem. Mach. Learn. 47(2–3), 235–256 (2002)
Krishna, R.A., et al.: Embracing error to enable rapid crowdsourcing. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, CHI 2016, pp. 3167–3179 (2016)
Lee, M.D., Welsh, M.: An empirical evaluation of models of text document similarity. In: CogSci 2005, pp. 1254–1259. Erlbaum (2005)
Losada, D.E., Parapar, J., Barreiro, A.: Multi-armed bandits for adjudicating documents in pooling-based evaluation of information retrieval systems. Inf. Process. Manag. 53(5), 1005–1025 (2017)
Mozafari, B., Sarkar, P., Franklin, M.J., Jordan, M.I., Madden, S.: Active learning for crowd-sourced databases. CoRR abs/1209.3686 (2012)
Pang, B., Lee, L.: A sentimental education: sentiment analysis using subjectivity summarization based on minimum cuts. In: Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics, ACL 2004 (2004)
Sheng, V.S., Provost, F., Ipeirotis, P.G.: Get another label? Improving data quality and data mining using multiple, noisy labelers. In: Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2008, pp. 614–622 (2008)
Snow, R., O’Connor, B., Jurafsky, D., Ng, A.Y.: Cheap and fast–but is it good? Evaluating non-expert annotations for natural language tasks. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP 2008, pp. 254–263 (2008)
Venanzi, M., Teacy, W., Rogers, A., Jennings, N.R.: Weather sentiment-Amazon Mechanical Turk dataset (2015)
Welinder, P., Branson, S., Belongie, S., Perona, P.: The multidimensional wisdom of crowds. In: Proceedings of the 23rd International Conference on Neural Information Processing Systems, NIPS 2010, pp. 2424–2432 (2010)
Whitehill, J., Ruvolo, P., Wu, T., Bergsma, J., Movellan, J.: Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In: Proceedings of the 22nd International Conference on Neural Information Processing Systems, NIPS 2009, pp. 2035–2043 (2009)
Zhong, J., Tang, K., Zhou, Z.H.: Active learning from crowds with unsure option. In: Proceedings of the 24th International Conference on Artificial Intelligence, IJCAI 2015, pp. 1061–1067 (2015)
Acknowledgments
This work was partially supported by JSPS KAKENHI Grant Number 19K20277.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Li, J. (2019). Budget Cost Reduction for Label Collection with Confusability Based Exploration. In: Gedeon, T., Wong, K., Lee, M. (eds) Neural Information Processing. ICONIP 2019. Communications in Computer and Information Science, vol 1143. Springer, Cham. https://doi.org/10.1007/978-3-030-36802-9_26
Download citation
DOI: https://doi.org/10.1007/978-3-030-36802-9_26
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-36801-2
Online ISBN: 978-3-030-36802-9
eBook Packages: Computer ScienceComputer Science (R0)