Abstract
Machine learning techniques require an enormous amount of high-quality data labeling for more naturally simulating human comprehension. Recently, mobile crowdsensing, as a new paradigm, makes it possible that a large number of instances can be often quickly labeled at low cost. Existing works only focus on the single labeling for supervised learning problems of traditional machine learning, where one instance associates with only label. However, in many real world applications, an instance may have more than one label. To the end, in this paper, we explore an incremental multi-labeling issue by incentivizing crowd users to label instances under the budget constraint, where each instance is composed of multiple labels. Considering both uncertainty and diversity of the number of each instance’s labels, this paper proposes two mechanisms for incremental multi-labeling crowdsensing by introducing both uncertainty and diversity. Through extensive simulations, we validate their theoretical properties and evaluate the performance.




Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Badanidiyuru, A., Kleinberg, R., & Singer, Y. (2012). Learning on a budget: Posted price mechanisms for online procurement. In Proceedings of the 13th ACM conference on electronic commerce, pp. 128–145.
Kapoor, A., Horvitz, E., & Basu, S. (2007). Selective supervision: Guiding supervised learning with decision-theoretic active learning. In Proceedings of international joint conference on artificial intelligence (IJCAI), pp. 877–882.
Krause, A., & Guestrin, C. (2007). Near-optimal observation selection using submodular function. In Proceedings of AAAI, pp. 1650–1654.
Settles, B., & Craven, M. (2008). An analysis of active learning strategies for sequence labeling tasks. In Proceedings of the conference on empirical methods in natural language processing (EMNLP), pp. 1069–1078.
Yang, B., Sun, J. T., Wang, T., & Chen, Z. (2009). Effective multilabel active learning for text classification. In Proceedings of KDD, pp. 917–926.
Baldridge, J., & Osborne, M. (2004). Active learning and the total cost of annotation. In Proceedings of the conference on empirical methods in natural language processing (EMNLP), pp. 9–16.
Chávez-Martínez, G., Ruiz-Correa, S., & Gatica-Perez, D. (2015). Happy and agreeable? Multi-label classification of impressions in social video. In Proceedings of the 14th international conference on mobile and ubiquitous multimedia, ACM, pp. 109–120.
Culotta, A., & McCallum, A. (2004). Reducing labeling effort for structured prediction tasks. In Proceedings of the national conference on artificial intelligence (AAAI), pp. 746–751.
Duan, L., Oyama, S., Sato, H., & Kurihara, M. (2014). Separate or joint? Estimation of multiple labels from crowdsourced annotations. Expert Systems with Applications, 41(13), 5723–5732.
Yang, D., Xue, G., Fang, X., & Tang, J. (2012). Crowdsourcing to smartphones: Incentive mechanism design for mobile phone sensing. In Proceedings of ACM MobiCom.
Zhou, D., Liu, Q., Platt, J. C., Meek, C., & Shah, N. B. (2016). Regularized minimax conditional entropy for crowdsourcing. arXiv, pp. 1–31.
Zhou, D., Liu, Q., Platt, J. C., Meek, C., & Shah, N. B. (2015). Regularized minimax conditional entropy for crowdsourcing. arXiv preprint arXiv:1503.07240.
Qi, G. J., Hua, X. S., Rui, Y., Tang, J., & Zhang, H. J. (2008). Twodimensional active learning for image classification. In Proceedings of CVPR, pp. 1–8.
Gan, X., Wang, X., Niu, W., Hang, G., Tian, X., Wang, X., & Xu, J. J. (2017). Incentivize multi-class crowd labeling under budget constraint. IEEE Journal on Selected Areas in Communications. doi:10.1109/JSAC.2017.2680838.
Lee, J. S., & Hoh, B. (2010). Dynamic pricing incentive for participatory sensing. Pervasive and Mobile Computing, 6(6), 693–708.
Karger, D. R., Oh, S., & Shah, D. (2013). Efficient crowdsourcing for multi-class labeling. ACM SIGMETRICS Performance Evaluation Review, 41(1), 81–92.
Krishna, R. A., Hata, K., Chen, S., Kravitz, J., Shamma, D. A., Fei-Fei, L., & Bernstein, M. S. (2016). Embracing error to enable rapid crowdsourcing. In Proceedings of the 2016 CHI conference on human factors in computing systems, ACM, pp. 3167–3179.
Duan, L., Kubo, T., Sugiyama, K., Huang, J., Hasegawa, T., & Walrand, J. (2012). Incentive mechanisms for smartphone collaboration in data acquisition and distributed computing. In Proceedings of IEEE INFOCOM, pp. 1701–1709.
Jaimes, L. G., Vergara-Laurens, I., & Labrador, M. A. (2012). A location-based incentive mechanism for participatory sensing systems with budget constraints. In 2012 IEEE international conference on pervasive computing and communications (PerCom), IEEE, pp. 103–108.
Li, X., & Guo, Y. (2013). Active learning with multi-label SVM classification. In IJCAI.
Chen, N., Gravin, N., & Lu, P. (2011). On the approximability of budget feasible mechanisms. In Proceedings of the twenty-second annual ACM-SIAM symposium on discrete algorithms, pp. 685–699.
Zhang, Q., Wen, Y., Tian, X., Gan, X., & Wang, X. (2015). Incentivize crowd labeling under budget constraint. In Proceedings of IEEE INFOCOM.
Raykar, V., & Agrawal, P. (2014). Sequential crowdsourced labeling as an epsilon-greedy exploration in a Markov decision process. In Artificial intelligence and statistics, pp. 832–840.
Chakraborty, S., Balasubramanian, V., & Panchanathan, S. (2011). Optimal batch selection for active learning in multi-label classification. In Proceedings of the 19th ACM international conference on multimedia, pp. 1413–1416.
Huang, S. J., & Zhou, Z. H. (2013). Active query driven by uncertainty and diversity for incremental multi-label learning. In Proceedings of IEEE 13th international conference on data mining, pp. 1079–1084.
Reddy, S., Estrin, D., & Srivastava, M. (2010). Recruitment framework for participatory sensing data collections. In Pervasive computing, Springer, pp. 138–155.
Singer, Y., & Mittal, M. (2013). Pricing mechanisms for crowdsourcing markets. In Proceedings of the 22nd international conference on World Wide Web, International World Wide Web Conferences Steering Committee, pp. 1157–1166.
Singer, Y., & Mittal, M. (2011). Pricing tasks in online labor markets. In Human computation.
Singla, A., & Krause, A. (2013). Truthful incentives in crowdsourcing tasks using regret minimization mechanisms. In Proceedings of ACM WWW, pp. 1167–1177.
Sun, J. (2016). Marginal quality-based long-term incentive mechanisms for crowd sensing. International Journal of Communication Systems, 29(5), 942–958.
Li, X., Wang, L., & Sung, E. (2004). Multilabel SVM active learning for image classification. In Proceedings of ICIP, pp. 2207–2210.
Singer, Y. (2010). Budget feasible mechanisms. IEEE Foundations of Computer Science (FOCS), 2010, 765–774.
Zhang, J., Wu, X., & Sheng, V. S. (2016). Learning from crowdsourced labeled data: A survey. Artificial Intelligence Review, 46(4), 543–576.
Acknowledgements
This work is supported by the National Natural Science Foundation of China under Grant No. 61375021.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Sun, J., Liu, N. & Wu, D. Budget-constraint mechanism for incremental multi-labeling crowdsensing. Telecommun Syst 67, 297–307 (2018). https://doi.org/10.1007/s11235-017-0339-7
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11235-017-0339-7