Skip to main content
SpringerLink
Log in

Learning Representations for Sparse Crowd Answers

  • Conference paper
  • First Online:
Neural Information Processing (ICONIP 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14452))

Included in the following conference series:

  • 442 Accesses

Abstract

When collecting answers from crowds, if there are many instances, each worker can only provide the answers to a small subset of the instances, and the instance-worker answer matrix is thus sparse. The solutions for improving the quality of crowd answers such as answer aggregation are usually proposed in an unsupervised fashion. In this paper, for enhancing the quality of crowd answers used for inferring true answers, we propose a solution with a self-supervised fashion to effectively learn the potential information in the sparse crowd answers. We propose a method named CrowdLR which first learns rich instance and worker representations from the crowd answers based on two types of self-supervised signals. We create a multi-task model with a Siamese structure to learn two classification tasks for two self-supervised signals in one framework. We then utilize the learned representations to complete the answers to fill the missing answers, and can utilize the answer aggregation methods to the complete answers. The experimental results based on real datasets show that our approach can effectively learn the representations from crowd answers and improve the performance of answer aggregation especially when the crowd answers are sparse.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 59.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 79.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Baba, Y., Li, J., Kashima, H.: Crowdea: multi-view idea prioritization with crowds. Proc. AAAI Conf. Human Comput. Crowdsourcing (HCOMP) 8(1), 23–32 (2020). https://doi.org/10.1609/hcomp.v8i1.7460

    Article  Google Scholar 

  2. Bachrach, Y., Minka, T., Guiver, J., Graepel, T.: How to grade a test without knowing the answers: a bayesian graphical model for adaptive crowdsourcing and aptitude testing. In: Proceedings of the 29th International Coference on International Conference on Machine Learning (ICML), pp. 819–826 (2012), https://dl.acm.org/doi/abs/10.5555/3042573.3042680

  3. Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: Proceedings of the 37th International Conference on Machine Learning (ICML) (2020). https://dl.acm.org/doi/abs/10.5555/3524938.3525087

  4. Dawid, A.P., Skene, A.M.: Maximum likelihood estimation of observer error-rates using the EM algorithm. J. Royal Stat. Society. Series C (Applied Statistics) 28(1), 20–28 (1979). https://doi.org/10.2307/2346806

  5. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4171–4186 (2019). https://doi.org/10.18653/v1/N19-1423

  6. Doersch, C., Gupta, A., Efros, A.A.: Unsupervised visual representation learning by context prediction. In: Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), pp. 1422–1430 (2015). https://doi.org/10.1109/ICCV.2015.167

  7. He, X., Liao, L., Zhang, H., Nie, L., Hu, X., Chua, T.S.: Neural collaborative filtering. In: Proceedings of the 26th International Conference on World Wide Web (WWW), pp. 173–182 (2017). https://doi.org/10.1145/3038912.3052569

  8. Hjelm, R.D., et al.: Learning deep representations by mutual information estimation and maximization. In: Proceedings of the Sixth International Conference on Learning Representations (ICLR) (2018). https://doi.org/10.48550/arXiv.1808.06670

  9. Jing, L., Tian, Y.: Self-supervised visual feature learning with deep neural networks: a survey. IEEE Trans. Pattern Anal. Mach. Intell. 43(11), 4037–4058 (2021). https://doi.org/10.1109/TPAMI.2020.2992393

    Article  Google Scholar 

  10. Kolesnikov, A., Zhai, X., Beyer, L.: Revisiting self-supervised visual representation learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1920–1929 (2019). https://doi.org/10.48550/arXiv.1901.09005

  11. Li, H., Zhao, B., Fuxman, A.: The wisdom of minority: discovering and targeting the right group of workers for crowdsourcing. In: Proceedings of the 23rd International Conference on World Wide Web (WWW), pp. 165–176 (2014). https://doi.org/10.1145/2566486.2568033

  12. Li, J., Sun, H., Li, J.: Beyond confusion matrix: learning from multiple annotators with awareness of instance features. Mach. Learn. 112(3), 1053–1075 (2022). https://doi.org/10.1007/s10994-022-06211-x

    Article  MathSciNet  MATH  Google Scholar 

  13. Li, J.: Budget cost reduction for label collection with confusability based exploration. In: Neural Information Processing, pp. 231–241 (2019). https://doi.org/10.1007/978-3-030-36802-9_26

  14. Li, J.: Crowdsourced text sequence aggregation based on hybrid reliability and representation. In: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), pp. 1761–1764 (2020). https://doi.org/10.1145/3397271.3401239

  15. Li, J.: Context-based collective preference aggregation for prioritizing crowd opinions in social decision-making. In: Proceedings of the ACM Web Conference 2022 (WWW), pp. 2657–2667 (2022). https://doi.org/10.1145/3485447.3512137

  16. Li, J., Baba, Y., Kashima, H.: Hyper questions: unsupervised targeting of a few experts in crowdsourcing. In: Proceedings of the 2017 ACM on Conference on Information and Knowledge Management (CIKM), pp. 1069–1078 (2017). https://doi.org/10.1145/3132847.3132971

  17. Li, J., Baba, Y., Kashima, H.: Incorporating worker similarity for label aggregation in crowdsourcing. In: Proceedings of the 27th International Conference on Artificial Neural Networks (ICANN), pp. 596–606 (2018). https://doi.org/10.1007/978-3-030-01421-6_57

  18. Li, J., Baba, Y., Kashima, H.: Simultaneous clustering and ranking from pairwise comparisons. In: Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence (IJCAI), pp. 1554–1560 (2018). https://doi.org/10.24963/ijcai.2018/215

  19. Li, J., Endo, L.R., Kashima, H.: Label aggregation for crowdsourced triplet similarity comparisons. In: Neural Information Processing, pp. 176–185 (2021). https://doi.org/10.1007/978-3-030-92310-5_21

  20. Li, J., Fukumoto, F.: A dataset of crowdsourced word sequences: collections and answer aggregation for ground truth creation. In: Proceedings of the First Workshop on Aggregating and Analysing Crowdsourced Annotations for NLP, pp. 24–28 (Nov 2019). https://doi.org/10.18653/v1/D19-5904

  21. Li, J., Kashima, H.: Iterative reduction worker filtering for crowdsourced label aggregation. In: Proceedings of the 18th International Conference on Web Information Systems Engineering (WISE), pp. 46–54 (2017). https://doi.org/10.1145/978-3-319-68786-5_4

  22. Li, J., Kawase, Y., Baba, Y., Kashima, H.: Performance as a constraint: an improved wisdom of crowds using performance regularization. In: Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence (IJCAI), pp. 1534–1541 (2020). https://doi.org/10.24963/ijcai.2020/213, main track

  23. Lin, T., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 2999–3007 (2017). arXiv:1708.02002

  24. Lu, X., Li, J., Takeuchi, K., Kashima, H.: Multiview representation learning from crowdsourced triplet comparisons. In: Proceedings of the ACM Web Conference 2023 (WWW), pp. 3827–3836 (2023). https://doi.org/10.1145/3543507.3583431

  25. Venanzi, M., Guiver, J., Kazai, G., Kohli, P., Shokouhi, M.: Community-based Bayesian aggregation models for crowdsourcing. In: Proceedings of the 23rd International Conference on World Wide Web (WWW), pp. 155–164 (2014). https://doi.org/10.1145/2566486.2567989

  26. Zhai, X., Oliver, A., Kolesnikov, A., Beyer, L.: S4l: self-supervised semi-supervised learning. In: Proceedings of the IEEE international conference on computer vision (CVPR), pp. 1476–1485 (2019). arXiv:1905.03670

  27. Zhang, G., Li, J., Kashima, H.: Improving pairwise rank aggregation via querying for rank difference. In: 2022 IEEE 9th International Conference on Data Science and Advanced Analytics (DSAA), pp. 1–9 (2022). https://doi.org/10.1109/DSAA54385.2022.10032454

  28. Zheng, Y., Li, G., Li, Y., Shan, C., Cheng, R.: Truth inference in crowdsourcing: is the problem solved? Proc. VLDB Endow. 10(5), 541–552 (2017). https://doi.org/10.14778/3055540.3055547

  29. Zhou, D., Platt, J.C., Basu, S., Mao, Y.: Learning from the wisdom of crowds by minimax entropy. In: Proceedings of the 25th International Conference on Neural Information Processing Systems (NIPS), pp. 2195–2203 (2012). https://dl.acm.org/doi/10.5555/2999325.2999380

  30. Zhou, Y., He, J.: Crowdsourcing via tensor augmentation and completion. In: Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence (IJCAI), pp. 2435–2441 (2016). https://dl.acm.org/doi/10.5555/3060832.3060962

  31. Zuo, X., Li, J., Zhou, Q., Li, J., Mao, X.: Affecti: a game for diverse, reliable, and efficient affective image annotation. In: Proceedings of the 28th ACM International Conference on Multimedia (MM), pp. 529–537 (2020). https://doi.org/10.1145/3394171.3413744

Download references

Acknowledgements

This work was partially supported by JKA Foundation and JSPS KAKENHI Grant Number 23H03402.

Author information

Authors and Affiliations

  1. University of Yamanashi, Kofu, Japan

    Jiyi Li

Authors
  1. Jiyi Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jiyi Li .

Editor information

Editors and Affiliations

  1. Central South University, Changsha, China

    Biao Luo

  2. Chinese Academy of Sciences, Beijing, China

    Long Cheng

  3. Zhejiang University, Hangzhou, China

    Zheng-Guang Wu

  4. Guangdong University of Technology, Guangzhou, China

    Hongyi Li

  5. UNSW Sydney, Sydney, NSW, Australia

    Chaojie Li

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Li, J. (2024). Learning Representations for Sparse Crowd Answers. In: Luo, B., Cheng, L., Wu, ZG., Li, H., Li, C. (eds) Neural Information Processing. ICONIP 2023. Lecture Notes in Computer Science, vol 14452. Springer, Singapore. https://doi.org/10.1007/978-981-99-8076-5_34

Download citation

  • DOI: https://doi.org/10.1007/978-981-99-8076-5_34

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-99-8075-8

  • Online ISBN: 978-981-99-8076-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation