Skip to main content
Springer Nature Link
Log in

Dynamic Labeling: A Control System for Labeling Styles in Image Annotation Tasks

  • Conference paper
  • First Online:
Human Interface and the Management of Information (HCII 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14689))

Included in the following conference series:

  • 528 Accesses

Abstract

Labeling style affects labeling efficiency and quality in image annotation tasks. For example, a “label quickly” style can increase labeling efficiency when the data are easy, and a “label carefully” style can increase label quality when the data are difficult. However, the selection of an appropriate labeling style is difficult as different annotators have different experiences and domain knowledge, affecting their subjective feelings of data difficulties (for example, User 1 feels Data A to be easy, while User 2 feels it difficult). In this paper, we propose “Dynamic Labeling” as a control system for labeling styles used in image-labeling tasks. Our control system analyzes the labeling behaviors of annotators (i.e., label selection time) and dynamically assigns an appropriate labeling style (label quickly or label carefully). We conducted a user study to compare a conventional “non-dynamic” and the proposed “dynamic” labeling approaches for an image-labeling task. The results suggest that Dynamic Labeling increased the label quality and labeling efficiency.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Abu-El-Haija, S., et al.: Youtube-8m: a large-scale video classification benchmark. arXiv preprint arXiv:1609.08675 (2016)

  2. Amershi, S., Fogarty, J., Kapoor, A., Tan, D.: Overview based example selection in end user interactive concept learning. In: Proceedings of the 22nd Annual ACM Symposium on User Interface Software and Technology, pp. 247–256 (2009)

    Google Scholar 

  3. Amershi, S., Fogarty, J., Kapoor, A., Tan, D.: Examining multiple potential models in end-user interactive concept learning. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 1357–1360 (2010)

    Google Scholar 

  4. Baba, Y.: Statistical quality control for human computation and crowdsourcing. In: IJCAI, pp. 5667–5671 (2018)

    Google Scholar 

  5. Bernstein, M. S., Brandt, J., Miller, R.C., Karger, D.R.: Crowds in two seconds: enabling realtime crowd-powered interfaces. In: Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, UIST 2011, pp. 33–42 (2011)

    Google Scholar 

  6. Bernstein, M.S., Karger, D.R., Miller, R.C., Brandt, J.: Analytic methods for optimizing realtime crowdsourcing. arXiv preprint arXiv:1204.2995 (2012)

  7. Bianco, S., Ciocca, G., Napoletano, P., Schettini, R.: An interactive tool for manual, semi-automatic and automatic video annotation. Comput. Vis. Image Underst. 131, 88–99 (2015)

    Article  Google Scholar 

  8. Bigham, J.P., et al.: VizWiz: nearly real-time answers to visual questions. In: Proceedings of the 23nd Annual ACM Symposium on User Interface Software and Technology, UIST 2010, pp. 333–342 (2010)

    Google Scholar 

  9. Boutsis, I., Kalogeraki, V.: On task assignment for real-time reliable crowdsourcing. In: 2014 IEEE 34th International Conference on Distributed Computing Systems, pp. 1–10 (2014)

    Google Scholar 

  10. Bragg, J., Weld, D.: Crowdsourcing multi-label classification for taxonomy creation. In: Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, vol. 1, pp. 25–33 (2013)

    Google Scholar 

  11. Burton, M.A., Brady, E., Brewer, R., Neylan, C., Bigham, J.P., Hurst, A.: Crowdsourcing subjective fashion advice using VizWiz: challenges and opportunities. In: Proceedings of the 14th International ACM SIGACCESS Conference on Computers and Accessibility, ASSETS 2012, pp, 135–142 (2012)

    Google Scholar 

  12. Chan, J., Dang, S., Dow, S.P.: Improving crowd innovation with expert facilitation. In: Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing, pp. 1223–1235 (2016)

    Google Scholar 

  13. Chang, C.M., Lee, C.H., Igarashi, T.: Spatial labeling: leveraging spatial layout for improving label quality in non-expert image annotation. In: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1–12 (2021)

    Google Scholar 

  14. Chang, C.M., Mishra, S.D., Igarashi, T.: A hierarchical task assignment for manual image labeling. In: 2019 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC), pp. 139–143. IEEE (2019)

    Google Scholar 

  15. Chang, C. M., Yang, X., Igarashi, T.: An empirical study on the effect of quick and careful labeling styles in image annotation. In: Graphics Interface (2022)

    Google Scholar 

  16. Chang, J.C., Amershi, S., Kamar, E.: Revolt: collaborative crowdsourcing for labeling machine learning datasets. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, pp. 2334–2346 (2017)

    Google Scholar 

  17. Dekel, O., Shamir, O.: Vox Populi: collecting high-quality labels from a crowd. In: COLT (2009)

    Google Scholar 

  18. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255. IEEE (2009)

    Google Scholar 

  19. Difallah, D.E., Demartini, G., Cudré-Mauroux, P.: Pick-a-crowd: tell me what you like, and I’ll tell you what to do. In: Proceedings of the 22nd International Conference on World Wide Web, pp. 367–374 (2013)

    Google Scholar 

  20. Doroudi, S., Kamar, E., Brunskill, E., Horvitz, E.: Toward a learning science for complex crowdsourcing tasks. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, pp. 2623–2634 (2016)

    Google Scholar 

  21. Dow, S., Kulkarni, A., Klemmer, S., Hartmann, B.: Shepherding the crowd yields better work. In Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work, pp. 1013–1022 (2012)

    Google Scholar 

  22. Ellero, A., Ferretti, P., Furlanetto, G.: Realtime crowdsourcing with payment of idle workers in the retainer model. Procedia Econ. Finan. 32, 20–26 (2015)

    Article  Google Scholar 

  23. Fang, Y.L., Sun, H.L., Chen, P.P., Deng, T.: Improving the quality of crowdsourced image labeling via label similarity. J. Comput. Sci. Technol. 32, 877–889 (2017)

    Article  Google Scholar 

  24. Feng, D., Besana, S., Zajac, R.: Acquiring high quality non-expert knowledge from on-demand workforce. In: Proceedings of the 2009 Workshop on The People’s Web Meets NLP: Collaboratively Constructed Semantic Resources (People’s Web), pp. 51–56 (2009)

    Google Scholar 

  25. Fogarty, J., Tan, D., Kapoor, A., Winder, S.: CueFlik: interactive concept learning in image search. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 29–38 (2008)

    Google Scholar 

  26. Gemmeke, J.F., et al.: Audio set: an ontology and human-labeled dataset for audio events. In: 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 776–780. IEEE (2017)

    Google Scholar 

  27. Hansen, D.L., Schone, P.J., Corey, D., Reid, M., Gehring, J.: Quality control mechanisms for crowdsourcing: peer review, arbitration, & expertise at family search indexing. In: Proceedings of the 2013 Conference on Computer Supported Cooperative Work, pp. 649–660 (2013)

    Google Scholar 

  28. He, J., van Ossenbruggen, J., de Vries, A.P.: Do you need experts in the crowd? A case study in image annotation for marine biology. In: Proceedings of the 10th Conference on Open Research Areas in Information Retrieval, pp. 57–60 (2013)

    Google Scholar 

  29. Hsueh, P.Y., Melville, P., Sindhwani, V.: Data quality from crowdsourcing: a study of annotation selection criteria. In: Proceedings of the NAACL HLT 2009 Workshop on Active Learning for Natural Language Processing, pp. 27–35 (2009)

    Google Scholar 

  30. Ipeirotis, P.G., Provost, F., Wang, J.: Quality management on amazon mechanical turk. In: Proceedings of the ACM SIGKDD Workshop on Human Computation, pp. 64–67 (2010)

    Google Scholar 

  31. Kamar, E., Hacker, S., Horvitz, E.: Combining human and machine intelligence in large-scale crowdsourcing. In: AAMAS, vol. 12, pp. 467–474 (2012)

    Google Scholar 

  32. Kim, J., Cheng, J., Bernstein, M.S.: Ensemble: exploring complementary strengths of leaders and crowds in creative collaboration. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing, pp. 745–755 (2014)

    Google Scholar 

  33. Kittur, A., Smus, B., Khamkar, S., Kraut, R.E.: CrowdForge: crowdsourcing complex work. In: Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, pp. 43–52 (2011)

    Google Scholar 

  34. Krishna, R.A., et al.: Embracing error to enable rapid crowdsourcing. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, CHI 2016, pp. 3167–3179 (2016)

    Google Scholar 

  35. Kulesza, T., Amershi, S., Caruana, R., Fisher, D., Charles, D.: Structured labeling for facilitating concept evolution in machine learning. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 3075–3084 (2014)

    Google Scholar 

  36. Kwitt, R., Hegenbart, S., Rasiwasia, N., Vécsei, A., Uhl, A.: Do we need annotation experts? A case study in celiac disease classification. In: Golland, P., Hata, N., Barillot, C., Hornegger, J., Howe, R. (eds.) MICCAI 2014. LNCS, vol. 8674, pp. 454–461. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10470-6_57

    Chapter  Google Scholar 

  37. Laput, G., Lasecki, W.S., Wiese, J., Xiao, R., Bigham, J.P., Harrison, C.: Zensors: adaptive, rapidly deployable, human-intelligent sensor feeds. In: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, CHI 2015, pp. 1935–1944 (2015)

    Google Scholar 

  38. Lasecki, W.S., Gordon, M., Koutra, D., Jung, M.F., Dow, S.P., Bigham, J.P.: Glance: rapidly coding behavioral video with the crowd. In: Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology. UIST 2014, pp. 551–562 (2014)

    Google Scholar 

  39. Lasecki, W.S., Murray, K.I., White, S., Miller, R.C., Bigham, J.P.: Real-time crowd control of existing interfaces. In: Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, UIST 2011, pp. 23–32 (2011)

    Google Scholar 

  40. Lasecki, W.S., Wesley, R., Nichols, J., Kulkarni, A., Allen, J.F., Bigham, J.P.: Chorus: a crowd- powered conversational assistant. In: Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology, UIST 2013, pp. 151–162 (2013)

    Google Scholar 

  41. Li, J., Baba, Y., Kashima, H.: Incorporating worker similarity for label aggregation in crowdsourcing. In: Kůrková, V., Manolopoulos, Y., Hammer, B., Iliadis, L., Maglogiannis, I. (eds.) ICANN 2018. LNCS, vol. 11140, pp. 596–606. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01421-6_57

    Chapter  Google Scholar 

  42. Liu, S., Chen, C., Lu, Y., Ouyang, F., Wang, B.: An interactive method to improve crowdsourced annotations. IEEE Trans. Vis. Comput. Graph. 25(1), 235–245 (2018)

    Article  Google Scholar 

  43. Marcus, M., Santorini, B., Marcinkiewicz, M.A.: Building a large annotated corpus of English: The Penn Treebank (1993)

    Google Scholar 

  44. Mitra, T., Hutto, C.J., Gilbert, E.: Comparing person-and process-centric strategies for obtaining quality data on amazon mechanical turk. In: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, pp. 1345–1354 (2015)

    Google Scholar 

  45. Nguyen, A.T., Lease, M., Wallace, B.C.: Explainable modeling of annotations in crowdsourcing. In: Proceedings of the 24th International Conference on Intelligent User Interfaces, IUI 2019, pp. 575–579 (2019)

    Google Scholar 

  46. Nowak, S., Rüger, S.: How reliable are annotations via crowdsourcing: a study about inter-annotator agreement for multi-label image annotation. In: Proceedings of the International Conference on Multimedia Information Retrieval, pp. 557–566 (2010)

    Google Scholar 

  47. Otani, N., Baba, Y., Kashima, H.: Quality control for crowdsourced hierarchical classification. In: 2015 IEEE International Conference on Data Mining, pp. 937–942. IEEE (2015)

    Google Scholar 

  48. Post, M., Callison-Burch, C., Osborne, M.: Constructing parallel corpora for six Indian languages via crowdsourcing. In: Proceedings of the Seventh Workshop on Statistical Machine Translation, pp. 401–409 (2012)

    Google Scholar 

  49. Rashtchian, C., Young, P., Hodosh, M., Hockenmaier, J.: Collecting image annotations using amazon’s mechanical turk. In: Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk, pp. 139–147 (2010)

    Google Scholar 

  50. Rogstadius, J., Kostakos, V., Kittur, A., Smus, B., Laredo, J., Vukovic, M.: An assessment of intrinsic and extrinsic motivation on task performance in crowdsourcing markets. In: Proceedings of the International AAAI Conference on Web and Social Media, vol. 5, pp. 321–328 (2011)

    Google Scholar 

  51. Russell, B.C., Torralba, A., Murphy, K.P., Freeman, W.T.: LabelMe: a database and web-based tool for image annotation. Int. J. Comput. Vis. 77(1), 157–173 (2008)

    Article  Google Scholar 

  52. Rzeszotarski, J., Kittur, A.: CrowdScape: interactively visualizing user behavior and output. In: Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology, pp. 55–62 (2012)

    Google Scholar 

  53. Safran, M., Che, D.: Real-time recommendation algorithms for crowdsourcing systems. Appl. Comput. Inf. 13(1), 47–56 (2017)

    Google Scholar 

  54. Schilling, M.P., et al.: Label assistant: a workflow for assisted data annotation in image segmentation tasks. In: Proceedings of the 31st Workshop Computational Intelligence, pp. 211–234 (2021)

    Google Scholar 

  55. Sunahase, T., Baba, Y., Kashima, H.: Pairwise hits: quality estimation from pairwise comparisons in creator- evaluator crowdsourcing process. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 31 (2017)

    Google Scholar 

  56. Torralba, A., Russell, B.C., Yuen, J.: Labelme: Online image annotation and applications. Proc. IEEE 98(8), 1467–1484 (2010)

    Article  Google Scholar 

  57. Tran-Thanh, L., Huynh, T.D., Rosenfeld, A., Ramchurn, S. Jennings, N.R.: BudgetFix: budget limited crowdsourcing for interdependent task allocation with quality guarantees. In: Proceedings of the 2014 International Conference on Autonomous Agents and Multi-Agent Systems, AAMAS 2014, pp. 477–484 (2014)

    Google Scholar 

  58. Von Ahn, L., Dabbish, L.: Labeling images with a computer game. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 319–326 (2004)

    Google Scholar 

  59. Zhang, Y., Wang, Y., Zhang, H., Zhu, B., Chen, S., Zhang, D.: OneLabeler: a flexible system for building data labeling tools. In: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, CHI 2022, pp. 1–22 (2022)

    Google Scholar 

  60. Parkhi, O.M., Vedaldi, A., Zisserman, A., Jawahar, C.V.: Cats and dogs. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3498–3505 (2012)

    Google Scholar 

  61. Buhrmester, M., Kwang, T., Gosling, S.D.: Amazon’s mechanical Turk: a new source of inexpensive, yet high-quality, data? Perspect. Psychol. Sci. 6(1), 3–5 (2011)

    Article  Google Scholar 

  62. Peer, E., Vosgerau, J., Acquisti, A.: Reputation as a sufficient condition for data quality on Amazon Mechanical Turk. Behav. Res. Meth. 46(4), 1023–1031 (2014)

    Article  Google Scholar 

  63. Mosheiov, G.: Parallel machine scheduling with a learning effect. J. Oper. Res. Soc. 52(10), 1165–1169 (2001)

    Article  Google Scholar 

  64. Sun, K.T., Lin, Y.C., Yu, C.J.: A study on learning effect among different learning styles in a Web-based lab of science for elementary school students. Comput. Educ. 50(4), 1411–1422 (2008)

    Article  Google Scholar 

  65. Kammerer, Y., Nairn, R., Pirolli, P., Chi, E.H.: Signpost from the masses: learning effects in an exploratory social tag search browser. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 625–634 (2009)

    Google Scholar 

  66. Chang, C.M., He, Y., Yang, X., Xie, H., Igarashi, T.: DualLabel: secondary Labels for Challenging Image Annotation. In: Graphics Interface 2022 (2022)

    Google Scholar 

  67. Miyata, S., Chang, C.M., Igarashi, T.: Trafne: a training framework for non-expert annotators with auto validation and expert feedback. In: Degen, H., Ntoa, S. (eds.) International Conference on Human-Computer Interaction, pp. 475–494. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-05643-7_31

  68. Lu, Y., Chang, C.M., Igarashi, T.: ConfLabeling: assisting image labeling with user and system confidence. In: Chen, J.Y.C., Fragomeni, G., Degen, H., Ntoa, S. (eds.) International Conference on Human-Computer Interaction, pp. 475–494. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-21707-4_26

  69. Zhou, Y., Lu, J., Xiang, C., Chang, C.M., Igarashi, T.: RelRoll: a relative elicitation mechanism for scoring annotation with a case study on speech emotion. In: Graphics Interface 2023 (2023)

    Google Scholar 

  70. Tang, Y., Chang, C.M., Yang, X.: PDFChatAnnotator: a human-LLM collaborative multi-modal data collection tool for PDF-format catalogs. In: Proceedings of the 29th International Conference on Intelligent User Interfaces, IUI 2024 (2024)

    Google Scholar 

Download references

Acknowledgments

This work was supported by JST ACT-X Grant Number JP-MJAX21AG, Japan.

Author information

Authors and Affiliations

  1. The University of Tokyo, Tokyo, Japan

    Chia-Ming Chang

  2. Japan Advanced Institute of Science and Technology, Ishikawa, Japan

    Yi He, Xusheng Du & Haoran Xie

  3. Jilin University, Jilin, China

    Xi Yang

Authors
  1. Chia-Ming Chang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yi He
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xusheng Du
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xi Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Haoran Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chia-Ming Chang .

Editor information

Editors and Affiliations

  1. Tokyo City University, Tokyo, Japan

    Hirohiko Mori

  2. Tokyo University of Science, Tokyo, Japan

    Yumi Asahi

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Chang, CM., He, Y., Du, X., Yang, X., Xie, H. (2024). Dynamic Labeling: A Control System for Labeling Styles in Image Annotation Tasks. In: Mori, H., Asahi, Y. (eds) Human Interface and the Management of Information. HCII 2024. Lecture Notes in Computer Science, vol 14689. Springer, Cham. https://doi.org/10.1007/978-3-031-60107-1_8

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-60107-1_8

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-60106-4

  • Online ISBN: 978-3-031-60107-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics