Abstract
Large-scale datasets play an important role in the application of deep learning methods to various practical tasks. Many crowdsourcing tools have been proposed for annotation tasks; however, these tasks are relatively easy. Non-obvious annotation tasks require professional knowledge (e.g., medical image annotation) and non-expert annotators need to be trained to perform such tasks. In this paper, we propose Trafne, a framework for the effective training of non-expert annotators by combining feedback from the system (auto validation) and human experts (expert validation). Subsequently, we present a prototype implementation designed for brain tumor image annotation. We perform a user study to evaluate the effectiveness of our framework compared to a traditional training method. The results demonstrate that our proposed approach can help non-expert annotators to complete a non-obvious annotation more accurately than the traditional method. In addition, we discuss the requirements of non-expert training on a non-obvious annotation and potential applications of the framework.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Brinker, T.J., et al.: Deep neural networks are superior to dermatologists in melanoma image classification. Eur. J. Cancer 119(2019), 11–17 (2019). https://doi.org/10.1016/j.ejca.2019.05.023
Chang, J.C., Amershi, S., Kamar, E.: Revolt: collaborative crowdsourcing for labeling machine learning datasets, pp. 2334–2346. Association for Computing Machinery, New York (2017). https://doi.org/10.1145/3025453.3026044
Clark, K., et al.: The Cancer Imaging Archive (TCIA): maintaining and operating a public information repository. J. Digit. Imaging 26(6), 1045–1057 (2013). https://doi.org/10.1007/s10278-013-9622-7
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database, pp. 248–255 (2009)
Dong, Z., Zhang, R., Shao, X.: Automatic annotation and segmentation of object instances with deep active curve network. IEEE Access 7(2019), 147501–147512 (2019). https://doi.org/10.1109/ACCESS.2019.2946650
Eickhoff, C., de Vries, A.: How crowd sourcable is your task? Mathematical Structures in Computer Science - MSCS (2011)
Ferreira, R., et al.: The virtual microscope. In: Proceedings: A Conference of the American Medical Informatics Association. AMIA Fall Symposium, vol. 4, pp. 49–453 (1997). https://pubmed.ncbi.nlm.nih.gov/9357666
Havaei, M., et al.: Brain tumor segmentation with Deep Neural Networks. Med. Image Anal. 35, 18–31 (2017). https://doi.org/10.1016/j.media.2016.05.004
He, J., Van Ossenbruggen, J., de Vries, A.: Do you need experts in the crowd?: a case study in image annotation for marine biology, pp. 57–60 (2013)
Heim, E., et al.: Large-scale medical image annotation with crowd-powered algorithms. J. Med. Imaging 5(092018), 1 (2018). https://doi.org/10.1117/1.JMI.5.3.034002
Hong, Y., et al.: Deep learning method for comet segmentation and comet assay image analysis. Sci. Rep. 10(1), 1–12 (2020)
Hu, E., Nosato, H., Sakanashi, H., Murakawa, M.: A modified anomaly detection method for capsule endoscopy images using non-linear color conversion and Higher-order Local Auto-Correlation (HLAC), pp. 5477–5480 (2013). https://doi.org/10.1109/EMBC.2013.6610789
Kae, A., Sohn, K., Lee, H., Learned-Miller, E.: Augmenting CRFs with Boltzmann machine shape priors for image labeling (2013)
Kittur, A., Smus, B., Khamkar, S., Kraut, R.: CrowdForge: crowdsourcing complex work. In: CHI 2011, pp. 43–52 (2011). https://doi.org/10.1145/2047196.2047202
Kumaravel, T.S., Vilhar, B., Faux, S., Jha, A.: Comet assay measurements: a perspective. Cell Biol. Toxicol. 25, 53–64 (2007). https://doi.org/10.1007/s10565-007-9043-9
The Medical Imaging Technology Association (MITA): Standard Digital Imaging and Communications in Medicine (2020). https://www.dicomstandard.org/current
Philbrick, K.A., et al.: RIL-contour: a medical imaging dataset annotation tool for and with deep learning. J. Digit. Imaging 32(4), 571–581 (2019). https://doi.org/10.1007/s10278-019-00232-0
Prah, M., Schmainda, K.M.: Data from brain-tumor-progression. Cancer Imaging Arch. (2018). https://doi.org/10.7937/K9/TCIA.2018.15quzvnb
Singla, A., Bogunovic, I., Bartók, G., Karbasi, A., Krause, A.: Near-optimally teaching the crowd to classify, pp. II-154–II-162 (2014)
Su, H., Deng, J., Fei-Fei, L.: Crowdsourcing annotations for visual object detection, pp. 40–46 (2012)
Suzuki, R., Igarashi, T.: Collaborative 3D modeling by the crowd, pp. 124–131 (2017)
Suzuki, S., Abe, K.: Topological structural analysis of digitized binary images by border following. Comput. Vis. Graph. Image Process. 30(1), 32–46 (1985). https://doi.org/10.1016/0734-189X(85)90016-7
Tschandl, P., Rosendahl, C., Kittler, H.: The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Sci. Data 5(2018), 180161 (2018)
van der Wal, R., Sharma, N., Mellish, C., Robinson, A., Siddharthan, A.: The role of automated feedback in training and retaining biological recorders for citizen science. Conserv. Biol. J. Soc. Conserv. Biol. 30, 550–561 (2016). https://doi.org/10.1111/cobi.12705
von Ahn, L.: Human computation, pp. 1–2 (2008). https://doi.org/10.1109/ICDE.2008.4497403
Chang, C.M., Mishra, S.D., Igarashi, T.: A hierarchical task assignment for manual image labeling. In: 2019 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC), pp. 139–143 (2019). https://doi.org/10.1109/VLHCC.2019.8818828
Otani, N., Baba, Y., Kashima, H.: Quality control for crowdsourced hierarchical classification. In: 2015 IEEE International Conference on Data Mining, pp. 937–942 (2015). https://doi.org/10.1109/ICDM.2015.83
Chang, C.M., Lee, C.H., Igarashi, T.: Spatial labeling: leveraging spatial layout for improving label quality in non-expert image annotation. In: CHI Conference on Human Factors in Computing Systems (CHI 2021), Yokohama, Japan, 8–13 May 2021. ACM, New York (2021). https://doi.org/10.1145/3411764.3445165
Steven, D., Kulkarni, A., Bunge, B., Nguyen, T., Klemmer, S., Hartmann, B.: Shepherding the crowd: managing and providing feedback to crowd workers. In: Conference on Human Factors in Computing Systems – Proceedings, pp. 1669–1674 (2011). https://doi.org/10.1145/1979742.1979826
Chang, C.M., Yang, X., Igarashi, T.: An empirical study on the effect of quick and careful labeling styles in image annotation. In: The 48th International Conference on Graphics Interface and Human-Computer Interaction (Gl 2022), Virtual Conference, 17–19 May 2022
Acknowledgements
This work was supported by JST CREST Grant Number JP- MJCR17A1, Japan.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Miyata, S., Chang, CM., Igarashi, T. (2022). Trafne: A Training Framework for Non-expert Annotators with Auto Validation and Expert Feedback. In: Degen, H., Ntoa, S. (eds) Artificial Intelligence in HCI. HCII 2022. Lecture Notes in Computer Science(), vol 13336. Springer, Cham. https://doi.org/10.1007/978-3-031-05643-7_31
Download citation
DOI: https://doi.org/10.1007/978-3-031-05643-7_31
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-05642-0
Online ISBN: 978-3-031-05643-7
eBook Packages: Computer ScienceComputer Science (R0)