Abstract
This paper discusses the CATEGORISE framework meant for establishing a supervised machine learning model without i) the requirement of training labels generated by experts, but by the crowd instead and ii) the labor-intensive manual management of crowdsourcing campaigns. When crowdworking is involved, quality control of results is essential. This control is an additional overhead for an expert diminishing the attractiveness of crowdsourcing. Hence, the requirement for an automated pipeline is that both quality control of labels received and the overall employment process of the crowd can run without the involvement of an expert. To further reduce the number of necessary labels and by this human labor (of the crowd), we make use of Active Learning. This also minimizes time and costs for annotation. Our framework is applied for semantic segmentation of 3D point clouds. We firstly focus on possibilities to overcome the aforementioned challenges by testing different measures for quality control in context of real crowd campaigns and develop the CATEGORISE framework for full automation capabilities, which leverages the microWorkers platform. We apply our approach to two different data sets of different characteristics to prove the feasibility of our method both in terms of accuracy and automation. We show that such a process results in an accuracy comparable to that of Passive Learning. Instead of labeling or administrative responsibilities, the operator solely monitors the progress of the iteration, which runs and terminates (using a proper stopping criterion) in an automated manner.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Bloodgood, M., Vijay-Shanker, K.: A method for stopping active learning based on stabilizing predictions and the need for user-adjustable stopping. In: Proceedings of the Thirteenth Conference on Computational Natural Language Learning (CoNLL-2009), pp. 39–47. Association for Computational Linguistics, Boulder, June 2009. https://www.aclweb.org/anthology/W09-1107
Branson, S., et al.: Visual recognition with humans in the loop. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6314, pp. 438–451. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15561-1_32
Breiman, L.: Random forests. Mach. Learn. 45(1), 5–32 (2001). https://doi.org/10.1023/A:1010933404324
Budhathoki, N.R., Haythornthwaite, C.: Motivation for open collaboration: crowd and community models and the case of OpenStreetMap. Am. Behav. Sci. 57(5), 548–575 (2012). https://doi.org/10.1177/0002764212469364
Buhrmester, M., Kwang, T., Gosling, S.D.: Amazon’s mechanical turk: a new source of inexpensive, yet high-quality, data? Perspect. Psychol. Sci. 6(1), 3–5 (2011). https://doi.org/10.1177/1745691610393980
Cramer, M.: The DGPF-test on digital airborne camera evaluation - overview and test design. Photogrammetr. - Fernerkundung - Geoinf. 2010(2), 73–82 (2010). https://doi.org/10.1127/1432-8364/2010/0041
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Li, F.F.: ImageNet: a large-scale hierarchical image database. In: CVPR 2009, pp. 248–255 (2009). https://doi.org/10.1109/CVPR.2009.5206848
Geiger, A., Lenz, P., Urtasun, R.: Are we ready for autonomous driving? The KITTI vision benchmark suite. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3354–3361 (2012). https://doi.org/10.1109/CVPR.2012.6248074
Google: AI platform data labeling service [WWW Document] (2021). https://cloud.google.com/ai-platform/data-labeling/docs. Accessed 2 June 2021
Graham, B., Engelcke, M., v. d. Maaten, L.: 3D semantic segmentation with submanifold sparse convolutional networks. In: CVPR 2018, pp. 9224–9232 (2018)
Haala, N., Kölle, M., Cramer, M., Laupheimer, D., Mandlburger, G., Glira, P.: Hybrid georeferencing, enhancement and classification of ultra-high resolution uav lidar and image point clouds for monitoring applications. ISPRS Ann. Photogr. Remote Sens. Spat. Inf. Sci. V-2-2020, 727–734 (2020). https://doi.org/10.5194/isprs-annals-V-2-2020-727-2020
Hirth, M., Hoßfeld, T., Tran-Gia, P.: Anatomy of a crowdsourcing platform - using the example of microworkers.com. In: IMIS 2011, pp. 322–329. IEEE Computer Society, Washington (2011). http://dx.doi.org/10.1109/IMIS.2011.89
Hou, J., Graham, B., Nießner, M., Xie, S.: Exploring data-efficient 3D scene understanding with contrastive scene contexts. ArXiv abs/2012.09165 (2020). http://arxiv.org/abs/2012.09165
Hui, Z., et al.: An active learning method for DEM extraction from airborne LiDAR point clouds. IEEE Access 7, 89366–89378 (2019)
Kölle, M., Walter, V., Schmohl, S., Soergel, U.: Hybrid acquisition of high quality training data for semantic segmentation of 3D point clouds using crowd-based active learning. ISPRS Ann. Photogr. Remote Sens. Spat. Inf. Sci. V-2-2020, 501–508 (2020). https://www.isprs-ann-photogramm-remote-sens-spatial-inf-sci.net/V-2-2020/501/2020/
Kölle, M., Walter, V., Schmohl, S., Soergel, U.: Remembering both the machine and the crowd when sampling points: active learning for semantic segmentation of ALS point clouds. In: Del Bimbo, A., et al. (eds.) ICPR 2021. LNCS, vol. 12667, pp. 505–520. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-68787-8_37
Krizhevsky, A.: Learning multiple layers of features from tiny images. Technical report TR-2009, University of Toronto, Toronto (2009)
Kölle, M., et al.: The Hessigheim 3D (H3D) benchmark on semantic segmentation of high-resolution 3D point clouds and textured meshes from UAV lidar and multi-view-stereo. ISPRS Open J. Photogr. Remote Sens. 1, 100001 (2021). https://doi.org/10.1016/j.ophoto.2021.100001
Li, N., Pfeifer, N.: Active learning to extend training data for large area airborne LiDAR classification. ISPRS - Int. Arch. Photogr. Remote Sens. Spat. Inf. Sci. XLII-2/W13, 1033–1037 (2019). https://doi.org/10.5194/isprs-archives-XLII-2-W13-1033-2019
Lin, Y., Vosselman, G., Cao, Y., Yang, M.Y.: Active and incremental learning for semantic ALS point cloud segmentation. ISPRS J. Photogramm. Remote. Sens. 169, 73–92 (2020). https://doi.org/10.1016/j.isprsjprs.2020.09.003
Luo, H., et al.: Semantic labeling of mobile LiDAR point clouds via active learning and higher order MRF. TGRS 56(7), 3631–3644 (2018)
Mackowiak, R., Lenz, P., Ghori, O., Diego, F., Lange, O., Rother, C.: CEREALS - Cost-Effective REgion-based Active Learning for Semantic Segmentation. In: BMVC 2018 (2018). http://arxiv.org/abs/1810.09726
Mandlburger, G., Lehner, H., Pfeifer, N.: A comparison of single photon and full waveform lidar. ISPRS Ann. Photogr. Remote Sens. Spat. Inf. Sci. IV-2/W5, 397–404 (2019). https://doi.org/10.5194/isprs-annals-IV-2-W5-397-2019
Niemeyer, J., Rottensteiner, F., Soergel, U.: Contextual classification of lidar data and building object detection in urban areas. ISPRS J. Photogramm. Remote. Sens. 87, 152–165 (2014). https://doi.org/10.1016/j.isprsjprs.2013.11.001
Penatti, O.A.B., Nogueira, K., dos Santos, J.A.: Do deep features generalize from everyday objects to remote sensing and aerial scenes domains? In: 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 44–51 (2015). https://doi.org/10.1109/CVPRW.2015.7301382
Qi, C.R., Yi, L., Su, H., Guibas, L.J.: PointNet++: deep hierarchical feature learning on point sets in a metric space. In: NIPS 2017, pp. 5105–5114. Curran Associates Inc., USA (2017). http://dl.acm.org/citation.cfm?id=3295222.3295263
Roscher, R., Volpi, M., Mallet, C., Drees, L., Wegner, J.D.: Semcity toulouse: a benchmark for building instance segmentation in satellite images. ISPRS Ann. Photogr. Remote Sens. Spat. Inf. Sci. V-5-2020, 109–116 (2020). https://doi.org/10.5194/isprs-annals-V-5-2020-109-2020
Roynard, X., Deschaud, J.E., Goulette, F.: Paris-Lille-3D: a large and high-quality ground-truth urban point cloud dataset for automatic segmentation and classification. Int. J. Robot. Res. 37(6), 545–557 (2018). https://doi.org/10.1177/0278364918767506
Settles, B.: Active learning literature survey. Computer Sciences Technical report 1648, University of Wisconsin-Madison (2009)
Surowiecki, J.: The Wisdom of Crowds. Anchor (2005)
Vaughan, J.W.: Making better use of the crowd: how crowdsourcing can advance machine learning research. Journ. Mach. Learn. Res. 18(193), 1–46 (2018). http://jmlr.org/papers/v18/17-234.html
Walter, V., Kölle, M., Yin, Y.: Evaluation and optimisation of crowd-based collection of trees from 3D point clouds. ISPRS Ann. Photogr. Remote Sens. Spat. Inf. Sci. V-4-2020, 49–56 (2020). https://doi.org/10.5194/isprs-annals-V-4-2020-49-2020
Walter, V., Soergel, U.: Implementation, results, and problems of paid crowd-based geospatial data collection. PFG 86, 187–197 (2018)
Zhang, J., Wu, X., Sheng, V.S.: Learning from crowdsourced labeled data: a survey. Artif. Intell. Rev. 46(4), 543–576 (2016). https://doi.org/10.1007/s10462-016-9491-9
Zhdanov, F.: Diverse mini-batch active learning. CoRR abs/1901.05954 (2019). http://arxiv.org/abs/1901.05954
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Kölle, M., Walter, V., Shiller, I., Soergel, U. (2021). CATEGORISE: An Automated Framework for Utilizing the Workforce of the Crowd for Semantic Segmentation of 3D Point Clouds. In: Bauckhage, C., Gall, J., Schwing, A. (eds) Pattern Recognition. DAGM GCPR 2021. Lecture Notes in Computer Science(), vol 13024. Springer, Cham. https://doi.org/10.1007/978-3-030-92659-5_41
Download citation
DOI: https://doi.org/10.1007/978-3-030-92659-5_41
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-92658-8
Online ISBN: 978-3-030-92659-5
eBook Packages: Computer ScienceComputer Science (R0)