Skip to main content
SpringerLink
Log in
Book cover

DAGM German Conference on Pattern Recognition

DAGM GCPR 2021: Pattern Recognition pp 633–648Cite as

CATEGORISE: An Automated Framework for Utilizing the Workforce of the Crowd for Semantic Segmentation of 3D Point Clouds

  • Conference paper
  • First Online:

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 13024))

Abstract

This paper discusses the CATEGORISE framework meant for establishing a supervised machine learning model without i) the requirement of training labels generated by experts, but by the crowd instead and ii) the labor-intensive manual management of crowdsourcing campaigns. When crowdworking is involved, quality control of results is essential. This control is an additional overhead for an expert diminishing the attractiveness of crowdsourcing. Hence, the requirement for an automated pipeline is that both quality control of labels received and the overall employment process of the crowd can run without the involvement of an expert. To further reduce the number of necessary labels and by this human labor (of the crowd), we make use of Active Learning. This also minimizes time and costs for annotation. Our framework is applied for semantic segmentation of 3D point clouds. We firstly focus on possibilities to overcome the aforementioned challenges by testing different measures for quality control in context of real crowd campaigns and develop the CATEGORISE framework for full automation capabilities, which leverages the microWorkers platform. We apply our approach to two different data sets of different characteristics to prove the feasibility of our method both in terms of accuracy and automation. We show that such a process results in an accuracy comparable to that of Passive Learning. Instead of labeling or administrative responsibilities, the operator solely monitors the progress of the iteration, which runs and terminates (using a proper stopping criterion) in an automated manner.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

References

  1. Bloodgood, M., Vijay-Shanker, K.: A method for stopping active learning based on stabilizing predictions and the need for user-adjustable stopping. In: Proceedings of the Thirteenth Conference on Computational Natural Language Learning (CoNLL-2009), pp. 39–47. Association for Computational Linguistics, Boulder, June 2009. https://www.aclweb.org/anthology/W09-1107

  2. Branson, S., et al.: Visual recognition with humans in the loop. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6314, pp. 438–451. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15561-1_32

    Chapter  Google Scholar 

  3. Breiman, L.: Random forests. Mach. Learn. 45(1), 5–32 (2001). https://doi.org/10.1023/A:1010933404324

    Article  MATH  Google Scholar 

  4. Budhathoki, N.R., Haythornthwaite, C.: Motivation for open collaboration: crowd and community models and the case of OpenStreetMap. Am. Behav. Sci. 57(5), 548–575 (2012). https://doi.org/10.1177/0002764212469364

    Article  Google Scholar 

  5. Buhrmester, M., Kwang, T., Gosling, S.D.: Amazon’s mechanical turk: a new source of inexpensive, yet high-quality, data? Perspect. Psychol. Sci. 6(1), 3–5 (2011). https://doi.org/10.1177/1745691610393980

    Article  Google Scholar 

  6. Cramer, M.: The DGPF-test on digital airborne camera evaluation - overview and test design. Photogrammetr. - Fernerkundung - Geoinf. 2010(2), 73–82 (2010). https://doi.org/10.1127/1432-8364/2010/0041

    Article  Google Scholar 

  7. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Li, F.F.: ImageNet: a large-scale hierarchical image database. In: CVPR 2009, pp. 248–255 (2009). https://doi.org/10.1109/CVPR.2009.5206848

  8. Geiger, A., Lenz, P., Urtasun, R.: Are we ready for autonomous driving? The KITTI vision benchmark suite. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3354–3361 (2012). https://doi.org/10.1109/CVPR.2012.6248074

  9. Google: AI platform data labeling service [WWW Document] (2021). https://cloud.google.com/ai-platform/data-labeling/docs. Accessed 2 June 2021

  10. Graham, B., Engelcke, M., v. d. Maaten, L.: 3D semantic segmentation with submanifold sparse convolutional networks. In: CVPR 2018, pp. 9224–9232 (2018)

    Google Scholar 

  11. Haala, N., Kölle, M., Cramer, M., Laupheimer, D., Mandlburger, G., Glira, P.: Hybrid georeferencing, enhancement and classification of ultra-high resolution uav lidar and image point clouds for monitoring applications. ISPRS Ann. Photogr. Remote Sens. Spat. Inf. Sci. V-2-2020, 727–734 (2020). https://doi.org/10.5194/isprs-annals-V-2-2020-727-2020

  12. Hirth, M., Hoßfeld, T., Tran-Gia, P.: Anatomy of a crowdsourcing platform - using the example of microworkers.com. In: IMIS 2011, pp. 322–329. IEEE Computer Society, Washington (2011). http://dx.doi.org/10.1109/IMIS.2011.89

  13. Hou, J., Graham, B., Nießner, M., Xie, S.: Exploring data-efficient 3D scene understanding with contrastive scene contexts. ArXiv abs/2012.09165 (2020). http://arxiv.org/abs/2012.09165

  14. Hui, Z., et al.: An active learning method for DEM extraction from airborne LiDAR point clouds. IEEE Access 7, 89366–89378 (2019)

    Article  Google Scholar 

  15. Kölle, M., Walter, V., Schmohl, S., Soergel, U.: Hybrid acquisition of high quality training data for semantic segmentation of 3D point clouds using crowd-based active learning. ISPRS Ann. Photogr. Remote Sens. Spat. Inf. Sci. V-2-2020, 501–508 (2020). https://www.isprs-ann-photogramm-remote-sens-spatial-inf-sci.net/V-2-2020/501/2020/

  16. Kölle, M., Walter, V., Schmohl, S., Soergel, U.: Remembering both the machine and the crowd when sampling points: active learning for semantic segmentation of ALS point clouds. In: Del Bimbo, A., et al. (eds.) ICPR 2021. LNCS, vol. 12667, pp. 505–520. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-68787-8_37

    Chapter  Google Scholar 

  17. Krizhevsky, A.: Learning multiple layers of features from tiny images. Technical report TR-2009, University of Toronto, Toronto (2009)

    Google Scholar 

  18. Kölle, M., et al.: The Hessigheim 3D (H3D) benchmark on semantic segmentation of high-resolution 3D point clouds and textured meshes from UAV lidar and multi-view-stereo. ISPRS Open J. Photogr. Remote Sens. 1, 100001 (2021). https://doi.org/10.1016/j.ophoto.2021.100001

    Article  Google Scholar 

  19. Li, N., Pfeifer, N.: Active learning to extend training data for large area airborne LiDAR classification. ISPRS - Int. Arch. Photogr. Remote Sens. Spat. Inf. Sci. XLII-2/W13, 1033–1037 (2019). https://doi.org/10.5194/isprs-archives-XLII-2-W13-1033-2019

  20. Lin, Y., Vosselman, G., Cao, Y., Yang, M.Y.: Active and incremental learning for semantic ALS point cloud segmentation. ISPRS J. Photogramm. Remote. Sens. 169, 73–92 (2020). https://doi.org/10.1016/j.isprsjprs.2020.09.003

    Article  Google Scholar 

  21. Luo, H., et al.: Semantic labeling of mobile LiDAR point clouds via active learning and higher order MRF. TGRS 56(7), 3631–3644 (2018)

    Google Scholar 

  22. Mackowiak, R., Lenz, P., Ghori, O., Diego, F., Lange, O., Rother, C.: CEREALS - Cost-Effective REgion-based Active Learning for Semantic Segmentation. In: BMVC 2018 (2018). http://arxiv.org/abs/1810.09726

  23. Mandlburger, G., Lehner, H., Pfeifer, N.: A comparison of single photon and full waveform lidar. ISPRS Ann. Photogr. Remote Sens. Spat. Inf. Sci. IV-2/W5, 397–404 (2019). https://doi.org/10.5194/isprs-annals-IV-2-W5-397-2019

  24. Niemeyer, J., Rottensteiner, F., Soergel, U.: Contextual classification of lidar data and building object detection in urban areas. ISPRS J. Photogramm. Remote. Sens. 87, 152–165 (2014). https://doi.org/10.1016/j.isprsjprs.2013.11.001

    Article  Google Scholar 

  25. Penatti, O.A.B., Nogueira, K., dos Santos, J.A.: Do deep features generalize from everyday objects to remote sensing and aerial scenes domains? In: 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 44–51 (2015). https://doi.org/10.1109/CVPRW.2015.7301382

  26. Qi, C.R., Yi, L., Su, H., Guibas, L.J.: PointNet++: deep hierarchical feature learning on point sets in a metric space. In: NIPS 2017, pp. 5105–5114. Curran Associates Inc., USA (2017). http://dl.acm.org/citation.cfm?id=3295222.3295263

  27. Roscher, R., Volpi, M., Mallet, C., Drees, L., Wegner, J.D.: Semcity toulouse: a benchmark for building instance segmentation in satellite images. ISPRS Ann. Photogr. Remote Sens. Spat. Inf. Sci. V-5-2020, 109–116 (2020). https://doi.org/10.5194/isprs-annals-V-5-2020-109-2020

  28. Roynard, X., Deschaud, J.E., Goulette, F.: Paris-Lille-3D: a large and high-quality ground-truth urban point cloud dataset for automatic segmentation and classification. Int. J. Robot. Res. 37(6), 545–557 (2018). https://doi.org/10.1177/0278364918767506

    Article  Google Scholar 

  29. Settles, B.: Active learning literature survey. Computer Sciences Technical report 1648, University of Wisconsin-Madison (2009)

    Google Scholar 

  30. Surowiecki, J.: The Wisdom of Crowds. Anchor (2005)

    Google Scholar 

  31. Vaughan, J.W.: Making better use of the crowd: how crowdsourcing can advance machine learning research. Journ. Mach. Learn. Res. 18(193), 1–46 (2018). http://jmlr.org/papers/v18/17-234.html

  32. Walter, V., Kölle, M., Yin, Y.: Evaluation and optimisation of crowd-based collection of trees from 3D point clouds. ISPRS Ann. Photogr. Remote Sens. Spat. Inf. Sci. V-4-2020, 49–56 (2020). https://doi.org/10.5194/isprs-annals-V-4-2020-49-2020

  33. Walter, V., Soergel, U.: Implementation, results, and problems of paid crowd-based geospatial data collection. PFG 86, 187–197 (2018)

    Article  Google Scholar 

  34. Zhang, J., Wu, X., Sheng, V.S.: Learning from crowdsourced labeled data: a survey. Artif. Intell. Rev. 46(4), 543–576 (2016). https://doi.org/10.1007/s10462-016-9491-9

    Article  Google Scholar 

  35. Zhdanov, F.: Diverse mini-batch active learning. CoRR abs/1901.05954 (2019). http://arxiv.org/abs/1901.05954

Download references

Author information

Authors and Affiliations

  1. Institute for Photogrammetry, University of Stuttgart, Geschwister-Scholl-Str. 24D, 70174, Stuttgart, Germany

    Michael Kölle, Volker Walter, Ivan Shiller & Uwe Soergel

Authors
  1. Michael Kölle
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Volker Walter
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ivan Shiller
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Uwe Soergel
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Michael Kölle .

Editor information

Editors and Affiliations

  1. Fraunhofer IAIS, Sankt Augustin, Germany

    Christian Bauckhage

  2. University of Bonn, Bonn, Germany

    Juergen Gall

  3. University of Illinois at Urbana-Champaign, Urbana, IL, USA

    Alexander Schwing

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Kölle, M., Walter, V., Shiller, I., Soergel, U. (2021). CATEGORISE: An Automated Framework for Utilizing the Workforce of the Crowd for Semantic Segmentation of 3D Point Clouds. In: Bauckhage, C., Gall, J., Schwing, A. (eds) Pattern Recognition. DAGM GCPR 2021. Lecture Notes in Computer Science(), vol 13024. Springer, Cham. https://doi.org/10.1007/978-3-030-92659-5_41

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-92659-5_41

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-92658-8

  • Online ISBN: 978-3-030-92659-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation