Abstract
Label noise is a primary point of interest for safety concerns in previous works as it affects the robustness of a machine learning system by a considerable amount. This paper studies the sensitivity of object detection loss functions to label noise in bounding box detection tasks. Although label noise has been widely studied in the classification context, less attention is paid to its effect on object detection. We characterize different types of label noise and concentrate on the most common type of annotation error, which is missing labels. We simulate missing labels by deliberately removing bounding boxes at training time and study its effect on different deep learning object detection architectures and their loss functions. Our primary focus is on comparing two particular loss functions: cross-entropy loss and focal loss. We also experiment on the effect of different focal loss hyperparameter values with varying amounts of noise in the datasets and discover that even up to 50% missing labels can be tolerated with an appropriate selection of hyperparameters. The results suggest that focal loss is more sensitive to label noise, but increasing the gamma value can boost its robustness.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Willers, O., Sudholt, S., Raafatnia, S., Abrecht, S.: Safety concerns and mitigation approaches regarding the use of deep learning in safety-critical perception tasks. In: Casimiro, A., Ortmeier, F., Schoitsch, E., Bitsch, F., Ferreira, P. (eds.) SAFECOMP 2020. LNCS, vol. 12235, pp. 336–350. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-55583-2_25
Wozniak, E., Cârlan, C., Acar-Celik, E., Putzer, H.J.: A safety case pattern for systems with machine learning components. In: Casimiro, A., Ortmeier, F., Schoitsch, E., Bitsch, F., Ferreira, P. (eds.) SAFECOMP 2020. LNCS, vol. 12235, pp. 370–382. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-55583-2_28
Schwalbe, G., et al.: Structuring the safety argumentation for deep neural network based perception in automotive applications. In: Casimiro, A., Ortmeier, F., Schoitsch, E., Bitsch, F., Ferreira, P. (eds.) SAFECOMP 2020. LNCS, vol. 12235, pp. 383–394. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-55583-2_29
Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), vol. 1, pp. 886–893 (2005)
Cortes, C., Vapnik, V.: Support-vector networks. Mach. Learn. 20, 273–297 (1995)
LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521, 436–444 (2015)
Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans. Pattern Anal. Mach. Intell. 39, 1137–1149 (2017)
Liu, W., et al.: SSD: single shot MultiBox detector. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9905, pp. 21–37. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46448-0_2
Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: unified, real-time object detection. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 779–788 (2016)
Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48
Everingham, M., Eslami, S., Van Gool, L., Williams, C., Winn, J., Zisserman, A.: The pascal visual object classes challenge: a retrospective. Int. J. Comput. Vision 111, 98–136 (2014)
Kuznetsova, A., et al.: The open images dataset V4. Int. J. Comput. Vision 128, 1956–1981 (2020)
Adhikari, B., Peltomaki, J., Puura, J., Huttunen, H.: Faster bounding box annotation for object detection in indoor scenes. In: 7th European Workshop on Visual Information Processing (EUVIP), pp. 1–6 (2018)
Zhang, X., Liu, C., Suen, C.: Towards robust pattern recognition: a review. Proc. IEEE 108, 894–922 (2020)
Frenay, B., Verleysen, M.: Classification in the presence of label noise: a survey. IEEE Trans. Neural Netw. Learn. Syst. 25, 845–869 (2014)
Li, C., Zhang, C., Ding, K., Li, G., Cheng, J., Lu, H.: BundleNet: learning with noisy label via sample correlations. IEEE Access. 6, 2367–2377 (2018)
Lee, K., He, X., Zhang, L., Yang, L.: CleanNet: transfer learning for scalable image classifier training with label noise. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5447–5456 (2018)
Su, H., Deng, J., Fei-Fei, L.: Crowdsourcing annotations for visual object detection. In: AAAI Human Computation Workshop, pp. 40–46 (2012)
Russakovsky, O., Li, L., Fei-Fei, L.: Best of both worlds: human-machine collaboration for object annotation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2121–2131 (2015)
Lin, T., Goyal, P., Girshick, R., He, K., Dollar, P.: Focal loss for dense object detection. IEEE Trans. Pattern Anal. Mach. Intell. 42, 318–327 (2020)
Howard, A.G., et al.: MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. arXiv (2017)
Jain, V., Learned-Miller, E.: FDDB: a benchmark for face detection in unconstrained settings. Department of Computer Science, University of Massachusetts. UM-CS-2010-009 (2010)
Acknowledgment
This work was financially supported by Business Finland project 408/31/2018 MIDAS.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Adhikari, B., Peltomäki, J., Germi, S.B., Rahtu, E., Huttunen, H. (2021). Effect of Label Noise on Robustness of Deep Neural Network Object Detectors. In: Habli, I., Sujan, M., Gerasimou, S., Schoitsch, E., Bitsch, F. (eds) Computer Safety, Reliability, and Security. SAFECOMP 2021 Workshops. SAFECOMP 2021. Lecture Notes in Computer Science(), vol 12853. Springer, Cham. https://doi.org/10.1007/978-3-030-83906-2_19
Download citation
DOI: https://doi.org/10.1007/978-3-030-83906-2_19
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-83905-5
Online ISBN: 978-3-030-83906-2
eBook Packages: Computer ScienceComputer Science (R0)