Abstract
We propose a novel means to improve the accuracy of semantic segmentation based on multi-task learning. More specifically, in our Multi-Task Semantic Segmentation and Super-Resolution (MT-SSSR) framework, we jointly train a super-resolution and semantic segmentation model in an end-to-end manner using the same task loss for both models. This allows us to optimize the super-resolution model towards producing images that are optimal for the segmentation task, rather than ones that are of high-fidelity. Simultaneously we adapt the segmentation model to better utilize the improved images and thereby improve the segmentation accuracy. We evaluate our approach on multiple public benchmark datasets, and our extensive experimental results show that our novel MT-SSSR framework outperforms other state-of-the-art approaches.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Agustsson, E., Timofte, R.: Ntire 2017 challenge on single image super-resolution: dataset and study. In: CVPR (2017)
Badrinarayanan, V., Kendall, A., Cipolla, R.: Segnet: a deep convolutional encoder-decoder architecture for image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 39(12), 2481–2495 (2017)
Baheti, B., Innani, S., Gajre, S., Talbar, S.N.: Eff-unet: a novel architecture for semantic segmentation in unstructured environment. In: CVPR, pp. 358–359 (2020)
Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: Deeplab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Trans. Pattern Anal. Mach. Intell. 40(4), 834–848 (2017)
Chen, L.C., Zhu, Y., Papandreou, G., Schroff, F., Adam, H.: Encoder-decoder with atrous separable convolution for semantic image segmentation. In: ECCV, pp. 801–818 (2018)
Cordts, M., et al.: The cityscapes dataset for semantic urban scene understanding. In: CVPR, pp. 3213–3223 (2016)
Dai, D., Wang, Y., Chen, Y., Van Gool, L.: Is image super-resolution helpful for other vision tasks? In: WACV, pp. 1–9 (2016)
Dong, C., Loy, C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2016)
Jha, A., Kumar, A., Pande, S., Banerjee, B., Chaudhuri, S.: MT-UNET: a novel u-net based multi-task architecture for visual scene understanding. In: ICIP, pp. 2191–2195 (2020)
Kampffmeyer, M., Salberg, A.B., Jenssen, R.: Semantic segmentation of small objects and modeling of uncertainty in urban remote sensing images using deep convolutional neural networks. In: CVPR-W, pp. 1–9 (2016)
Maninis, K.K., Radosavovic, I., Kokkinos, I.: Attentive single-tasking of multiple tasks. In: CVPR, pp. 1851–1860 (2019)
Minaee, S., Boykov, Y., Porikli, F., Plaza, A., Kehtarnavaz, N., Terzopoulos, D.: Image segmentation using deep learning: A survey. arXiv preprint (2020)
Mishra, A., Kumar, S., Kalluri, T., Varma, G., Subramaian, A., Chandraker, M., Jawahar, C.V.: Semantic segmentation datasets for resource constrained training. In: NCVPRIPG, vol. 2, p. 6 (2020)
Na, B., Fox, G.C.: Object classifications by image super-resolution preprocessing for convolutional neural networks. ASTESJ 5(2), 476–483 (2020)
Robert, V., Talbot, H.: Does super-resolution improve OCR performance in the real world? ICIP, a case study on images of receipts. In: 2020 IEEE International Conference on Image Processing (ICIP), pp. 548–552 (2020)
Romera, E., Alvarez, J.M., Bergasa, L.M., Arroyo, R.: Erfnet: efficient residual factorized convnet for real-time semantic segmentation. T-ITS 19(1), 263–272 (2018)
Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28
Shermeyer, J., Van Etten, A.: The effects of super-resolution on object detection performance in satellite imagery. In: CVPR-W (2019)
Shrivastava, A., Gupta, A., Girshick, R.: Training region-based object detectors with online hard example mining. In: CVPR, pp. 761–769 (2016)
Tan, M., Le, Q.V.: Efficientnet: rethinking model scaling for convolutional neural networks. In: ICML, pp. 6105–6114 (2019)
Varma, G., Subramanian, A., Namboodiri, A.M., Chandraker, M., Jawahar, C.V.: IDD: a dataset for exploring problems of autonomous navigation in unconstrained environments. In: WACV, pp. 1743–1751 (2019)
Wang, J., et al.: Deep high-resolution representation learning for visual recognition. TPAMI (2019)
Wang, L., Li, D., Zhu, Y., Tian, L., Shan, Y.: Dual super-resolution learning for semantic segmentation. In: CVPR, pp. 3774–3783 (2020)
Wang, X., et al.: Esrgan: enhanced super-resolution generative adversarial networks. In: ECCV, vol. 38, pp. 295–307 (2019)
Yuan, Y., Liu, S., Zhang, J., Zhang, Y., Dong, C., Lin, L.: Unsupervised image super-resolution using cycle-in-cycle generative adversarial networks. In: CVPR (2018)
Zhao, B., Feng, J., Wu, X., Yan, S.: A survey on deep learning-based fine-grained object classification and semantic segmentation. Int. J. Autom. Comput. 14(2), 119–135 (2017). https://doi.org/10.1007/s11633-017-1053-3
Zhao, H., Shi, J., Qi, X., Wang, X., Jia, J.: Pyramid scene parsing network. In: CVPR (2017)
Zhao, S., Wang, Y., Yang, Z., Cai, D.: Region mutual information loss for semantic segmentation. In: NIPS (2019)
Acknowledgements
This work was partially supported by the Milestone Research Programme at Aalborg University (MRPA) and Danmarks Frie Forskningsfond (DFF 8022-00360B).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Aakerberg, A., Johansen, A.S., Nasrollahi, K., Moeslund, T.B. (2021). Single-Loss Multi-task Learning For Improving Semantic Segmentation Using Super-Resolution. In: Tsapatsoulis, N., Panayides, A., Theocharides, T., Lanitis, A., Pattichis, C., Vento, M. (eds) Computer Analysis of Images and Patterns. CAIP 2021. Lecture Notes in Computer Science(), vol 13053. Springer, Cham. https://doi.org/10.1007/978-3-030-89131-2_37
Download citation
DOI: https://doi.org/10.1007/978-3-030-89131-2_37
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-89130-5
Online ISBN: 978-3-030-89131-2
eBook Packages: Computer ScienceComputer Science (R0)