Abstract
Classifying urban land use/cover types poses significant challenges due to the complex and heterogeneous nature of urban landscapes. Recent years have witnessed notable advancements in land use/cover classification, driven by improvements in classification methods and the utilization of data from multiple sources. Deep learning networks, especially, have gained prominence in various image analysis tasks, including land use/cover classification. However, when it comes to urban areas, the classification of urban land use/cover encounters additional obstacles, including the complexity of classes, limited training data, and the presence of numerous urban categories. To overcome the limitations arising from similar classes and insufficient training data, we propose a novel approach that integrates hyperspectral and LiDAR data through a Conditional Generative Adversarial Network (CGAN) for semantic segmentation. Our methodology leverages the UNet + + generator and PatchGAN discriminator to achieve accurate segmentation. The CGAN-generated segmented images are then processed by a fully connected neural network (FCN) to classify 20 land use/cover classes. By validating our approach on the 2018 GRSS Data Fusion dataset, our study demonstrates its exceptional operational performance. The CGAN architecture outperforms previous algorithms in terms of class diversity and training data volume. By generating synthetic data that closely resembles the ground truth, the CGAN enhances the classification performance. Clear visual distinctions are observed among various urban features, such as vegetation, trees, buildings, roads, and cars. Classes associated with healthy grass, stressed grass, bare earth, and stadium seats achieve high accuracy. However, road and railway classes exhibit poorer performance due to their similarity with sidewalk, highway, major thoroughfare, and crosswalk classes. Overall, our study showcases a significant improvement in classification accuracy, achieving an approximate accuracy of 96.98% compared to the winning articles presented in the 2018 competition, which achieved accuracies of 64.95% and 76.54%, respectively. This improvement in accuracy can be attributed to the effective extraction and combination of high and low-level urban land cover/land use features within our proposed architecture.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data availability
Data will be made available on request.
Code availability
Code will be made available on request.
References
Akar O, TuncGormus E (2022) Land use/land cover mapping from airborne hyperspectral images with machine learning algorithms and contextual information. Geocarto Int 37(14):3963–3990. https://doi.org/10.1080/10106049.2021.1945149
Bousias Alexakis E, Armenakis C (2020) Evaluation of UNet and UNet++ architectures in high resolution image change detection applications. Int Arch Photogramm Remote Sens Spat Inf Sci 43:1507–1514. https://doi.org/10.5194/isprs-archives-XLIII-B3-2020-1507-2020
Chen XL, Zhao HM, Li PX, Yin ZY (2006) Remote sensing image-based analysis of the relationship between urban heat island and land use/cover changes. Remote Sens Environ 104(2):133–146. https://doi.org/10.1016/j.rse.2005.11.016
Chen Y, Li C, Ghamisi P, Shi C, Gu Y (2016) Deep fusion of hyperspectral and LiDAR data for thematic classification. In: 2016 IEEE International Geoscience and Remote Sensing Symposium (IGARSS), pp 3591–3594. IEEE. https://doi.org/10.1109/IGARSS.2016.7729930
Chen G, Zhang G, Yang Z, Liu W (2023) Multi-scale patch-GAN with edge detection for image inpainting. Appl Intell 53(4):3917–3932. https://doi.org/10.1007/s10489-022-03577-2
Debes C, Merentitis A, Heremans R, Hahn J, Frangiadakis N, van Kasteren T, Pacifici F (2014) Hyperspectral and LiDAR data fusion: outcome of the 2013 GRSS data fusion contest. IEEE J Sel Top Appl Earth Obs Remote Sens 7(6):2405–2418. https://doi.org/10.1109/JSTARS.2014.2305441
Feng Q, Zhu D, Yang J, Li B (2019) Multisource hyperspectral and lidar data fusion for urban land-use mapping based on a modified two-branch convolutional neural network. ISPRS Int J Geo Inf 8(1):28. https://doi.org/10.3390/ijgi8010028
Herold M, Scepan J, Müller A, Günther S (2002) Object-oriented mapping and analysis of urban land use / cover using IKONOS data. In: 22nd Earsel Symposium Geoinformation for European-Wide Integration, pp 4–6
Herold M, Roberts DA, Gardner ME, Dennison PE (2004) Spectrometry for urban area remote sensing—development and analysis of a spectral library from 350 to 2400 nm. Remote Sens Environ 91(3–4):304–319. https://doi.org/10.1016/j.rse.2004.02.013
Huang X, Zhang L, Gong W (2011) Information fusion of aerial images and LIDAR data in urban areas: vector-stacking, re-classification and post-processing approaches. Int J Remote Sens 32(1):69–84. https://doi.org/10.1080/01431160903439882
Huang G, Liu Z, Van Der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4700–4708. https://doi.org/10.48550/arXiv.1608.06993
Isola P, Zhu JY, Zhou T, Efros AA (2017) Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1125–1134. https://doi.org/10.48550/arXiv.1611.07004
Jin L, Wang Z, Lu X (2019) Deep residual U-Net for imbalanced segmentation. IEEE Access 7:73589–73597. https://doi.org/10.1109/ACCESS.2019.2928483
Kervadec H, Bouchtiba R, Desrosiers C, Granger É (2019) Boundary loss for highly unbalanced segmentation. In: Deep learning in medical image analysis and multimodal learning for clinical decision support. Springer, pp 179–187. https://doi.org/10.1007/978-3-030-32245-8_17
Khan M, Wang Q, Bennamoun M, Sohel F (2021) Self-adaptive semi-supervised learning for imbalanced segmentation. IEEE Trans Image Process 30:4305–4318. https://doi.org/10.1109/TIP.2021.3051579
Koetz B, Sun G, Morsdorf F, Ranson KJ, Kneubühler M, Itten K, Allgöwer B (2007) Fusion of imaging spectrometer and LIDAR data over combined radiative transfer models for forest canopy characterization. Remote Sens Environ 106(4):449–459. https://doi.org/10.1016/j.rse.2006.09.013
Li J, Zhao X, Li Y, Du Q, Xi B, Hu J (2018) Classification of hyperspectral imagery using a new fully convolutional neural network. IEEE Geosci Remote Sens Lett 15(2):292–296. https://doi.org/10.1109/LGRS.2017.2786272
Liu P (2021a) a review on remote sensing data fusion with Generative Adversarial Networks (GAN).
Liu P (2021b) A review on remote sens-ing data fusion with Generative Adversarial Networks (GAN). TechRxiv. Preprint. https://doi.org/10.36227/techrxiv.16607261.v1
Liu X, He J, Yao Y, Zhang J, Liang H, Wang H, Hong Y (2017) Classifying urban land use by integrating remote sensing and social media data. Int J Geogr Inf Sci 31:1675–1696. https://doi.org/10.1080/13658816.2017.1324976
Ma J, Shang C, Yu Z, Yu J (2020) Understanding class imbalance in semantic segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp 292–293. https://doi.org/10.1109/CVPRW50498.2020.00043
Man Q, Dong P, Guo H (2015) Pixel-and feature-level fusion of hyperspectral and lidar data for urban land-use classification. Int J Remote Sens 36(6):1618–1644. https://doi.org/10.1080/01431161.2015.1015657
Mishra K, Garg RD (2023) Assessing variations in land cover-land use and surface temperature dynamics for Dehradun, India, using multi-time and multi-sensor landsat data. Environ Monit Assess 195(3):373. https://doi.org/10.1007/s10661-023-10945-z
Moharram MA, Sundaram DM (2023) Land use and land cover classification with hyperspectral data: a comprehensive review of methods, challenges and future directions. Neurocomputing. https://doi.org/10.1016/j.neucom.2023.03.025
Moran EF (2010) Land cover classification in a complex urban-rural landscape with QuickBird imagery. Photogramm Eng Remote Sens 76(10):1159. https://doi.org/10.14358/pers.76.10.1159
Myint SW, Gober P, Brazel A, Grossman-Clarke S, Weng Q (2011) Per-pixel vs. object-based classification of urban land cover extraction using high spatial resolution imagery. Remote Sens Environ 115(5):1145–1161. https://doi.org/10.1016/j.rse.2010.12.017
Nduati E, Sofue Y, Matniyaz A, Park JG, Yang W, Kondoh A (2019) Cropland mapping using fusion of multi-sensor data in a complex urban/peri-urban area. Remote Sensing 11(2):207. https://doi.org/10.3390/rs11020207
Pedergnana M, Marpu PR, Dalla Mura M, Benediktsson JA, Bruzzone L (2012) Classification of remote sensing optical and LiDAR data using extended attribute profiles. IEEE J Sel Top Signal Process 6(7):856–865. https://doi.org/10.1109/JSTSP.2012.2208177
Petropoulos GP, Arvanitis K, Sigrimis N (2012) Hyperion hyperspectral imagery analysis combined with machine learning classifiers for land use/cover mapping. Expert Syst Appl 39(3):3800–3809. https://doi.org/10.1016/j.eswa.2011.09.083
Rahman MT (2016) Detection of land use/land cover changes and urban sprawl in Al-Khobar, Saudi Arabia: an analysis of multi-temporal remote sensing data. ISPRS Int J Geo Inf 5(2):15. https://doi.org/10.3390/ijgi5020015
Saboori M, Homayouni S, Shah-Hosseini R, Zhang Y (2022) Optimum feature and classifier selection for accurate urban land use/cover mapping from very high resolution satellite imagery. Remote Sens 14(9):2097. https://doi.org/10.3390/rs14092097
Shafri HZ (2017) Machine learning in hyperspectral and multispectral remote sensing data analysis. In: Artificial Intelligence Science and Technology: Proceedings of the 2016 International Conference (AIST2016), pp 3–9. https://doi.org/10.1142/9789813206823_0001
Silveyra Gonzalez R, Latifi H, Weinacker H, Dees M, Koch B, Heurich M (2018) Integrating LiDAR and high-resolution imagery for object-based mapping of forest habitats in a heterogeneous temperate forest landscape. Int J Remote Sens 39(23):8859–8884. https://doi.org/10.1080/01431161.2018.1500071
Verhulp J, Van Niekerk A (2016) Effect of inter-image spectral variation on land cover separability in heterogeneous areas. Int J Remote Sens 37(7):1639–1657. https://doi.org/10.1080/01431161.2016.1160300
Wang C, Xu C, Wang C, Tao D (2018) Perceptual adversarial networks for image-to-image transformation. IEEE Trans Image Process 27(8):4066–4079. https://doi.org/10.1109/TIP.2018.2836316
Xu X, Li W, Ran Q, Du Q, Gao L, Zhang B (2017) Multisource remote sensing data classification based on convolutional neural network. IEEE Trans Geosci Remote Sens 56(2):937–949. https://doi.org/10.1109/TGRS.2017.2756851
Xu Y, Du B, Zhang L (2018) Multi-source remote sensing data classification via fully convolutional networks and post-classification processing. In: IGARSS 2018-2018 IEEE International Geoscience and Remote Sensing Symposium. IEEE, pp 3852–3855. https://doi.org/10.1109/IGARSS.2018.8518295.
Xu Y, Du B, Zhang L, Cerra D, Pato M, Carmona E, Le Saux B (2019) Advanced multi-sensor optical remote sensing for urban land use and land cover classification: outcome of the 2018 IEEE GRSS data fusion contest. IEEE J Sel Top Appl Earth Obs Remote Sens 12(6):1709–1724. https://doi.org/10.1109/JSTARS.2019.2911113
Zhang C, Sargent I, Pan X, Li H, Gardiner A, Hare J, Atkinson PM (2019) Joint deep learning for land cover and land use classification. Remote Sens Environ 221:173–187. https://doi.org/10.1016/j.rse.2018.11.014
Zhang Z, Li Z, Wu J, Ye H (2020) Balanced loss for image segmentation. Pattern Recognit 106:107404. https://doi.org/10.1016/j.patcog.2020.107404
Zhou T, Li Z, Pan J (2018a) Multi-feature classification of multi-sensor satellite imagery based on dual-polarimetric sentinel-1A, landsat-8 OLI, and Hyperion images for urban land-cover classification. Sensors 18(2):373. https://doi.org/10.3390/s18020373
Zhou Z, Rahman Siddiquee MM, Tajbakhsh N, Liang J (2018b) Unet++: a nested u-net architecture for medical image segmentation. In: Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support: 4th International Workshop, DLMIA 2018b, and 8th International Workshop, ML-CDS 2018b, Held in Conjunction with MICCAI 2018b, Granada, Spain, September 20, 2018, Proceedings 4. Springer International Publishing, pp 3–11. https://doi.org/10.1007/978-3-030-00889-5_1
Zhu J, Fang L, Ghamisi P (2018) Deformable convolutional neural networks for hyperspectral image classification. IEEE Geosci Remote Sens Lett 15(8):1254–1258. https://doi.org/10.1109/LGRS.2018.2830403
Acknowledgements
The authors are grateful to the committee of the 2018 IEEE GRSS Data Fusion Contest for providing the hyperspectral and LiDAR data, and their efforts in promoting the development of multisource remote-sensing data fusion.
Author information
Authors and Affiliations
Contributions
Conceptualization, all; methodology, A.S. and M.S.; programming, A.S.; validation, all; formal analysis, all; writing—original draft preparation, M.p.; writing—review and editing, M.p., M.S. and M.M.; supervision, M.S., M.M.S. All authors have read and, agreed to the published version of the manuscript.
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing interests.
Additional information
Communicated by: H. Babaie
Publisher's note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Sirous, A., Satari, M., Shahraki, M.M. et al. A Conditional Generative Adversarial Network for urban area classification using multi-source data. Earth Sci Inform 16, 2529–2543 (2023). https://doi.org/10.1007/s12145-023-01050-3
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12145-023-01050-3