Skip to main content
SpringerLink
Log in

Scene change detection: semantic and depth information

  • 1182: Deep Processing of Multimedia Data
  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Change detection is widely used in city construction, remote image detection, autonomous driving and so on. The most important challenge in change detection is how to define the real change, that is to say, how to detect the real changes while ignoring the unexpected changes? such as varying illumination, shadows, seasonal scene changes. Motivated by the key issue, in this paper, we propose a novel two-classified method that utilizes the depth information to assist the semantic information for better detecting changes. A gradual modification strategy is also designed to combine the high-level semantics and the low-level edge-sensitive features to achieve better depth estimation. The proposed framework was evaluated by using the VL-CMU-CD streetscape change detection dataset. Both quantitative and qualitative experiments have been implemented for evaluating the performance of the framework under different light and seasons. Experimental results show that the proposed method outperforms most of the current state-of-the-art results.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Aach T, Dümbgen L, Mester R, Toth D (2001) Bayesian illumination-invariant motion detection. In: International conference on image processing

  2. Alcantarilla PF, Stent S, Ros G, Arroyo R, Gherardi R (2018) Street-view change detection with deconvolutional networks. Auton Robot 42(7):1301–1322

    Article  Google Scholar 

  3. Badino H, Huber D, Kanade T (2011) Visual topometric localization. In: 2011 IEEE Intelligent vehicles symposium (IV), pp 794–799. IEEE

  4. Badino H, Huber D, Kanade T (2012) Real-time topometric localization. In: 2012 IEEE International conference on robotics and automation, pp 1635–1642. IEEE

  5. Bovolo F, Bruzzone L (2006) A theoretical framework for unsupervised change detection based on change vector analysis in the polar domain. IEEE Transactions on Geoscience & Remote Sensing 45(1):218–236

    Article  Google Scholar 

  6. Chen W, Qian S, Deng J (2019) Learning single-image depth from videos using quality assessment networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5604–5613

  7. Chu Y, Cao G, Hayat H (2016) Change detection of remote sensing image based on deep neural networks. In: International conference on artificial intelligence & industrial engineering

  8. Eigen D, Fergus R (2015) Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture. In: IEEE International conference on computer vision

  9. Eigen D, Puhrsch C, Fergus R (2014) Depth map prediction from a single image using a multi-scale deep network

  10. Furukawa Y, Suzuki K, Hamaguchi R, Onishi M, Sakurada K (2020) Self-supervised simultaneous alignment and change detection

  11. Garg R, B.G. V K, Carneiro G, Reid I (2016) Unsupervised cnn for single view depth estimation: Geometry to the rescue. In: European conference on computer vision

  12. Geiger A (2012) Are we ready for autonomous driving? the kitti vision benchmark suite. In: IEEE Conference on computer vision & pattern recognition

  13. Godard C, Aodha OM, Brostow GJ (2017) Unsupervised monocular depth estimation with left-right consistency. In: Computer vision & pattern recognition

  14. Guo E, Fu X, Zhu J, Deng M, Liu Y, Zhu Q, Li H (2018) Learning to measure change: Fully convolutional siamese metric networks for scene change detection. arXiv:1810.09111

  15. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  16. Jaderberg M, Simonyan K, Zisserman A, Kavukcuoglu K (2015) Spatial transformer networks. arXiv:1506.02025

  17. Khan M, Li M, Teoh A, Leng L (2014) A remote cancelable palmprint authentication protocol based on multi-directional two-dimensional palmphasor-fusion. Security and Communication Networks

  18. Khan S, He X, Porikli F, Bennamoun M, Sohel F, Togneri R (2017) Learning deep structured network for weakly supervised change detection. In: Twenty-sixth international joint conference on artificial intelligence

  19. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105

  20. Laina I, Rupprecht C, Belagiannis V, Tombari F, Navab N (2016) Deeper depth prediction with fully convolutional residual networks. In: 3D Vision (3DV), 2016 fourth international conference on, pp 239–248. IEEE

  21. Leng L, Li M, Kim C, Bi X (2017) Dual-source discrimination power analysis for multi-instance contactless palmprint recognition. Multi Tools App 76(1):333–354

    Article  Google Scholar 

  22. Leng L, Li M, Teoh ABJ (2014) Conjugate 2dpalmhash code for secure palm-print-vein verification. In: International congress on image and signal processing

  23. Leng L, Teoh ABJ (2015) Alignment-free row-co-occurrence cancelable palmprint fuzzy vault. Pattern Recogn 48(7):2290–2303

    Article  Google Scholar 

  24. Leng L, Zhang J (2011) Dual-key-binding cancelable palmprint cryptosystem for palmprint protection and information security. J Netw Comput Appl 34 (6):1979–1989

    Article  Google Scholar 

  25. Leng L, Zhang J (2013) Palmhash code vs. palmphasor code. Neurocomputing 108(may 2):1–12

    Article  Google Scholar 

  26. Li B, Shen C, Dai Y, Hengel AVD, He M (2015) Depth and surface normal estimation from monocular images using regression on deep features and hierarchical crfs. In: Computer vision & pattern recognition

  27. Lillestrand R L (1972) Techniques ror change detection. IEEE Trans Comput C-21(7):654–659. https://doi.org/10.1109/T-C.1972.223570

    Article  Google Scholar 

  28. Liu F, Shen C, Lin G (2014) Deep convolutional neural fields for depth estimation from a single image. arXiv:1411.6387

  29. Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: 2015 IEEE Conference on computer vision and pattern recognition (CVPR), pp 3431–3440

  30. Lorenzo B, Diego Fernndez P (2002) An adaptive semiparametric and context-based approach to unsupervised change detection in multitemporal remote-sensing images. IEEE Transactions on Image Processing A Publication of the IEEE Signal Processing Society 11(4):452–66

    Article  Google Scholar 

  31. Luo Y, Ren JSJ, Lin M, Pang J, Sun W, Li H, Lin L (2018) Single view stereo matching. arXiv:1803.02612

  32. Ma W, Xiong Y, Wu Y, Yang H, Zhang X, Jiao L (2019) Change detection in remote sensing images based on image mapping and a deep capsule network. Remote Sens 11(6):626

    Article  Google Scholar 

  33. Mahmoodzadeh H (2007) Digital change detection using remotely sensed data for monitoring green space destruction in tabriz. Int J Environ Res 1 (1):35–411735

    Google Scholar 

  34. Mayer N, Ilg E, Hausser P, Fischer P, Cremers D, Dosovitskiy A, Brox T (2016) A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4040–4048

  35. Moser G, Angiati E, Serpico S B (2011) Multiscale unsupervised change detection on optical images by markov random fields and wavelets. IEEE Geosci Remote Sens Lett 8(4):725–729. https://doi.org/10.1109/LGRS.2010.2102333

    Article  Google Scholar 

  36. Noh H, Hong S, Han B (2015) Learning deconvolution network for semantic segmentation. arXiv:1505.04366

  37. Peng D, Zhang Y, Guan H (2019) End-to-end change detection for high resolution satellite images using improved unet++. Remote Sens 11(11):1382

    Article  Google Scholar 

  38. Radke RJ, Srinivas A, Omar AK, Badrinath R (2005) Image change detection algorithms: a systematic survey. IEEE Trans Image Process 14 (3):294–307

    Article  MathSciNet  Google Scholar 

  39. Ronneberger O, Fischer P, Brox T (2015) U-net: Convolutional networks for biomedical image segmentation. In: Medical image computing and computer-assisted intervention (MICCAI), Springer, LNCS, vol 9351, pp 234–241, http://www.lmb.informatik.uni-freiburg.de/Publications/2015/RFB15a. (available on arXiv:1505.04597 [cs.CV])

  40. Rosin PL (1998) Thresholding for change detection. In: International conference on computer vision

  41. Rosin PL, Ioannidis E (2003) Evaluation of global image thresholding for change detection. Pattern Recogn Lett 24(14):2345–2356

    Article  Google Scholar 

  42. Saha S, Bovolo F, Bruzzone L (2019) Unsupervised deep change vector analysis for multiple-change detection in vhr images. IEEE Trans Geosci Remote Sens 57(6):3677–3693

    Article  Google Scholar 

  43. Sakurada K, Shibuya M, Wang W (2020) Weakly supervised silhouette-based semantic scene change detection. In: 2020 IEEE International conference on robotics and automation (ICRA), pp 6861–6867. IEEE

  44. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556

  45. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1–9

  46. Taneja A, Ballan L, Pollefeys M (2011) Image based detection of geometric changes in urban environments. In: IEEE International conference on computer vision

  47. Toth D, Aach T, Metzler V (2000) Illumination-invariant change detection. In: 4th IEEE Southwest symposium on image analysis and interpretation, pp 3–7

  48. Ulstad MS (1973) An algorithm for estimating small scale differences between two digital images. Pattern Recogn 5(4):323,IN1,331–330,IN2,333

    Article  Google Scholar 

  49. Xian K, Shen C, Cao Z, Lu H, Xiao Y, Li R, Luo Z (2018) Monocular relative depth perception with web stereo data supervision. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)

  50. Zhao J, Gong M, Jia L, Jiao L (2014) Deep learning to classify difference image for image change detection. In: International joint conference on neural networks

Download references

Acknowledgements

This work was supported in part by National Science Fund of China no.61871170 and The National Defense Basic Research Program of JCKY2017210A001.

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, Hangzhou Dianzi University, Hangzhou, China

    Jianjun Li, Peiqi Tang, Yong Wu & Mian Pan

  2. Key Laboratory of Brain Machine Collaborative Intelligence of Zhejiang Province, Hangzhou, China

    Jianjun Li, Peiqi Tang, Yong Wu & Mian Pan

  3. Key Laboratory of Data Link Technology of CETC, Xi’An, China

    Zheng Tang & Guobao Hui

Authors
  1. Jianjun Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Peiqi Tang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yong Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Mian Pan
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zheng Tang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Guobao Hui
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jianjun Li.

Additional information

Declarations

The authors declare that they have no conflict of interest.

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, J., Tang, P., Wu, Y. et al. Scene change detection: semantic and depth information. Multimed Tools Appl 81, 19301–19319 (2022). https://doi.org/10.1007/s11042-021-10793-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-021-10793-4

Keywords

Search

Navigation