Skip to main content
SpringerLink
Log in

All-day Image Alignment for PTZ Surveillance Based on Correlated Siamese Neural Network

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

Image alignment is a highly researched topic in computer vision, which aligns a pair of images due to image changes. Despite the numerous studies conducted on this topic, large object transformation and huge illumination changes between a pair of images are still commonly encountered in real-world scenes, making the task of image alignment very challenging. In this paper, a novel image alignment algorithm is proposed. By inputting a pair of images that need to be aligned into the correlated siamese neural network, a series of blocks are extracted in feature layers from the reference image, and those blocks are correlated in the feature layers of the target image. Finally, the homography parameters between images are then regressed from the correlate layers. Compared with the classical image alignment algorithms, supervised deep homography, and unsupervised deep homography, the experimental results of our method demonstrate a superior performance on the image alignment tasks involving illumination changes, camera translation, and rotation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Availability of data and materials

The data used in this research involve sensitive information; however, we are willing to consider requests for access to a limited portion of the data to support transparency and reproducibility of our findings.

References

  1. Zhao, Q., Ma, Y.K., Zhu, C., Yao, C.F., Feng, B.L., Dai, F.: Image stitching via deep homography estimation. Neurocomputing 450, 219–229 (1995)

    Article  Google Scholar 

  2. Brown, M., Lowe, D.G.: Automatic panoramic image stitching using invariant features. Int. J. Comput. Vis. 74, 59–73 (2007)

    Article  Google Scholar 

  3. Ban, Y., Wang, Y., Liu, S., Yang, B., Liu, M., Yin, L., Zheng, W.: 2D/3D multimode medical image alignment based on spatial histograms. Appl. Sci. 12, 8261 (2022)

    Article  Google Scholar 

  4. Xu, M., Savolainen, T., Anderson, J., Zubko, N., Schuh, H.: Impacts of the image alignment over frequency for vlbi global observing system. Astron. Astrophys. 663 (2022)

  5. Suzuki, K., Inoue, T., Nagata, T., Kasai, M., Nonomura, T., Matsuda, Y.: Markerless image alignment method for pressure-sensitive paint image. Sensors 22, 453 (2022)

    Article  Google Scholar 

  6. Zeng, X., Xu, M.: Gum-Net: Unsupervised geometric matching for fast and accurate 3D subtomogram image alignment and averaging. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recogn. 2020, 4072–4083 (2020)

    Google Scholar 

  7. López-Rubio, E., Molina-Cabello, M.A., Castro, F.M., Luque-Baena, R.M., Marín-Jiménez, M.J., Guil, N.: Anomalous object detection by active search with PTZ cameras. Expert Syst. Appl. 181, 115150 (2021)

    Article  Google Scholar 

  8. Xu, Z., Li, J., Meng, Y., Zhang, X.: CAP-YOLO: channel attention based pruning YOLO for coal mine real-time intelligent monitoring. Sensors 22 (2022)

  9. Xu, X., Chen, X., Wu, B., Wang, Z., Zhen, J.: Exploiting high-fidelity kinematic information from port surveillance videos via a YOLO-based framework. Ocean Coast. Manag. 222, 106117 (2022)

  10. Lowe, D.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60, 91–110 (2004)

    Article  Google Scholar 

  11. Bay, H., Ess, A., Tuytelaars, T., Gool, L.V.: Speeded-up robust features (SURF). In: Computer Vision and Image Understanding, vol. 110, No. 3, pp. 346–359. Similarity Matching in Computer Vision and Multimedia (2008)

  12. Rublee, E., Rabaud, V., Konolige, K., Bradski, G.: ORB: an efficient alternative to SIFT or SURF. In: IEEE International Conference on Computer Vision, pp. 2564–2571 (2011)

  13. Smith, S., Brady, M.: SUSAN-a new approach to low level image processing. Int. J. Comput. Vis. 23, 45–78 (1997)

    Article  Google Scholar 

  14. Dai, W., Kan, H., Tan, R., Yang, B., Guan, Q., Zhu, N., Xiao, W., Dong, Z.: Multisource forest point cloud registration with semantic-guided keypoints and robust ransac mechanisms. Int. J. Appl. Earth Observ. Geoinf. 115, 103105 (2022)

    Google Scholar 

  15. Shi, G., Xu, X., Dai, Y.: SIFT feature point matching based on improved RANSAC algorithm. In: 5th International Conference on Intelligent Human-Machine Systems and Cybernetics, vol. 1, pp. 474–477 (2013)

  16. Li, B., Wu, F., Liu, S., Tang, J., Li, G., Zhong, M., Guan, X.: CA-Unet++: An improved structure for medical CT scanning based on the Unet++ architecture. Int. J. Intell. Syst. 37 (2022)

  17. Dhanya, V.G., Subeesh, A., Kushwaha, N.L., Vishwakarma, D.K., Kumar, T.N., Ritika, G., Singh, A.N.: Deep learning based computer vision approaches for smart agricultural applications. Artif. Intell. Agric. 6, 211–229 (2022)

  18. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. Commun. ACM 60(6), 84–90 (2017)

    Article  Google Scholar 

  19. DeTone, D., Malisiewicz, T., Rabinovich, A.: Deep image homography estimation. abs/1606.03798 (2016)

  20. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. abs/1409.1556 (2014)

  21. Nan, S.N.L.: Feature extraction and segmentation processing of images based on convolutional neural networks. Optical Memory Neural Netw. 30, 67–73 (2021)

    Article  Google Scholar 

  22. Le, H., Liu, F., Zhang, S., Agarwala, A.: Deep homography estimation for dynamic scenes. In: The IEEE Conference on Computer Vision and Pattern Recognition (2020)

  23. Chen, Z., Fang, X.-N., Zhang, S.-H.: Local homography estimation on user-specified textureless regions. J. Comput. Sci. Technol. 37, 615–625 (2022)

    Article  Google Scholar 

  24. Nguyen, T., Chen, S., Shivakumar, S., Taylor, C., Kondepogu, V.: Unsupervised deep homography: a fast and robust homography estimation model. IEEE Robot. Autom. Lett. 3(3), 2346–2353 (2018)

  25. Zhang, J., Wang, C., Liu, S., Jia, L., Wang, J., Zhou, J., Sun, J.: Content-aware unsupervised deep homography estimation. In: European Conference on Computer Vision. Springer (2020)

  26. Gonzalez, R., Faisal, Z.: Digital Image Processing, 2nd Edn (2019)

  27. Lin, Z., Jia, J., Huang, F., Gao, W.: Feature correlation-steered capsule network for object detection. Neural Netw. 147, 25–41 (2022)

    Article  Google Scholar 

  28. Yang, L., Kong, C., Chang, X., Zhao, S., Cao, Y., Zhang, S.: Correlation filters with adaptive convolution response fusion for object tracking. Knowl. Based Syst. 228, 107314 (2021)

  29. Valmadre, J., Bertinetto, L., Henriques, J., Vedaldi, A., Torr, P.H.S.: End-to-end representation learning for correlation filter based tracking. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5000–5008 (2017)

  30. Li, B., Yan, J., Wu, W., Zhu, Z., Hu, X.: High performance visual tracking with siamese region proposal network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018)

  31. Cen, M., Jung, C.: Fully convolutional siamese fusion networks for object tracking. In: 2018 25th IEEE International Conference on Image Processing (ICIP), pp. 3718–3722 (2018)

Download references

Funding

This research did not receive any funding support.

Author information

Authors and Affiliations

  1. Hangzhou Dianzi University, Xiasha Higher Education Zone, Hangzhou, 310018, Zhejiang, China

    Ziteng Hu, Xiaolong Zheng, Shuai Wang, Liang Zheng & Chenggang Yan

  2. Hangzhou Rayin Technology Co., Ltd., No. 399 Danfeng Road, Hangzhou, 310052, Zhejiang, China

    Guangming Xu & Huanhuan Wu

Authors
  1. Ziteng Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiaolong Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shuai Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Guangming Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Huanhuan Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Liang Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Chenggang Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

ZH Validation, Data curation, Formal analysis, Investigation, Writing—original draft. XZ Methodology, Investigation, Formal analysis, Visualization, Writing—original draft, Writing—review and editing. SW Investigation, Formal analysis, Writing—review. GX Data curation, Investigation. HW Data curation, Investigation. LZ Conceptualization, Investigation, Formal analysis. CY Investigation, Formal analysis. All authors reviewed the manuscript.

Corresponding authors

Correspondence to Xiaolong Zheng or Liang Zheng.

Ethics declarations

Conflict of interest

The authors declare that they have no competing interests.

Ethical approval

Not applicable.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hu, Z., Zheng, X., Wang, S. et al. All-day Image Alignment for PTZ Surveillance Based on Correlated Siamese Neural Network. SIViP 18, 615–624 (2024). https://doi.org/10.1007/s11760-023-02720-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-023-02720-x

Keywords

Search

Navigation