Skip to main content
SpringerLink
Log in

SuperGlue-based accurate feature matching via outlier filtering

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

The feature matching algorithm based on deep learning has achieved superior performance compared to traditional algorithms in terms of both matching quantity and accuracy, but there are still some high-error matching results in complex scenes, which adversely affects the subsequent work. Based on SuperGlue, we propose an accurate feature matching algorithm via outlier filtering. Firstly, DBSCAN real-time superpixel segmentation (RTSS-DBSCAN) is used to divide the image into regions, and then the outlier filtering module is designed according to the local similarity principle of feature matching. On the premise of not affecting the correct matching results, the matching results with high errors will be filtered to improve the matching accuracy. Meanwhile, due to the lag of traditional Exponential Moving Average algorithm (EMA), an adaptive EMA is designed and integrated into the SuperGlue training process to further improve the training speed and matching accuracy. We evaluate the overall performance of the matching method using the AUC of pose error at the thresholds (5°, 10°, 20°), a common evaluation metric, to provide a more detailed and intuitive evaluation of the matching effectiveness using precision and recall. The experimental results show that the method in this paper can effectively filter the matching results with large errors and has high accuracy and robustness. The AUC of pose error at thresholds (5°, 10°, 20°) reaches 36.53, 56.23, and 73.68, and the precision and recall reach 80.07 and 91.52, respectively, which have better matching results compared with other algorithms.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Availability of data and materials

The datasets used or analyzed during the current study are available from the corresponding author on reasonable request.

Code availability

The codes used during the current study are available from the corresponding author on reasonable request.

References

  1. Ma, J., Jiang, X., Fan, A., et al.: Image matching from handcrafted to deep features: a survey. Int. J. Comput. Vis. 129, 23–79 (2021). https://doi.org/10.1007/s11263-020-01359-2

    Article  MathSciNet  Google Scholar 

  2. Campos, C., Elvira, R., Rodríguez, J.J.G., et al.: Orb-slam3: an accurate open-source library for visual, visual–inertial, and multimap slam. IEEE Trans. Robot. 37(6), 1874–1890 (2021)

    Article  Google Scholar 

  3. He, M., Zhu, C., Huang, Q., et al.: A review of monocular visual odometry. Vis. Comput. 36, 1053–1065 (2020). https://doi.org/10.1007/s00371-019-01714-6

    Article  Google Scholar 

  4. Fu, Y., Yan, Q., Liao, J., et al.: Real-time dense 3D reconstruction and camera tracking via embedded planes representation. Vis. Comput. 36, 2215–2226 (2020). https://doi.org/10.1007/s00371-020-01899-1

    Article  Google Scholar 

  5. Cai, Y., Li, L., Wang, D., et al.: GlcMatch: global and local constraints for reliable feature matching. Vis. Comput. (2022). https://doi.org/10.1007/s00371-022-02478-2

    Article  Google Scholar 

  6. Balntas, V., Riba, E., Ponsa, D., et al.: Learning local feature descriptors with triplets and shallow convolutional neural networks. Bmvc. 1(2), 3 (2016)

    Google Scholar 

  7. Demarche, C., Harari, D.: Duality for complexes of tori over a global field of positive characteristic. Journal de l’École Polytechnique-Mathématiques 7, 831–870 (2020)

    Article  MathSciNet  Google Scholar 

  8. Corso, M.P., Perez, F.L., Stefenon, S.F., et al.: Classification of contaminated insulators using k-nearest neighbors based on computer vision. Computers 10(9), 112 (2021). https://doi.org/10.3390/computers10090112

    Article  Google Scholar 

  9. Georgiou, T., Liu, Y., Chen, W., et al.: A survey of traditional and deep learning-based feature descriptors for high dimensional data in computer vision. Int. J. Multimed. Info. Retr. 9, 135–170 (2020). https://doi.org/10.1007/s13735-019-00183-w

    Article  Google Scholar 

  10. Ma, S., Guo, P., You, H., et al.: An image matching optimization algorithm based on pixel shift clustering RANSAC. Inf. Sci. 562, 452–474 (2021). https://doi.org/10.1016/j.ins.2021.03.023

    Article  MathSciNet  Google Scholar 

  11. Rahman, M., Li, X., Yin, X.: DL-RANSAC: An improved RANSAC with modified sampling strategy based on the likelihood. In: 2019 IEEE 4th International Conference on Image, Vision and Computing (ICIVC). IEEE, pp. 463–468 (2019)

  12. Sarlin, P, E., DeTone, D., Malisiewicz, T. et al.: Superglue: Learning feature matching with graph neural networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4938–4947 (2020). https://doi.org/10.1109/CVPR42600.2020.00499

  13. Chen, H., Luo, Z., Zhang, J. et al.: Learning to match features with seeded graph matching network. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6301–6310 (2021). https://doi.org/10.1109/ICCV48922.2021.00624

  14. Shi, Y., Cai, J, X., Shavit, Y. et al.: ClusterGNN: Cluster-based coarse-to-fine graph neural network for efficient feature matching. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12517–12526 (2022). https://doi.org/10.1109/CVPR52688.2022.01219

  15. Vaswani, A., Shazeer, N., Parmar, N., et al.: Attention is all you need. Adv. Neural Inf. Process. Syst. (2017). https://doi.org/10.48550/arXiv.1706.03762

    Article  Google Scholar 

  16. Sinaga, K.P., Yang, M.S.: Unsupervised K-means clustering algorithm. IEEE Access. 8, 80716–80727 (2022). https://doi.org/10.1109/ACCESS.2020.2988796

    Article  Google Scholar 

  17. Zhang, H., Goodfellow, I., Metaxas, D. et al.: Self-attention generative adversarial networks. In: International Conference on Machine Learning. PMLR, pp. 7354–7363 (2019). https://doi.org/10.48550/arXiv.1805.08318

  18. Gadipudi, N., Elamvazuthi, I., Izhar, L.I., et al.: A review on monocular tracking and mapping: from model-based to data-driven methods. Vis. Comput. (2022). https://doi.org/10.1007/s00371-022-02702-z

    Article  Google Scholar 

  19. Toft, C., Maddern, W., Torii, A., et al.: Long-term visual localization revisited. IEEE Trans. Pattern Anal. Mach. Intell. 44(4), 2074–2088 (2020). https://doi.org/10.1109/TPAMI.2020.3032010

    Article  Google Scholar 

  20. Chen, C., Wang, B., Lu, C.X. et al.: A survey on deep learning for localization and mapping: Towards the age of spatial machine intelligence. arXiv preprint. (2020). https://doi.org/10.48550/arXiv.2006.12567

  21. Carrasco, M., Barbot, A.: Spatial attention alters visual appearance. Curr. Opin. Psychol. 29, 56–64 (2019). https://doi.org/10.1016/j.copsyc.2018.10.010

    Article  Google Scholar 

  22. Shi, W., Rajkumar, R.: Point-gnn: Graph neural network for 3D object detection in a point cloud. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1711–1719 (2020). https://doi.org/10.48550/arXiv.2003.01251

  23. Luo, H., Li, L., Zhang, Y., et al.: Link prediction in multiplex networks using a novel multiple-attribute decision-making approach. Knowl. Based Syst. 219, 106904 (2021). https://doi.org/10.1016/j.knosys.2021.106904

    Article  Google Scholar 

  24. Ngo, D., Lee, S., Kang, B.: Robust single-image haze removal using optimal transmission map and adaptive atmospheric light. Remote Sens. 12(14), 2233 (2020). https://doi.org/10.3390/rs12142233

    Article  Google Scholar 

  25. Chizat, L., Roussillon, P., Léger, F., et al.: Faster Wasserstein distance estimation with the Sinkhorn divergence. Adv. Neural Inf. Process. Syst. 33, 2257–2269 (2020)

    Google Scholar 

  26. DeTone, D., Malisiewicz, T., Rabinovich, A.: SuperPoint: self-supervised interest point detection and description. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). (2018). https://doi.org/10.48550/arXiv.1712.07629

  27. Viniavskyi, O., Dobko, M., Mishkin, D. et al.: Openglue: Open source graph neural net based pipeline for image matching. arXiv preprint. (2022). https://doi.org/10.48550/arXiv.2204.08870

  28. Bian, J., Lin, W, Y., Matsushita, Y. et al.: GMS: grid-based motion statistics for fast, ultra-robust feature correspondence. In: IEEE Conference on Computer Vision & Pattern Recognition. IEEE. (2017). https://doi.org/10.1109/CVPR.2017.302

  29. Xu, Q., Zeng, Y., Tang, W., et al.: Multi-task joint learning model for segmenting and classifying tongue images using a deep neural network. IEEE J. Biomed. Health Inform. 24(9), 2481–2489 (2020). https://doi.org/10.1109/JBHI.2020.2986376

    Article  Google Scholar 

  30. Deng, D.: DBSCAN clustering algorithm based on density. In: 2020 7th IEEE International Forum on Electrical Engineering and Automation (IFEEA), pp 949–953 (2020). https://doi.org/10.1109/IFEEA51475.2020.00199

  31. Shen, J., Hao, X., Liang, Z., et al.: Real-Time Superpixel Segmentation by DBSCAN Clustering Algorithm[J]. IEEE Trans. Image Process. 25(12), 5933–5942 (2016). https://doi.org/10.1109/TIP.2016.2616302

    Article  MathSciNet  Google Scholar 

  32. Van den Bergh, M., Boix, X., Roig, G., et al.: SEEDS: superpixels extracted via energy-driven sampling. Int. J. Comput. Vis. 111, 298–314 (2015). https://doi.org/10.1007/s11263-014-0744-2

    Article  MathSciNet  Google Scholar 

  33. Achanta, R., Shaji, A., Smith, K., et al.: SLIC superpixels compared to state-of-the-art superpixel methods. IEEE Trans. Pattern Anal. Mach. Intell. 34(11), 2274–2282 (2012). https://doi.org/10.1109/TPAMI.2012.120

    Article  Google Scholar 

  34. Li, Z., Chen, J.: Superpixel segmentation using linear spectral clustering. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE. (2015). https://doi.org/10.1109/CVPR.2015.7298741

  35. Zhang, Y., Hartley, R.I., Mashford, J. et al.: Superpixels via pseudo-Boolean optimization. In: IEEE International Conference on Computer Vision. (2012). https://doi.org/10.1109/ICCV.2011.6126393

  36. Liu, M.Y., Tuzel, O., Ramalingam, S. et al.: Entropy rate superpixel segmentation. In: The 24th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2011, pp 20–25 (2011). https://doi.org/10.1109/CVPR.2011.5995323

  37. Shen, J., Du, Y., Wang, W., et al.: Lazy random walks for superpixel segmentation. IEEE Trans. Image Process. 23(4), 1451–1462 (2014). https://doi.org/10.1109/TIP.2014.2302892

    Article  MathSciNet  Google Scholar 

  38. Lin, TY. et al. (2014). Microsoft COCO: Common Objects in Context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds) Computer Vision–ECCV 2014. ECCV 2014. Lecture Notes in Computer Science, vol 8693. Springer, Cham. https://doi.org/10.1007/978-3-319-10602-1_48

Download references

Funding

This work was partially supported by China Postdoctoral Science Foundation (Grant No. 2021M702030) and Shandong Provincial Transportation Science and Technology Project (Grant No. 2021B120).

Author information

Authors and Affiliations

  1. School of Information Science and Electrical Engineering, Shan Dong Jiao Tong University, Jinan, 250357, China

    Weilong Hao, Peng Wang, Cui Ni, Guangyuan Zhang & Wenjun Huangfu

  2. Institute of Automation, Shandong Academy of Sciences, Jinan, 250013, China

    Peng Wang

Authors
  1. Weilong Hao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Peng Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Cui Ni
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Guangyuan Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Wenjun Huangfu
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

WH contributed significantly to analysis and wrote the manuscript, PW contributed to the conception of the study, CN and GZ contributed to performed the data analyses and manuscript preparation, and WH performed the experiment.

Corresponding author

Correspondence to Peng Wang.

Ethics declarations

Conflict of interest

The authors declare that there is no conflict of interests or competing interests regarding the publication of this article.

Ethics approval

Not applicable.

Consent to participate

Not applicable.

Consent for publication

The work described has not been published before, and its publication has been approved by the responsible authorities at the institution where the work is carried out.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hao, W., Wang, P., Ni, C. et al. SuperGlue-based accurate feature matching via outlier filtering. Vis Comput 40, 3137–3150 (2024). https://doi.org/10.1007/s00371-023-03015-5

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-023-03015-5

Keywords

Search

Navigation