Skip to main content
SpringerLink
Log in

Estimating the fundamental matrix based on the end-to-end convolutional network

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Estimating the fundamental matrix (F-matrix) is a basic problem in computer vision. The traditional algorithms are highly based on correspondences. By imprecise detecting and matching correspondences, the F-matrix is estimated incredibly. An end-to-end network (F-net) is provided in the present work without detecting and matching correspondences. To ensure estimation of an accurate F-matrix which is rank-2 with 7 degrees of freedom and scale invariance, we used the Improved convolutional block attention module (Improved-CBAM), and two self-define layers in this network. The experiments were conducted on the KITTI dataset. Two metrics, MMABS (Epipolar Constraint with Mean Absolute Value) and MMSQR (Epipolar Constraint with Mean Squared Value) were used to measure how well the epipolar constraint is satisfied by the estimated F-matrix. MMSQR and MMABS of the F-net are 0.21 and 0.11, respectively, and are 95.32 and 37.36 in the eight-point algorithm, respectively. For another end-to-end network, they are 3.48 and 2.77, respectively. F-net outperforms the other algorithms. The results demonstrated that the F-matrix can be successfully estimated by the F-net.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Kittisak J, Parinya S, Ludmila AS, Wahidah H, Robbi R, Andino M, Abdurrahman A (2020) Pattern recognition and features selection for speech emotion recognition model using deep learning. Int J Speech Technol 23:799–806

  2. Lei W, Juan G, Yize C, Yuanbo L, Weijie Z, Jiantao P, Hao C (2021) Automated segmentation of the optic disc from fundus images using an asymmetric deep learning network. Pattern Recogn 112(107810):1–12

    Google Scholar 

  3. Intisar RIH, Jeremiah N (2020) Deep learning approaches to biomedical image segmentation. Inform Med Unlocked 18(100297):1–12

    Google Scholar 

  4. Xulei Y, Zeng Z, Sin GT, Li W, Vijay C, Steven H (2018) Deep learning for practical image recognition:case study on Kaggle competitions. In: Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery & Data Mining ACM, pp 923–931

  5. Xiao Y, Yuhua L, Qiang G (2021) Pipeline image diagnosis algorithm based on neural immune ensemble learning. Int J Press Vessels Pip 189(104249):1–8

    Google Scholar 

  6. Scott W, Connor G, Menghua Z, Ryan B, Nathan J (2015) Deepfocal: a method for direct focal length estimation. In: Processing of IEEE International Conference on Image Processing, pp 1369–1373

  7. Oleksandr B, Francois R, Viktor E, Jean-Chales B (2018) Deepcalib: A deep learning Approach for Automatic Intrinsic calibration of wide field-of-view cameras. In: proceedings of European Conference on Visual Media Production, pp 1–10

  8. Chaoning Z, Francois R, Junsik J, Dawit MA (2020) DeepPTZ: Deep self-calibration for PTZ cameras. In: Proceedings of Conference on Applications of Computer Vision, pp 1030–1038

  9. Iaroslav M, Juha Y, Juho K, Esa R (2017) Relative camera pose estimation using convolutional neural networks. In: proceedings of Springer Conference on Advanced Concepts for Intelligent Vision Systems, pp 675–687

  10. Zakaria L, Iaroslav M, Surya K, Juho K (2017) Camera relocalization by computing pairwise relative poses using convolutional neural network. arXiv:1707.09733

  11. Daniel D, Tomasz M, Andrew R (2016) Deep image homography estimation. arXiv:1606.03798

  12. Farzan EN, Robert L, Nathalie J (2017) Homography estimation from image pairs with hierarchical convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 913–920

  13. Daniel D, Tomasz M, Andrew R (2017) Toward geometric deep slam. arXiv:1707.07410

  14. Ty N, Steven WC, Shreyas SS, Camillo JT, Vijay K (2018) Unsupervised deep homography: A fast and robust homography estimation model. IEEE Robot Autom Lett 3(3):2346–2353

    Article  Google Scholar 

  15. Ignacio R, Relja A, Josef S (2017) Convolutional neural network architecture for geometric matching. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 40–48

  16. Xinwang L, Xinzhong Z, Miaomiao L, Lei W, En Z, Tongliang L, Marius K, Dinggang S, Jianping Y, Wen G (2020) Multiple kernel K-means with incomplete kernels. IEEE Trans Pattern Anal Mach Intell 5(42):1191–1204

    Google Scholar 

  17. Vladlen K, Rene R (2018) Deep fundamental matrix estimation. In: Proceedings of the European Conference on Computer Vision. pp1–16

  18. Omind P, Guandao Y, Aditya P, Qiuren F, Hanqing J, Bharath H, Serge B (2018) Deep F matrix estimation without correspondences. arXiv: 1810.01575

  19. Yesheng Z, Xu Z, Dahong Q (2020) An end to end network architecture for fundamental matrix estimation. arXiv: 2010. 15528

  20. Tsung-Yi L, Piotr D, Ross G, Kaiming H, Bharath H, Serge B (2017) Feature pyramid networks for object detection. arXiv: 1612.03144

  21. Sanghyun W, Jongchan P, Joon-Young L (2018) CBAM: convolutional block attention module. In: Proceedings of the European Conference on Computer Vision, pp 3–19

  22. Andreas G, Philip L, Raquel U (2012) Are we ready for. Autonomous Driving? The KITTI vision benchmark suite. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3354–3361

  23. Zhengyou Z (1998) Determining the epipolar geometry and its uncertainty: A review. Int J Comput Vision 27:161–195

    Article  Google Scholar 

  24. Quan-Tuan L, Oliver DF (1996) The fundamental matrix: Theory, algorithms, and stability analysis. Int J Comput Vision 17:43–75

    Article  Google Scholar 

  25. Lili Z, Binjie H, Jingguang Q, Manman C (2020) A deep-learning-based self-calibration time-reversal fingerprinting localization approach on Wi-Fi platform. IEEE Internet Things J 7(8):7072–7082

    Article  Google Scholar 

  26. Zahra K, Hamid S, Ahmand RNN, Mehran M (2020) Fast, yet robust end-to-end camera pose estimation for robotic applications. Appl Intell 3:1–19

    Google Scholar 

  27. Tao Z (2021) Research on environmental landscape design based on virtual reality technology and deep learning. Microprocess Microsyst 8(103796):1–6

    Google Scholar 

  28. George F, Khalid A, Sameh Z (2021) Single-View 3D reconstruction: A Survey of deep learning methods. Computers & Graphics, p 127

  29. Kui F, Jiansheng P, Qiwen H, Hanxiao Z (2020) Single image 3D object reconstruction based on deep learning: A review. Multimed Tools Appl 80:463–498

    Google Scholar 

  30. Mateusz M, Anders E, Engene B, Mahsa B (2020) A simple and scalable shape representation for 3D reconstruction. arXiv:2005.04623

  31. Longuet-Higgins HC (1981) A computer algorithm for reconstructing a scene from two projections. Nature 293(5828):133–135

    Article  Google Scholar 

  32. Ricard IR (1994) Projective reconstruction and invariants from multiple images. IEEE Trans Pattern Anal Mach Intell 16(10):1036–1040

    Article  Google Scholar 

  33. Michel D, Marc R, Jean-Thierry L, Gerard R (1989) Determination of the attitude of 3-D objects from a single perspective view. IEEE Trans Pattern Anal Mach Intell 11(12):1265–1278

    Article  Google Scholar 

  34. Nister D (2003) An efficient solution to the five-point relative pose problem. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 951–764

  35. Fuchao W, Zhanyi H (2003) 5-points and 4-point algorithm to determine of the F matrix. Acta Autom Sin 29(2):175–180

    MathSciNet  Google Scholar 

  36. Luong QT, Faugeras OD (1993) Determining the F matrix with planes: unstability and new algorithms. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 489–494

  37. Crepey S (2003) Calibration of the local volatility in a generalized black—scholes model using Tikhonov regularization. SIAM 1 Math Anal 34(5):1183–1206

    Article  MathSciNet  Google Scholar 

  38. Chengyuan T, YiLeh W, Yuehhuang L (2008) F matrix estimation using evolutionary algorithms with Multi-Objective functions. J Inf Sci Eng 24(3):785–800

    Google Scholar 

  39. Wojciech C, Michel JB, Darren G, Anton VDH (2002) A new approach to constrained parameter estimation applicable to some computer vision problems. In: Proceedings of statistical methods in video processing workshop, pp 1–2

  40. Wojciech C, Michel JB, Darren G, Anton VDH (2000) What value covariance information in estimating vision parameters? In: Proceedings of the Eighth IEEE International Conference on Computer Vision, pp 302–308

  41. Hanzi W, David S (2004) Robust adaptive-scale parametric model estimation for computer vision. IEEE Trans Pattern Anal Mach Intell 26(11):1459–1474

    Article  Google Scholar 

  42. Maria T, Ebroul I (2012) Improving the efficiency of a least median of squares schema for the estimation of the F matrix. Int J Pattern Recognit Artif Intell 20(05):633–648

    Google Scholar 

  43. Martin AF, Robert CB (1981) Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Commun ACM 24(6):381–385

    Article  MathSciNet  Google Scholar 

  44. Torr PHS, Zisserman A (2000) MLESAC: a new robust estimator with application to estimating image geometry. Comput Vis Image Underst 78(1):138–156

    Article  Google Scholar 

  45. Torr PHS (2002) Bayesian model estimation and selection for epipolar geometry and generic manifold fitting. Int J Comput Vision 50(1):35–61

    Article  Google Scholar 

  46. Kun Y, Rujin Z, Enhai L, Yuemao M (2019) A robust fundamental matrix estimation method based on epipolar geometric error criterion. IEEE Access 7(2019):147523-147533

  47. Yong C, Lopez JA, Camps O, Sznaier M (2015) A convex optimization approach to robust fundamental matrix estimation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2170-2178

  48. Fan Z, Can Z, Qi Z (2015) Method for fundamental matrix estimation combined with feature lines. Neurocomputing 160(2015): 300-307

  49. Chunbao X, Dazheng F, Mingdong Y (2019) Soft decision optimization method for robust fundamental matrix estimation. Mach Vis Appl 30(2019):657–669

  50. Lindeberg T (2012) Scale invariant feature transform. Scholarpedia 7(5):2012–2021

    Article  Google Scholar 

  51. Baofeng Z, Yingkui J, Zhijun M, Yongchen L (2014) An efficient image matching method using Speed Up Robust Features. In: Proceedings of IEEE International Conference on Mechatronics & Automation, pp 553–558

  52. Xuebing B, Jin C, Xiaokai M, Ying Z (2006) Improved feature points matching algorithm based on speed-up robust feature and oriented fast and rotated brief. J Comput Appl 36(7):1923–1926

    Google Scholar 

Download references

Acknowledgements

The authors would like to express their gratitude to EditSprings (https://www.editsprings.com/) for the expert linguistic services provided.

This work is supported by the National Natural Science Foundation of China (62063034).

This work is supported by the Research and innovation project fund of Yunnan University (2020Z76).

Author information

Authors and Affiliations

  1. Department of Electronic Engineering, Yunnan University, Kunming, China

    Ruiqi Yang, Junhua Zhang & Bo Li

Authors
  1. Ruiqi Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Junhua Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bo Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Junhua Zhang.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yang, R., Zhang, J. & Li, B. Estimating the fundamental matrix based on the end-to-end convolutional network. Appl Intell 52, 15517–15528 (2022). https://doi.org/10.1007/s10489-021-03103-w

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-021-03103-w

Keywords

Search

Navigation