Skip to main content
Springer Nature Link
Log in

Towards learning line descriptors from patches: a new paradigm and large-scale dataset

  • Original Article
  • Published:
International Journal of Machine Learning and Cybernetics Aims and scope Submit manuscript

Abstract

Line feature description is important for image matching. However, its development is much slow compared to point description, and is still in the stage of manual design, which suffers from the problem of weak distinguish ability and poor robustness under complex conditions. To improve on this situation, this paper proposes to learn the line feature description based on convolutional neural network. First, a large-scale dataset consisting of about 229,000 labeled pairs of matched lines is built for training and testing. Then, a paradigm for learning the line descriptors based on the constructed line dataset is proposed. Specifically, the line is represented uniquely by the stacked mean and standard deviation patches of the support regions of those points lying on the line, which is subsequently fed into the L2Net to output the required line descriptors directly. Based on the line matching principals, the network is also trained with the triplet loss that is widely used for learning point descriptors. Experimental results for line matching and curve matching both demonstrate the superiority and effectiveness of the proposed learning-based descriptor, especially, averaged increases of 4.66 ~ 5.7% mAPs, 10.59 ~ 12.10% mAPs, 0.96 ~ 3.75% mAPs and 3.73% mAP on testing subset, Oxford dataset, line dataset and curve dataset are obtained compared to handcrafted descriptors. As an application, we apply the learned line descriptor to image stitching and also obtain good results.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  1. Clement M, Poulenard A, Kurtz C, Wendling L (2017) Directional enlacement histograms for the description of complex spatial configurations between objects. IEEE Trans Pattern Anal Mach Intell 39(12):2366–2380

    Article  Google Scholar 

  2. Song X, Jiang S, Wang B, Chen C, Chen G (2020) Image representations with spatial object-to-object relations for RGB-D scene recognition. IEEE Trans Image Process 29:525–537

    Article  MathSciNet  Google Scholar 

  3. Sulaiman A, Omar K, Nasrudin MF, Arram A (2019) Length independent writer identification based on the fusion of deep and hand-crafted descriptors. IEEE Access 7:91772–91784

    Article  Google Scholar 

  4. Cheng G, Zhou P, Han J (2016) Learning rotation-invariant convolutional neural networks for object detection in VHR optical remote sensing images. IEEE Trans Geosci Remote Sens 54(12):7405–7415

    Article  Google Scholar 

  5. Han J, Zhang D, Cheng G, Guo L, Ren J (2015a) Object detection in optical remote sensing images based on weakly supervised learning and high-level feature learning. IEEE Trans Geosci Remote Sens 53(6):3325–3337

    Article  Google Scholar 

  6. Han J, Zhang D, Hu X, Guo L, Ren J, Wu F (2015b) Background prior-based salient object detection via deep reconstruction residual. IEEE Trans Circuits Syst Video Technol 25(8):1309–1321

    Article  Google Scholar 

  7. Liu S, Chai Q (2019) Shape-optimizing and illumination-smoothing image stitching. IEEE Trans Multimed 21(3):690–703

    Article  Google Scholar 

  8. Hu F, Li Y, Feng M (2019) Continuous point cloud stitch based on image feature matching constraint and score. IEEE Trans Intell Vehicles 4(3):363–374

    Article  Google Scholar 

  9. Wei X, Wang H, Scotney B, Wan H (2019) Selective multi-descriptor fusion for face identification. Int J Mach Learn Cybern 10(12):3417–3429

    Article  Google Scholar 

  10. Yang W, Zhang X, Li J (2020) A local multiple patterns feature descriptor for face recognition. Neurocomputing 373:109–122

    Article  Google Scholar 

  11. Tang Y, Gao Z, Lin F, Li Y, Wen F (2018) Visual adaptive tracking for monocular omnidirectional camera. J Vis Commun Image Represent 55:253–262

    Article  Google Scholar 

  12. Schmid BJ, Adhami RR (2018) Building descriptors from local feature neighborhoods for applications in semi-dense 3D reconstruction. IEEE Trans Image Process 27(11):5491–5500

    Article  MathSciNet  Google Scholar 

  13. Papadopoulos GT, Daras P (2018) Human action recognition using 3D reconstruction data. IEEE Trans Circuits Syst Video Technol 28(8):1807–1823

    Article  Google Scholar 

  14. Fan B, Kong Q, Wang X, Wang Z, Xiang S, Pan C, Fua P (2019) A performance evaluation of local features for image based 3D reconstruction. IEEE Trans Image Process 28(10):4774–4789

    Article  MathSciNet  Google Scholar 

  15. Guan Z, Xie F, Zhao W, Wang X, Chen L, Zhao W, Peng J (2018) Tag-based weakly-supervised hashing for image retrieval. Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, pp 3776—3782

  16. Liu H, Zhang Q, Fan B, Wang Z, Han J (2019) Features combined binary descriptor based on voted ring-sampling pattern. IEEE Trans Circuits Syst Video Technol. https://doi.org/10.1109/TCSVT.2019.2943595

    Article  Google Scholar 

  17. Krizhevsky A, Sutskever I, Hinton GE (2012) ImageNet classification with deep convolutional neural networks. Neuralin in neural Information Processing Systems, pp 1097–1105

  18. Han X, Leung T, Jia Y, Sukthankar R, Berg AC (2015) MatchNet: unifying feature and metric learning for patch-based matching. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 3279–3286

  19. Yi KM, Trulls E, Lepetit V, Fua P (2016) LIFT: learned invariant feature transform. In European Conference on Computer Vision. Springer, pp 467–483

  20. Zagoruyko S, Komodakis N (2015) Learning to compare image patches via convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 4353–4361

  21. Kumar BG V, Carneiro G, Reid I (2016) Learning local image descriptors with deep siamese and triplet convolutional networks by minimising global loss functions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 5385–5394

  22. Balntas V, Riba E, Ponsa D, Mikolajczyk K (2016) Learning local feature descriptors with triplets and shallow convolutional neural networks. In Proceedings of the British Machine Vision Conference, pp 1–11

  23. Tian Y, Fan B, Wu F (2017) L2-Net: deep learning of discriminative patch descriptor in Euclidean space. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 661–669

  24. Mishchuk A, Mishkin D, Radenovic F, Matas J (2017) Working hard to know your neighbor’s margins: Local descriptor learning loss. In Advances in Neural Information Processing Systems, pp 4826–4837

  25. Lin K, Lu J, Chen C, Zhou J, Sun M (2019) Unsupervised deep learning of compact binary descriptors. IEEE Trans Pattern Anal Mach Intell 41(6):1501–1514

    Article  Google Scholar 

  26. Duan Y, Lu J, Wang Z, Feng J, Zhou J (2017) Learning deep binary descriptor with multi-quantization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 1183–1192

  27. Geng L, Jodoin PM, Su S, Li S (2016) CBDF: compressed binary discriminative feature. Neurocomputing 184:43–54

    Article  Google Scholar 

  28. Tang AWK, Ng TP, Hung YS, Leung CH (2006) Projective reconstruction from line-correspondences in multiple uncalibrated images. Pattern Recogn 39(5):889–896

    Article  Google Scholar 

  29. Polewski P, Yao W (2019) Scale invariant line-based co-registration of multimodal aerial data using L1 minimization of spatial and angular deviations. ISPRS Journal of Photogrammetry and Remote Sensing 152:79–93

    Article  Google Scholar 

  30. Wang Z, Wu F, Hu Z (2009) MSLD: a robust descriptor for line matching. Pattern Recogn 42(5):941–953

    Article  Google Scholar 

  31. Zhang L, Koch R (2013) An efficient and robust line segment matching approach based on LBD descriptor and pairwise geometric consistency. J Vis Commun Image Represent 24(7):794–805

    Article  Google Scholar 

  32. Lourakis MI, Halkidis ST, Orphanoudakis SC (2000) Matching disparate views of planar surfaces using projective invariants. Image Vis Comput 18(9):673–683

    Article  Google Scholar 

  33. Fan B, Wu F, Hu Z (2012) Robust line matching through line-point invariants. Pattern Recogn 45(2):794–805

    Article  Google Scholar 

  34. Zeng J, Yu S, Fu X, Li C (2011) A line segments matching method based on epipolar-line constraint and line segment features. Journal of Software 6(9):1746–1754

    Article  Google Scholar 

  35. Zuliani M, Bertelli L, Kenney C, Chandrasekaran S, Manjunath BS (2008) Drums, curve descriptors and affine invariant region matching. Image Vis Comput 26(3):347–360

    Article  Google Scholar 

  36. Schmid C, Zisserman A (2000) The geometry and matching of lines and curves over multiple views. Int J Comput Vision 40(3):199–233

    Article  Google Scholar 

  37. Bay H, Ferraris V, Van Gool L (2005) Wide-baseline stereo matching with line segments. IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp 329–336

  38. Verhagen B, Timofte R, Van Gool L (2014) Scale-invariant line descriptors for wide baseline matching. Proceedings of the IEEE Winter Conference on Applications of Computer Vision. Springs, pp 493–500

  39. Liu H, Zhi S, Wang Z (2013) IOCD: intensity order curve descriptor. Int J Pattern Recognit Artif Intell 27(7):1355011–2135037

    Article  Google Scholar 

  40. López J, Santos R, Fdez-Vidal XR, Pardo XM (2015) Two-view line matching algorithm based on context and appearance in low-textured images. Pattern Recogn 48(7):2164–2184

    Article  Google Scholar 

  41. Philbin J, Arandjelović R, Zisserman A (2007) The oxford buildings dataset. https://www.robots.ox.ac.uk/~vgg/data/oxbuildings/. Accessed 10 Apr 2020

  42. Philbin J, Zisserman A (2008) The paris dataset. http://www.robots.ox.ac.uk/~vgg/data/parisbuildings/. Accessed 10 Apr 2020

  43. Stewénius H (2006) UKBench database. https://www.vis.uky.edu/~stewe/ukbench/. Accessed 10 Apr 2020

  44. Wang Z, Fan B, Wu F(2011) Local intensity order pattern for feature description. International Conference on Computer Vision, pp 603–610

  45. Mikolajczyk K, Schmid C (2005) A performance evaluation of local descriptors. IEEE Trans Pattern Anal Mach Intell 27(10):1615–1630

    Article  Google Scholar 

  46. Gioi RGV, Jérémie J, Morel JM, Gregory R (2012) LSD: a line segment detector. Image Processing On Line 2(4):35–55

    Article  Google Scholar 

  47. Huo Z, Liu Y, Fu M, Qiao Y (2020) Research on line description method based on convolution neural network. Comput Eng. https://doi.org/10.19678/j.issn.1000-3428.0057686

    Article  Google Scholar 

Download references

Acknowledgements

This work is supported by Henan University Scientific and Technological Innovation Team Support Program (19IRTSTHN012).

Author information

Authors and Affiliations

  1. School of Automation and Electrical Engineering, University of Science and Technology Beijing, Beijing, 100083, China

    Hongmin Liu

  2. School of Computer Science and Technique, Henan Polytechnic University, Jiaozuo, 454003, China

    Hongmin Liu, Yujie Liu, Miaomiao Fu, Yuhui Wei, Zhanqiang Huo & Yingxu Qiao

Authors
  1. Hongmin Liu
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Yujie Liu
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Miaomiao Fu
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Yuhui Wei
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Zhanqiang Huo
    View author publications

    You can also search for this author inPubMed Google Scholar

  6. Yingxu Qiao
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding authors

Correspondence to Zhanqiang Huo or Yingxu Qiao.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, H., Liu, Y., Fu, M. et al. Towards learning line descriptors from patches: a new paradigm and large-scale dataset. Int. J. Mach. Learn. & Cyber. 12, 877–890 (2021). https://doi.org/10.1007/s13042-020-01207-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13042-020-01207-2

Keywords