Skip to main content
SpringerLink
Log in

NRSfPP: non-rigid structure-from-perspective projection

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

A state-of-the-art algorithm for perspective projection reconstruction of non-rigid surfaces from single-view and realistic videos is proposed. It overcomes the limitations arising from the usage of orthographic camera model and also the complexity and non-linearity issues of perspective projection equation. Unlike traditional non-rigid structure-from-motion (NRSfM) methods, which have been studied only on synthetic datasets and controlled lab environments that require some prior constraints (such as manually segmented objects, limited rotations and occlusions, and full-length trajectories); the proposed method can be used in realistic video sequences. In addition, contrary to previous methods that use multiple cameras with different relative viewing angles, only a single-view video is required to reconstruct the 3D structures. By only using the 2D frames of incoming video stream, the proposed method extracts the projective depth coefficients of each point in each input frame, rotation matrix, translation vector, varying camera parameters (such as focal lengths for each input frame), and finally reconstructs the 3D deformable shape. Due to the high number of unknowns, the problem has been divided into two parts of projective depth coefficients extraction and 3D shape reconstruction. Perspective reconstruction of non-rigid surfaces has been extended to be certainly converged which leads to a significant increase in execution frequency of the iterated algorithm that has been presented for projective depth coefficients extraction. As such, it produces promising results for perspective projection reconstruction of non-rigid surfaces from single-view and realistic videos. The accuracy and robustness of the proposed method is demonstrated quantitatively on synthetic data and qualitatively on real image sequences. The experimental results show that NRSfPP provides the state-of-the-art results and resolves the failures of previous approaches by eliminating all of the predefined situations and constraints of applying perspective projection as the camera model.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  1. Aanæs H, Kahl F (2002) Estimation of deformable structure and motion. In Proceedings of the Vision and Modelling of Dynamic Scenes Workshop vol 2, p 3

  2. Agudo A, Agapito L, Calvo B, Montiel JM (2014) Good vibrations: a modal analysis approach for sequential non-rigid structure from motion. In proceedings of the IEEE conference on computer vision and pattern recognition pp 1558-1565

  3. Agudo A, Moreno-Noguer F (2015) Simultaneous pose and non-rigid shape with particle dynamics. In proceedings of the IEEE conference on computer vision and pattern recognition pp 2179-2187

  4. Agudo A, Moreno-Noguer F, Calvo B, Montiel JMM (2015) Sequential non-rigid structure from motion using physical priors. IEEE Trans Pattern Anal Mach Intell 38(5):979–994

    Article  Google Scholar 

  5. Akhter I, Sheikh Y, Khan S, Kanade T (2009) Nonrigid structure from motion in trajectory space. In advances in neural information processing systems pp 41-48

  6. Akhter I, Sheikh Y, Khan S, Kanade T (2010) Trajectory space: a dual representation for nonrigid structure from motion. IEEE Trans Pattern Anal Mach Intell 33(7):1442–1456

    Article  Google Scholar 

  7. Brand M (2005) A direct method for 3D factorization of nonrigid motion observed in 2D. In: 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR'05), vol 2. IEEE, pp 122-128

  8. Bregler C, Hertzmann A, Biermann H (2000) Recovering non-rigid 3D shape from image streams. In proceedings IEEE conference on computer vision and pattern recognition. CVPR 2000 (cat. No. PR00662), vol 2. IEEE, pp 690-696

  9. Bronte S, Bergasa LM, Pizarro D, Barea R (2017) Model-Based Real-Time Non-Rigid Tracking. Sensors 17(10):2342

    Article  Google Scholar 

  10. Cha G, Lee M, Cho J, Oh S (2018) Non-rigid surface recovery with a robust local-rigidity prior. Pattern Recogn Lett 110:51–57

    Article  Google Scholar 

  11. Chhatkuli A, Pizarro D, Bartoli A (2014) Non-Rigid Shape-from-Motion for Isometric Surfaces using Infinitesimal Planarity. In BMVC

  12. Chhatkuli A, Pizarro D, Collins T, Bartoli A (2017) Inextensible non-rigid structure-from-motion by second-order cone programming. IEEE Trans Pattern Anal Mach Intell 40(10):2428–2441

    Article  Google Scholar 

  13. Dai Y, Li H, He M (2014) A simple prior-free method for non-rigid structure-from-motion factorization. Int J Comput Vis 107(2):101–122

    Article  MathSciNet  Google Scholar 

  14. Del Bue A, Llad X, Agapito L (2006) Non-rigid metric shape and motion recovery from uncalibrated images using priors. In 2006 IEEE computer society conference on computer vision and pattern recognition (CVPR'06), vol 1. IEEE, pp 1191-1198

  15. Fragkiadaki K, Salas M, Arbelaez P, Malik J (2014) Grouping-based low-rank trajectory completion and 3D reconstruction. In advances in neural information processing systems pp 55-63

  16. Garg R, Roussos A, Agapito L (2013a) A variational approach to video registration with subspace constraints. Int J Comput Vis 104(3):286–314

    Article  MathSciNet  Google Scholar 

  17. Garg R, Roussos A, Agapito L (2013b) Dense variational reconstruction of non-rigid surfaces from monocular video. In proceedings of the IEEE conference on computer vision and pattern recognition pp 1272-1279

  18. Gotardo PF, Martinez AM (2011) Computing smooth time trajectories for camera and deformable shape in structure from motion with occlusion. IEEE Trans Pattern Anal Mach Intell 33(10):2051–2065

    Article  Google Scholar 

  19. Hartley R, Vidal R (2008) Perspective nonrigid shape and motion recovery. In: European conference on computer vision. Springer, Berlin, pp 276–289

    Google Scholar 

  20. Kong C, Lucey S (2019) Deep interpretable non-rigid structure from motion. arXiv preprint arXiv:1902.10840

  21. Kumar S (2020) Non-rigid structure from motion: prior-free factorization method revisited. In: The IEEE winter conference on applications of computer vision pp 51-60

  22. Kumar S, Dai Y, Li H (2017) Spatio-temporal union of subspaces for multi-body non-rigid structure-from-motion. Pattern Recogn 71:428–443

    Article  Google Scholar 

  23. Paladini M, Del Bue A, Xavier J, Agapito L, Stošić M, Dodig M (2012) Optimal metric projections for deformable and articulated structure-from-motion. Int J Comput Vis 96(2):252–276

    Article  MathSciNet  Google Scholar 

  24. Parashar S, Bartoli A, Pizarro D (2018) Self-calibrating isometric non-rigid structure-from-motion. In proceedings of the European conference on computer vision (ECCV) pp 252-267

  25. Probst T, Pani Paudel D, Chhatkuli A, Van Gool L (2018) Incremental non-rigid structure-from-motion with unknown focal length. In: Proceedings of the European conference on computer vision (ECCV) pp 756-771

  26. Rehan A, Zaheer A, Akhter I, Saeed A, Usmani MH, Mahmood B, Khan S (2014) Nrsfm using local rigidity. In: IEEE winter conference on applications of computer vision. IEEE, pp 69-74

  27. Russell C, Yu R, Agapito L (2014) Video pop-up: monocular 3d reconstruction of dynamic scenes. In: European conference on computer vision. Springer, Cham, pp 583–598

    Google Scholar 

  28. Sepehrinour M, Kasaei S (2015) 3D reconstruction of non-rigid surfaces from realistic monocular video. In: 2015 9th Iranian conference on machine vision and image processing (MVIP). IEEE, pp 199-202

  29. Sepehrinour M, Kasaei S (2017) Perspective reconstruction of non-rigid surfaces from single-view videos. In: 2017 Iranian conference on electrical engineering (ICEE). IEEE, pp 1452-1458

  30. Simon T, Valmadre J, Matthews I, Sheikh Y (2014) Separable spatiotemporal priors for convex reconstruction of time-varying 3D point clouds. In: European conference on computer vision. Springer, Cham, pp 204–219

    Google Scholar 

  31. Taylor J, Jepson AD, Kutulakos KN (2010) Non-rigid structure from locally-rigid motion. In: 2010 IEEE computer society conference on computer vision and pattern recognition. IEEE, pp 2761-2768

  32. Tomasi C, Kanade T (1992) Shape and motion from image streams under orthography: a factorization method. Int J Comput Vis 9(2):137–154

    Article  Google Scholar 

  33. Torresani L, Hertzmann A, Bregler C (2004) Learning non-rigid 3D shape from 2D motion. In: Advances in neural information processing systems pp 1555-1562

  34. Torresani L, Hertzmann A, Bregler C (2008) Nonrigid structure-from-motion: estimating shape and motion with hierarchical priors. IEEE Trans Pattern Anal Mach Intell 30(5):878–892

    Article  Google Scholar 

  35. Varol A, Salzmann M, Tola E, Fua P (2009) Template-free monocular reconstruction of deformable surfaces. In: 2009 IEEE 12th international conference on computer vision. IEEE, pp 1811-1818

  36. Vicente S, Agapito L (2012) Soft inextensibility constraints for template-free non-rigid reconstruction. In: European conference on computer vision. Springer, Berlin, pp 426–440

    Google Scholar 

  37. Wang X, Salzmann M, Wang F, Zhao J (2016) Template-free 3d reconstruction of poorly-textured nonrigid surfaces. In: European conference on computer vision. Springer, Cham, pp 648–663

    Google Scholar 

  38. Wang Y, Tong L, Jiang M, Zheng J (2015a) Non-rigid structure estimation in trajectory space from monocular vision. Sensors 15(10):25730–25745

    Article  Google Scholar 

  39. Wang Y, Yan X, Jiang M, Zheng J (2015b) Research on non-rigid structure from motion: a literature review. J Fiber BioEng Inform 8(4):751–760

    Article  Google Scholar 

  40. Xiao J, Chai J, Kanade T (2006) A closed-form solution to non-rigid shape and motion recovery. Int J Comput Vis 67(2):233–246

    Article  Google Scholar 

  41. Xiao J, Kanade T (2005) Uncalibrated perspective reconstruction of deformable structures. In: Tenth IEEE international conference on computer vision (ICCV'05) volume 1, vol 2. IEEE, pp 1075-1082

  42. Yu R, Russell C, Campbell ND, Agapito L (2015) Direct, dense, and deformable: template-based non-rigid 3d reconstruction from rgb video. In: Proceedings of the IEEE international conference on computer vision pp 918-926

Download references

Author information

Authors and Affiliations

  1. Department of Computer Engineering, Sharif University of Technology, Tehran, Iran

    Maryam Sepehrinour & Shohreh Kasaei

Authors
  1. Maryam Sepehrinour
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shohreh Kasaei
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shohreh Kasaei.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sepehrinour, M., Kasaei, S. NRSfPP: non-rigid structure-from-perspective projection. Multimed Tools Appl 80, 9093–9108 (2021). https://doi.org/10.1007/s11042-020-10068-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-10068-4

Keywords

Search

Navigation