Skip to main content
Springer Nature Link
Log in

Non-rigid Tracking Using RGB-D Data

  • Chapter
  • First Online:
Robot Dynamic Manipulation

Part of the book series: Springer Tracts in Advanced Robotics ((STAR,volume 144))

  • 712 Accesses

Abstract

In this chapter, the real-time non-rigid tracking system to continuously estimate the deformations of the manipulated objects is described, using visual and range data provided by an RGB-D sensor. Based on the models described in the previous chapter, the method enables to deal with various deformations (elastic and plastic), fractures, and contacts, while ensuring physical consistency, handling rigid motions, occlusions, and addressing these tasks in real-time. It relies on a prior visual segmentation of the object in the RGB image. The mesh is registered first in a rigid manner with a classical ICP algorithm between the visible surface of the mesh and the segmented point cloud. A non-rigid fitting phase is then performed by determining geometrical point-to-point correspondences with the point cloud, used to compute external forces exerted on the mesh. Deformations are computed by solving mechanical equations balancing these external forces with internal forces provided by the FEM models. A technique to estimate the elastic parameters of the object is proposed by minimizing a fitting error between the simulated deformations, actuated by the input operator force provided by a force sensor, and the deformations captured by the RGB-D camera. Conversely, estimating a contact force exerted on the object can be carried out using point cloud data by minimizing the deviation between the registered and the simulated deformations. The system has been evaluated on synthetic and real data, with various objects, deformation, and interaction scenarios, and by integrating it into manipulation experiments on the RoDyMan humanoid robotic platform. This chapter is based on the works presented in [1,2,3].

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    http://www.cgal.org.

References

  1. A. Petit, S. Cotin, V. Lippiello, B. Siciliano. Capturing deformations of interacting non-rigid objects using RGB-D data, in 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 491–497, Madrid, E (2018)

    Google Scholar 

  2. A. Petit, V. Lippiello, G.A. Fontanelli, B. Siciliano, Tracking elastic deformable objects with an RGB-D sensor for a pizza chef robot. Robot. Auton. Syst. 88, 187–201 (2017)

    Article  Google Scholar 

  3. A. Petit, V. Lippiello, B. Siciliano, Tracking fractures of deformable objects in real-time with an RGB-D sensor, in 2015 International Conference on 3D Vision, pp. 632–639, Lyon, F (2015)

    Google Scholar 

  4. C. Elbrechter, R. Haschke, H. Ritter, Bi-manual robotic paper manipulation based on real-time marker tracking and physical modelling, in 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 1427–1432 (2011)

    Google Scholar 

  5. N. Haouchine, J. Dequidt, M.-O. Berger, S. Cotin, Monocular 3D reconstruction and augmentation of elastic surfaces with self-occlusion handling. IEEE Trans. Vis. Comput. Gr. 21(12), 1363–1376 (2015)

    Article  Google Scholar 

  6. N. Haouchine, J. Dequidt, I. Peterlik, E. Kerrien, M.-O. Berger, S. Cotin, Image-guided simulation of heterogeneous tissue deformation for augmented reality during hepatic surgery, in 2013 IEEE International Symposium on Mixed and Augmented Reality, pp. 199–208 (2013)

    Google Scholar 

  7. J. Schulman, A. Lee, J. Ho, P. Abbeel, Tracking deformable objects with point clouds, in 2013 IEEE International Conference on Robotics and Automation, pp. 1130–1137 (2013)

    Google Scholar 

  8. A. Bartoli, V. Gay-Bellile, U. Castellani, J. Peyras, S. Olsen, P. Sayd, Coarse-to-fine low-rank structure-from-motion in 2008 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8 (2008)

    Google Scholar 

  9. A. Weiss, D. Hirshberg, M.J. Black, Home 3D body scans from noisy image and range data, in 2011 IEEE International Conference on Computer Vision, pp. 1951–1958 (2011)

    Google Scholar 

  10. A. Bartoli, A. Zisserman, Direct estimation of non-rigid registrations, in British Machine Vision Conference, pp. 899–908 (2004)

    Google Scholar 

  11. M. Kass, A. Witkin, D. Terzopoulos, Snakes: active contour models. Int. J. Comput. Vis. 1(4), 321–331 (1988)

    Article  Google Scholar 

  12. J. Pilet, V. Lepetit, P. Fua, Fast non-rigid surface detection, registration and realistic augmentation. Int. J. Comput. Vis. 76(2), 109–122 (2007)

    Article  Google Scholar 

  13. D. Terzopoulos, A. Witkin, M. Kass, Constraints on deformable models: recovering 3D shape and nonrigid motion. Artif. Intell. 36(1), 91–123 (1988)

    Article  Google Scholar 

  14. M. Salzmann, J. Pilet, S. Ilic, P. Fua, Surface deformation models for nonrigid 3D shape recovery. IEEE Trans. Pattern Anal. Mach. Intell. 29(8), 1481–1487 (2007)

    Article  Google Scholar 

  15. A. Jordt, R. Koch, Direct model-based tracking of 3D object deformations in depth and color video. Int. J. Comput. Vis. 102, 1–17 (2013)

    Article  MathSciNet  Google Scholar 

  16. M. Zollhöfer, M. Nießner, S. Izadi, C. Rehmann, C. Zach, M. Fisher, C. Wu, A. Fitzgibbon, C. Loop, C. Theobalt, M. Stamminger, Real-time non-rigid reconstruction using an RGB-D camera. ACM Trans. Gr. 33(4), 1–12 (2014)

    Article  Google Scholar 

  17. B. Allain, J.-S. Franco, E. Boyer, An efficient volumetric framework for shape tracking, in 2015 IEEE Conference on Computer Vision and Pattern Recognition, pp. 268–276 (2015)

    Google Scholar 

  18. R.A. Newcombe, D. Fox, S.M. Seitz, Dynamic fusion: reconstruction and tracking of non-rigid scenes in real-time, in 2015 IEEE Conference on Computer Vision and Pattern Recognition, pp. 343–352 (2015)

    Google Scholar 

  19. V. Lippiello, F. Ruggiero, B. Siciliano, Floating visual grasp of unknown objects using an elastic reconstruction surface, in Robotics Research: The Fourteenth International Symposium, in Springer Tracts in Advanced Robotics 70 ed. by C. Pradalier, R. Siegwart, G. Hirzinger (Springer, 2011), pp. 329–344

    Google Scholar 

  20. L.D. Cohen, I. Cohen, Deformable models for 3-D medical images using finite elements and balloons, in 1992 IEEE Conference on Computer Vision and Pattern Recognition, pp. 592–598 (1992)

    Google Scholar 

  21. T. McInerney, D. Terzopoulos, A finite element model for 3D shape reconstruction and nonrigid motion tracking, in 1993 IEEE International Conference on Computer Vision, pp. 518–523 (1993)

    Google Scholar 

  22. A. Malti, R. Hartley, A. Bartoli, J.-H. Kim. Monocular template-based 3D reconstruction of extensible surfaces with local linear elasticity, in 2013 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1522–1529 (2013)

    Google Scholar 

  23. K. Varanasi, A. Zaharescu, E. Boyer, R. Horaud, Temporal surface tracking using mesh evolution, in European Conference on Computer Vision, pp. 30–43 (2008)

    Google Scholar 

  24. A. Zaharescu, E. Boyer, R. Horaud, Topology-adaptive mesh deformation for surface evolution, morphing, and multiview reconstruction. IEEE Trans. Pattern Anal. Mach. Intell. 33(4), 823–837 (2011)

    Article  Google Scholar 

  25. C.J. Paulus, N. Haouchine, S.-H. Kong, R.V. Soares, D. Cazier, S. Cotin, Handling topological changes during elastic registration. Int. J. Comput. Assist. Radiol. Surg. 12(3), 461–470 (2017)

    Article  Google Scholar 

  26. A. Tsoli, A.A. Argyros, Tracking deformable surfaces that undergo topological changes using an RGB-D camera, in 2016 Fourth International Conference on 3D Vision, pp. 333–341 (2016)

    Google Scholar 

  27. I. Badami, J. Stückler, S. Behnke, Depth-enhanced Hough Forests for object-class detection and continuous pose estimation, in Workshop on Semantic Perception, Mapping and Exploration (2013)

    Google Scholar 

  28. A. Tejani, D. Tang, R. Kouskouridas, T.-K. Kim, Latent-Class Hough Forests for 3D object detection and pose estimation, in European Conference on Computer Vision, pp. 462–477 (2014)

    Google Scholar 

  29. H. Hamer, K. Schindler, E. Koller-Meier, L. Van Gool, Tracking a hand manipulating an object, in 2009 IEEE International Conference On Computer Vision, pp. 1475–1482 (2009)

    Google Scholar 

  30. K. Kim, V. Lepetit, W. Woo, Keyframe-based modeling and tracking of multiple 3d objects, in 2010 9th IEEE International Symposium on Mixed and Augmented Reality, pp. 193–198 (2010)

    Google Scholar 

  31. N. Kyriazis, A. Argyros, Scalable 3D tracking of multiple interacting objects, in 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3430–3437 (2014)

    Google Scholar 

  32. I. Oikonomidis, N. Kyriazis, A.A. Argyros, Full DoF tracking of a hand interacting with an object by modeling occlusions and physical constraints, in 2011 IEEE International Conference on Computer Vision, pp. 2088–2095 (2011)

    Google Scholar 

  33. I. Oikonomidis, N. Kyriazis, A.A. Argyros, Tracking the articulated motion of two strongly interacting hands, in 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1862–1869 (2012)

    Google Scholar 

  34. T.-H. Pham, A. Kheddar, A. Qammaz, A.A. Argyros, Towards force sensing from vision: Observing hand-object interactions to infer manipulation forces, in 2015 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2810–2819 (2015)

    Google Scholar 

  35. M. Salzmann, R. Urtasun, Physically-based motion models for 3D tracking: A convex formulation, in 2011 IEEE International Conference on Computer Vision, pp. 2064–2071 (2011)

    Google Scholar 

  36. S. Sridhar, F. Mueller, M. Zollhöfer, D. Casas, A. Oulasvirta, C. Theobalt, Real-time joint tracking of a hand manipulating an object from RGB-D input, in European Conference on Computer Vision, pp. 294–310 (2016)

    Google Scholar 

  37. K. Guo, F. Xu, T. Yu, X. Liu, Q. Dai, Y. Liu, Real-time geometry, albedo, and motion reconstruction using a single RGB-D camera. ACM Trans. Gr. 36(4), 32:1–32:13 (2017)

    Google Scholar 

  38. C. Rother, V. Kolmogorov, A. Blake, Grabcut: interactive foreground extraction using iterated graph cuts. ACM Trans. Gr. 23, 309–314 (2004)

    Article  Google Scholar 

  39. Y. Boykov, O. Veksler, R. Zabih, Fast approximate energy minimization via graph cuts. IEEE Trans. Pattern Anal. Mach. Intell. 23(11), 1222–1239 (2001)

    Article  Google Scholar 

  40. Y. Chen, G. Medioni, Object modelling by registration of multiple range images. Image Vis. Comput. 10(3), 145–155 (1992)

    Article  Google Scholar 

  41. F. Faure, C. Duriez, H. Delingette, J. Allard, B. Gilles, S. Marchesseau, H. Talbot, H. Courtecuisse, G. Bousquet, I. Peterlik, S. Cotin, Sofa: A multi-model framework for interactive physical simulation, in Soft Tissue Biomechanical Modeling for Computer Assisted Surgery, pp. 283–321 (2012)

    Google Scholar 

  42. A.V. Gelder, Approximate simulation of elastic membranes by triangulated spring meshes. J. Gr. Tools 3(2), 21–41 (1998)

    Article  Google Scholar 

  43. B.A. Lloyd, G. Székely, M. Harders, Identification of spring parameters for deformable object simulation. IEEE Trans. Vis. Comput. Gr. 13(5), 1081–1094 (2007)

    Article  Google Scholar 

  44. A. Petit, V. Lippiello, B. Siciliano, Real-time tracking of 3D elastic objects with an RGB-D sensor, in 2015 IEEE International Conference on Intelligent Robots and Systems, pp. 3914–3921 (2015)

    Google Scholar 

  45. R.A. Newcombe, A.J. Davison, S. Izadi, P. Kohli, O. Hilliges, J. Shotton, D. Molyneaux, S. Hodges, D. Kim, A. Fitzgibbon. Kinect Fusion: real-time dense surface mapping and tracking, in 10th IEEE International Symposium on Mixed and Augmented Reality, pp. 127–136 (2011)

    Google Scholar 

  46. O. Kähler, V.A. Prisacariu, D.W. Murray, Real-time large-scale dense 3d reconstruction with loop closure, in European Conference on Computer Vision, pp. 500–516 (2016)

    Google Scholar 

  47. J. Allard, F. Faure, H. Courtecuisse, F. Falipou, C. Duriez, P.G. Kry, Volume contact constraints at arbitrary resolution, in ACM SIGGRAPH 2010 Papers, pp. 1–10 (2010)

    Google Scholar 

  48. B. Siciliano, O. Khatib, Springer Handbook of Robotics, 2nd edn. (Springer Science & Business Media, 2016)

    Google Scholar 

  49. R.S. Dahiya, G. Metta, M. Valle, G. Sandini, Tactile sensing-from humans to humanoids. IEEE Trans. Rob. 26(1), 1–20 (2010)

    Article  Google Scholar 

  50. N. Wettels, J.A. Fishel, Z. Su, C.H. Lin, G.E. Loeb, Multi-modal synergistic tactile sensing, in Tactile Sensing in Humanoids-Tactile Sensors and Beyond Workshop, 9th IEEE-RAS International Conference on Humanoid Robots (2009)

    Google Scholar 

  51. H. Yousef, M. Boukallel, K. Althoefer, Tactile sensing for dexterous in-hand manipulation in robotics-A review. Sens. Actuat. A 167(2), 171–187 (2011)

    Article  Google Scholar 

  52. T.R. Grieve, J.M. Hollerbach, S.A. Mascaro, Force prediction by fingernail imaging using active appearance models, in 2013 World Haptics Conference, pp. 181–186 (2013)

    Google Scholar 

  53. D. Hristu, N. Ferrier, R.W. Brockett, The performance of a deformable-membrane tactile sensor: Basic results on geometrically-defined tasks, in 2000 IEEE International Conference on Robotics and Automation, pp. 508–513 (2000)

    Google Scholar 

  54. S.A. Mascaro, H.H. Asada, Photoplethysmograph fingernail sensors for measuring finger forces without haptic obstruction. IEEE Trans. Robot. Autom. 17(5), 698–708 (2001)

    Article  Google Scholar 

  55. Y. Sun, J.M. Hollerbach, S.A. Mascaro, Predicting fingertip forces by imaging coloration changes in the fingernail and surrounding skin. IEEE Trans. Biomed. Eng. 55(10), 2363–2371 (2008)

    Article  Google Scholar 

  56. Y. Sun, J.M. Hollerbach, S.A. Mascaro, Estimation of fingertip force direction with computer vision. IEEE Trans. Rob. 25(6), 1356–1369 (2009)

    Article  Google Scholar 

  57. S. Urban, J. Bayer, C. Osendorfer, G. Westling, B.B. Edin, P. Van Der Smagt, Computing grip force and torque from finger nail images using gaussian processes, in 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 4034–4039 (2013)

    Google Scholar 

  58. W. Yuan, R: Li, M.A. Srinivasan, and E.H. Adelson. Measurement of shear and slip with a GelSight tactile sensor, in 2015 IEEE International Conference on Robotics and Automation, pp. 304–311 (2015)

    Google Scholar 

  59. K. Sato, K. Kamiyama, N. Kawakami, S. Tachi, Finger-shaped gelforce: Sensor for measuring surface traction fields for robotic hand. IEEE Trans. Haptics 3(1), 37–47 (2010)

    Article  Google Scholar 

  60. L. Ballan, A. Taneja, J. Gall, L. Van Gool, and M. Pollefeys. Motion capture of hands in action using discriminative salient points, in European Conference on Computer Vision, pp. 640–653 (2012)

    Google Scholar 

  61. N. Kyriazis, A. Argyros, Physically plausible 3D scene tracking: the single actor hypothesis. In 2013 IEEE Conference on Computer Vision and Pattern Recognition, pp. 9–16 (2013)

    Google Scholar 

  62. Y. Wang, J. Min, J. Zhang, Y. Liu, F. Xu, Q. Dai, J. Chai, Video-based hand manipulation capture through composite motion control. ACM Trans. Gr. 32(4), 43 (2013)

    Article  Google Scholar 

  63. W. Zhao, J. Zhang, J. Min, J. Chai, Robust realtime physics-based motion control for human grasping. ACM Trans. Gr. 32(6), 1–12 (2013)

    Article  Google Scholar 

  64. M.A. Brubaker, L. Sigal, D.J. Fleet, Estimating contact dynamics, in 2009 IEEE International Conference on Computer Vision, pp. 2389–2396 (2009)

    Google Scholar 

  65. B. Frank, R. Schmedding, C. Stachniss, M. Teschner, W. Burgard, Learning the elasticity parameters of deformable objects with a manipulation robot, in 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 1877–1883 (2010)

    Google Scholar 

  66. B. Wang, L. Wu, K. Yin, U. Ascher, L. Liu, H. Huang, Deformation capture and modeling of soft objects. ACM Trans. Gr. 34(4), 94:1–94:12 (2015)

    Google Scholar 

  67. F. Largilliere, V. Verona, E. Coevoet, M. Sanz-Lopez, J. Dequidt, C. Duriez, Real-time control of soft-robots using asynchronous finite element modeling, in 2015 IEEE International Conference on Robotics and Automation, pp. 2550–2555 (2015)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Therenva, 74F rue de Paris, 35000, Rennes, France

    Antoine Petit

  2. Department of Electrical Engineering and Information Technology, CREATE Consortium & University of Naples Federico II, PRISMA Lab, Via Claudio 21, 80125, Naples, Italy

    Vincenzo Lippiello & Bruno Siciliano

Authors
  1. Antoine Petit
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Vincenzo Lippiello
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bruno Siciliano
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Department of Electrical Engineering, University of Naples Federico II, Naples, Italy

    Bruno Siciliano

  2. Department of Electrical Engineering, University of Naples Federico II, Naples, Italy

    Fabio Ruggiero

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Petit, A., Lippiello, V., Siciliano, B. (2022). Non-rigid Tracking Using RGB-D Data. In: Siciliano, B., Ruggiero, F. (eds) Robot Dynamic Manipulation. Springer Tracts in Advanced Robotics, vol 144. Springer, Cham. https://doi.org/10.1007/978-3-030-93290-9_2

Download citation

Publish with us

Policies and ethics