Skip to main content
SpringerLink
Log in

End-to-End Surface Reconstruction for Touching Trajectories

  • Conference paper
  • First Online:
Computer Vision – ACCV 2022 (ACCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13847))

Included in the following conference series:

  • 374 Accesses

Abstract

Whereas vision based 3D reconstruction strategies have progressed substantially with the abundance of visual data and emerging machine-learning tools, there are as yet no equivalent work or datasets with which to probe the use of the touching information. Unlike vision data organized in regularly arranged pixels or point clouds evenly distributed in space, touching trajectories are composed of continuous basic lines, which brings more sparsity and ambiguity. In this paper we address this problem by proposing the first end-to-end haptic reconstruction network, which takes any arbitrary touching trajectory as input, learns an implicit representation of the underling shape and outputs a watertight triangle surface. It is composed of three modules, namely trajectory feature extraction, 3D feature interpolation, as well as implicit surface validation. Our key insight is that formulating the haptic reconstruction process into an implicit surface learning problem not only brings the ability to reconstruct shapes, but also improves the fitting ability of the network in small datasets. To tackle the sparsity of the trajectories, we use a spatial gridding operator to assign features of touching trajectories into grids. A surface validation module is used to tackle the dilemma of computing resources and calculation accuracy. We also build the first touching trajectory dataset, formulating touching process under the guide of Gaussian Process. We demonstrate that our method performs favorably against other methods both in qualitive and quantitative way. Insights from the tactile signatures of the touching will aid the future design of virtual-reality and human-robot interactions.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Dataset and source code can be found: https://github.com/LiuLiuJerry/TouchNet.

References

  1. Alldieck, T., Magnor, M., Bhatnagar, B.L., Theobalt, C., Pons-Moll, G.: Learning to reconstruct people in clothing from a single rgb camera. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1175–1186 (2019). https://doi.org/10.1109/CVPR.2019.00127

  2. Chu, V., et al.: Robotic learning of haptic adjectives through physical interaction. Robot. Auton. Syst. 63, 279–292 (2015)

    Article  Google Scholar 

  3. Dai, A., Qi, C.R., Nießner, M.: Shape completion using 3d-encoder-predictor cnns and shape synthesis. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6545–6554 (2017). https://doi.org/10.1109/CVPR.2017.693

  4. Dallaire, P., Giguère, P., Émond, D., Chaib-Draa, B.: Autonomous tactile perception: A combined improved sensing and bayesian nonparametric approach. Robot. Auton. Syst. 62(4), 422–435 (2014)

    Article  Google Scholar 

  5. Driess, D., Englert, P., Toussaint, M.: Active learning with query paths for tactile object shape exploration. In: 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 65–72 (2017). https://doi.org/10.1109/IROS.2017.8202139

  6. Egger, B., et al.: 3d morphable face models - past, present and future. arXiv: 1909.01815 (2019)

  7. Erickson, Z., Chernova, S., Kemp, C.C.: Semi-supervised haptic material recognition for robots using generative adversarial networks. In: Conference on Robot Learning, pp. 157–166. PMLR (2017)

    Google Scholar 

  8. Gecer, B., Ploumpis, S., Kotsia, I., Zafeiriou, S.: GANFIT: generative adversarial network fitting for high fidelity 3d face reconstruction. arXiv: 1902.05978 (2019)

  9. Giguere, P., Dudek, G.: A simple tactile probe for surface identification by mobile robots. IEEE Trans. Rob. 27(3), 534–544 (2011)

    Article  Google Scholar 

  10. Han, X., Li, Z., Huang, H., Kalogerakis, E., Yu, Y.: High-resolution shape completion using deep neural networks for global structure and local geometry inference. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 85–93 (2017). https://doi.org/10.1109/ICCV.2017.19

  11. Huang, J., Zhou, Y., Guibas, L.: Manifoldplus: A robust and scalable watertight manifold surface generation method for triangle soups. arXiv preprint arXiv:2005.11621 (2020)

  12. Kanazawa, A., Tulsiani, S., Efros, A.A., Malik, J.: Learning category-specific mesh reconstruction from image collections. arXiv: 1803.07549 (2018)

  13. Kolotouros, N., Pavlakos, G., Daniilidis, K.: Convolutional mesh regression for single-image human shape reconstruction. In: CVPR (2019)

    Google Scholar 

  14. Kursun, O., Patooghy, A.: An embedded system for collection and real-time classification of a tactile dataset. IEEE Access 8, 97462–97473 (2020)

    Article  Google Scholar 

  15. Li, D., Shao, T., Wu, H., Zhou, K.: Shape completion from a single rgbd image. IEEE Trans. Visual Comput. Graphics 23(7), 1809–1822 (2017). https://doi.org/10.1109/TVCG.2016.2553102

    Article  Google Scholar 

  16. Li, J., Niu, C., Xu, K.: Learning part generation and assembly for structure-aware shape synthesis. arXiv: 1906.06693 (2019)

  17. Lin, C., et al.: Photometric mesh optimization for video-aligned 3d object reconstruction. arXiv: 1903.08642 (2019)

  18. Liu, J., Xia, Q., Li, S., Hao, A., Qin, H.: Quantitative and flexible 3d shape dataset augmentation via latent space embedding and deformation learning. Comput. Aided Geometric Design 71, 63–76 (2019). https://doi.org/10.1016/j.cagd.2019.04.017, https://www.sciencedirect.com/science/article/pii/S0167839619300330

  19. Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., Black, M.J.: Smpl: a skinned multi-person linear model. ACM Trans. Graph. 34, 248:1–248:16 (2015)

    Google Scholar 

  20. Mandikal, P., Babu, R.V.: Dense 3d point cloud reconstruction using a deep pyramid network. arXiv: 1901.08906 (2019)

  21. Mi, Z., Luo, Y., Tao, W.: Tsrnet: Scalable 3d surface reconstruction network for point clouds using tangent convolution. arXiv: 1911.07401 (2019)

  22. Oddo, C.M., Controzzi, M., Beccai, L., Cipriani, C., Carrozza, M.C.: Roughness encoding for discrimination of surfaces in artificial active-touch. IEEE Trans. Rob. 27(3), 522–533 (2011)

    Article  Google Scholar 

  23. Paszke, A., et al.: Pytorch: An imperative style, high-performance deep learning library. arXiv: 1912.01703 (2019)

  24. Qi, C.R., Su, H., Mo, K., Guibas, L.J.: Pointnet: Deep learning on point sets for 3d classification and segmentation. arXiv: 1612.00593 (2016)

  25. Qi, C.R., Yi, L., Su, H., Guibas, L.J.: Pointnet++: Deep hierarchical feature learning on point sets in a metric space. arXiv: 1706.02413 (2017)

  26. R., M.B., Tewari, A., Seidel, H., Elgharib, M., Theobalt, C.: Learning complete 3d morphable face models from images and videos. arXiv: 2010.01679 (2020).

  27. Richardson, B.A., Kuchenbecker, K.J.: Improving haptic adjective recognition with unsupervised feature learning. In: 2019 International Conference on Robotics and Automation (ICRA), pp. 3804–3810. IEEE (2019)

    Google Scholar 

  28. Saito, S., Huang, Z., Natsume, R., Morishima, S., Li, H., Kanazawa, A.: Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 2304–2314 (2019). https://doi.org/10.1109/ICCV.2019.00239

  29. Sundaram, S., Kellnhofer, P., Li, Y., Zhu, J.Y., Torralba, A., Matusik, W.: Learning the signatures of the human grasp using a scalable tactile glove. Nature 569, 698–702 (2019). https://doi.org/10.1038/s41586-019-1234-z

    Article  Google Scholar 

  30. Tulbure, A., Bäuml, B.: Superhuman performance in tactile material classification and differentiation with a flexible pressure-sensitive skin. In: 2018 IEEE-RAS 18th International Conference on Humanoid Robots (Humanoids), pp. 1–9. IEEE (2018)

    Google Scholar 

  31. Varley, J., DeChant, C., Richardson, A., Nair, A., Ruales, J., Allen, P.K.: Shape completion enabled robotic grasping. arXiv: 1609.08546 (2016)

  32. Varley, J., Watkins-Valls, D., Allen, P.K.: Multi-modal geometric learning for grasping and manipulation. arXiv: 1803.07671 (2018)

  33. Wang, K., Chen, K., Jia, K.: Deep cascade generation on point sets. In: Proceedings of the 28th International Joint Conference on Artificial Intelligence, IJCAI 2019, pp. 3726–3732. AAAI Press (2019)

    Google Scholar 

  34. Wang, Y., Sun, Y., Liu, Z., Sarma, S.E., Bronstein, M.M., Solomon, J.M.: Dynamic graph CNN for learning on point clouds. arXiv: 1801.07829 (2018)

  35. Wen, Y., Liu, W., Raj, B., Singh, R.: Self-supervised 3d face reconstruction via conditional estimation. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 13269–13278 (2021). https://doi.org/10.1109/ICCV48922.2021.01304

  36. Windau, J., Shen, W.M.: An inertia-based surface identification system. In: 2010 IEEE International Conference on Robotics and Automation, pp. 2330–2335. IEEE (2010)

    Google Scholar 

  37. Wu, Z., Song, S., Khosla, A., Tang, X., Xiao, J.: 3d shapenets for 2.5d object recognition and next-best-view prediction. arXiv: 1406.5670 (2014)

  38. Xie, H., Yao, H., Sun, X., Zhou, S., Zhang, S., Tong, X.: Pix2vox: Context-aware 3d reconstruction from single and multi-view images. arXiv: 1901.11153 (2019)

  39. Xie, H., Yao, H., Zhou, S., Mao, J., Zhang, S., Sun, W.: Grnet: Gridding residual network for dense point cloud completion. arXiv: 2006.03761 (2020)

  40. Yi, Z., et al.: Active tactile object exploration with gaussian processes. In: 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4925–4930 (2016). https://doi.org/10.1109/IROS.2016.7759723

  41. Yin, K., Huang, H., Cohen-Or, D., Zhang, H.R.: P2P-NET: bidirectional point displacement network for shape transform. arXiv: 1803.09263 (2018)

Download references

Acknowledgements

This work is supported by the National Science and Technology Major Project from Minister of Science and Technology, China (Grant No. 2018AAA0103100), the Guangdong Provincial Key Research and Development Plan(Grant No. 2019B090917009), the Science and Technology Program of Guangzhou, China(Grant No. 202201000009).

Author information

Authors and Affiliations

  1. Institute of Automation, Chinese Academy of Sciences, Beijing, China

    Jiarui Liu, Yuanpei Zhang, Zhuojun Zou & Jie Hao

  2. Guangdong Institute of Artificial Intelligence and Advanced Computing, Guangzhou, China

    Jiarui Liu & Jie Hao

Authors
  1. Jiarui Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yuanpei Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhuojun Zou
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jie Hao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jie Hao .

Editor information

Editors and Affiliations

  1. University of Wollongong, Wollongong, NSW, Australia

    Lei Wang

  2. University of Bonn, Bonn, Germany

    Juergen Gall

  3. University of Adelaide, Adelaide, SA, Australia

    Tat-Jun Chin

  4. National Institute of Informatics, Tokyo, Japan

    Imari Sato

  5. Johns Hopkins University, Baltimore, MD, USA

    Rama Chellappa

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Liu, J., Zhang, Y., Zou, Z., Hao, J. (2023). End-to-End Surface Reconstruction for Touching Trajectories. In: Wang, L., Gall, J., Chin, TJ., Sato, I., Chellappa, R. (eds) Computer Vision – ACCV 2022. ACCV 2022. Lecture Notes in Computer Science, vol 13847. Springer, Cham. https://doi.org/10.1007/978-3-031-26293-7_7

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-26293-7_7

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-26292-0

  • Online ISBN: 978-3-031-26293-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation