Skip to main content
Springer Nature Link
Log in

Self-supervised Interactive Object Segmentation Through a Singulation-and-Grasping Approach

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 (ECCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13699))

Included in the following conference series:

  • 3522 Accesses

Abstract

Instance segmentation with unseen objects is a challenging problem in unstructured environments. To solve this problem, we propose a robot learning approach to actively interact with novel objects and collect each object’s training label for further fine-tuning to improve the segmentation model performance, while avoiding the time-consuming process of manually labeling a dataset. Given a cluttered pile of objects, our approach chooses pushing and grasping motions to break the clutter and conducts object-agnostic grasping for which the Singulation-and-Grasping (SaG) policy takes as input the visual observations and imperfect segmentation. We decompose the problem into three subtasks: (1) the object singulation subtask aims to separate the objects from each other, which creates more space that alleviates the difficulty of (2) the collision-free grasping subtask; (3) the mask generation subtask obtains the self-labeled ground truth masks by using an optical flow-based binary classifier and motion cue post-processing for transfer learning. Our system achieves \(70\%\) singulation success rate in simulated cluttered scenes. The interactive segmentation of our system achieves \(87.8\%\), \(73.9\%\), and \(69.3\%\) average precision for toy blocks, YCB objects in simulation, and real-world novel objects, respectively, which outperforms the compared baselines. Please refer to our project page for more information: https://z.umn.edu/sag-interactive-segmentation.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Boerdijk, W., Sundermeyer, M., Durner, M., Triebel, R.: Self-supervised object-in-gripper segmentation from robotic motions. arXiv preprint arXiv:2002.04487 (2020)

  2. Bohg, J.: Interactive perception: Leveraging action in perception and perception in action. IEEE Trans. Rob. 33(6), 1273–1291 (2017)

    Article  Google Scholar 

  3. Byravan, A., Fox, D.: Se3-nets: Learning rigid body motion using deep neural networks. In: 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 173–180. IEEE (2017)

    Google Scholar 

  4. Calli, B., Walsman, A., Singh, A., Srinivasa, S., Abbeel, P., Dollar, A.M.: Benchmarking in manipulation research: Using the yale-cmu-berkeley object and model set. IEEE Robotics Autom. Mag. 22(3), 36–52 (2015). https://doi.org/10.1109/MRA.2015.2448951

    Article  Google Scholar 

  5. Chaudhary, K., et al.: Retrieving unknown objects using robot in-the-loop based interactive segmentation. In: 2016 IEEE/SICE International Symposium on System Integration (SII), pp. 75–80. IEEE (2016)

    Google Scholar 

  6. Chen, Y., Ju, Z., Yang, C.: Combining reinforcement learning and rule-based method to manipulate objects in clutter. In: 2020 International Joint Conference on Neural Networks (IJCNN), pp. 1–6. IEEE (2020)

    Google Scholar 

  7. Coleman, T.F., Moré, J.J.: Estimation of sparse Jacobian matrices and graph coloring problems. SIAM J. Numer. Anal. 20(1), 187–209 (1983)

    Article  MathSciNet  Google Scholar 

  8. Dave, A., Tokmakov, P., Ramanan, D.: Towards segmenting anything that moves. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, Oct 2019

    Google Scholar 

  9. Deng, Y., et al.: Deep reinforcement learning for robotic pushing and picking in cluttered environment. In: 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 619–626. IEEE (2019)

    Google Scholar 

  10. Eitel, A., Hauff, N., Burgard, W.: Self-supervised transfer learning for instance segmentation through physical interaction. In: 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4020–4026. IEEE (2019)

    Google Scholar 

  11. Eitel, A., Hauff, N., Burgard, W.: Learning to singulate objects using a push proposal network. In: Amato, N.M., Hager, G., Thomas, S., Torres-Torriti, M. (eds.) Robotics Research. SPAR, vol. 10, pp. 405–419. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-28619-4_32

    Chapter  Google Scholar 

  12. Fang, K., Bai, Y., Hinterstoisser, S., Savarese, S., Kalakrishnan, M.: Multi-task domain adaptation for deep learning of instance grasping from simulation. In: 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 3516–3523. IEEE (2018)

    Google Scholar 

  13. Fitzpatrick, P.: First contact: an active vision approach to segmentation. In: Proceedings 2003 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2003) (Cat. No. 03CH37453), vol. 3, pp. 2161–2166. IEEE (2003)

    Google Scholar 

  14. He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask r-cnn. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2961–2969 (2017)

    Google Scholar 

  15. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

    Google Scholar 

  16. Hermans, T., Rehg, J.M., Bobick, A.: Guided pushing for object singulation. In: 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 4783–4790. IEEE (2012)

    Google Scholar 

  17. Huang, B., Han, S.D., Boularias, A., Yu, J.: Dipn: Deep interaction prediction network with application to clutter removal. In: 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 4694–4701. IEEE (2021)

    Google Scholar 

  18. Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4700–4708 (2017)

    Google Scholar 

  19. Ilg, E., Mayer, N., Saikia, T., Keuper, M., Dosovitskiy, A., Brox, T.: Flownet 2.0: Evolution of optical flow estimation with deep networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2462–2470 (2017)

    Google Scholar 

  20. Kenney, J., Buckley, T., Brock, O.: Interactive segmentation for manipulation in unstructured environments. In: 2009 IEEE International Conference on Robotics and Automation, pp. 1377–1382. IEEE (2009)

    Google Scholar 

  21. Kiatos, M., Malassiotis, S.: Robust object grasping in clutter via singulation. In: 2019 International Conference on Robotics and Automation (ICRA), pp. 1596–1600. IEEE (2019)

    Google Scholar 

  22. Kurenkov, A., et al.: Visuomotor mechanical search: Learning to retrieve target objects in clutter. In: 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 8408–8414. IEEE (2020)

    Google Scholar 

  23. Kuzmič, E.S., Ude, A.: Object segmentation and learning through feature grouping and manipulation. In: 2010 10th IEEE-RAS International Conference on Humanoid Robots, pp. 371–378. IEEE (2010)

    Google Scholar 

  24. Le Goff, L.K., Mukhtar, G., Le Fur, P.H., Doncieux, S.: Segmenting objects through an autonomous agnostic exploration conducted by a robot. In: 2017 First IEEE International Conference on Robotic Computing (IRC), pp. 284–291. IEEE (2017)

    Google Scholar 

  25. Liang, H., Lou, X., Yang, Y., Choi, C.: Learning visual affordances with target-orientated deep q-network to grasp objects by harnessing environmental fixtures. In: 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 2562–2568. IEEE (2021)

    Google Scholar 

  26. Lin, T.-Y., et al.: Microsoft coco: Common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48

    Chapter  Google Scholar 

  27. Mnih, V., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529 (2015)

    Google Scholar 

  28. O Pinheiro, P.O., Collobert, R., Dollár, P.: Learning to segment object candidates. In: Advances in Neural Information Processing Systems, vol. 28 (2015)

    Google Scholar 

  29. Pathak, D., et al.: Learning instance segmentation by interaction. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 2042–2045 (2018)

    Google Scholar 

  30. Rohmer, E., Singh, S.P., Freese, M.: V-rep: A versatile and scalable robot simulation framework. In: 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 1321–1326. IEEE (2013)

    Google Scholar 

  31. Russakovsky, O., et al.: Imagenet large scale visual recognition challenge. Int. J. Comput. Vision 115(3), 211–252 (2015)

    Article  MathSciNet  Google Scholar 

  32. Sarantopoulos, I., Kiatos, M., Doulgeri, Z., Malassiotis, S.: Split deep q-learning for robust object singulation. In: 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 6225–6231. IEEE (2020)

    Google Scholar 

  33. Schiebener, D., Ude, A., Asfour, T.: Physical interaction for segmentation of unknown textured and non-textured rigid objects. In: 2014 IEEE International Conference on Robotics and Automation (ICRA), pp. 4959–4966. IEEE (2014)

    Google Scholar 

  34. Spelke, E.S.: Principles of object perception. Cogn. Sci. 14(1), 29–56 (1990)

    Article  Google Scholar 

  35. Wu, Y., Kirillov, A., Massa, F., Lo, W.Y., Girshick, R.: Detectron2. https://github.com/facebookresearch/detectron2 (2019)

  36. Xie, C., Xiang, Y., Mousavian, A., Fox, D.: The best of both modes: Separately leveraging rgb and depth for unseen object instance segmentation. In: Conference on Robot Learning, pp. 1369–1378. PMLR (2020)

    Google Scholar 

  37. Xie, C., Xiang, Y., Mousavian, A., Fox, D.: Unseen object instance segmentation for robotic environments. IEEE Trans. Robot. 1–17 (2021)

    Google Scholar 

  38. Xu, K., Yu, H., Lai, Q., Wang, Y., Xiong, R.: Efficient learning of goal-oriented push-grasping synergy in clutter. IEEE Robot. Autom. Lett. 6(4), 6337–6344 (2021)

    Article  Google Scholar 

  39. Yang, Y., Liang, H., Choi, C.: A deep learning approach to grasping the invisible. IEEE Robot. Autom. Lett. 5(2), 2232–2239 (2020)

    Article  Google Scholar 

  40. Zeng, A., Song, S., Welker, S., Lee, J., Rodriguez, A., Funkhouser, T.: Learning synergies between pushing and grasping with self-supervised deep reinforcement learning. In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4238–4245. IEEE (2018)

    Google Scholar 

Download references

Acknowledgements

This work was supported in part by the Sony Research Award Program and NSF Award 2143730.

Author information

Authors and Affiliations

  1. Department of Electrical and Computer Engineering, University of Minnesota, Minneapolis, USA

    Houjian Yu & Changhyun Choi

Authors
  1. Houjian Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Changhyun Choi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Houjian Yu .

Editor information

Editors and Affiliations

  1. Tel Aviv University, Tel Aviv, Israel

    Shai Avidan

  2. University College London, London, UK

    Gabriel Brostow

  3. Google AI, Accra, Ghana

    Moustapha Cissé

  4. University of Catania, Catania, Italy

    Giovanni Maria Farinella

  5. Facebook (United States), Menlo Park, CA, USA

    Tal Hassner

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 1255 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Yu, H., Choi, C. (2022). Self-supervised Interactive Object Segmentation Through a Singulation-and-Grasping Approach. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13699. Springer, Cham. https://doi.org/10.1007/978-3-031-19842-7_36

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-19842-7_36

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-19841-0

  • Online ISBN: 978-3-031-19842-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics