Skip to main content
SpringerLink
Log in

Objects Relocation in Clutter with Robot Manipulators via Tree-based Q-Learning Algorithm: Analysis and Experiments

  • Regular paper
  • Published:
Journal of Intelligent & Robotic Systems Aims and scope Submit manuscript

Abstract

This work addresses the problem of retrieving a target object from cluttered environment using a robot manipulator. In the details, the proposed solution relies on a Task and Motion Planning approach based on a two-level architecture: the high-level is a Task Planner aimed at finding the optimal objects sequence to relocate, according to a metric based on the objects weight; the low-level is a Motion Planner in charge of planning the end-effector path for reaching the specific objects taking into account the robot physical constraints. The high-level task planner is a Reinforcement Learning agent, trained using the information coming from the low-level Motion Planner. In this work we consider the Q-Tree algorithm, which is based on a dynamic tree structure inspired by the Q-learning technique. Three different RL-policies with two kinds of tree exploration techniques (Breadth and Depth) are compared in simulation scenarios with different complexity. Moreover, the proposed learning methods are experimentally validated in a real scenario by adopting a KINOVA Jaco\(^{2}\) 7-DoFs robot manipulator.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Correll, N., Bekris, K.E., Berenson, D., Brock, O., Causo, A., Hauser, K., Okada, K., Rodriguez, A., Romano, J.M., Wurman, P.R.: Analysis and observations from the first amazon picking challenge. IEEE Trans. Autom. Sci. Eng. 15(1), 172 (2016)

    Article  Google Scholar 

  2. Ceola, F., Tosello, E., Tagliapietra, L., Nicola, G., Ghidoni S.: Robot task planning via deep reinforcement learning: a tabletop object sorting application. In: 2019 IEEE Int. Conf. on Systems, Man and Cybernetics (SMC), pp. 486–492. IEEE (2019)

  3. Nam, C., Lee, J., Cheong, S.H., Cho, B.Y., Kim C.: Fast and resilient manipulation planning for target retrieval in clutter. In: 2020 IEEE Int. Conf. on Robotics and Automation (ICRA), pp. 3777–3783. IEEE (2020)

  4. Lee, J., Cho, Y., Nam, C., Park, J., Kim, C.: Efficient obstacle rearrangement for object manipulation tasks in cluttered environments. In: 2019 Int. Conf. on Robotics and Automation (ICRA), pp. 183–189. IEEE (2019)

  5. Stilman, M., Kuffner, J.: Planning among movable obstacles with artificial constraints. Int. Journ. Robot. Res. 27(11–12), 1295 (2008)

    Article  Google Scholar 

  6. Hang, K., Stork, J.A., Pokorny, F.T., Kragic, D.: Combinatorial optimization for hierarchical contact-level grasping. In: 2014 IEEE Int. Conf. on Robotics and Automation (ICRA), pp. 381–388. IEEE (2014)

  7. Stilman, M., Schamburek, J.U., Kuffner, J., Asfour, T.: Manipulation planning among movable obstacles. In Proceedings 2007 IEEE Int. Conf. on Robotics and Automation, pp. 3327–3332. IEEE (2007)

  8. Yuan, W., Hang, K. Kragic, D., Wang, M.Y., Stork, J.A.: End-to-end nonprehensile rearrangement with deep reinforcement learning and simulation-to-reality transfer. Robotics and Autonomous Systems 119 (2019)

  9. Haustein, J.A., King, J., Srinivasa, S.S., Asfour, T.: Kinodynamic randomized rearrangement planning via dynamic transitions between statically stable states. In: 2015 IEEE Int. Conf. on Robotics and Automation (ICRA), pp. 3075–3082. IEEE (2015)

  10. Dantam, N.T., Kingston, Z.K., Chaudhuri, S., Kavraki, L.E.: Incremental task and motion planning: A constraint-based approach. In Robotics: Science and systems, vol. 12, p. 00052. Ann Arbor, MI, USA (2016)

  11. Havur, G., Ozbilgin, G., Erdem, E., Patoglu, V.: Geometric rearrangement of multiple movable objects on cluttered surfaces: A hybrid reasoning approach. In: 2014 IEEE Int. Conf. on Robotics and Automation (ICRA), pp. 445–452. IEEE (2014)

  12. Karami, H., Thomas, A., Mastrogiovanni, F.: A task-motion planning framework using iteratively deepened and/or graph networks. arXiv:2104.01549 (2021)

  13. Eppe, M., Nguyen, P.D., Wermter, S.: From semantics to execution: Integrating action planning with reinforcement learning for robotic causal problemsolving. Front. Robot. AI 6, 123 (2019)

    Article  Google Scholar 

  14. Bonet, B., Geffner, H.: Planning as heuristic search. Artif. Intell. 129(1–2), 5 (2001)

    Article  MathSciNet  Google Scholar 

  15. Qureshi, A.H., Mousavian, A., Paxton, C., Yip, M.C., Fox, D.: Nerp: Neural rearrangement planning for unknown objects. arXiv:2106.01352 (2021)

  16. Mohammed, M.Q., Chung, K.L., Chyi, C.S.: Review of deep reinforcement learning-based object grasping: Techniques, open challenges and recommendations. IEEE Access (2020)

  17. Kormushev, P., Calinon, S., Caldwell, D.G.: Reinforcement learning in robotics: Applications and real-world challenges. Robotics 2(3), 122 (2013)

    Article  Google Scholar 

  18. Bejjani, W., Agboh, W.C., Dogar, M.R., Leonetti, M.: Occlusion-aware search for object retrieval in clutter. In: 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4678–4685. IEEE (2021)

  19. Deng, Y., Guo, X., Wei, Y., Lu, K., Fang, B., Guo, D., Liu, H., Sun, F.: Deep reinforcement learning for robotic pushing and picking in cluttered environment. In 2019 IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS), pp. 619–626. IEEE (2019)

  20. Wu, B., Akinola, I., Allen, P.K.: Pixel-attentive policy gradient for multi-fingered grasping in cluttered scenes. In: 2019 IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS), pp. 1789–1796. IEEE (2019)

  21. LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436 (2015)

    Article  Google Scholar 

  22. Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., Riedmiller, M.: Playing atari with deep reinforcement learning. arXiv:1312.5602 (2013)

  23. Mnih, V., Badia, A.P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., Kavukcuoglu, K.: Asynchronous methods for deep reinforcement learning. In: Int. Conf. on Machine Learning, pp. 1928–1937. PMLR (2016)

  24. Golluccio, G., Di Vito, D., Marino, A., Bria, A., Antonelli, G.: Task-motion planning via tree-based q-learning approach for robotic object displacement in cluttered spaces. In: Proceedings of the 18th Int. Conf. on Informatics in Control, Automation and Robotics - ICINCO. INSTICC, pp. 130–137. SciTePress (2021) https://doi.org/10.5220/0010542601300137

  25. Golluccio, G., Di Vito, D., Marino, A., Antonelli, G.: Robotic weight-based object relocation in clutter via tree-based q-learning approach using breadth and depth search techniques. In 2021 20th Int. Conf. on Advanced Robotics (ICAR), pp. 676–681. IEEE (2021)

  26. Watkins, C.J., Dayan, P.: Q-learning. Mach. Learn. 8(3–4), 279 (1992)

    Article  Google Scholar 

  27. Di Vito, D., Bergeron, M., Meger, D., Dudek, G., Antonelli, G.: Dynamic planning of redundant robots within a set-based task-priority inverse kinematics framework. In 2020 IEEE Conf. on Control Technology and Applications (CCTA), pp. 549–554. IEEE (2020)

  28. Kuffner, J.J., LaValle, S.M.: Rrt-connect: An efficient approach to singlequery path planning. In: Proceedings 2000 ICRA. Millennium Conference. IEEE Int. Conf. on Robotics and Automation. Symposia Proceedings (Cat. No. 00CH37065), vol. 2, pp. 995–1001. IEEE (2000),

  29. Chiaverini, S.: Singularity-robust task-priority redundancy resolution for real-time kinematic control of robot manipulators. IEEE Trans. Robot. Autom. 13(3), 398 (1997)

    Article  Google Scholar 

  30. Siciliano, B., Slotine, J.J.E.: A general framework for managing multiple tasks in highly redundant robotic systems. In: Proc. Fifth Int. Conf. on Advanced Robotics (ICAR), pp. 1211–1216. IEEE, Pisa (1991) https://doi.org/10.1109/ICAR.1991.240390

  31. Di Lillo, P., Arrichiello, F., Di Vito, D., Antonelli, G.: BCIcontrolled assistive manipulator: developed architecture and experimental results. IEEE Trans. Cognitive Development. Syst. pp. 1–1 (2020). https://doi.org/10.1109/TCDS.2020.2979375

  32. Di Lillo, P., Simetti, E., Wanderlingh, F., Casalino, G., Antonelli, G.: Underwater intervention with remote supervision via satellite communication: Developed control architecture and experimental results within the dexrov project. IEEE Trans. Control Syst. Technol. 29(1), 108 (2021). https://doi.org/10.1109/TCST.2020.2971440

    Article  Google Scholar 

  33. Garrido-Jurado, S., Muñoz-Salinas, R., Madrid-Cuevas, F.J., Marín-Jiménez, M.J.: Automatic generation and detection of highly reliable fiducial markers under occlusion. Patt. Recognit. 47(6), 2280 (2014)

    Article  Google Scholar 

  34. Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: Unified, real-time object detection. In: Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition, pp. 779–788 (2016)

Download references

Funding

The Authors declare that this work was supported by Dipartimento di Eccellenza granted to DIEI Department, University of Cassino and Southern Lazio, by H2020-ICT project CANOPIES (Grant Agreement N. 101016906) and by POR FSE LAZIO 2014-2020, Project DE G06374/2021.

Author information

Authors and Affiliations

  1. Department of Electrical and Information Engineering (DIEI), University of Cassino and Southern Lazio, Cassino, Italy

    Giacomo Golluccio, Paolo Di Lillo, Daniele Di Vito, Alessandro Marino & Gianluca Antonelli

Authors
  1. Giacomo Golluccio
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Paolo Di Lillo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Daniele Di Vito
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Alessandro Marino
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Gianluca Antonelli
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All Authors have contributed equally to the ideas, theories and analysis of results. The first draft of the manuscript was written by Giacomo Golluccio. Paolo Di Lillo, Daniele Di Vito, Alessandro Marino and Gianluca Antonelli commented and revised this first version. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Giacomo Golluccio.

Ethics declarations

Conflicts of interest

The Authors declare that they have no conflict of interest.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Golluccio, G., Di Lillo, P., Di Vito, D. et al. Objects Relocation in Clutter with Robot Manipulators via Tree-based Q-Learning Algorithm: Analysis and Experiments. J Intell Robot Syst 106, 44 (2022). https://doi.org/10.1007/s10846-022-01719-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10846-022-01719-9

Keywords

Search

Navigation