Skip to main content

Advertisement

SpringerLink
Log in

Inference of Manipulation Intent in Teleoperation for Robotic Assistance

  • Published:
Journal of Intelligent & Robotic Systems Aims and scope Submit manuscript

Abstract

In teleoperation, predicting an operator’s intent and providing subsequent assistance have demonstrated great advantages in reducing an operator’s workload and a task’s difficulty as well as enhancing the task performance. Current research aims to tackle target-approaching intent, while our work focus on inferring manipulation (task) intent after the user grasps the object. We model how an object is grasped when being utilized in different manipulation tasks (intents) and then adopt this object grasping model in teleoperation for the intent inference. Our paper focuses on determining if direct interaction models can be used for indirect interaction. As the nature of one’s grasping pose may satisfy multiple tasks (intents), we explore a form of classification modeling known as multi-label classification for multiple broad categories of tasks and objects. We also comprehensively compare classification techniques to determine the most suitable method for determining manipulation intent. With knowing the manipulation intent, future robot control algorithms can provide more helpful and appropriate assistance to facilitate task accomplishment.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Lum, M.J., Friedman, D.C., Sankaranarayanan, G., King, H., Fodero, K., Leuschke, R., Hannaford, B., Rosen, J., Sinanan, M.N.: The raven: Design and validation of a telesurgery system. Int. J. Robot. Res. 28(9), 1183–1197 (2009)

    Article  Google Scholar 

  2. Hochberg, L.R., Bacher, D., Jarosiewicz, B., Masse, N.Y., Simeral, J.D., Vogel, J., Haddadin, S., Liu, J., Cash, S.S., van der Smagt, P., et al.: Reach and grasp by people with tetraplegia using a neurally controlled robotic arm. Nature 485(7398), 372 (2012)

    Article  Google Scholar 

  3. Bodner, J., Wykypiel, H., Wetscher, G., Schmid, T.: First experiences with the da vinci™operating robot in thoracic surgery. Europ. J. Cardio-thoracic Surg. 25(5), 844–851 (2004)

    Article  Google Scholar 

  4. Rybarczyk, Y., Colle, E., Hoppenot, P.: Contribution of neuroscience to the teleoperation of rehabilitation robot. In: 2002 IEEE International Conference on Systems, Man and Cybernetics, vol. 4, pp. 6–pp. IEEE (2002)

  5. Healey, A.N.: Speculation on the neuropsychology of teleoperation: Implications for presence research and minimally invasive surgery. Presence 17(2), 199–211 (2008)

    Article  Google Scholar 

  6. Li, Y., Tee, K.P., Chan, W.L., Yan, R., Chua, Y., Limbu, D.K.: Continuous role adaptation for human–robot shared control. IEEE Trans. Robot. 31(3), 672–681 (2015)

    Article  Google Scholar 

  7. Webb, J.D., Li, S., Zhang, X.: Using visuomotor tendencies to increase control performance in teleoperation. In: American Control Conference (ACC), 2016, pp. 7110–7116. IEEE (2016)

  8. Dragan, A.D., Srinivasa, S.S.: A policy-blending formalism for shared control. Int. J. Robot. Res. 32(7), 790–805 (2013)

    Article  Google Scholar 

  9. Javdani, S., Srinivasa, S.S., Bagnell, J.A.: Shared autonomy via hindsight optimization. arXiv:1503.07619 (2015)

  10. Mylonas, G.P., Kwok, K.-W., James, D.R., Leff, D., Orihuela-Espina, F., Darzi, A., Yang, G.-Z.: Gaze-contingent motor channelling, haptic constraints and associated cognitive demand for robotic mis. Medi. Image Anal. 16(3), 612–631 (2012)

    Article  Google Scholar 

  11. Ren, J., Patel, R.V., McIsaac, K.A., Guiraudon, G., Peters, T.M.: Dynamic 3-d virtual fixtures for minimally invasive beating heart procedures. IEEE Trans. Med. Imag. 27(8), 1061–1070 (2008)

    Article  Google Scholar 

  12. Muelling, K., Venkatraman, A., Valois, J.-S., Downey, J., Weiss, J., Javdani, S., Hebert, M., Schwartz, A.B., Collinger, J.L., Bagnell, J.A.: Autonomy infused teleoperation with application to bci manipulation. arXiv:1503.05451 (2015)

  13. Kim, H.K., Biggs, J., Schloerb, W., Carmena, M., Lebedev, M.A., Nicolelis, M.A., Srinivasan, M.A.: Continuous shared control for stabilizing reaching and grasping with brain-machine interfaces. IEEE Trans. Biomed. Eng. 53(6), 1164–1173 (2006)

    Article  Google Scholar 

  14. Li, S., Zhang, X., Kim, F.J., da Silva, R.D., Gustafson, D., Molina, W.R.: Attention-aware robotic laparoscope based on fuzzy interpretation of eye-gaze patterns. J. Med. Dev. 9(4), 041007 (2015)

    Article  Google Scholar 

  15. Nikolaidis, S., Zhu, Y.X., Hsu, D., Srinivasa, S.: Human-robot mutual adaptation in shared autonomy. arXiv:1701.07851 (2017)

  16. Romano, J.M., Hsiao, K., Niemeyer, G., Chitta, S., Kuchenbecker, K.J.: Human-inspired robotic grasp control with tactile sensing. IEEE Trans. Robot. 27(6), 1067–1079 (2011)

    Article  Google Scholar 

  17. Lenz, I., Lee, H., Saxena, A.: Deep learning for detecting robotic grasps. Int. J. Robot. Res. 34(4-5), 705–724 (2015)

    Article  Google Scholar 

  18. Montesano, L., Lopes, M., Bernardino, A., Santos-Victor, J.: Learning object affordances: From sensory–motor coordination to imitation. IEEE Trans. Robot. 24(1), 15–26 (2008)

    Article  Google Scholar 

  19. Fischinger, D., Vincze, M.: Empty the basket-a shape based learning approach for grasping piles of unknown objects. In: 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2051–2057. IEEE (2012)

  20. Trinkle, J.C.: On the stability and instantaneous velocity of grasped frictionless objects. IEEE Trans. Robot. Autom. 8(5), 560–572 (1992)

    Article  Google Scholar 

  21. Song, D., Huebner, K., Kyrki, V., Kragic, D.: Learning task constraints for robot grasping using graphical models. In: 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1579–1585. IEEE (2010)

  22. Balasubramanian, R., Xu, L., Brook, P.D., Smith, J.R., Matsuoka, Y.: Physical human interactive guidance: Identifying grasping principles from human-planned grasps. IEEE Trans. Robot. 28(4), 899–910 (2012)

    Article  Google Scholar 

  23. Huaman Quispe, A., Ben Amor, H., Christensen, H., Stilman, M.: Grasping for a purpose: Using task goals for efficient manipulation planning, arXiv:1603.04338 (2016)

  24. Ng, A.Y., Jordan, M.I.: On discriminative vs. generative classifiers: A comparison of logistic regression and naive Bayes. In: Proceedings of the 14th International Conference on Neural Information Processing Systems: Natural and Synthetic, ser. NIPS’01, pp. 841–848. MIT Press, Cambridge (2001). [Online]. Available: http://dl.acm.org/citation.cfm?id=2980539.2980648

  25. Kohonen, T.: The self-organizing map. Neurocomputing 21(1-3), 1–6 (1998)

    Article  Google Scholar 

  26. Wehrens, R., Buydens, L.M., et al.: Self-and super-organizing maps in r: The kohonen package. J Stat Softw 21(5), 1–19 (2007)

    Article  Google Scholar 

  27. Aliferis, C.F., Tsamardinos, I., Statnikov, A.: Hiton: A novel Markov blanket algorithm for optimal variable selection. In: AMIA Annual Symposium Proceedings, vol. 2003, p 21. American Medical Informatics Association (2003)

  28. Scutari, M.: Learning bayesian networks with the bnlearn R package. J. Stat. Softw. 35(3), 1–22 (2010)

    Article  Google Scholar 

  29. Kruskal, J.B.: On the shortest spanning subtree of a graph and the traveling salesman problem. Proc. Am. Math. Soc. 7(1), 48–50 (1956)

    Article  MathSciNet  Google Scholar 

  30. Cooper, G.F., Herskovits, E.: A bayesian method for the induction of probabilistic networks from data. Mach. Learn. 9(4), 309–347 (1992)

    MATH  Google Scholar 

  31. Tsoumakas, G., Katakis, I.: Multi-label classification: An overview. Int. J. Data Warehousing Mining 3, 3 (2006)

    Google Scholar 

  32. Alvares-Cherman, E., Metz, J., Monard, M.C.: Incorporating label dependency into the binary relevance framework for multi-label classification. Expert Syst. Appl. 39(2), 1647–1655 (2012)

    Article  Google Scholar 

  33. Elisseeff, A., Weston, J.: A kernel method for multi-labelled classification. In: Advances in Neural Information Processing Systems (2002)

  34. Cooper, G.F.: The computational complexity of probabilistic inference using Bayesian belief networks. Artif. Intell. 42(2–3), 393–405 (1990)

    Article  MathSciNet  Google Scholar 

  35. Abdiansah, A., Wardoyo, R.: Time complexity analysis of support vector machines (svm) in libsvm. International Journal Computer and Application (2015)

  36. Rojas, R.: Neural Networks: A Systematic Introduction. Springer Science and Business Media (2013)

  37. Baranitha, R., Mohajerpoor, R., Rakkiyappan, R.: Bilateral teleoperation of single-master multislave systems with semi-Markovian jump stochastic interval time-varying delayed communication channels. IEEE Trans. Cybern., 1–11 (2019)

  38. Mohajerpoor, R., Sharifi, I., Talebi, H.A., Rezaei, S.M.: Adaptive bilateral teleoperation of an unknown object handled by multiple robots under unknown communication delay. In: 2013 IEEE/ASME International Conference on Advanced Intelligent Mechatronics, pp. 1158–1163 (2013)

Download references

Acknowledgements

This material is based on work supported by the US NSF under grant 1652454. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect those of the National Science Foundation.

Author information

Authors and Affiliations

  1. Department of Electrical and Computer Engineering, Duke University, Durham, NC, 27708, USA

    Songpo Li

  2. Department of Mechanical Engineering, Colorado School of Mines, Golden, CO, 80401, USA

    Michael Bowman, Hamed Nobarani & Xiaoli Zhang

Authors
  1. Songpo Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Michael Bowman
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hamed Nobarani
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xiaoli Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiaoli Zhang.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, S., Bowman, M., Nobarani, H. et al. Inference of Manipulation Intent in Teleoperation for Robotic Assistance. J Intell Robot Syst 99, 29–43 (2020). https://doi.org/10.1007/s10846-019-01125-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10846-019-01125-8

Keywords

Search

Navigation