skip to main content
10.1145/3313831.3376705acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article
Public Access

Dynamic Motor Skill Synthesis with Human-Machine Mutual Actuation

Published: 23 April 2020 Publication History

Abstract

This paper presents an approach for coupling robotic capability with human ability in dynamic motor skills, called "Human-Machine Mutual Actuation (HMMA)." We focus specifically on throwing motions and propose a method to control the release timing computationally. A system we developed achieves our concept, HMMA, by a robotic handheld device that acts as a release controller. We conducted user studies to validate the feasibility of the concept and clarify related technical issues to be tackled. We recognized that the system successfully performs on throwing according to the target while it exploits human ability. These empirical experiments suggest that robotic capability can be embedded into the users' motions without losing their senses of control. Throughout the user study, we also revealed several issues to be tackled in further research contributing to HMMA.

Supplemental Material

MP4 File
MP4 File
Preview video

References

[1]
Ismail Ben Abdallah, Yassine Bouteraa, and Chokri Rekik. 2017. Design and development of 3D printed myoelectric robotic exoskeleton for hand rehabilitation. International Journal on Smart Sensing & Intelligent Systems 10, 2 (2017).
[2]
Lung-Pan Cheng, Patrick Lühne, Pedro Lopes, Christoph Sterz, and Patrick Baudisch. 2014. Haptic turk: a motion platform based on people. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 3463--3472.
[3]
Lung-Pan Cheng, Sebastian Marwecki, and Patrick Baudisch. 2017. Mutual human actuation. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology. ACM, 797--805.
[4]
Lung-Pan Cheng, Thijs Roumen, Hannes Rantzsch, Sven Köhler, Patrick Schmidt, Robert Kovacs, Johannes Jasper, Jonas Kemper, and Patrick Baudisch. 2015. Turkdeck: Physical virtual reality based on people. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology. ACM, 417--426.
[5]
Seongkook Heo, Christina Chung, Geehyuk Lee, and Daniel Wigdor. 2018. Thor's hammer: An ungrounded force feedback device utilizing propeller-induced propulsive force. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 525.
[6]
Jon Hore and Sherry Watts. 2011. Skilled throwers use physics to time ball release to the nearest millisecond. Journal of Neurophysiology 106, 4 (2011), 2024--2033.
[7]
Yuuki Horiuchi, Yasutoshi Makino, and Hiroyuki Shinoda. 2017. Computational foresight: forecasting human body motion in real-time for reducing delays in interactive system. In Proceedings of the 2017 ACM International Conference on Interactive Surfaces and Spaces. ACM, 312--317.
[8]
Jian Huang, Weiguang Huo, Wenxia Xu, Samer Mohammed, and Yacine Amirat. 2015. Control of upper-limb power-assist exoskeleton using a human-robot interface based on motion intention recognition. IEEE transactions on automation science and engineering 12, 4 (2015), 1257--1270.
[9]
Yuta Itoh, Jason Orlosky, Kiyoshi Kiyokawa, and Gudrun Klinker. 2016. Laplacian vision: Augmenting motion prediction via optical see-through head-mounted displays. In Proceedings of the 7th Augmented Human International Conference 2016. ACM, 16.
[10]
Urs Keller, Hubertus JA van Hedel, Verena Klamroth-Marganska, and Robert Riener. 2016. ChARMin: The frst actuated exoskeleton robot for pediatric arm rehabilitation. IEEE/ASME Transactions on Mechatronics 21, 5 (2016), 2201--2213.
[11]
Jinsoo Kim, Giuk Lee, Roman Heimgartner, Dheepak Arumukhom Revi, Nikos Karavas, Danielle Nathanson, Ignacio Galiana, Asa Eckert-Erdheim, Patrick Murphy, David Perry, and others. 2019. Reducing the metabolic rate of walking and running with a versatile, portable exosuit. Science 365, 6454 (2019), 668--672.
[12]
Jan Koprnický, Petr Najman, and Jií afka. 2017. 3D printed bionic prosthetic hands. In 2017 IEEE International Workshop of Electronics, Control, Measurement, Signals and their Application to Mechatronics (ECMSM). IEEE, 1--6.
[13]
Benjamin Lafreniere, Tovi Grossman, Fraser Anderson, Justin Matejka, Heather Kerrick, Danil Nagy, Lauren Vasey, Evan Atherton, Nicholas Beirne, Marcelo H Coelho, and others. 2016. Crowdsourced fabrication. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. ACM, 15--28.
[14]
Ian Lenz, Honglak Lee, and Ashutosh Saxena. 2015. Deep learning for detecting robotic grasps. The International Journal of Robotics Research 34, 4--5 (2015), 705--724.
[15]
Baldin Llorens-Bonilla, Federico Parietti, and H Harry Asada. 2012. Demonstration-based control of supernumerary robotic limbs. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, 3936--3942.
[16]
Sebastian Madgwick. 2010. An effcient orientation flter for inertial and inertial/magnetic sensor arrays. Report x-io and University of Bristol (UK) 25 (2010), 113--118.
[17]
Azumi Maekawa, Seito Matsubara, Atsushi Hiyama, and Masahiko Inami. 2019a. PickHits: hitting experience generation with throwing motion via a handheld mechanical device. In ACM SIGGRAPH 2019 Emerging Technologies. ACM, 20.
[18]
Azumi Maekawa, Shota Takahashi, MHD Saraiji, Sohei Wakisaka, Hiroyasu Iwata, and Masahiko Inami. 2019b. Naviarm: Augmenting the Learning of Motor Skills using a Backpack-type Robotic Arm System. In Proceedings of the 10th Augmented Human International Conference 2019. ACM, 38.
[19]
Julieta Martinez, Michael J Black, and Javier Romero. 2017. On Human Motion Prediction Using Recurrent Neural Networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 4674--4683.
[20]
Kenichi Murakami, Yuji Yamakawa, Taku Senoo, and Masatoshi Ishikawa. 2015. Motion planning for catching a light-weight ball with high-speed visual feedback. In 2015 IEEE International Conference on Robotics and Biomimetics (ROBIO). IEEE, 339--344.
[21]
Jun Nishida, Shunichi Kasahara, and Kenji Suzuki. 2017. Wired muscle: generating faster kinesthetic reaction by inter-personally connecting muscles. In ACM SIGGRAPH 2017 Emerging Technologies. ACM, 26.
[22]
Kei Nitta, Keita Higuchi, and Jun Rekimoto. 2014. HoverBall: augmented sports with a fying ball. In Proceedings of the 5th Augmented Human International Conference. ACM, 13.
[23]
Kei Nitta, Keita Higuchi, Yuichi Tadokoro, and Jun Rekimoto. 2015. Shepherd pass: ability tuning for augmented sports using ball-shaped quadcopter. In Proceedings of the 12th International Conference on Advances in Computer Entertainment Technology. ACM, 11.
[24]
Takuya Nojima, Dairoku Sekiguchi, Masahiko Inami, and Susumu Tachi. 2002. The SmartTool: A system for augmented reality of haptics. In Proceedings IEEE Virtual Reality 2002. IEEE, 67--72.
[25]
Tomoya Ohta, Shumpei Yamakawa, Takashi Ichikawa, and Takuya Nojima. 2014. TAMA: development of trajectory changeable ball for future entertainment. In Proceedings of the 5th Augmented Human International Conference. ACM, 50.
[26]
Federico Parietti, Kameron Chan, and H Harry Asada. 2014. Bracing the human body with supernumerary robotic limbs for physical assistance and load reduction. In 2014 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 141--148.
[27]
Luka Peternel, Tadej Petri?. c, Erhan Oztop, and Jan Babi?. 2014. Teaching robots to cooperate with humans in dynamic manipulation tasks based on multi-modal human-in-the-loop approach. Autonomous robots 36, 1--2 (2014), 123--136.
[28]
Domenico Prattichizzo, Monica Malvezzi, Irfan Hussain, and Gionata Salvietti. 2014. The sixth-fnger: a modular extra-fnger to enhance human hand capabilities. In The 23rd IEEE International Symposium on Robot and Human Interactive Communication. IEEE, 993--998.
[29]
RJ Rajesh and P Kavitha. 2015. Camera gimbal stabilization using conventional PID controller and evolutionary algorithms. In 2015 International Conference on Computer, Communication and Control (IC4). IEEE, 1--6.
[30]
Robert Riener. 2016. The Cybathlon promotes the development of assistive technology for people with physical disabilities. Journal of neuroengineering and rehabilitation 13, 1 (2016), 49.
[31]
Alec Rivers, Ilan E Moyer, and Frédo Durand. 2012. Position-correcting tools for 2D digital fabrication. ACM Transactions on Graphics (TOG) 31, 4 (2012), 88.
[32]
Mark Rober. 2017. Automatic Bullseye, MOVING DARTBOARD. https://www.youtube.com/watch?v=MHTizZ_XcUM. (2017). Accessed: 2019-09--10.
[33]
MHD Saraiji, Tomoya Sasaki, Kai Kunze, Kouta Minamizawa, and Masahiko Inami. 2018a. MetaArmS: Body remapping using feet-controlled artifcial arms. In The 31st Annual ACM Symposium on User Interface Software and Technology. ACM, 65--74.
[34]
MHD Saraiji, Tomoya Sasaki, Reo Matsumura, Kouta Minamizawa, and Masahiko Inami. 2018b. Fusion: full body surrogacy for collaborative communication. In ACM SIGGRAPH 2018 Emerging Technologies. ACM, 7.
[35]
Koya Sato, Yuji Sano, Mai Otsuki, Mizuki Oka, and Kazuhiko Kato. 2019. Augmented Recreational Volleyball Court: Supporting the Beginners' Landing Position Prediction Skill by Providing Peripheral Visual Feedback. In Proceedings of the 10th Augmented Human International Conference 2019. ACM, 15.
[36]
Taku Senoo, Yuuki Horiuchi, Yoshinobu Nakanishi, Kenichi Murakami, and Masatoshi Ishikawa. 2016. Robotic pitching by rolling ball on fngers for a randomly located target. In 2016 IEEE International Conference on Robotics and Biomimetics (ROBIO). IEEE, 325--330.
[37]
Taku Senoo, Akio Namiki, and Masatoshi Ishikawa. 2004. High-speed batting using a multi-jointed manipulator. In IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA'04. 2004, Vol. 2. IEEE, 1191--1196.
[38]
Taku Senoo, Akio Namiki, and Masatoshi Ishikawa. 2006. Ball control in high-speed batting motion using hybrid trajectory generator. In Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006. IEEE, 1762--1767.
[39]
Taku Senoo, Akio Namiki, and Masatoshi Ishikawa. 2008. High-speed throwing motion based on kinetic chain approach. In 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, 3206--3211.
[40]
Tatsuya Teramae, Koji Ishihara, Jan Babi?, Jun Morimoto, and Erhan Oztop. 2018. Human-in-the-loop control and task learning for pneumatically actuated muscle based robots. Frontiers in neurorobotics 12 (2018).
[41]
Junichi Yamaoka and Yasuaki Kakehi. 2013. dePENd: augmented handwriting system using ferromagnetism of a ballpoint pen. In Proceedings of the 26th annual ACM symposium on User interface software and technology. ACM, 203--210.
[42]
Hiroaki Yano, Masayuki Yoshie, and Hiroo Iwata. 2003. Development of a non-grounded haptic interface using the gyro effect. In 11th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, 2003. HAPTICS 2003. Proceedings. IEEE, 32--39.
[43]
Hong Kai Yap, Jeong Hoon Lim, Fatima Nasrallah, James CH Goh, and Raye CH Yeow. 2015. A soft exoskeleton for hand assistive and rehabilitation application using pneumatic actuators with variable stiffness. In 2015 IEEE international conference on robotics and automation (ICRA). IEEE, 4967--4972.
[44]
Hironori Yoshida, Takeo Igarashi, Yusuke Obuchi, Yosuke Takami, Jun Sato, Mika Araki, Masaaki Miki, Kosuke Nagata, Kazuhide Sakai, and Syunsuke Igarashi. 2015. Architecture-scale human-assisted additive manufacturing. ACM Transactions on Graphics (TOG) 34, 4 (2015), 88.
[45]
Andy Zeng, Shuran Song, Kuan-Ting Yu, Elliott Donlon, Francois R Hogan, Maria Bauza, Daolin Ma, Orion Taylor, Melody Liu, Eudald Romo, and others. 2018. Robotic pick-and-place of novel objects in clutter with multi-affordance grasping and cross-domain image matching. In 2018 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 1--8.
[46]
Juanjuan Zhang, Pieter Fiers, Kirby A Witte, Rachel W Jackson, Katherine L Poggensee, Christopher G Atkeson, and Steven H Collins. 2017. Human-in-the-loop optimization of exoskeleton assistance during walking. Science 356, 6344 (2017), 1280--1284.
[47]
Amit Zoran and Joseph A Paradiso. 2013. FreeD: a freehand digital sculpting tool. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2613--2616.
[48]
Amit Zoran, Roy Shilkrot, and Joseph Paradiso. 2013. Human-computer interaction for hybrid carving. In Proceedings of the 26th annual ACM symposium on User interface software and technology. ACM, 433--440.

Cited By

View all
  • (2023)MI-PoserProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36108917:3(1-24)Online publication date: 27-Sep-2023
  • (2023)STAMPER: Human-machine Integrated DrummingExtended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544549.3585619(1-5)Online publication date: 19-Apr-2023
  • (2023)Social Digital Cyborgs: The Collaborative Design Process of JIZAI ARMSProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581169(1-19)Online publication date: 19-Apr-2023
  • Show More Cited By

Index Terms

  1. Dynamic Motor Skill Synthesis with Human-Machine Mutual Actuation

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    CHI '20: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems
    April 2020
    10688 pages
    ISBN:9781450367080
    DOI:10.1145/3313831
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 23 April 2020

    Permissions

    Request permissions for this article.

    Check for updates

    Badges

    • Honorable Mention

    Author Tags

    1. human augmentation
    2. human-machine mutual actuation
    3. motion sensing
    4. motor skill
    5. robotic device

    Qualifiers

    • Research-article

    Funding Sources

    Conference

    CHI '20
    Sponsor:

    Acceptance Rates

    Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

    Upcoming Conference

    CHI 2025
    ACM CHI Conference on Human Factors in Computing Systems
    April 26 - May 1, 2025
    Yokohama , Japan

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)360
    • Downloads (Last 6 weeks)59
    Reflects downloads up to 23 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2023)MI-PoserProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36108917:3(1-24)Online publication date: 27-Sep-2023
    • (2023)STAMPER: Human-machine Integrated DrummingExtended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544549.3585619(1-5)Online publication date: 19-Apr-2023
    • (2023)Social Digital Cyborgs: The Collaborative Design Process of JIZAI ARMSProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581169(1-19)Online publication date: 19-Apr-2023
    • (2023)Optimal Motor Point Search Using Mm-Order Electrode ArraysIEEE Access10.1109/ACCESS.2023.328542211(58970-58981)Online publication date: 2023
    • (2022)Augmented Humanity: A Systematic Mapping ReviewSensors10.3390/s2202051422:2(514)Online publication date: 10-Jan-2022
    • (2022)Cyborgs, Human Augmentation, Cybernetics, and JIZAI BodyProceedings of the Augmented Humans International Conference 202210.1145/3519391.3519401(230-242)Online publication date: 13-Mar-2022
    • (2022)Machine-Mediated Teaming: Mixture of Human and Machine in Physical Gaming ExperienceProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3517555(1-11)Online publication date: 29-Apr-2022
    • (2022)Electrical Head Actuation: Enabling Interactive Systems to Directly Manipulate Head OrientationProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3501910(1-15)Online publication date: 29-Apr-2022
    • (2021)Introduction to JIZAI BODY and Our Future Visions自在化身体とその展望Journal of the Robotics Society of Japan10.7210/jrsj.39.68539:8(685-692)Online publication date: 2021
    • (2021)A COMPUTATIONAL FRAMEWORK OF GOAL DIRECTED VOLUNTARY MOTION GENERATION AND CONTROL LOOP IN HUMANOID ROBOTSThe Journal of Cognitive Systems10.52876/jcs.9357736:1(13-17)Online publication date: 29-Jun-2021
    • Show More Cited By

    View Options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    HTML Format

    View this article in HTML Format.

    HTML Format

    Login options

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media