Hostname: page-component-76fb5796d-dfsvx Total loading time: 0 Render date: 2024-04-29T03:53:34.750Z Has data issue: false hasContentIssue false
  • English
  • Français

Face-tracking algorithm for large-amplitude head motions with a 7-DOF manipulator

Published online by Cambridge University Press:  20 July 2023

Shuai Zhang Open the ORCID record for Shuai Zhang [Opens in a new window] ,
Cancan Zhao ,
Xin Yuan ,
Bo Ouyang  and
Shanlin Yang
Shuai Zhang
Affiliation:
The School of Management, Hefei University of Technology, Hefei, China The Key Laboratory of Process Optimization and Intelligent Decision-Making (Ministry of Education), Hefei University of Technology, Hefei, China
Cancan Zhao
Affiliation:
The School of Management, Hefei University of Technology, Hefei, China The Key Laboratory of Process Optimization and Intelligent Decision-Making (Ministry of Education), Hefei University of Technology, Hefei, China
Xin Yuan
Affiliation:
The School of Management, Hefei University of Technology, Hefei, China The Key Laboratory of Process Optimization and Intelligent Decision-Making (Ministry of Education), Hefei University of Technology, Hefei, China
Bo Ouyang*
Affiliation:
The School of Management, Hefei University of Technology, Hefei, China The Key Laboratory of Process Optimization and Intelligent Decision-Making (Ministry of Education), Hefei University of Technology, Hefei, China
Shanlin Yang
Affiliation:
The School of Management, Hefei University of Technology, Hefei, China The Key Laboratory of Process Optimization and Intelligent Decision-Making (Ministry of Education), Hefei University of Technology, Hefei, China
*
Corresponding author: Bo Ouyang; Email: boouyang@hfut.edu.cn
Get access Rights & Permissions [Opens in a new window]

Abstract

The collection of facial action data is essential for the accurate evaluation of a patient’s condition in the intensive care unit, such as pain evaluation. An automatic face-tracking system is demanded to reduce the burden of data collection on the medical staff. However, many previous studies assume that the optimal trajectory of a robotic tracking system is reachable which is inapplicable for large-amplitude head motions. To tackle this problem, we propose a region-based face-tracking algorithm for large-amplitude head motion with a 7-DOF manipulator. A configuration-based optimization algorithm is proposed to trade-off between theoretical optimal pose and workspace constraints through the assignment of importance weights. To increase the probability of recapturing the face exceeding the reachable workspace of the manipulator, the camera is directed toward the center of the head, named the facial orientation center (FOC) constraint. Furthermore, a region-based tracking approach is designed to stabilize the manipulator for small amplitude head motions and smooth the tracking trajectory by adjusting the joint angle in the null space of the 7-DOF manipulator. Experimental results demonstrate the effectiveness of the proposed algorithm in tracking performance and finding an appropriate configuration for the unreachable theoretical optimal configuration. Moreover, the proposed algorithm with FOC constraint can successfully follow the head motion as losing 33.2% of the face during the tracking.

Keywords

face trackingintensive care unitvisual servohead motionlarge-amplitude
Type
Research Article
Information
Robotica , Volume 41 , Issue 11 , November 2023 , pp. 3314 - 3332
Copyright
© The Author(s), 2023. Published by Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Rodriguez, P., Cucurull, G., Gonzlez, J. and Gonfaus, J. M., “Deep pain: Exploiting long short-term memory networks for facial expression classification,” IEEE Trans. Cybern. 52(5), 3314–3324 (2022).CrossRefGoogle ScholarPubMed
Miriam, K., Meixner, D. and Lautenbacher, S., “Facial muscle movements encoding pain a systematic review,” Pain 160(3), 535–549 (2019).Google Scholar
Drennan, M. V. and Fiona, R., “Global nurse shortages: The facts, the impact and action for change,” Br. Med. Bull. 130(1), 25–37 (2019).CrossRefGoogle ScholarPubMed
Stretch, B. and Shepherd, S. J., “Criteria for intensive care unit admission and severity of illness,” Surgery 39(1), 22–28 (2021).Google Scholar
Tian, S., Hu, N. and Lou, J., “Characteristics of COVID-19 infection in Beijing,” J. Infect. 80(4), 401–406 (2020).CrossRefGoogle ScholarPubMed
Roy, S. D., Bhowmik, M. K. and Saha, P., “An appr oach for automatic pain detection through facial expression,” Proc. Comput. Sci. 84(2016), 99–106 (2016).CrossRefGoogle Scholar
Yuan, X., Zhang, S., Zhao, C. C., He, X., Ouyang, B. and Yang, S., “Pain Intensity Recognition from Masked Facial Expressions Using Swin-Transformer,” In: Proceedings of the 2022 IEEE International Conference on Robotics and Biomimetics (ROBIO), Xishuangbanna, China (2022) pp. 723–728.CrossRefGoogle Scholar
Mohammad, T. and Hadid, A., “A spatiotemporal convolutional neural network for automatic pain intensity estimation from facial dynamics,” Int. J. Comput. Vis. 127(10), 1413–1425 (2019).Google Scholar
Qi, R., Tang, Y. and Zhang, K., “An optimal visual servo trajectory planning method for manipulators based on system nondeterministic model,” Robotica 40(6), 1665–1681 (2022).CrossRefGoogle Scholar
Chaumette, F. and Hutchinson, S., “Visual servo control. I. Basic approaches,” IEEE Robot. Autom. Mag. 13(4), 82–90 (2006).CrossRefGoogle Scholar
Gans, N. R. and Hutchinson, S. A., “Stable visual servoing through hybrid switched-system control,” IEEE Trans. Robot. 23(3), 530–540 (2007).CrossRefGoogle Scholar
Dong, G. Q. and Zhu, Z. H., “Position-based visual servo control of autonomous robotic manipulators,” Acta Astronaut. 115(2015), 291–302 (2015).CrossRefGoogle Scholar
Zhang, H., Fan, B., Zhang, X. and Zhan, H., “A Six-degree-of-freedom Face Tracking Method for Non-contact Physiological Detection Robot,” In: Proceedings of the 2021 IEEE International Conference on Mechatronics and Automation (ICMA), Takamatsu, Japan (2021) pp. 950–955.CrossRefGoogle Scholar
Zhang, S., Ouyang, B., He, X., Yuan, X. and Yang, S., “Face Tracking Strategy Based on Manipulability of a 7-DOF Robot Arm and Head Motion Intention Ellipsoids,” In: Proceedings of the 2022 IEEE International Conference on Real-time Computing and Robotics (RCAR), Guiyang, China (2022) pp. 290–295.CrossRefGoogle Scholar
Rubio, J. D. J., “Bat algorithm based control to decrease the control energy consumption and modified bat algorithm based control to increase the trajectory tracking accuracy in robots,” Neural Netw. 161(2023), 437–448 (2023).CrossRefGoogle ScholarPubMed
Rubio, J. D. J., Orozco, E., Cordova, D. A. and Islas, M. A., “Modified linear technique for the controllability and observability of robotic arms,” IEEE Access 10(2022), 3366–3377 (2022).CrossRefGoogle Scholar
Lughofer, E. and Skrjanc, I., “Evolving error feedback fuzzy model for improved robustness under measurement noise,” IEEE Trans. Fuzzy Syst. 31(3), 997–1008 (2023).CrossRefGoogle Scholar
Balcazar, R., Rubio, J. D. J., Orozco, E., Andres Cordova, D., Ochoa, G., Garcia, E., Pacheco, J., Gutierrez, G. J., Mujica-Vargas, D. and Aguilar-Ibañez, C., “The regulation of an electric oven and an inverted pendulum,” Symmetry 14(4), 759 (2022).CrossRefGoogle Scholar
Soriano, L. A., Zamora, E., Vazquez-Nicolas, J. M., Hernandez, G., Madrigal, J. A. B. and Balderas, D., “PD control compensation based on a cascade neural network applied to a robot manipulator,” Front. Neurorobot. 14(2020), 577749 (2020).CrossRefGoogle ScholarPubMed
Silva-Ortigoza, R., Hernandez-Marquez, E., Roldan-Caballero, A., Tavera-Mosqueda, S., Marciano-Melchor, M., Garcia-Sanchez, J. R., Hernandez-Guzman, V. M. and Silva-Ortigoza, G., “Sensorless tracking control for a “full-bridge buck inverter-dc motor” system: Passivity and flatness-based design,” IEEE Access 9(2021), 132191–132204 (2021).CrossRefGoogle Scholar
Sadeghian, H., Zokaei, F. and Hadian Jazi, S., “Constrained kinematic control in minimally invasive robotic surgery subject to remote center of motion constraint,” J. Intell. Robot. Syst. 95(3), 901–913 (2019).CrossRefGoogle Scholar
Tahri, O. and Chaumette, F., “Point-based and region-based image moments for visual servoing of planar objects,” IEEE Trans. Robot. 21(6), 1116–1127 (2005).CrossRefGoogle Scholar
M., C., Tsai, K. Y., Chen, M. Y. and Cheng, M. Y., “Implementation of a real-time moving object tracking system using visual servoing,” Robotica 21(6), 615–625 (2003).Google Scholar
Chen, D., Li, S. and Wu, Q., “New disturbance rejection constraint for redundant robot manipulators: An optimization perspective,” IEEE Trans. Ind. Inform. 16(4), 2221–2232 (2019).CrossRefGoogle Scholar
Zhang, J. Z., Long, J. and Long, C., “RNN for perturbed manipulability optimization of manipulators based on a distributed scheme: A game-theoretic perspective,” IEEE Trans. Neural Netw. Learn. Syst. 31(12), 5116–5126 (2020).CrossRefGoogle ScholarPubMed
Tarokh, M. and Zhang, X., “Real-time motion tracking of robot manipulators using adaptive genetic algorithms,” J. Intell. Robot. Syst. 74(3), 697–708 (2014).CrossRefGoogle Scholar
Tsai, C. C., Hung, C. C. and Chang, C. F., “Trajectory Planning and Control of a 7-DOF Robotic Manipulator,” In: Proceedings of the 2014 International Conference on Advanced Robotics and Intelligent Systems, Taipei, Taiwan (2014) pp. 78–84.Google Scholar
Pan, L., Gao, T. and Xu, F., “Enhanced robust motion tracking control for 6 degree-of-freedom industrial assembly robot with disturbance adaption,” Int. J. Control Autom. Syst. 16(2), 921–928 (2018).CrossRefGoogle Scholar
Semwal, A. and Londhe, N. D., “Computer aided pain detection and intensity estimation using compact CNN based fusion network,” Appl. Soft. Comput. 112(2021), 107780 (2021).CrossRefGoogle Scholar
Riker, R. R., Jean, T. P. and Gilles, L. F., “Prospective evaluation of the Sedation-Agitation Scale for adult critically ill patients,” Crit. Care Med. 27(7), 1325–1329 (1999).CrossRefGoogle ScholarPubMed
Devlin, J. W., Boleski, G. and Mlynarek, M., “Motor Activity Assessment Scale: A valid and reliable sedation scale for use with mechanically ventilated patients in an adult surgical intensive care unit,” Crit. Care Med. 27(7), 1271–1275 (1999).CrossRefGoogle Scholar
Weinert, C. and McFarland, L., “The state of intubated ICU patients: Development of a two-dimensional sedation rating scale for critically ill adults,” Chest 126(6), 1883–1890 (2004).CrossRefGoogle ScholarPubMed
Chevrolet, J.-C. and Jolliet, P., “Clinical review: Agitation and delirium in the critically ill significance and management,” Crit. Care 11(3), 1–5 (2007).CrossRefGoogle ScholarPubMed
Shimizu, M., Kakuya, H. and Yoon, W. K., “Analytical inverse kinematic computation for 7-DOF redundant manipulators with joint limits and its application to redundancy resolution,” IEEE Trans. Robot. 24(5), 1131–1142 (2008).CrossRefGoogle Scholar
Fu, Z. T. and Pan, J. B., “A Lie-theory-based dynamic parameter identification methodology for serial manipulators,” IEEE/ASME Trans. Mechatron. 26(5), 2688–2699 (2021).CrossRefGoogle Scholar
Corke, P. I., “A simple and systematic approach to assigning Denavit-Hartenberg parameters,” IEEE Trans. Robot. 23(3), 590–594 (2007).CrossRefGoogle Scholar
Zhang, S., Zhao, C. C., He, X., Ouyang, B. and Yang, S., “Face Tracking Strategy of 7-DOF Manipulator Using Optimization Algorithm Based on Velocity Analysis,” In: Proceedings of the 2022 IEEE International Conference on Robotics and Biomimetics (ROBIO), Xishuangbanna, China (2022) pp. 327–332.Google Scholar
Corke, P., Robotics, Vision and Control: Fundamental Algorithms in MATLAB ® Second, Completely Revised, vol. 118 (Springer, Cham, 2017).CrossRefGoogle Scholar
Liao, B. and Liu, W., “Pseudoinverse-type bi-criteria minimization scheme for redundancy resolution of robot manipulators,” Robotica 33(10), 2100–2113 (2015).CrossRefGoogle Scholar