Abstract
Video-based grasp classification can enhance robotics and prosthetics. However, its accuracy is low when compared to e-skin based systems. This paper improves video-based grasp classification systems by including an automatic annotation of the frames that highlights the joints of the hand. Experiments on real-world data prove that the proposed system obtains higher accuracy with respect to the previous solutions. In addition, the framework is implemented on a NVIDIA Jetson TX2, achieving real-time performances.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Anwar, S.M., Majid, M., Qayyum, A., Awais, M., Alnowami, M., Khan, M.K.: Medical image analysis using convolutional neural networks: a review. J. Med. Syst. 42(11), 226 (2018)
Bambach, S., Lee, S., Crandall, D.J., Yu, C.: Lending a hand: detecting hands and recognizing activities in complex egocentric interactions. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1949–1957 (2015)
Bullock, I.M., Feix, T., Dollar, A.M.: The Yale human grasping dataset: grasp, object, and task data in household and machine shop environments. Int. J. Robot. Res. 34(3), 251–255 (2015)
Cai, M., Kitani, K.M., Sato, Y.: An ego-vision system for hand grasp analysis. IEEE Trans. Hum.-Mach. Syst. 47(4), 524–535 (2017)
Chortos, A., Liu, J., Bao, Z.: Pursuing prosthetic electronic skin. Nat. Mater. 15(9), 937 (2016)
Fan, Q., Shen, X., Hu, Y., Yu, C.: Simple very deep convolutional network for robust hand pose regression from a single depth image. Pattern Recogn. Lett. 119, 205–213 (2017)
Feix, T., Romero, J., Schmiedmayer, H.B., Dollar, A.M., Kragic, D.: The grasp taxonomy of human grasp types. IEEE Trans. Hum.-Mach. Syst. 46(1), 66–77 (2015)
Gao, Q., Liu, J., Ju, Z., Zhang, X.: Dual-hand detection for human-robot interaction by a parallel network based on hand detection and body pose estimation. IEEE Trans. Ind. Electron. 66, 9663–9672 (2019)
Ghazaei, G., Alameer, A., Degenaar, P., Morgan, G., Nazarpour, K.: Deep learning-based artificial vision for grasp classification in myoelectric hands. J. Neural Eng. 14(3), 036025 (2017)
Huang, Y.C., Liao, I.N., Chen, C.H., İk, T.U., Peng, W.C.: TrackNet: a deep learning network for tracking high-speed and tiny objects in sports applications. In: 2019 16th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), pp. 1–8. IEEE (2019)
Ibrahim, A., Valle, M.: Real-time embedded machine learning for tensorial tactile data processing. IEEE Trans. Circuits Syst. I Regul. Pap. 99, 1–10 (2018)
Li, Y., Wang, Y., Yue, Y., Xu, D., Case, M., Chang, S.F., Grinspun, E., Allen, P.K.: Model-driven feedforward prediction for manipulation of deformable objects. IEEE Trans. Autom. Sci. Eng. 99, 1–18 (2018)
Markovic, M., Dosen, S., Popovic, D., Graimann, B., Farina, D.: Sensor fusion and computer vision for context-aware control of a multi degree-of-freedom prosthesis. J. Neural Eng. 12(6), 066022 (2015)
Mittal, A., Zisserman, A., Torr, P.H.: Hand detection using multiple proposals. In: BMVC, pp. 1–11. Citeseer (2011)
Pham, T.H., Kyriazis, N., Argyros, A.A., Kheddar, A.: Hand-object contact force estimation from markerless visual tracking. IEEE Trans. Pattern Anal. Mach. Intell. 40(12), 2883–2896 (2017)
Ragusa, E., Cambria, E., Zunino, R., Gastaldo, P.: A survey on deep learning in image polarity detection: balancing generalization performances and computational costs. Electronics 8(7), 783 (2019)
Ragusa, E., Gianoglio, C., Zunino, R., Gastaldo, P.: Data-driven video grasping classification for low-power embedded system. In: 2019 26th IEEE International Conference on Electronics, Circuits and Systems (ICECS), pp. 871–874. IEEE (2019)
Ragusa, E., Gianoglio, C., Zunino, R., Gastaldo, P.: Image polarity detection on resource-constrained devices. IEEE Intell. Syst. 35, 50–57 (2020)
Saudabayev, A., Rysbek, Z., Khassenova, R., Varol, H.A.: Human grasping database for activities of daily living with depth, color and kinematic data streams. Sci. Data 5, 180101 (2018)
Simon, T., Joo, H., Matthews, I., Sheikh, Y.: Hand keypoint detection in single images using multiview bootstrapping. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1145–1153 (2017)
Sundaram, S., Kellnhofer, P., Li, Y., Zhu, J.Y., Torralba, A., Matusik, W.: Learning the signatures of the human grasp using a scalable tactile glove. Nature 569(7758), 698 (2019)
Wang, T., Li, Y., Hu, J., Khan, A., Liu, L., Li, C., Hashmi, A., Ran, M.: A survey on vision-based hand gesture recognition. In: International Conference on Smart Multimedia, pp. 219–231. Springer (2018)
Wei, S.E., Ramakrishna, V., Kanade, T., Sheikh, Y.: Convolutional pose machines. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4724–4732 (2016)
Yang, Y., Fermuller, C., Li, Y., Aloimonos, Y.: Grasp type revisited: a modern perspective on a classical feature for vision. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 400–408 (2015)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Ragusa, E., Gianoglio, C., Dalmonte, F., Gastaldo, P. (2021). Video Grasping Classification Enhanced with Automatic Annotations. In: Saponara, S., De Gloria, A. (eds) Applications in Electronics Pervading Industry, Environment and Society. ApplePies 2020. Lecture Notes in Electrical Engineering, vol 738. Springer, Cham. https://doi.org/10.1007/978-3-030-66729-0_3
Download citation
DOI: https://doi.org/10.1007/978-3-030-66729-0_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-66728-3
Online ISBN: 978-3-030-66729-0
eBook Packages: EngineeringEngineering (R0)