Skip to main content

Advertisement

SpringerLink
Log in

A CNN-LSTM model for the effects of object temperature, object hardness, and grip strength on human sensation grasped by a prosthetic hand

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

In the previous literature, the sensory feedback obtained by grasping objects with the prosthetic hand in patients with upper extremity disabilities is usually a single vibration or electrical stimulation sensation obtained from the grip strength of the prosthetic hand. However, this kind of human feeling does not consider the influence of various factors such as the hardness of the object, the grip strength of the artificial hand, and the temperature of the object when the prosthetic hand grasps the object. Therefore, this paper uses the CNN-LSTM model, sensory voting and probability mass function methods to study the effects of various grasping factors on human perception and predict them. First, laboratory tests were conducted to collect human sensory votes on object hardness, human grip strength, and object temperature, which were mapped to prosthetic hand grasping. Then, using the collected data, a CNN-LSTM model was developed and trained to predict different human sensory votes under the stimulus of object hardness, prosthetic hand grip, and object temperature when the prosthetic hand grasped an object. Finally, the model generalization is applied to predict human sensation under different conditions within the range of different factors. The results show that the probability quality function predicted by the trained neural network model is in good agreement with the collected sensory voting proportion distribution. By comparing the sensory voting expectations under different definition conditions, it is confirmed that there is a cross interaction between different object factors on human perception when grasping objects. In addition, the direction and magnitude of the influence of object hardness, object temperature, and prosthetic hand grip force on human perception through variable control are discussed. The CNN-LSTM model proposed in this paper can be applied to the sensory feedback of prosthetic hands, and a reasonable and accurate human sensory feedback effect can be formulated by predicting different human sensory votes.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Code or Data Availability

The raw/processed data required to reproduce these findings cannot be shared at this time as the data also forms part of an ongoing study.

References

  1. Ayad HG, Kamel MS (2007) Cumulative voting consensus method for partitions with variable number of clusters. IEEE Trans Pattern Anal Mach Intell 30(1):160–173

    Article  Google Scholar 

  2. Bai J, Li B, Wang H, Guo Y (2022) Tactile perception information recognition of prosthetic hand based on dnn-lstm. IEEE Trans Instrum Meas 71:1–10. https://doi.org/10.1109/TIM.2022.3189644

    Article  Google Scholar 

  3. Baishya SS, Bäuml B (2016) Robust material classification with a tactile skin using deep learning. In: 2016 IEEE/RSJ international conference on intelligent robots and systems (IROS). IEEE, pp 8–15

  4. Buescher G, Meier M, Walck G, Haschke R, Ritter HJ (2015) Augmenting curved robot surfaces with soft tactile skin. In: 2015 IEEE/RSJ international conference on intelligent robots and systems (IROS). IEEE, pp 1514–1519

  5. Cao L, Kotagiri R, Sun F, Li H, Huang W, Aye ZMM (2016) Efficient spatio-temporal tactile object recognition with randomized tiling convolutional networks in a hierarchical fusion strategy. In: Proceedings of the AAAI conference on artificial intelligence, vol 30

  6. Costantini G, Casali D, Perfetti R (2006) Associative memory design for 256 gray-level images using a multilayer neural network. IEEE Trans Neural Networks 17(2):519–522

    Article  Google Scholar 

  7. Dua N, Singh SN, Semwal VB, Challa SK (2022) Inception inspired cnn-gru hybrid network for human activity recognition. Multimed Tools Appl:1–35

  8. Fradi M, Khriji L, Machhout M (2021) Real-time arrhythmia heart disease detection system using cnn architecture based various optimizers-networks. Multimed Tools Appl:1–22

  9. Fujimoto T, Fujii N, Dobashi K, Cao Y, Matsutake R, Takayanagi M, Kondo N, Nishiyasu T (2021) Effects of low-intensity exercise on local skin and whole-body thermal sensation in hypothermic young males. Physiol Behav 240:113531. https://doi.org/10.1016/j.physbeh.2021.113531

    Article  Google Scholar 

  10. Higashi K, Okamoto S, Yamada Y, Nagano H, Konyo M (2017) Hardness perception by tapping: effect of dynamic stiffness of objects. In: 2017 IEEE world haptics conference (WHC). IEEE, pp 37–41

  11. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780

    Article  Google Scholar 

  12. Hu W-S, Li H-C, Pan L, Li W, Tao R, Du Q (2020) Spatial–spectral feature extraction via deep convlstm neural networks for hyperspectral image classification. IEEE Trans Geosci Remote Sens 58(6):4237–4250

    Article  Google Scholar 

  13. Kao C (2010) Fuzzy data standardization. IEEE Trans Fuzzy Syst 18(4):745–754

    Article  Google Scholar 

  14. Kasuya M, Seki M, Kawamura K, Kobayashi Y, Fujie MG, Yokoi H (2013) Robust grip force estimation under electric feedback using muscle stiffness and electromyography for powered prosthetic hand. In: 2013 IEEE international conference on robotics and automation. IEEE, pp 93–98

  15. Kataoka K, Yamamoto T, Otsuki M, Shibata F, Kimura A (2019) A new interactive haptic device for getting physical contact feeling of virtual objects. In: 2019 IEEE conference on virtual reality and 3D user interfaces (VR). IEEE, pp 1323–1324

  16. Kim J, Park H, Park JI (2020) Cnn-based image steganalysis using additional data embedding. Multimed Tools Appl 79(3):1–18

    Google Scholar 

  17. Kojima K (2008) Prediction of individual thermal sensation using unspecified sensors in sensor networks. In: 2008 international conference on control, automation and systems. IEEE, pp 123–126

  18. Lecun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436

    Article  Google Scholar 

  19. Lepora NF, Lloyd J (2020) Optimal deep learning for robot touch: training accurate pose models of 3d surfaces and edges. IEEE Robot Autom Mag 27(2):66–77

    Article  Google Scholar 

  20. Lu X, Wang W, Shen J, Crandall D, Van Gool L (2021) Segmenting objects from relational visual data. IEEE Trans Pattern Anal Mach Intell:1–1. https://doi.org/10.1109/TPAMI.2021.3115815

  21. Mastinu E, Engels LF, Clemente F, Dione M, Sassu P, Aszmann O, Brånemark R, Håkansson B, Controzzi M, Wessberg J et al (2020) Neural feedback strategies to improve grasping coordination in neuromusculoskeletal prostheses. Sci Rep 10(1):1–14

    Article  Google Scholar 

  22. Meier M, Patzelt F, Haschke R, Ritter HJ (2016) Tactile convolutional networks for online slip and rotation detection. In: International conference on artificial neural networks. Springer, pp 12–19

  23. Miyahara Y, Kato R (2021) Development of thin vibration sheets using a shape memory alloy actuator for the tactile feedback of myoelectric prosthetic hands. In: 2021 43rd annual international conference of the IEEE engineering in medicine & biology society (EMBC). IEEE, pp 6255–6258

  24. Moradzadeh A, Zakeri S, Shoaran M, Mohammadi-Ivatloo B, Mohammadi F (2020) Short-term load forecasting of microgrid via hybrid support vector regression and long short-term memory algorithms. Sustainability 12(17):7076

    Article  Google Scholar 

  25. Moradzadeh A, Mohammadi-Ivatloo B, Abapour M, Anvari-Moghaddam A, Gholami Farkoush S, Rhee S-B (2021) A practical solution based on convolutional neural network for non-intrusive load monitoring. J Ambient Intell Humaniz Comput 12(10):9775–9789

    Article  Google Scholar 

  26. Porquis LB, Konyo M, Tadokoro S (2010) Can multiple tactile pressure stimulation in gripping position induce virtual force directions?. In: 2010 IEEE/SICE international symposium on system integration. IEEE, pp 402–407

  27. Schmitz A, Bansho Y, Noda K, Iwata H, Ogata T, Sugano S (2014) Tactile object recognition using deep learning and dropout. In: 2014 IEEE-RAS international conference on humanoid robots. IEEE, pp 1044–1050

  28. Singh P, Sehgal P (2021) G.v black dental caries classification and preparation technique using optimal cnn-lstm classifier. Multimed Tools Appl 80(4):1–18

    Article  Google Scholar 

  29. Sugimoto Y, Hoshino T, Watanebe K, Matsumoto K (2019) Research on display that reproduces the thermal sensation perceived by human. In: IECON 2019-45th annual conference of the IEEE industrial electronics society, vol 1. IEEE, pp 6902–6907

  30. Tan HZ, Durlach NI, Beauregard GL, Srinivasan MA (1995) Manual discrimination of compliance using active pinch grasp: the roles of force and work cues. Percept Psychophys 57(4):495–510

    Article  Google Scholar 

  31. Wang W, Lu X, Shen J, Crandall D, Shao L (2019) Zero-shot video object segmentation via attentive graph neural networks. In: 2019 IEEE/CVF international conference on computer vision (ICCV), pp 9235–9244. https://doi.org/10.1109/ICCV.2019.00933

  32. Yeredor A, Haardt M (2019) Maximum likelihood estimation of a low-rank probability mass tensor from partial observations. IEEE Signal Process Lett 26(10):1551–1555

    Article  Google Scholar 

  33. Zhong J, Liu Z, Han Z, Han Y, Zhang W (2018) A cnn-based defect inspection method for catenary split pins in high-speed railway. IEEE Trans Instrum Meas 68(8):2849–2860

    Article  Google Scholar 

  34. Zolfaghari A, Maerefat M (2010) Thermal response of cutaneous thermoreceptors: a new criterion for the human body thermal sensation. In: 2010 17th Iranian conference of biomedical engineering (ICBME). IEEE, pp 1–4

Download references

Funding

This work was supported by Su Yan Yuan(“Development and industrialization of intelligent multi degree of freedom arm based on perceptual fusion and collaborative control” (Su Yan Yuan [2019] No. 107)) and Shanghai DianJi University(“Research on flexible joint and adaptive control technology for new upper limb prosthesis” (scientific research start-up fund project of Shanghai DianJi University) and “Research on robot intelligent grasping technology based on visual touch fusion in unstructured environment” (Science and technology [2020] No. 79 of Shanghai DianJi University))

Author information

Authors and Affiliations

  1. The School of Electrical Engineering, Shanghai Dianji University, Shanghai, 201306, China

    Jibo Bai, Baojiang Li, Haiyan Wang & Yutin Guo

Authors
  1. Jibo Bai
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Baojiang Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Haiyan Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yutin Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors contributed to the study conception and design. Material preparation, data collection and analysis were performed by Jibo Bai, Yutin Guo, Haiyan Wang and Baojiang Li. The first draft of the manuscript was written by Jibo Bai and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Baojiang Li.

Ethics declarations

Ethics approval

The study did not address ethical issues.

Consent to participate

Informed consent was obtained from all individual participants included in the study.

Consent for Publication

The authors affirm that human research participants provided informed consent for publication of the images in this paper.

Competing interests

Financial interests: Author Jibo Bai, Haiyan Wang and Yutin Guo declare they have no financial interests. Author Baojiang Li received research funding from Su Yan Yuan and Shanghai DianJi University.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Baojiang Li, Haiyan Wang and Yutin Guo contributed equally to this work.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bai, J., Li, B., Wang, H. et al. A CNN-LSTM model for the effects of object temperature, object hardness, and grip strength on human sensation grasped by a prosthetic hand. Multimed Tools Appl 82, 17009–17031 (2023). https://doi.org/10.1007/s11042-022-14086-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-14086-2

Keywords

Search

Navigation