Skip to main content
SpringerLink
Log in

On the Design and Development of Vision-based Tactile Sensors

  • Short Paper
  • Published:
Journal of Intelligent & Robotic Systems Aims and scope Submit manuscript

Abstract

This paper reviews the existing vision-based tactile sensor (VBTS) designs reported in the literature. Although some reviews on VBTSs already exist in the literature. We believe it is necessary to review existing VBTS designs to formulate a guideline for developing such systems considering recent developments in the manufacturing and imaging technologies. Therefore, the main emphasis of this paper is to investigate current manufacturing trends and component selection criteria for developing a complete VBTS system. Further, the motivation behind this review is to identify the shortcomings in the current VBTS development technology and to furnish viable solutions to overcome such challenges. First, three different modalities of VBTSs are discussed: i) Waveguide-type designs, ii) marker-tracking based designs, and ii) reflective membrane designs. Next, a detailed discussion on various design aspects, like manufacturing, selection, and arrangements of the various sensor components, of the VBTSs is included. Then, a discussion on the validation/testing of various VBTSs is presented. Finally, based on the review, several challenges related to the development of VBTS are presented and the future research directions to overcome such challenges are recommended. This will serve the research community in determining the future research directions in the area of VBTS development.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Abbreviations

CCD:

Charge-coupled device

CNN:

Convolutional neural network

DOF:

Degrees of freedom

FEM:

Finite element methods

HD:

High definition

KNN:

K-nearest neighbor

LED:

Light emitting diode

LSTM:

Long short-term memory

RGB:

Red, green, and blue

VBTS:

Vision-based tactile sensor

2D:

Two dimensional

3D:

Three dimensional

References

  1. Lee, M, Nicholls, H: Tactile sensing for mechatronics-a state of the art survey, Mechatronics, vol. 9 (1999)

  2. Dahiya, R.S., Metta, G., Valle, M., Sandini, G.: Tactile sensing—from humans to humanoids. IEEE Trans. Robot 26(1), 1–20 (2009)

    Article  Google Scholar 

  3. Chi, C., Sun, X., Xue, N., Li, T., Liu, C.: Recent progress in technologies for tactile sensors. Sensors 18(4), 948 (2018)

    Article  Google Scholar 

  4. Howe, R.D.: Tactile sensing and control of robotic manipulation. Adv. Robot. 8(3), 245–261 (1993)

    Article  Google Scholar 

  5. Tegin, J., Wikander, J.: Tactile sensing in intelligent robotic manipulation–a review. Industrial Robot: An International Journal (2005)

  6. Wan, Y., Wang, Y., Guo, C.F.: Recent progresses on flexible tactile sensors. Mater. Today Phys. 1, 61–73 (2017)

    Article  Google Scholar 

  7. Yamaguchi, A., Atkeson, C.G.: Recent progress in tactile sensing and sensors for robotic manipulation: can we turn tactile sensing into vision? Adv. Robot. 33(14), 661–673 (2019)

    Article  Google Scholar 

  8. Naeini, F.B., AlAli, A.M., Al-Husari, R., Rigi, A., Al-Sharman, M.K., Makris, D., Zweiri, Y.: A novel dynamic-vision-based approach for tactile sensing applications. IEEE Trans. Instrum. Meas. 69 (5), 1881–1893 (2019)

    Article  Google Scholar 

  9. Sferrazza, C, D’Andrea, R: Transfer learning for vision-based tactile sensing. arXiv:181203163 (2018)

  10. Sferrazza, C., D’Andrea, R.: Design, motivation and evaluation of a full-resolution optical tactile sensor. Sensors 19(4), 928 (2019a)

    Article  Google Scholar 

  11. Begej, S.: Planar and finger-shaped optical tactile sensors for robotic applications. IEEE J. Robot. Autom. 4(5), 472–484 (1988)

    Article  Google Scholar 

  12. Hristu, D., Ferrier, N., Brockett, R.W.: The performance of a deformable-membrane tactile sensor: basic results on geometrically-defined tasks. In: Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No. 00CH37065), vol. 1, pp 508–513. IEEE (2000)

  13. Nagata, K., Ooki, M., Kakikur, M.: Feature detection with an image based compliant tactile sensor. In: Proceedings 1999 IEEE/RSJ International Conference on Intelligent Robots and Systems. Human and Environment Friendly Robots with High Intelligence and Emotional Quotients (Cat. No. 99CH36289), vol. 2, pp 838–843. IEEE (1999)

  14. Sferrazza, C., Wahlsten, A., Trueeb, C., D’Andrea, R.: Ground truth force distribution for learning-based tactile sensing: a finite element approach. IEEE Access 7, 173,438–173,449 (2019b)

    Article  Google Scholar 

  15. Lang, P.: Optical tactile sensors for medical palpation. Canada-Wide Science Fair (2004)

  16. Dong, S., Yuan, W., Adelson, E.H.: Improved gelsight tactile sensor for measuring geometry and slip. In: 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp 137–144. IEEE (2017)

  17. Donlon, E., Dong, S., Liu, M., Li, J., Adelson, E., Rodriguez, A.: Gelslim: A high-resolution, compact, robust, and calibrated tactile-sensing finger. In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp 1927–1934. IEEE (2018)

  18. Nozu, K., Shimonomura, K.: Robotic bolt insertion and tightening based on in-hand object localization and force sensing. In: 2018 IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM), pp 310–315. IEEE (2018)

  19. Yuan, W., Li, R., Srinivasan, M.A., Adelson, E.H.: Measurement of shear and slip with a gelsight tactile sensor. In: 2015 IEEE International Conference on Robotics and Automation (ICRA), pp 304–311. IEEE (2015)

  20. Yuan, W., Dong, S., Adelson, E.H.: Gelsight: High-resolution robot tactile sensors for estimating geometry and force. Sensors 17(12), 2762 (2017a)

    Article  Google Scholar 

  21. Abad, A.C., Ranasinghe A: Visuotactile sensors with emphasis on gelsight sensor: A review. IEEE Sensors J. (2020)

  22. Shimonomura, K.: Tactile image sensors employing camera: A review. Sensors 19(18), 3933 (2019)

    Article  Google Scholar 

  23. Liu, P., Yu, H., Cang, S.: Adaptive neural network tracking control for underactuated systems with matched and mismatched disturbances. Nonlinear Dyn. 98(2), 1447–1464 (2019)

    Article  Google Scholar 

  24. Elgeneidy, K., Liu, P., Pearson, S., Lohse, N., Neumann, G., et al: Printable soft grippers with integrated bend sensing for handling of crops. In: 19th Annual Conference Towards Autonomous Robotic Systems, vol. 10965, pp 479–480 (2018)

  25. James, JW, Church, A, Cramphorn, L, Lepora, NF: Tactile model o: Fabrication and testing of a 3d-printed, three-fingered tactile robot hand. Soft Robotics (2020)

  26. Tang, Z., Yu, H., Lu, C., Liu, P., Jin, X.: Single-trial classification of different movements on one arm based on erd/ers and corticomuscular coherence. IEEE Access 7, 128,185–128,197 (2019)

    Article  Google Scholar 

  27. Maekawa, H., Tanie, K., Komoriya, K., Kaneko, M., Horiguchi, C., Sugawara, T: Development of a finger-shaped tactile sensor and its evaluation by active touch. In: Proceedings 1992 IEEE International Conference on Robotics and Automation, pp 1327–1328. IEEE Computer Society (1992)

  28. Ohka, M., Mitsuya, Y., Matsunaga, Y., Takeuchi, S.: Sensing characteristics of an optical three-axis tactile sensor under combined loading. Robotica 22(2), 213 (2004)

    Article  Google Scholar 

  29. Ikai, T., Kamiya, S., Ohka, M.: Robot control using natural instructions via visual and tactile sensations. J. Comput. Sci. 12(5), 246–254 (2016)

    Article  Google Scholar 

  30. Ohka, M., Kobayashi, H., Mitsuya, Y.: Sensing characteristics of an optical three-axis tactile sensor mounted on a multi-fingered robotic hand. In: 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp 493–498. IEEE (2005)

  31. Yussof, H., Wada, J.: Sensorization of robotic hand using optical three-axis tactile sensor: Evaluation with grasping and twisting motions (2010)

  32. Shimonomura, K, Nakashima, H: A combined tactile and proximity sensing employing a compound-eye camera. In: Sensors, 2013 IEEE, pp 1–2. IEEE (2013)

  33. Shimonomura, K., Nakashima, H., Nozu, K.: Robotic grasp control with high-resolution combined tactile and proximity sensing. In: 2016 IEEE International Conference on Robotics and Automation (ICRA), pp 138–143. IEEE (2016)

  34. Xie, H., Jiang, A., Wurdemann, H.A., Liu, H., Seneviratne, L.D., Althoefer, K.: Magnetic resonance-compatible tactile force sensor using fiber optics and vision sensor. IEEE Sensors J. 14(3), 829–838 (2013)

    Article  Google Scholar 

  35. Kamiyama, K, Kajimoto, H, Kawakami, N, Tachi, S: Evaluation of a vision-based tactile sensor. In: IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA’04. 2004, vol. 2, pp 1542–1547. IEEE (2004)

  36. Sato, K., Shinoda, H., Tachi, S.: Finger-shaped thermal sensor using thermo-sensitive paint and camera for telexistence. In: 2011 IEEE International Conference on Robotics and Automation, pp 1120–1125. IEEE (2011)

  37. Obinata, G, Dutta, A, Watanabe, N, Moriyama, N: Vision Based Tactile Sensor using Transparent Elastic Fingertip for Dexterous Handling. In Mobile Robots: Perception & Navigation. IntechOpen (2007)

  38. Ito, Y., Kim, Y., Nagai, C., Obinata, G.: Shape sensing by vision-based tactile sensor for dexterous handling of robot hands. In: 2010 IEEE International Conference on Automation Science and Engineering, pp 574–579. IEEE (2010)

  39. Ito, Y., Kim, Y., Nagai, C., Obinata, G.: Vision-based tactile sensing and shape estimation using a fluid-type touchpad. IEEE Trans. Autom. Sci. Eng. 9(4), 734–744 (2012)

    Article  Google Scholar 

  40. Ito, Y., Kim, Y., Obinata, G.: Robust slippage degree estimation based on reference update of vision-based tactile sensor. IEEE Sensors J. 11(9), 2037–2047 (2011)

    Article  Google Scholar 

  41. Ward-Cherrier, B., Pestell, N., Cramphorn, L., Winstone, B., Giannaccini, M.E., Rossiter, J., Lepora, N.F.: The tactip family: Soft optical tactile sensors with 3d-printed biomimetic morphologies. Soft Robot. 5(2), 216–227 (2018)

    Article  Google Scholar 

  42. Chorley, C., Melhuish, C., Pipe, T., Rossiter, J: Development of a tactile sensor based on biologically inspired edge encoding. In: 2009 International Conference on Advanced Robotics, pp 1–6. IEEE (2009)

  43. Winstone, B., Griffiths, G., Melhuish, C., Pipe, T., Rossiter, J.: Tactip—tactile fingertip device, challenges in reduction of size to ready for robot hand integration. In: 2012 IEEE International Conference on Robotics and Biomimetics (ROBIO), pp 160–166. IEEE (2012)

  44. Winstone, B., Griffiths, G., Pipe, T., Melhuish, C., Rossiter, J.: Tactip-tactile fingertip device, texture analysis through optical tracking of skin features. In: Conference on Biomimetic and Biohybrid Systems, pp 323–334 (2013)

  45. Assaf, T., Roke, C., Rossiter, J., Pipe, T., Melhuish, C.: Seeing by touch: Evaluation of a soft biologically-inspired artificial fingertip in real-time active touch. Sensors 14(2), 2561–2577 (2014)

    Article  Google Scholar 

  46. Lepora, N.F., Ward-Cherrier, B.: Superresolution with an optical tactile sensor. In: 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp 2686–2691. IEEE (2015)

  47. Yamaguchi, A., Atkeson, C.G.: Combining finger vision and optical tactile sensing: Reducing and handling errors while cutting vegetables. In: 2016 IEEE-RAS 16th International Conference on Humanoid Robots (Humanoids), pp 1045–1051. IEEE (2016)

  48. Trueeb, C., Sferrazza, C., D’Andrea, R.: Towards vision-based robotic skins: a data-driven, multi-camera tactile sensor. In: 2020 3rd IEEE International Conference on Soft Robotics (RoboSoft), pp 333–338. IEEE (2020)

  49. Ward-Cherrier, B., Cramphorn, L., Lepora, N.F.: Exploiting sensor symmetry for generalized tactile perception in biomimetic touch. IEEE Robot. Autom. Lett. 2(2), 1218–1225 (2017)

    Article  Google Scholar 

  50. Winstone, B., Pipe, T., Melhuish, C., Dogramadzi, S., Callaway, M.: Biomimetic tactile sensing capsule. In: Conference on Biomimetic and Biohybrid Systems, pp 113–122. Springer (2015)

  51. Winstone, B., Melhuish, C., Pipe, T., Callaway, M., Dogramadzi, S.: Toward bio-inspired tactile sensing capsule endoscopy for detection of submucosal tumors. IEEE Sensors J. 17(3), 848–857 (2016)

    Article  Google Scholar 

  52. Rojas, N., Ma, R.R., Dollar, A.M.: The gr2 gripper: An underactuated hand for open-loop in-hand planar manipulation. IEEE Trans. Robot. 32(3), 763–770 (2016)

    Article  Google Scholar 

  53. Ma, R.R., Spiers, A., Dollar, A.M.: M 2 gripper: Extending the dexterity of a simple, underactuated gripper. In: Advances in reconfigurable mechanisms and robots II, pp 795–805. Springer (2016)

  54. Kumagai, K., Shimonomura, K.: Event-based tactile image sensor for detecting spatio-temporal fast phenomena in contacts. In: 2019 IEEE World Haptics Conference (WHC), pp 343–348. IEEE (2019)

  55. Johnson, M.K., Adelson, E.H.: Retrographic sensing for the measurement of surface texture and shape. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp 1070–1077. IEEE (2009)

  56. Li, R., Platt, R., Yuan, W., ten Pas, A., Roscup, N., Srinivasan, M.A., Adelson, E.: Localization and manipulation of small parts using gelsight tactile sensing. In: 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp 3988–3993. IEEE (2014)

  57. Zhang, T., Cong, Y., Li, X., Peng, Y.: Robot tactile sensing: Vision based tactile sensor for force perception. In: 2018 IEEE 8th Annual International Conference on CYBER Technology in Automation, Control, and Intelligent Systems (CYBER), pp 360–1365. IEEE (2018)

  58. Zhang Y., Kan Z., Yang Y., Tse Y. A., Wang M. Y.: Effective estimation of contact force and torque for vision-based tactile sensors with helmholtz–hodge decomposition. IEEE Robotics and Automation Letters 4(4), 4094–4101 (2019)

    Article  Google Scholar 

  59. Li, R., Adelson, E.H.: Sensing and recognizing surface textures using a gelsight sensor. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 1241–1247 (2013)

  60. Yuan, W., Zhu, C., Owens, A., Srinivasan, M.A., Adelson, E.H.: Shape-independent hardness estimation using deep learning and a gelsight tactile sensor. In: 2017 IEEE International Conference on Robotics and Automation (ICRA), pp 951–958. IEEE (2017b)

  61. Yuan, W., Mo, Y., Wang, S., Adelson, E.H.: Active clothing material perception using tactile sensing and deep learning. In: 2018 IEEE International Conference on Robotics and Automation (ICRA), pp 1–8. IEEE (2018)

  62. Lepora, N.F., Church, A., De Kerckhove, C., Hadsell, R., Lloyd, J.: From pixels to percepts: Highly robust edge perception and contour following using deep learning and an optical biomimetic tactile sensor. IEEE Robot. Autom. Lett. 4(2), 2101–2107 (2019)

    Article  Google Scholar 

  63. Church A, James J, Cramphorn L, Lepora N: Tactile model o: Fabrication and testing of a 3d-printed, three-fingered tactile robot hand. arXiv:190707535 (2019)

  64. Polic, M., Krajacic, I., Lepora, N., Orsag, M.: Convolutional autoencoder for feature extraction in tactile sensing. IEEE Robot. Autom. Lett. 4(4), 3671–3678 (2019)

    Article  Google Scholar 

  65. Calandra, R, Owens, A, Upadhyaya, M, Yuan, W, Lin, J, Adelson, EH, Levine, S: The feeling of success: Does touch sensing help predict grasp outcomes? arXiv:171005512 (2017)

  66. Tian, S., Ebert, F., Jayaraman, D., Mudigonda, M., Finn, C., Calandra, R., Levine, S.: Manipulation by feel: Touch-based control with deep predictive models. In: 2019 International Conference on Robotics and Automation (ICRA), pp 818–824. IEEE (2019)

  67. Ward-Cherrier, B, Pestell, N, Lepora, NF: Neurotac: A neuromorphic optical tactile sensor applied to texture recognition. arXiv:200300467 (2020)

  68. Muthusamy, R., Huang, X., Zweiri, Y., Seneviratne, L., Gan, D.: Neuromorphic event-based slip detection and suppression in robotic grasping and manipulation. IEEE Access 8, 153,364–153,384 (2020)

    Article  Google Scholar 

  69. Baghaei Naeini, F., Makris, D., Gan, D., Zweiri, Y.: Dynamic-vision-based force measurements using convolutional recurrent neural networks. Sensors 20(16), 4469 (2020)

    Article  Google Scholar 

  70. Huang, X., Muthusamy, R., Hassan, E., Niu, Z., Seneviratne, L., Gan, D., Zweiri, Y.: Neuromorphic vision based contact-level classification in robotic grasping applications. Sensors 20(17), 4724 (2020)

    Article  Google Scholar 

  71. Sun, L., Zhao, C., Yan, Z., Liu, P., Duckett, T., Stolkin, R.: A novel weakly-supervised approach for rgb-d-based nuclear waste object detection. IEEE Sensors J. 19(9), 3487–3500 (2019). https://doi.org/10.1109/JSEN.2018.2888815

    Article  Google Scholar 

Download references

Acknowledgements

We would like to acknowledge the efforts of graduate student Huda Alyammahi for developing figures and schematics of the sensor systems discussed in this review.

Funding

This work is supported by the Khalifa University of Science and Technology under Award No. CIRA-2018-55 and RC1-2018-KUCARS.

Author information

Authors and Affiliations

  1. Khalifa University Center for Autonomous Robotic Systems, Khalifa University of Science and Technology, Abu Dhabi, UAE

    Umer Hameed Shah, Rajkumar Muthusamy, Yahya Zweiri & Lakmal Seneviratne

  2. School of Engineering Technology, Purdue University, 47907, West Lafayette, IN, USA

    Dongming Gan

  3. Faculty of Science, Engineering and Computing, Kingston University London, London, SW15 3DW, UK

    Yahya Zweiri

Authors
  1. Umer Hameed Shah
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rajkumar Muthusamy
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Dongming Gan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yahya Zweiri
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Lakmal Seneviratne
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

U.H. Shah proposed the idea of this article; U.H. Shah and R. Muthusamy conducted literature survey and drafted the manuscript; D. Gan, Y. Zweiri, and L. Seneviratne critically revised the manuscript.

Corresponding author

Correspondence to Umer Hameed Shah.

Ethics declarations

Competing interests

The authors have no relevant financial or non-financial interests to disclose.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Shah, U.H., Muthusamy, R., Gan, D. et al. On the Design and Development of Vision-based Tactile Sensors. J Intell Robot Syst 102, 82 (2021). https://doi.org/10.1007/s10846-021-01431-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10846-021-01431-0

Keywords

Search

Navigation