1 Introduction

Technological progress in network and mobile phone allows users to remotely share their voice or vision with high quality. Beside the voice and vision, sharing the sensation of touch is also important for a local user to explore the property of a remote material. Transmission of tactile sensation is widely study for tele-operation, tele-training or tele-touch communication [1, 2]. Such a tactile transmission is also required to immerse the sensation of being in a place other than where a person actually exists. This is called telexistence [3]. However, there are still challenges for the tactile transmission with realistic sensation of touch due to the lack of material information that measured by the tactile sensor and a small range of tactile feeling that reproduced by the tactile display.

Accelerometer, microphone or pressure sensor is commonly used for measuring the property of a material [4,5,6]. However, each of these sensors can measure in a limited range of spatial or temporal resolution. Temperature is also an important information for exploring the material property. Therefore, the tactile sensor is required to be able to simultaneously measure the vibration, pressure and temperature with high spatiotemporal resolution. Moreover, the sensor need to be lightweight and compact enough for mounting to the fingertip. On the other hand, tactile display is also required to be able to reproduce tactile sensation with rich information. In principle, we might be able to reproduce any tactile sensation if we could drive the skin with sufficient spatial (up to 1.5 mm at fingertip) and temporal (0 to 1 kHz) resolution, but as the skin has large mass and damper, it is still quite difficult to develop such versatile micro machine for tactile display.

In this study, we developed both tactile sensor and tactile display with high spatiotemporal resolution. The tactile sensor embedded inside a glove can simultaneously measure the temperature and the pressure distribution when a tactile sender wears the glove and touches the object with three fingers. The tactile display reproduces the touch sensation based on the data that measured and transmitted from the tactile sender (Fig. 1). Each module of tactile display consisted of electrotactile, vibrotactile and thermal tactile actuators, and was embedded inside a glove. With these three primary-tactile colors, our display can theoretically activate Merkel cells, Meissner corpuscles, Pacinian corpuscles and thermo-receptors, all of these are important for reproduce any tactile feeling.

Fig. 1.
figure 1

The tactile-sender wears the sensor glove on the right hand and touches or grasps an object (left). The tactile-receiver perceives touch feedback sensation on the right hand when the tactile-sender touches the material.

Full size image

2 Related Work

2.1 Wearable Tactile Sensor

Tactile sensors are widely used, including touch panels as input devices for information equipment, robot fingers for giving tactile abilities to the robots, pressure distribution for industrial applications and human behavior measurement [6,7,8]. In our study we focus on a wearable tactile sensor that can record various tactile sense in some tasks that use fingertips for operation. Such a wearable tactile sensor can be used for sharing tactile information of a remote user by combining with a tactile display.

It is desirable that the wearable tactile sensor does not hinder the original tactile sensation of the fingertip. For this reason, such a tactile sensor can be constructed by, for example, arranging a strain gauge on the side of the finger [9], measuring the color change of the nail [10] and the vibration on the nail [11, 12], which were designed not to cover the finger pad. However, with such a method, although it is possible to measure the force and vibration applied to the entire finger, it is not possible to measure the pressure distribution. Moreover, since the temperature perception is important when a person judges the material and surface quality of an object [13, 14], the measurement of the temperature distribution applied to the surface is also required.

Many glove-type tactile sensors have been developed, and many of which use pressure-sensitive conductive inks or sheets [15]. However, most of these did not evaluate the sensor’s responsiveness, thus, it is difficult to directly compare with temporal and spatial resolution of human tactile perception. In recent years, for example, the High-Speed I-SCAN system developed by Nitta Co., Ltd. can measure at 720 Hz [16]. However, to the best of the author’s knowledge, there were no wearable tactile sensor that combines the measurements of temperature distribution and pressure distribution together.

2.2 Tactile Feedback on the Fingertip

Many wearable tactile devices with different presentation methods have been proposed for fingertip interaction with virtual world. Vibration is commonly used for presenting texture sensation [17,18,19]. However, such a method can present a limited range of object properties due to its low spatial resolution. Several studies have developed a pin matrix to simulate shape sensation to users [20,21,22]. Though, the device can provide higher spatial resolution compare with vibrotactile system, the issues of low temporal resolution and large size still remain. Some wearable devices provide force feedback to the finger pads by presenting a pressure or skin deformation sensation [23,24,25]. These devices are mainly used for touch or surface exploration interactions in the virtual environment. Several other studies have focused on the illusion of softness or stickiness that can be induced by changing the contact area of the force applied to the skin [26, 27]. However, these devices can reproduce only some sort of tactile feeling.

Several studies have proposed using electrical stimulation to directly activate tactile receptors on the fingertip [28, 29]. Though this method achieves high responsiveness and small in size, reproducing realistic tactile sensation remains a challenge. In our previous study, we propose to combine electrical and mechanical stimulation to optimize the size and weight of a versatile tactile feedback device [30, 31]. Based on previous finding we considered that this method can selectively activate four kinds of mechanoreceptors. However, temperature presentation was not considered. Thermal tactile display has been widely studied [32], but a small-size tactile display that includes all of electrical, vibration and thermal tactile stimulation was not revealed. This paper also introduces our tactile display that can present all of the stimulation on the fingertip.

3 System

3.1 Tactile Sensor for the Sender

We developed a glove-type tactile sensor that is compatible with wearing on multiple fingers (Fig. 2). Figure 2 (left) shows the internal structure of the pressure and temperature distributions for three fingers. The pressure sensitive conductive sheet was used for pressure distribution sensing and the chip thermistor for temperature sensing. For each finger, the pressure distribution has 50 points (5 × 10) and the temperature distribution has 8 points as the measurement elements. The distance between the sensor elements is 2 mm. The flexible board meanders to correspond to the bending of the finger.

Fig. 2.
figure 2

Overview of the pressure and temperature distribution sensor (left) and the glove with the sensor inside (right) for three fingers.

Full size image

Figure 3 (left) shows the structure of the sensor. The conductive layer has a pressure-sensitive property whose resistance value varies with pressure, and the change in resistance value is measured by two electrodes. The electrodes were formed on a flexible substrate; the chip thermistors (TDK, B 57232 V 5103 F 360) were arranged on the opposite side of this substrate. Figure 3 (right) shows the configuration of the data reading circuit. Both the pressure-sensitive film and the chip thermistor are the elements that change the resistance values. For this reason, we constructed a general resistance matrix detection circuit, which one row of the sensor matrix was selected by a digital output from a microprocessor (mbed NXP LPC 1768), and the voltage output divided by the fixed resistance was detected by a multi-channel AD converter (Texas Instruments, ADS 7953).

Fig. 3.
figure 3

Structure of the sensor and circuit diagram.

Full size image

We used preliminary prototypes of pressure distribution (Fig. 4 (left)) for evaluation. The number of sensor elements is 256 points (16 × 16), and the sensor interval is 4 mm. Except for the influence of communication overhead with the PC, in order to obtain the upper limit value of the sensor reading speed, the measurement data was temporarily stored in the memory of the microprocessor. As a result, the pressure distribution of 16 × 16 could be measured 100 times (37 ms) at 2.7 kHz. This means that the time required for one measurement of 256 points was about 0.37 ms. Figure 4 (right) shows the time change of the pressure distribution when dropping the rubber ball onto the sensor. It can be observed how the pressure distribution spreads. Although the main measurement result does not indicate the physical responsiveness of the sensor element itself, since at least at the beginning and the end of the contact, a clear change is seen between consecutive frames (for example, frames 1 to 3, 13 to 15); which shows that it has high responsiveness.

Fig. 4.
figure 4

Pressure distribution for measurement (left); measurement of rubber ball hitting the surface at 2.7 kHz sampling rate (right).

Full size image

3.2 Three Primary Color Tactile Display for the Receiver

Figure 5 shows the glove of tactile display for the receiver. The size of this glove can be adjusted to fit to any size of the fingertips. Three tactile modules and the module controller were embedded inside the glove.

Fig. 5.
figure 5

The glove of tactile display (left). This glove can be adjusted to fit to any size of the fingertip (right).

Full size image

Figure 6 shows a tactile module that consisted of an array of electrode for electrotactile, a vibration actuator for vibrotactile, and a Peltier and a heater for thermal tactile presentation. Three modules were used for tactile presenting to the thumb, index and middle finger. The vibration actuator, Peltier and heater, and module controllers were developed by one of joint group researches of Alps Alpine Co., Ltd. [33]. Though, Peltier can provide both coolness and hotness sensation, to efficiently reduce energy consumption we used it for cooling only and used heater made from a resistant instead for heating.

Fig. 6.
figure 6

A module of the tactile display (left) and controller (right).

Full size image

3.3 Tactile Transmission System and Algorithm

As Fig. 1 shows, when a sender touches or grasps an object with the thumb, index or middle finger, the tactile sensor spatiotemporally measures the pressure and temperature of that object. Due to the tactile display cannot directly replay the measured data, algorithm of replay tactile sense is required (e.g. converting from pressure value to vibration density). Electrotactile stimulation can mainly present the movement of the pattern of a shape by changing the stimulation points of electrode array, vibrotactile stimulation for tapping or scrolling sensation, and thermal tactile stimulation for temperature sensation.

The data of pressure sensors were spatially filtered by using Eq. (1). We used Eq. (2) to convert spatiotemporal pressure values to the audio signal for vibration input. Though, there are many points of pressure distribution were contacted to the object, only maximum pressure value of each time frame were considered in this equation. For the electrical stimulation, the current intensity of each point was calculated with the Eq. (3).

$$ p_{i,j} = \frac{{p_{i,j} + p_{i + 1,j} + p_{i,j + 1 + } p_{i + 1,j + 1} }}{4} $$
(1)
$$ A_{t} = k_{1} (p_{max,t} - p_{max, t - 1} ) $$
(2)
$$ I_{i,j} = I_{sense} + k_{2} \times p_{i,j} $$
(3)

where p is pressure value, i and j are order number on the axis of width and height of the array. \( A_{t} \) and \( p_{max,t} \) are vibration signal amplitude and maximum pressure at time t, \( p_{max, t - 1} \) is maximum pressure at previous time. \( I_{sense} \) is the sensation threshold of electrical intensity, \( k_{1} \) and \( k_{2} \) are constant.

To stabilize and to prevent pain sensation of electrical stimulation, we do not stimulate all points of electrode array even when all points of the pressure sensor were contacted to the object. As Fig. 7 shows, when the number of stimulation points becomes more than the maximum number (e.g. four), we keep stimulating the electrodes that correspond to the points of maximum pressure and its surrounding in which the total stimulated points are maximum number of the stimulation points.

Fig. 7.
figure 7

Reducing the stimulated points of electrical stimulation.

Full size image

For thermal tactile stimulation, we do not present the degree of the temperature the same as that of the object but the degree of the temperature difference after touching the object.

4 System Evaluation

4.1 Purpose and Conditions

We conducted a preliminary experiment to evaluate the ability of our system for tactile transmission. The main purpose of this experiment is to observe the ability of our proposed algorithm for converting the sensor values to the parameters of tactile display. Three performances operated by a tactile sender were used in this experiment: scrolling a hexagonal pencil, tapping between the thumb and the index finger, and holding a paper cup that one with hot water and other with cold water (Fig. 8). The purpose of using hot and cold water is to observe the speed of the temperature transmission. We developed a GUI to directly observe the pattern change of pressure distribution and audio wave form, these are for electrical and vibration stimulation.

Fig. 8.
figure 8

Three performances operated by a tactile sender for system evaluation.

Full size image

4.2 Result and Discussion

  • Scrolling a hexagonal pencil

Figure 9 shows the pattern changing of the pressure distribution and audio waveform while scrolling a hexagonal pencil. Each frame was captured at 10 Hz sampling rate. The scrolling patterns with three fingers can be observed. The patterns were transmitted and reproduced at the receiver side by the electrode array, and the receiver can perceive the movement of the shape. This result indicated that our algorithm can remain the scrolling shape even when the tactile sender strongly pressed the pencil and all points of pressure distribution detected.

Fig. 9.
figure 9

Pressure distributions and audio waveforms while scrolling a pencil.

Full size image

On the other hand, each audio waveform was observed to be a continuous vibration with a small amplitude. Due to the shape of the pencil is hexagon, we feel the vibration like discrete accordingly to the rotation of the pencil. In our algorithm, we focused on the temporally changing the maximum pressure. To create such discrete-like vibration, the spatially changing the contact area also should be considered.

  • Tapping between the thumb and index finger

Figure 10 shows the pressure distribution and audio waveform while tapping between two fingers. Each frame was captured at 40 Hz sampling rate. The tapping patterns and vibration waveforms can be observed.

Fig. 10.
figure 10

Pressure distributions and audio waveforms while tapping.

Full size image

It is widely known that the vibration of tapping process can be modeled to be as damped sinusoidal waveform, and the frequency represents the hardness of the material. The waveforms by the proposed algorithm look like to impulse rather than sinusoidal waveform. Similar to scrolling performance, to reproduce tactile sensation with more realistic, the changing of contact area that is related to the object property should also be considered.

  • Holding a paper cup with hot water and cold water

Figure 11 shows the temperature change of each finger at both sender and receiver sides in the process of grasping a paper cup of hot water, releasing, and grasping a paper cup of cold water. In our algorithm, we did not present the actual temperature of the object but the temperature different of the sender after touching the object. There was about 5 s latency because heating or cooling by heater or Peltier required time. During cooling by Peltier, we observed that the temperature unstably vibrated. It is because the other side of Peltier produced heat and transferred to the cool side.

Fig. 11.
figure 11

Temperature transmission of each finger while holding a paper cup with hot water and cold water.

Full size image

5 Conclusion

We developed a tactile transmission system that can transmit a tactile sense occurs while operating the object with three fingers: thumb, index and middle fingers. For tactile sender side, we developed a glove-type tactile sensor that can measure pressure distribution and temperature distribution simultaneously. For tactile receiver side, we developed a glove-type tactile display with three kinds of primary tactile color: electrical stimulation, thermal stimulation and vibration stimulation. We also developed the algorithm for reproducing tactile sense measured by tactile sensor (e.g. converting pressure distribution changing values to the parameters of vibration presentation).

We tested our system with three performances: scrolling a hexagonal pencil, tapping between the thumb and the index finger, and holding a paper cup that one with hot water and other with cold water. The results confirmed the ability of our system to transmit some sorts of tactile sense with rich information. In the future work we will redevelop more effective algorithm that can present the object property in wider range.