Keywords

1 Introduction

In the field of tactile display, many researchers are developing systems that employ recorded vibration as a tactile signal, and these systems present high-quality tactile sensations. To enhance this type of display method, it is necessary to collect and classify recorded vibration during haptic behaviors. Consequently, various studies of the collection and classification of recorded vibration in haptic behaviors have been performed [1, 2]. However, most of these studies have collected tactile information under limited experimental environments using devices with many sensors. Therefore, it is difficult to collect haptic information outside of the experiment environment, for example in daily behavior.

In a previous research, we proposed a solution to the problem of collecting haptic information without complicated devices [3]. Using this approach, we collected only acceleration data as haptic information. Using a ZigBee-based microcomputer with an accelerometer, we collected haptic information more easily than by conventional research methods. We previously reported that our method can collect acceleration data from rubbing haptic behaviors on a table. We also proposed applying collected data to classify haptic information. Our previously implementation used Support Vector Machine (SVM) and classified eight types of data with an accuracy of about 80%. However, if we change machine learning method, there is a possibility of getting higher accuracy.

Therefore, we implemented a method in the current study to classify haptic information collected by our method using machine learning. We attached a ZigBee-based wireless microcomputer with an accelerometer to the experimenter’s finger or pen while they traced various objects, and we collected accelerations during haptic behaviors on a table. As a machine learning method, we used convolutional neural network (CNN) to classify the haptic information with high precision. We succeeded in classifying 18 types of data with an accuracy of about 93%. We also performed an experiment to classify rubbed objects in real-time. The CNN was able to classify five types of objects, about 67.7% on average.

2 A Method of Collecting and Classifying Haptic Information

In this study, we demonstrate our method of collecting and classifying haptic information. Acceleration in haptic behaviors is collected by a ZigBee-based microcomputer and then classified by machine learning using CNN. In the following sections, this method will be described in detail.

2.1 ZigBee-Based Microcomputer with Acceleration Sensor

Many studies have used mechanical vibrations in the band of 0–1 kHz frequency band as haptic information for haptic display, and their implementations have produced highly realistic information, such as the TECHTILE toolkit [4]. Therefore, we also collected mechanical vibration information within 0–1 kHz as haptic information. For that purpose, our implementation was based on a small ZigBee-based microcomputer equipped with an accelerometer capable of collecting vibration information in the 0–1 kHz range.

Fig. 1.
figure 1

TWE-Lite-2525A, Mono Wireless Inc. [5]. This sensor node includes a 3-axis accelerometer, ZigBee transfer module, and battery cell (CR2032).

Full size image

In this study, we used a ready-made device, TWE-Lite-2525A (Mono Wireless Inc. [5]), shown in Fig. 1. This device consists of a 3-axis accelerometer and a ZigBee wireless communication module with a small battery cell as a power supply. The device was 25 mm square and its weight was 6.5g; it is capable of wireless communication using ZigBee, such that it can be installed and operate on various objects. To save power, we added a sleep mode to the device, during which it measures acceleration at low frequencies until the measured acceleration exceeds the set threshold, and then begins measurement at high frequencies after the threshold is exceeded.

2.2 Data Transmission

To classify haptic information by machine learning, the collected acceleration data was transmitted to a personal computer (PC) by wireless communication using ZigBee. The ADXL345 accelerometer installed on the device can measure vibrations at 800 Hz. However, I2C communication between the accelerometer and the ZigBee wireless module, as well as ZigBee communication between the wireless module and the computer, must be taken into consideration; thus, the actual value was considerably lower than 800 Hz. The measurement rate of the microcomputer was 33 Hz using the program as written at the time of purchase; this rate is insufficient for the transmission of haptic information.

Therefore, we adopted a method to pack 10 sequentially measured values into one packet. We implemented this method as a new program in a microcomputer to facilitate the measurement of 3-axis acceleration at about 330 Hz.

2.3 Classification by Machine Learning

In this subsection, we describe a method to classify collected data by machine learning. In a previous study [3], we classified data by machine learning using a support vector machine (SVM). In the current study, to improve the number of classes, we employed a CNN-based machine learning method. We used a total of 13 CNN layers for data classification. The configuration is shown in Fig. 2.

Fig. 2.
figure 2

Composition of CNN.

Full size image

As the input data to this CNN, we used clusters of 200 consecutive points (each point includes 3-axis data: x, y, z), which were randomly extracted from the acceleration data. This method increase the number of learning data and improve generalization performance of the model. The output data of this CNN was the probability of the type of haptic information input.

We used TensorFlow [6], a machine learning library provided by Google, to build the CNN. We built the CNN with reference to the model of VGG [7] which is a typical CNN composition; we reduced the amount of data to be convoluted by one convolution layer, and increased the convolution layer to improve the accuracy of classification. We used the ReLU function [8] as the activation function and a convolution filter of size \(1\times 5\). We doubled the number of filters for each pooling to improve generalization. In the pooling layer, maximum pooling was performed at \(1\times 2\), and data was compressed to half so that CNN would be robust against shape of data. In addition, overfitting was suppressed using the Batch Normalization algorithm [9] after calculating the activation function in each convolution layer and in the fully connected layers.

3 Classification Experiment

We collected and classified haptic information using the method described in the previous section and verified the accuracy of the classification of this collected information. In addition, we performed an experiment to classify rubbed objects in real-time.

3.1 Classification of 18 Types of Haptic Information

We performed an experiment to classify 18 types of haptic information. We will describe this experiment in detail below.

Settings. The haptic information was collected by the experimenter. Attaching the ZigBee-based microcomputer to their finger or a pen, the experimenter traced the surfaces of objects with various textures, thus collecting many types of haptic information. This collection process is shown in Fig. 3. The textures used in this experiment are shown in Fig. 4.

Fig. 3.
figure 3

Collection of tactile data. The microcomputer was put into the 3D printed case, attached to experimenter’s finger or pen and operated to collect haptic data.

Full size image

Each texture to be examined was a plate-like object of about 100 mm in length and width. In Fig. 4, carpet1, carpet2, carpet3 were pieces of carpet made with different materials; sponge-g and sponge-y were the front and back of a household sponge. Objects with a texture similar to that of sponge-b were made of Styrofoam; stonetile1, stonetile2, and stonetile3 were stone tiles made with materials of different texture; whitetile1, whitetile2, and whitetile3 were white flooring tiles with different textures; woodtile1, woodtile2, and woodtile3 were wooden plates with different textures; sandpaper40, sandpaper80 were sandpapers with different roughness.

Fig. 4.
figure 4

Textures. These are plate-like object with length 70–100 mm and width 100–130 mm.

Full size image

Haptic information was collected by rubbing these objects with a finger or pen at an almost constant speed of 400 mm/s. At the surface of each object, acceleration data were collected during three minutes of back-and-forth rubbing movement. By performing this operation three times per object, nine minutes of acceleration data were collected per object. As shown in Fig. 4, 10 types of objects were used in this experiment. We rubbed each object with a finger and rubbed each object except ‘‘sandpaper40"and ‘‘sandpaper80" with a pen. We collected a total of 18 types of data (10 types of finger data\(\times \)8 types of pen data). We could not collect accurate data from sandpaper rubbed by pen, because the data collection process damaged the tip of the pen; therefore, we discarded these data.

We evaluated classification by machine learning using the CNN. The CNN input included \(3\times 200\) values of acceleration data, and the output was a probability of \(1\times 18\), the probability of an object having been rubbed to create the input data. During classification, to confirm the generalization performance of the model created by this CNN, we divided all data by 10 and performed 10-fold cross-validation.

Table 1. Confusion matrix for the classification of 18 types of data.
Full size table

Result. We describe the accuracy of the data classification experiment in Table 1, which shows the confusion matrix obtained by classifying 18 types of data. We obtained an accuracy of 93.2% on average, indicating that we succeeded in classifying 18 types of information with high precision.

As shown in the confusion matrix, we obtained an accuracy exceeding 80% for all results. Therefore, in this experiment, we found that critical overfitting did not occur. However, this result does not mean that there was no overfitting. The best result shown in the confusion matrix was woodtile-pen, at 97.2%; however the worst result was sandpaper40, at 83.1%. There was a difference of 14.1% between the pen and finger results, and a small amount of overfitting occurred.

For sandpaper40, the correct result was 83.1%, and the incorrect result was 17.9%. The largest of the incorrect results for sandpaper80 was 6%. Among the incorrect results for sandpaper80, sandpaper40 was incorrectly classified at 7.8%, the largest value among incorrect results. From these results, we conclude that it was relatively difficult to classify fine textures.

3.2 Object Classification in Real-Time

We performed an experiment to classify rubbed objects in real-time. We classified five types of rubbed object using the CNN model. We will describe this experiment in detail below.

Settings. An experimenter collected haptic information by attaching the ZigBee-based microcomputer to their finger and rubbing the surfaces of objects with various textures. The textures used in this experiment were carpet1, carpet2, carpet3, sponge-g, and sponge-y (Fig. 4). Haptic information was collected by rubbing these objects with a finger or pen at an almost constant speed of 400 mm/s. At the surface of each object, acceleration data were collected during five seconds of back-and-forth rubbing movement. We performed this operation 15 times per object each day, on six consecutive days. We collected a total of 90 observations per object.

We collected new data for this experiment because we had obtained less than 10% accuracy on average in the nine-minute data collected in Sect. 3.1. We determined that the rubbing method (e.g., finger angle, rubbing speed) changed slightly each time data was collected during that experiment. When we collected nine-minutes data, we collected data only three times per objects. Therefore, when the nine-minute data was used, CNN model learned only three types of data per object; thus, the CNN model did not attain adequate generalization performance. To solve this problem, we used six-days data. If we use six-days data representing 90 times collections per object, CNN model can learned 90 types of data per object and get better generalization performance. Therefore, CNN model using six-days data was better than using nine-minutes data.

We created the CNN model to identify objects by rubbing. The CNN input was \(3\times 200\) acceleration measurements, and the output was a probability of \(1\times 5\), the probability of an object having been rubbed to create the input data. We used 80% of all data as training data and the remaining data as test data. We used the test data to confirm the progress of CNN learning.

Using this CNN model, we classified objects rubbed by an experimenter in real-time. The instant he collected data, we inputted data and classified. He collected a total of 20 5-s observations as new data. Using this method, we classified each input value 10 times; that is, we conducted 200 classification experiments per object. The output of this CNN was the object from which the haptic information had been derived.

Result. Table 2 shows the accuracy of the experimental results. These accuracies represent averages of 20 classifications. Table 3 shows the confusion matrix for the classification of five objects.

Table 2. Accuracy of classification of haptic data collected by touching five objects.
Full size table
Table 3. Confusion matrix for the classification of haptic data collected by touching five objects.
Full size table

In Table 2, we obtained an accuracy of 67.7% on average. The best accuracy was 83.5% (sponge-g), and the worst was 41.5% (carpet3). There was a large difference between the best and the worst accuracies. In Table 3, the accuracy for detecting carpet3 was 71.4% and this represented the least accurate result, which is consistent with the trend shown in Table 2. As shown in Table 3, the accuracies for detecting sponge-g and sponge-y were 100%. Because these values are very high and there was a large difference between the best and worst accuracy results, we conclude that overfitting occurred.

The insufficient generalization performance of the CNN model may explain these results. To solve this problem in the future, we will design the CNN model using more observations than used in this experiment. In addition, we will review the CNN settings to improve the results.

4 Conclusion and Future Work

In this research, we implemented a method in the current study to classify haptic information collected by our method using CNN.

Using a compact wireless microcomputer with an acceleration sensor applied to the tip of a finger or ballpoint pen, an experimenter rubbed various objects at approximately 400 mm/s, collecting 18 types of haptic information. We then classified these data, obtaining an accuracy of 93.2%. We concluded that we were able to classify haptic information with high accuracy. In addition, we performed an experiment in which haptic data from five types of touched objects were classified using a CNN model, and obtained an accuracy of 67.2% on average. We determined that the rubbing method (e.g., finger angle, rubbing speed) may change slightly each time data is collected.

In the future, we will review our CNN settings to explore the possibility of improving classification accuracy, for example, by changing the number of convolution layers. In addition, we will apply our method in different settings.