1 Introduction

Locomotion ability is diverse arise from several factors such as sex, ages and medical histories, and many people have to use wheelchair. Moreover, advanced personal mobility devices consisting of multiple wheels and computers are starting to enter daily life. It is presumed that the use of wheelchair type devices will only increase. The surface conditions of paths and roads such as roughness or inclines may interfere with wheelchair users’ convenience while bipedal walkers experience no such difficulties. By gathering such surface information, we can improve movement ease for wheelchair users. MLIT(Ministry of Land, Infrastructure, Transport and Tourism) is distributing barrier information such as the existence of the roughness or gradients of roads [1]. Typical, barrier information is gathered by the office in charge of the facilities such as stations and parks. However, the amount of barrier information is insufficient, and much barrier information too old. In order to better support user movement, information must be gathered often from various areas and in more detail.

Many projects use the crowd-sourcing approach for collecting information about road surface state. For example, PADM(NPO Corporation) is running the project “Let’s join together to create a Barrier Free Map” which uses smartphones to gather barrier information. This project won a grand prize in Google Impact Challenge [2]. As detailed in “Basic Program for Persons with Disabilities” which is published by the Japanese Cabinet, “Barrier” includes not only physical barriers such as roughness or slopes, but also psychological barriers such as a feeling of pressure or feeling of fear [3]. For example, areas that have cars passing nearby make wheelchair users afraid. Such areas represent a significant psychological barrier. Psychological barriers for wheelchair users are not considered by most researchers and most technologies focus on physical barriers such as road state.

“Ride Comfort” is commonly used as a metric related to psychological barriers [4, 5]. Sawada et al. studied the ride comfort of wheelchair users in collaboration with medical and welfare institutions, and showed that ride comfort can be identified by terms such “Relief”, “Safety”, “Comfort”, “Stability”[4]. In accordance with the findings of Sawada et al., we define “ride comfort” as the psychological barrier metric. Since many current technologies exist for gathering physical barrier information, our goal is to create a methodology for gathering psychological barrier information. This paper proposes a method that can estimate ride comfort based on inertial and vital data. It is confirmed by an experiment involving 12 subjects and 4 different courses. Our proposal will improve the comfort of wheelchair users by allowing physical and psychological barriers to be gathered and disseminated. In terms of HCI, user adaptive system is important. If user feels fear, computer should calculate and recommend different ways dynamically. The proposed method is expected to extract information for system’s decision making. Therefore, this paper contributes to the user adaptive navigation or recommendation system.

2 Related Works

Many studies have focused on “Ride comfort” as a metric to evaluate physical and psychological factors and how they impact users. Liu et al. provided a factor analysis of the ride comfort of automobile users [5]. They found that ride comfort consisted of 3 groups of factors; vibratory stimulation factors such as vibration strength and frequency, physiological factors such as physical condition and alertness, and psychological factors such as mood and sense of stability. Sawada et al. studied the ride comfort of wheelchair users in collaboration with medical and welfare institutions, and introduced the semantic differential method which assesses ride comfort in terms of “Relief”, “Safety”, “Comfort”, “Stability” [4, 6].

Evaluating the surface state of roads, which is assumed to impact ride comfort, has been the goal of many researchers. Mounting acceleration sensors on wheelchairs and user responses are common techniques for evaluating the surface state of roads [7–9]. Some works tried to find barriers in urban areas by mounting acceleration and gyro sensors on wheelchairs [2, 10, 11]. Iwasawa et al. applied a learning support vector machine to acceleration data to detect roughness and slopes [10]. Kuwahara et al. showed how to detect road surface state with 85 % accuracy by applying the k-nearest neighbor method to acceleration data [11].

Some researchers have studied mental stress, which is assumed to be related to ride comfort. Mental stress estimation methods based on vital data have been proposed. Yokoyama et al. evaluate driver alertness from heartbeat rhythms [12]. Imai et al. proposed a way to quantitatively estimate the driver’s mental workload from heart rate variability [11].

Fig. 1.
figure 1

The concept of the proposed method

Full size image

3 Ride Comfort Estimation Based on Inertial and Vital Data

This paper classifies barriers as physical or psychological, and proposes a method for estimating psychological barrier. As the ground truth of mental stress, we adopt the semantic differential method of Sawada et al. Wheelchair users have to respond to a questionnaire to assess the factors of “Relief”, “Safety”, “Comfort”, “Stability”. It is difficult for users to respond a questionnaire in real environments. Therefore, we propose a methodology for estimating ride comfort automatically, based on sensor data. The users are assumed to manipulate wheelchairs carefully, and to feel stress or fear in areas that impose high psychological burdens on the user. The burden is identified by gathering the physical movements of the wheelchair under the control of the user. Feelings of stress or fear are assumed to be reflected in the users’ vital data such as heartbeat. Figure 1 shows the concept of the proposed method.

The ride comfort estimation method consists of a learning phase and an estimation phase. In the learning phase, sensor data vector \({\varvec{d}} = (s, {\varvec{acc}}, {\varvec{gyro}}, {\varvec{rri}})\) is used; s is the degree of ride comfort, time-series 3-axis acceleration data \({\varvec{acc}}\), time-series 3-axis gyro data \({\varvec{gyro}}\), and RRI data rri. \({\varvec{acc}} = ({\varvec{ax}}, {\varvec{ay}}, {\varvec{az}})\); time-series x-axis, y-axis, z-axis data is \({\varvec{ax}}, {\varvec{ay}}, {\varvec{az}}\), respectively. \({\varvec{gyro}} = ({\varvec{gx}}, {\varvec{gy}}, {\varvec{gz}})\); time-series x-axis, y-axis, z-axis data is \({\varvec{gx}}, {\varvec{gy}}, {\varvec{gz}}\), respectively. From the heartbeat data we extract pulse shape, which includes P, Q, R, S, T waves. Time-series RRI data \({\varvec{rri}}\) is obtained by calculating R wave intervals. x, y, z-axis acceleration data and x, y, z-axis gyro data are used as inertial data. Inertial features, shown in Table 1 are extracted from the inertial data. Statistical features(Minimum, Maximum, Amplitude, Median, Average, Standard deviation, Variance) and Frequency features(the intensity of the frequency of from 0 to 15[Hz] of 6-axis data at 1[Hz] intervals) are extracted.

Vital features are extracted as shown in Table 2. Bauer et al. investigated the relationship between heartbeat variability and the nervous system [14]. Heart rate variability(HRV) features are used as vital features.

Table 1. Inertial features
Full size table
Table 2. Vital features
Full size table

mRR, SDNN, RMSSD, SDSD, pNN50, LF Norm, HF Norm, LFHF Ratio are calculated by following equations.

$$\begin{aligned} mRR= & {} \frac{1}{N} \sum _{n = 1}^N RRI_i \end{aligned}$$
(1)
$$\begin{aligned} SDNN= & {} \sqrt{\frac{1}{N - 1}{\sum _{n = 1}^N (RRI_i - mRR})^2} \end{aligned}$$
(2)
$$\begin{aligned} RMSSDN= & {} \sqrt{\frac{1}{N}{\sum _{n = 1}^{N-1} (RRI_{i+1} - RRI_{i}})^2} \end{aligned}$$
(3)
$$\begin{aligned} mA= & {} \frac{1}{N - 1} \sum _{n = 1}^{N-1} (RRI_{i+1} - RRI_{i}) \end{aligned}$$
(4)
$$\begin{aligned} SDSD= & {} \frac{1}{N - 1} \sum _{n = 1}^{N-1} \{(RRI_{i+1} - RRI_{i}) - mA\}^2 \nonumber \\ \end{aligned}$$
(5)
$$\begin{aligned} pNN50= & {} \frac{num(RRI > 50)}{num(RRI)} \end{aligned}$$
(6)
$$\begin{aligned} LF Norm= & {} \frac{LF}{TotalPower} \end{aligned}$$
(7)
$$\begin{aligned} HF Norm= & {} \frac{HF}{TotalPower} \end{aligned}$$
(8)
$$\begin{aligned} LFHF Ratio= & {} \frac{LF}{HF} \end{aligned}$$
(9)

144 dimension feature \({\varvec{f}}\) is extracted from each datum \({\varvec{d}}\), and each feature is normalized so that the average is 0.0 and variation is 1.0. Estimator \({\varvec{M}}\) is created through machine learning, where the score(degree) of ride comfort, s, is the objective variable, and feature f is the explanatory variable. In the estimation phase, 144 dimension feature \({\varvec{f}}\) is extracted from each datum \({\varvec{d}}\). Ride comfort score s is calculated by using Estimator \({\varvec{M}}\) and \({\varvec{f}}\) as follows.

$$\begin{aligned} s = {\varvec{M}}({\varvec{f}}) \end{aligned}$$
(10)

4 Experiment

4.1 Purpose and Setup

The purpose of the experiment is to verify the accuracy of the proposed method. To verify the validity of the proposed method, Baseline1:only inertial data based method and Baseline2:only vital data based method were compared, 12 males participated (average age:28.3).

Fig. 2.
figure 2

Experimental environment

Full size image
Fig. 3.
figure 3

Experimental landscape

Full size image
Fig. 4.
figure 4

Axis coordination of the inertial sensor

Full size image
Table 3. Questionnaire for ride comfort score
Full size table

In order to collect a wide range of ride comfort scores, 4 courses were set as shown in Fig. 2. The environments were designed to alter each subjects psychological state. The narrow roads demand careful operation, while the uneven roads impose feelings of discomfort or disgust. In order to narrow course width while ensuring safety, we used paper cups. Subjects were asked not to damage/upset paper cups in Courses B and D. Woodblocks (height:4cm) and cable guards (height:3cm) were used in Courses C and D to impose a feeling of discomfort. Course D is shown in Fig. 3. Course A was designed to more clearly identify the psychological changes triggered by Courses B, C, and D.

In this experiment, WHILL(WHILL Corp.; max speed:6km/h, width:60cm) was used as a wheelchair. Acceleration and gyro data were acquired at 30[Hz] by mounting an Xperia A(Sony; Android OS 4.2) to the wheelchair with the axis setting shown in Fig. 4. RRI data was acquired by using myBeat(UNION Tool Corp.).

Table 3 shows the questionnaires used for acquiring ride comfort scores. Each subject drove around each course 3 times, and answered the questionnaires using a 7 point scale(6:very positive, 0:very negative) after circuit. The ride comfort score of each circuit, s, was calculated by the following equation, where \(s_k\) is the score of the k-that circuit.

$$\begin{aligned} s = \frac{\sum _{k=1}^{9} s_k}{9} \end{aligned}$$
(11)

12 data sets \({\varvec{d}}\) were obtained from each subject. Finally, 144 data \({\varvec{d}}_0, \dots , {\varvec{d}}_{143}\) were obtained from all subjects. Features \({\varvec{f}}\) described in Sect. 3 were calculated from each datum. Features obtained by using only inertial data \({\varvec{f}}_{inertial}\), and those obtained by using only vital data \({\varvec{f}}_{vital}\) were used for Baseline1, Baseline2, respectively. In the learning phase, estimator \({\varvec{M}}\) was created. The objective variable was the ride comfort score, s, the explanatory variable is the feature f. In the estimating phase, the correlation between obtained score and estimated score was determined by 10-fold cross validation. For learning, Random Forest Regressor(Python Scikit-learn) was used, the number of trees in the forest was 10, the number of features to consider when looking for the best split was 10.

Fig. 5.
figure 5

Ride comfort score of each course

Full size image
Fig. 6.
figure 6

Difference of the score of 1st time and 3rd time

Full size image

4.2 Results and Discussions

Figure 5 shows the averaged ride comfort score of each course. The courses (A, B, C, D) are shown in Fig. 2; the number is the number of circuits. For example, “Course C:3” is the 3rd circuit of Course C. The averaged scores of 3 times circuits are Course A:4.55, Course B:3.15, Course C:2.92, Course D:2.56. Comparing the scores of each trial of each course, the ride comfort score increased with the number of trials. As shown in Fig. 6, the differences on comfort score between the 1st trial and 3rd trial are Course A:0.78, Course B:0.96, Course C:0.80, Course D:0.43. The scores increased in later trials because the subjects became accustomed to the courses. This result showed that “ride comfort” alters with habituation. Of course, ride comfort is also assumed to depend on the road state. Course D was the most complex course and so the habituation effect was weakest in Course D.

Table 4. Correlation coefficient between obtained score and estimated score
Full size table
Fig. 7.
figure 7

Averaged importance of each features

Full size image

Ride Comfort Estimation. In order to verify the validity of the proposed method, we determined the correlation between the obtained score and estimated score. As shown in Table 4, correlation coefficient of the proposed method was 0.759, that of the baseline1 was 0.684, that of the baseline2 was 0.543. We used bonferroni correction as a multiple comparison technique. There was a significant difference between the proposed method and the baseline2 according to the t-test(t(18) = 3.93, p < 0.05 / 3). However, there was no significant difference between the proposed method and the baseline1 (t(18) = 1.31, p > 0.05 / 3). Although the correlation coefficient of the proposed method was higher than that of others, further studies are needed for validating the acceptable accuracy.

In order to investigate the contribution of each feature, the importance of each feature was calculated and compared by averaging each sensor type(x-axis acceleration, y-axis acceleration, z-axis acceleration, x-axis gyro, y-axis gyro, z-axis gyro, HRV). In the example of x-axis acceleration, the average of all x-axis features, shown in Table 1, was calculated. The result is shown in Fig. 7. “AX”, “AY”, “AZ” refer to the importances of x-axis, y-axis, z-axis acceleration data, respectively. “GX”, “GY”, “GZ” represent x-axis, y-axis, z-axis gyro importances, respectively. “HRV” is HRV importance. It can be seen that importance order is x-axis acceleration, y-axis gyro, x-axis gyro, and HRV.

We assume that there was no significant difference between proposed method and baseline1 because of the environment used in this experiment. As shown in Fig. 2, Courses C and D had rough surfaces to shake the subject. Such roughness mainly impacts the component of the x-axis gyro and y-axis gyro. Considering the high importance of the x-axis gyro and y-axis gyro in terms of ride comfort, x-axis and y-axis gyro highly contributed to estimation accuracy. As Courses B and D had limited width, many subjects adjusted their speed when entering and exiting the courses. Such manipulation characteristics altered the acceleration components on the front-back axis, which caused the high importance of x-axis acceleration. The importance of the HRV was next highest after the x-axis acceleration, x-axis and y-axis gyro. Further measurements and analysis are needed confirm why HRV features contributed to the estimation so strongly. The proposed method was validated by comparing the baselines, and the importance of both inertial and vital features was confirmed.

5 Conclusion

We classified the barriers facing users of personal mobility devices into physical and psychological barriers, and focused on the latter. As the psychological barrier metric, we adopted the “ride comfort” from Sawada et al. [4]. In order to obtain ground truth “ride comfort” data, the subjects had to reply to a questionnaire. Our goal is to estimate “ride comfort” from sensor data automatically so as to minimize users’ burden. We hypothesized that the users’ psychological state can be extracted from inertial and vital information, and we proposed an inertial and vital data based ride comfort estimation method. A verification experiment conducted with 12 subjects found that the proposed method had an accuracy of 0.75, higher than that of baseline methods that used only inertial or vital data. There was a significant difference between the proposed method and vital-data-based method. There was no significant difference between the proposed method and the inertial-data-based method because of the environment used in the experiment.

Some issues remain to be addressed. In this experiment, we used healthy subjects who do not normal use wheelchairs as the expected users of future personal mobility devices. The proposed method also can be applied to daily wheelchair users. It is unknown if the daily wheelchair users will exhibit the same psychological effects seen in the experiment. Further trials are needed with such subjects. In this experiment, the environments examined were artificial course constructed indoors. While it is difficult to perform extensive trials outdoors, such tests are necessary.