Keywords

1 Introduction

Mobility serves as a key issue for many people to guarantee independence in everyday situations and to participate in social and working life. However, the ability to operate a vehicle can be limited due to high age or physical impairments caused by disease or accident. The ability to drive despite any physical limitations can be maintained by suitable modifications of the vehicle. Even if physical constraints need to be considered, the driver must remain in full control over the vehicle. Not only persons with physical difficulties find themselves confronted with several issues when driving, but also physically fit young drivers with a lack of driving experience form a risk group regarding traffic safety. The manual operation of a vehicle can be increasingly difficult depending on the vehicle operator’s degree of the limiting factors, the implemented operating elements and the individual training level. Comfort, accessibility and safety in driving could therefore be increased by recent developments in automated driving and the broader availability of advanced drivers’ assistant systems. In today’s traffic, some vehicles are already capable of driving at least partially automated. Thus, in some vehicles drivers can choose certain maneuvers to be conducted by the automation system depending on the current driving situation. Moreover, such vehicles can, for example, intervene by initiating a braking maneuver in a dangerous driving situation. The functionality of the automation is meant to be influenced by the vehicle operator in partially and conditional automated driving, thus, the vehicle operator may execute a driving maneuver with an intuitive command or gesture. The idea of the vehicle operator controlling the automated system by gestures origins in the H(orse)-Metaphor, where the natural example of rider (or horse cart driver) and horse is used to describe the role and interaction between a driver and an automated vehicle [6]. Generally, this concept can be described as cooperative guidance and control [7], an extension of the shared control concept. To analyze different modalities for input gestures, a multimodal human vehicle interface for conducting primary driving tasks, this paper presents an experimental study. The experiment is conducted in a static driving simulator at RWTH Aachen University. The driving simulator is based on a professional driving simulation software SILAB and self developed software modules emulating a vehicle automation [1]. A concept of three different modalities was developed as input method for the activation of a maneuver under consideration that vehicle operators with different levels of impairments will be using the system throughout the research. The conducted study presented in this research considers non-impaired operators as a proof of concept. The HMI concepts in use are a haptic steering wheel operated by hand (contact), a sidestick operated by hand (contact), and a touchscreen operated by hand (contact). The research is subdivided into multiple packages where the presented results are covering an investigation of input gestures using a haptic side stick with different guidance transition methods. The automated system gives visual and audible feedback about the current state of the input and visual-only feedback about the driving state. Visual feedback was integrated as trajectories directly in the simulated world as well as feedback on a mid-console display next to the vehicle operator. The gesture-based control of the driving maneuvers was developed in an iterative design process under participation of the target group of impaired and non-impaired vehicle operators. The study was conducted with untrained non-impaired participants as a first user group to validate the consistency of the driving maneuvers where the different modalities for transition between driving modes are set as independent variables.

2 Related Work

The use of gestures for steering a vehicle is particularly eligible to be used in combination with automated driving maneuvers. For that purpose the driver should be able to intuitively give commands in automated driving while user input should intuitively be understood by the automated system without the driver having to learn circumstantial explanations. The driver’s input has to be correctly interpreted by the system in critical and non-critical situations likewise. Recent studies about in-car interfaces using gestural input merely suggested to be used for secondary tasks. Secondary tasks are defined to have no relevance for the driving task i.e. are tasks that are not critical, e.g. interacting with a multimedia interface for changing the radio station or change the volume of a music player. Cairnie et al. (2000) developed a prototype finger-pointing method for operating secondary controls [5]. They replaced the physical controls by a computer interface and thus achieved to situate the interface much closer to the driver’s normal line of sight. The interface is operated by pointing gestures that are processed by a computer vision system. The system implicates a gain of safety in dangerous driving situations since driver distraction through the operation of operating secondary tasks while driving is a major cause for accidents. Another system using gestural input for in-car secondary tasks was developed by Zobl et al. [12]. The system’s concept allows drivers to effectively operate on a variety of multimedia and infotainment tasks with hand poses and dynamic hand gestures. The gesture inventory consisted of 22 dynamic gestures which were grouped to twelve gesture classes which were e.g. pointing, kinemimic gestures (e.g. waving to the left/right/up/down), symbolic (e.g. ‘pointing’ for “engage”) and mimic (e.g. ‘lift virtual phone’). Handposes like ‘grab’, ‘open hand’ or ‘relaxed’ were added to the inventory to allow additional functionality inside the user interface. These gestures enabled drivers to operate on a navigation system as well as multimedia and communication devices. The system was investigated for its recognition rate where the results show that the gesture recognition worked very well for both handposes and dynamic gesture recognition when it was adapted to a single user. [12] Zobl et al. suggest that a gesture controlled in-car human machine interface should be part of a multimodal interface, i.e. the driver should have choices on selecting the best suitable modality for an appropriate situation while driving a vehicle. Other concepts provide interaction commands to be conducted on the steering wheel. Angelini et al. developed a prototype for tangible gestures on the steering wheel for in-car natural interaction [3]. Pressure sensors in the turntable of the steering wheel are used to detect gestural input of the vehicle driver. However, no haptic feedback was given through the steering wheel for a confirming input for an input gesture since the input was used to conduct secondary tasks. Bach et al. [4] suggested not to use tactile or haptic feedback for primary and secondary tasks since tactile force is already applied on the steering wheel by the road. Thus, the tactile feedback channel is already allocated by primary tasks and should not be occupied by other tasks to distract the driver (Fig. 1).

Fig. 1.
figure 1

(Shared and) Cooperative guidance and control [7]

Full size image

Kienle et al. [9] developed a concept of automated driving to provide an active side stick for gesture input for primary driving tasks in automated driving based on the so-called H-Metaphor [6, 8, 10, 11] (cf. Sect. 3.1). The concept suggests to establish haptic communication between the driver and the automated system which is contributing benefits that cannot be achieved through a conventional interface system. A first user study indicated that using a force feedback side stick is a promising implementation to realize the idea of the cooperative concept to be used in vehicles.

3 Method

3.1 Human Machine Interface

The human machine interface system used in the driving simulator applies a version where different levels of automation can be picked by the driver to match the desired driving experience. The H-Metaphor metaphor by [7] constitutes the basis of the gestural human machine interface to control the automated system. In this specific case the driver uses an active side stick to interact with the automated vehicle which allows to control the vehicle’s automation laterally and longitudinally at the same time using the same input device. This type of input is common for altered barrier free cars to fit the requirements of drivers with impairments. The H-Metaphor’s paradigm origins in nature and describes a transitive relationship between the human driver and the system in which the system behaves like a horse attached to reins. The metaphor can be interpreted to the human driver who drags the reins tighter if more control over the vehicle is desired and vice versa. This metaphor maintains different modes of control which conform with the levels of automation defined by the Society of Automotive Engineers and can be switched while driving (Figs. 2 and 3):

  1. 1.

    Tight Rein. Conforms approximately with SAE level 1: Driver Assistance e.g. with lateral or longitudinal assistance.

  2. 2.

    Loose Rein. Conforms approximately with SAE level 2: Partial Automation e.g. with lateral and longitudinal assistance.

  3. 3.

    Secured Rein. Conforms approximately with SAE level 4: Conditional Automation or autonomous driving expecting the driver to intervene in certain situations.

Fig. 2.
figure 2

Simulator environment: clear section example

Full size image
Fig. 3.
figure 3

Support and automation scale [1]

Full size image

Interaction Modalities. The conducted study incorporates the investigation of two different interaction methods, each having a specific method for communicating a transition between different levels of automation. They both are conversed using a haptical side stick as input method and originate from the H-Metaphor. The driver is allowed to switch the modes of automation while driving but a limitation or boundary of driver control is implemented. The boundary typically occurs when driving situations become unclear and decisions have to be made or when the driver manually switches the level of automation. In case of an unclear situation the system communicates its boundaries to the driver when she or he is driving in Secured-Rein mode and switches to Tight-Rein mode in one of the two described ways (Fig. 4):

Fig. 4.
figure 4

Simulator environment: urban section example

Full size image
  1. 1.

    Tactile feedback is given and the system conducts the transition automatically. The human machine interface visualizes an animation which signals the transition in between two seconds.

  2. 2.

    Tactile feedback is given and the system awaits input of the user to perform the transition. If no action is taken the vehicle slows down until a complete stop which only can be prevented by manually switching the mode from Loose-Rein to Tight-Rein. The allowed transition time is limited to two seconds before the vehicle slows down to zero.

The transition is carried out automatically by the system in a predefined situation in each of the specific modes. This is announced by a the described vibration of the side stick and by a short animation on the human machine interface, in which the button of the new mode is automatically stored in blue. Both interaction variants are automatically switching back to the Loose-Rein mode when the system is pulled out of the unclear situation. A subsequent change to the Loose-Rein mode is also possible at any time if the system has not exceeded the system limits. The user was allowed to perform a manual change from Loose-Rein and Tight-Rein at all times. The situations that are investigated in the study are the passing of cross road sections.

Fig. 5.
figure 5

Simulator environment: hardware implementation and setup

Full size image

3.2 Driving Simulator

Simulation Environment. The software SILAB which is developed by the Wuerzburg Institute for Traffic Sciences (WIVW) is used as a traffic simulation environment and deployed in a static driving simulator concept with a seat box. The tools provided by the software facilitates to simulate traffic scenarios including the design of the road network, the programming of other road users and the landscaping so that a realistic riding experience can be established in any multi display setup. The simulation software was extended by a ROS interfaceFootnote 1 which is used for communication between the simulator software and hardware components that are installed in the seat box. The automation which conducts the conditional driving maneuvers is a software module which operates the vehicle in the simulation environment and is also connected through ROS. The simulation environment is capable of recording environmental data and driver data and allows precise statements concerning e.g. lane keeping, acceleration, distance to the car in front, capacity to react and the exact driving time (Fig. 5).

Road Structure. The applied test route is split into an urban and non-urban section. The software allows to define track layouts easily as they are written as a text file whereby very long, monotonous roads can be created quickly. A visual editor can be used for complex traffic situations like they were used in the applied urban scenario. The software emulates real road courses based on satellite images and provides realistic road sign setups to emulate a preferably realistic user experience of a traffic scenario.

4 Experimental Design

A factorial design was used where the two different modalities of feedback in automated driving mode described in Sect. 3 are set as independent variables. The drivers were meant to drive the complete track manually by also using the side stick which is set as a third independent variable. A questionnaire developed by Altendorf et al. [2] was used to provide dimensions for dependent variables. The dimensions taken into account are “Perceived Utility”, “Perceived Safety”, “User Satisfaction” and “Perceived Usability”, which later were investigated for correlations and difference of mean values. Before the test the participants gave information about demographic data, driving performance and experience with driver assistance systems. The items are evaluated used a Likert scale with up to seven evaluation steps.

4.1 Use Cases

The use-cases are intended to represent a balanced mix of realistic driving situations that automation can handle, and those that can no longer be handled by the automated system. The changes between the H-modes Loose-Rein and Tight-Rein should be forced by reaching certain points on the track. In order to achieve a comparable situation for all subjects, a forced route guidance by directory signs is necessary. To achieve a more realistic design, oncoming traffic is added on the route. In order to create confusing or complex forced situations, crossroads in particular have been selected in several variants. Thus, a takeover situation for the test persons is to be enforced in order to collect data on the functionality and user-friendliness of these transitions. In total, 17 transitions are artificially brought about on the test track of the urban scenario. The driving situation took approximately 20 min. The use cases can be divided into three groups. The first group includes crossings and bending situations. The second group can be titled obstacle. The last scenario is a two-lane roundabout.

Experimental Phase. The study was conducted in the laboratory facilities of the Institut fuer Arbeitswissenschaften (IAW) in Aachen. After the welcome and explanation of the test procedure by the study leader and the declaration of consent of the test person to participate in the study, the first questionnaire for the collection of personal data and the experiences with driver assistance systems was completed. After an explanation of the side-stick and the interface on the touch screen, the seats and side-stick were adapted to the needs of the test person. Subsequently, the participants was asked to take the test seriously and to stick as well as possible to the road traffic regulations. The driver should familiarize himself with the side-stick in the first drive, meantime no driver assistance systems have been available yet. This trip lasted between ten and fifteen minutes, depending on the speeds traveled. During the journey, the study leader logged the behavior and statements of the participants. The test was evaluated by the test person with a questionnaire afterwards. In the second ride the function and the handling of the activated driver assistance system was explained. The drive led the participant through a city center scenario, a country road and a highway. The study leader has recorded comments and the behavior of the test person while driving. This trip was evaluated with the questionnaire, which questions were arranged in a different order. Then the journey was repeated on the same route with the second test system. After the third trip, there was a short interview of the participants in which they could express subjective impressions and feelings and system improvements.

5 Results

The pilot study was conducted with twelve participants aged between 23 and 41 years (mean = 27, SD = 4.8 years). All participants owned at least a valid German driver’s license for regular cars (European type B). In average, the participants had 8.7 years of driving practice with a standard distribution of 4.8 years. Their median driving distance was 6000 Km per year with nine of the twelve participants (75%) driving at least on a monthly basis. Some participants used driving assistance systems such as front collision warning systems, cruise control systems and parking assistance systems on a regular basis, while others reported only the use of GPS systems and standard features such as anti-lock brakes (ABS) and stability control systems (ESP). Due to the sample size, we conducted the statistical analysis using the Wilcoxon signed-rank test. The perceived control significantly differs between the two driving blocks (p = .045). For both driving conditions, the scales for “Perceived Safety”, “User Satisfaction”, “Perceived Usability” and “Perceived Control” correlate with “Perceived Utility”. This can be interpreted as indication that the overall perception of the respective system is consistent for each individual driving block. Notably, “Perceived Safety” and “Perceived Utility” do not correlate for the manual driving block.

Table 1. Correlation of the scales for perceived safety, user satisfaction, perceived usability, and perceived control with perceived utility.
Full size table

In general, the participants reported a positive attitude towards the use of an automation system during the interviews after each driving block. This result is fully consistent with previous studies conducted with a similar automation approach [1]. All participants were able to perceive (feel) the provided haptic feedback. The feedback was individually adjusted for each participant in such a way that it could be well perceived without being too intense. Nonetheless, the automation setup with a fallback to the human driver on a two second notice received lower ratings. As a reason for this, the participants reported that the harsh fallback, even though they anticipated the situation in most cases correctly, had a negative impact on their judgment. When analyzing the actual driving data, no significant impact regarding safety between the two test conditions can be found. In this experiment, the fallback to the human driver with a warning of two seconds did have an impact only on perceived safety, not on actual safety. An explanation for this is that the participants were familiar with the system behavior and the danger of degradations in the level of automation. Thus, in real driving situations, an impact on actual safety can be expected due to the limitations in the ability of human drivers to take over control in such situations.

6 Discussion and Conclusion

Joysticks or side-sticks are among the most commonly used input devices for impaired drivers. In this paper, we focus on such interfaces and test the users’ interaction with it in combination with an automation system. In a pilot study with non-impaired participants, we find that drivers who showed a positive attitude toward manual driving with an active side-stick rated the automation slightly worse in comparison to participants who were more skeptical in using an active side-stick. Most notably, the participants mentioned that controlling two degrees of freedom with one single input device led to imprecise steering actions. This indicates that degree of automation and the input device might influence each other, and that with automation, innovate input technologies might emerge. On the other hand, if these actuators also have to be used in manual conditions, they have to be designed in such a way that human drivers have a chance of taking over control when and if necessary. Especially in safety critical driving situations, such as driving through complex intersections, drivers might want to see a clear advantage in using automation technology before expressing an intention to use it. In the case of our pilot study, the participants were not used to the presented driving interface, i.e. side-stick, which might intensify a certain critical attitude towards the entire system. Future research would focus especially on the design of adequate HMI systems and user requirements regarding driving automation for people with special needs. A follow-up study will be conducted with participants who might already be more used to driving with joysticks from their driving experience in special cars. This also implies that the simulator will be equipped with the in the group of drivers with special needs more commonly known joysticks instead of sidesticks. Future research can also look into an extension of the available control gestures.