Abstract
Todays, the electric wheelchair has been an essential tool for handicapped people. It may be difficult, however for hand-impaired people to operate the electric wheelchair with the conventional joystick. For hand-impaired people, we have to provide other operations such as eye-tracking. Traditional eye-tracking method, however, is tiresome, because the user has to fix his or her eyes to the direction of the destination. In this paper, we propose an user interface with eye tracking extended by the augmented reality (AR) technology. In our user interface, the sight in front of the electric wheelchair is displayed on the PC screen. Furthermore, an oval is overlaid at user’s view point in the sight on the screen as if a search light spots focus at the specified location. The user can decide the location at the oval as a temporary destination, to which the electric wheelchair moves. Repeating the process, the user can intuitively drive his or her electric wheelchair to the final destination without the burden of operations.
Keywords
You have full access to this open access chapter, Download conference paper PDF
Similar content being viewed by others
1 Introduction
The electric wheelchair is an indispensable tool for handicapped people. It especially has a great value for ones with impaired legs. It is not, however, always easy to use for those who have invalidity with hands. Most electric wheelchairs are designed to be operated through a joystick, which requires operations with hand; therefore, other operation manners have to be given for hand-impaired people. One of such improvements of an electric wheelchair is intelligence. The purpose of intelligence is to realize safer and friendlier driving assistance by giving intuitive operations for driving electric wheelchairs.
Eye tracking is one of the most leading candidates of hands-free operations [1, 2]. In traditional eye tracking operations, users have to focus their attentions to the direction by their eyes. When driving a electric wheelchair, the user has to fix his or her eyes to the destination or goal of the electric wheelchair. It is not only unintuitive, but also tiresome. In this paper we propose a user interface that allows users to intuitively control their electric wheelchairs through eye tracking. Our electric wheelchair has a laptop PC, which displays the sight in front of the electric wheelchair with a web camera. The user can observe the sight as if he or she looks through the window.
The user interface for driving uses AR technique. AR is a technique and a method for expanding reality by adding “something different information” to human-perceivable information in the real world. In order to provide feedback to the user so that he or she can confirm the destination of the electric wheelchair, the front sight displayed on the PC includes an oval shape as a destination sign. The highlighted oval shape indicates the spot on the floor the user looks at, as if he or she plays search light on the floor. This feature let the user confirm he or she is driving the wheelchair in the right direction. Thus, iteration of AR based specification of the destination and move to it contributes to intuitively driving electric wheelchair without using hands or legs.
2 Related Work
There are roughly two approaches proposed for making of electric wheelchairs intelligent. The first approach is autonomous movement such as giving map information or recognizing surrounding environments through several sensors, so that the electric wheelchair can autonomously avoid obstacles and traveling along the wall without troublesome operations. This approach is effective for prevention of accidents due to user’s mistakes in judgment or operation errors. The second one is improvement of the user interface, which introduces an alternative method to replace the joystick which is a conventional operation for input interface. This approach is effective for users such as a hand-impaired people.
2.1 Autonomous Traveling
Many intelligent electric wheelchairs have been developed to realize autonomous traveling by recognizing the surrounding environment using multiple sensors [3,4,5]. They have infrared sensors and ultrasonic sensors as distance sensors, and touch sensors as contact sensors. Once these sensors recognize some obstacles, the electric wheelchairs calculate suitable roundabout routes. Furthermore, some systems that drive a wheelchair through visualizing a destination based on electroencephalogram [6]. In these driving systems, users specify the destination instead of the traveling direction; the specification manner reduces the burden on the user. Our user interface also aims at reduction of users’ burden in the same direction.
2.2 User Interface Improvements
Some control utilize facial expression methods through recognizing gestures of user’s face with the web camera [7], and the inclination of the user’s face [8]. Also, some electric wheelchairs are equipped with an interface that selects and manipulates buttons on a GUI (Graphical User Interface) built on the laptop PC screen with gaze [9, 10]. In addition, electric wheelchairs with a voice input interface have been developed [11]. In our proposal, we mainly improve the user interface, but we have also included autonomous driving features in two aspects.
3 Device Overview
3.1 Concept of Our Electric Wheelchair
Figure 1 shows the concept of our electric wheelchair. Users drive the wheelchair with a laptop PC on his knee. As shown in the figure, the laptop PC has an eye tracker sensor and a web camera. The web camera takes the sight in front of the electric wheelchair, and displays it on the screen of the PC. Also, an eye tracking sensor is equipped at the joint of the laptop so as it faces with the user to give the coordinates of the viewpoint on the PC screen. In order to track the user’s eye correctly, the display of the PC is inclined with right angle as the display and the user’s line of sight crosses perpendicularly.
3.2 AR Based User Interface
The sight in front of the user, which is a floor in most cases, is displayed on the screen of PC. The user can see the sight on the screen as if he or she is looking at the front sight through a physical window. On the screen, the user can see the spot corresponding to the destination of a line of sight on the floor, as shown in Fig. 2. It looks as if a small oval area of the floor is lighted up by a search light. Since the spot is on the line of sight, the user feels as if light would be emitted from his or her eyes to the floor as shown by the dotted line in Fig. 1.
Even though the destination spot moves along the line of sight, once the user determine the destination, it is not necessary to make the destination spot move. We provide a means to fix the destination. For example, the user can fix the spot through a trigger like blinking his or her eyes. In our current prototype, we tentatively use a specific key on the PC to fix the destination spot. When the destination spot is fixed, the wheelchair moves forward until the destination spot comes under its wheels, as shown in the right side of Fig. 1.
3.3 Operations of the Electric Wheelchair
We can summarize the manner for driving our electric wheelchair using our user interface as follows:
-
1.
Move the oval on the screen with the line of sight to the coordinates of the user’s destination.
-
2.
When the user matches the oval to the destination, he or she blinks one eye for 0.5 s or more.
-
3.
A signal is sent from the PC to the electric wheelchair, and the electric wheelchair move to the destination.
-
4.
Once the electric wheelchair reaches to the destination, return to Operation 1. The user can reach the final destination through repeating this set of operations. If the user wants to stop halfway, he or she blink one eye.
Basically, the operations that the user is responsible are only to determine the destination with gaze, and to blink one eye for start and stop.
Notice that eye tracking is not performed during movement of the wheelchair. In general, continuous eye tracking imposes a burden on users. In our user interface, on the other hand, the user does not need to fix his or her stare at a point for operations. Our user interface liberates the users from concentration of eye-tracking. The user can see anything during the movement of wheelchair once he or she decides the destination.
It is rare for users to directly specify the final destination, because the display of laptop PC is not big enough. Also, there may be some obstacles on the path to the destination. Thus, we have designed our user interface to let the user reset partial destinations repeatedly while approaching to the final destination.
4 System Overview
4.1 Overall System Configuration
Figure 3 is an overview of our system. The image obtained through the web camera is displayed on the screen, and the depth value obtained by the sensor is sent to the user program of the host PC. Also, the ellipse is displayed at the coordinates of the line of sight on the screen, which are obtained by the eye tracker connected to the PC. Once a gesture (blink) is recognized with an eye tracker, a signal for moving to the specified coordinates is sent to the RRC of the electric wheelchair from the user program of the host PC. RRC is a device that adjusts the contents of the communication signal between the electric wheelchair and the host PC.
4.2 State Transition of Electric Wheelchair
Figure 4 is a state transition diagram of the electric wheelchair with our user interface. Basically, when we use interface, state of our system is Run mode, and then, the state transits to output mode of (vi) in Fig. 4. In this state, the host PC sends a signal with output command to the RRC. Table 1 describes the output mode and output command.
4.3 Driving Controls
Figure 5 shows flows of data or commands in our system. In order to make an electric wheelchair move to the destination on the floor, the degree \(\theta \) of the destination for the front of the wheelchair, and the distance d from the wheelchair to the destination on the floor (DoF). Therefore, our system calculates them based on the coordinates of the destination on the screen (DoS), which is given by an eye tracker as a view point on the screen, and depth information, which is given by a depth sensor.
Once user’s Gesture is recognized, the turning direction, i.e. the left turn or the right turn, is first determined. The direction can easily be determined based on DoS. The turn off degree \(\theta \) can be calculated as follows: \(\theta = \arctan (Gx / depth)\) as shown in Fig. 6(b). Gx is the x-coordinate in global coordinates obtained from the depth sensor with depth value. As well, the distance from the Web camera (WtoD) to DoF can also be calculated using Gx and depth as follows: \(\text{ WtoD } = \sqrt{Gx^2+\text{ depth }^2}\).
Once WtoD is obtained, assuming that the height from the floor to the web camera is fixed at \(height = 0.85\) [m], as shown in Fig. 6(a), the distance to the destination can be calculated as follows: \(d = \sqrt{\text{ WtoD }^2 - 0.7225}\)
Finally, PC sends a command for turning in degree \(\theta \) and then a command for going straight distance d to the electric wheelchair.
5 System Implementation
As shown in Fig. 7, we have implemented our user interface on a real electric wheelchair. Main components are as follows:
-
A laptop PC with the proposed user interface
This is the core of the entire intelligent wheelchair control system. All the data are processed on this laptop PC, which send all the control signals to the electric wheelchair driving system (RRC).
-
Web camera with depth sensor
This is mounted on a rack at the front of electric wheelchair. The depth sensor is used to measure the distance to the destination.
-
Eye tracker
This is equipped on the joint of the laptop PC. It recognizes the user’s line of sight coordinates and gesture.
-
RRC (driving system)
This is used to supply energy to the motors of wheelchair based on control signals sent by PC.
Figure 8(a) shows a screen shot of the destination on the floor and the screen of overlaid oval. Figure 8(b) is a screen shot of the destination specified by an oval. Notice that the size of oval varies in proportion to the distance. This feature gives users intuitiveness similar to physical perspective.
Figure 9(a) shows a screen when one eye is closed and a command for fixing the destination, and the signal is transmitted to the electric wheelchair. At this time, the color of the oval turns to red, and the color is kept until the electric wheelchair reaches the destination or stops. The color representation of the oval contributes to intuitive operations. Also, Fig. 9(b) shows the screen after arriving at the destination. As mentioned above, when one of the eyes is closed while traveling, it stops and returns to the state of Fig. 8.
6 Experimental Results
In order to demonstrate the effectiveness of our AR based user interface, we have conducted experiments, and measured accuracy for arriving at the destination and tracing cranked route, and the number of executed operations.
-
1.
error for reaching point
As shown in Fig. 10, for some cases, where the direction is set to the right turn and the left turn, the distances are set to 1.2 [m], 2.4 [m], 3.6 [m], we have measured the difference of the positions between the destination and the electric wheelchair after arrival five times. Table 2 shows the results. As shown in the table, the farther the destination is, the more errors occur.
-
2.
tracing cranked route and the number of operations
Figure 11 shows the cranked route we employed for the electric wheelchair passes through, and how many times the user had to perform the operations. The dotted line is the path through which the electric wheelchair passed. Although we conducted the experiment several times, the number of operations until reaching the final destination was roughly four times.
7 Conclusion and Future Work
There are some electric wheelchairs (EWC) controlled with gaze have already been developed, but most of them lack intuitiveness or impose a burden through gazing at the direction during operation. Therefore, we have proposed and developed an ergonomic user interface for controlling an electric wheelchair. As the interface, we have adopted the operation manner to designate the destination instead of the direction to control the electric wheelchair. This reduces user’s fatigue during operation. In addition, the gaze based approach enables handicapped people to operate the electric wheelchair easily and intuitively through AR technology. However, the results of the experiments showed that Table 2 shows that specifying far positions tend to cause certain errors. Also, we have observed that complex routes such as a crank intend to require too many operations. In future works, we plan to combine our approach with the traditional obstacle avoidance algorithm in order to enhance autonomous driving. This should allow the user to reach the destination while avoiding obstacles in an unexpected environment and reduce the number of operations. Also, since the operating range of the oval is restricted on the screen of the PC, the use of wearable device would make operation for turning around easy, and enable to decrease the number of operations.
References
Arai, K., Mardiyanto, R.: A prototype of electric wheelchair controlled by eye-only for paralyzed user. JRM 23(1), 66–74 (2011)
Al-Haddad, A., Sudirman, R., Camallil, O.: Gaze at desired destination, and wheelchair will navigate towards it.new technique to guide wheelchair motion based on EOG signals. In: First International Conference on Informatics and Computational Intelligence (2011)
Miller, D.P., Slack, M.G.: Design and testing of a lo-cost robotic wheelchair prototype. Auton. Robot. 2, 77–80 (1995)
Nakanishi, S., Kuno, Y.: Robotic wheelchair based on observations of both user and environment. IEEE Intell. Robot. Syst. 2, 177–188 (2002)
Eid, A.M., Giakoumidis, N.: A novel eye-gaze-controlled wheelchair system for navigating unknown environments: case study with a person with ALS. IEEE Access 4, 558–573 (2016)
Takahashi, K., Nakazawa, M., Abe, T.: Wheelchair robot control system using EEG. IPSJ Interact. (2015). (in Japanese)
Adachi, Y., Shimada, N.: Intelligent wheelchair using visual information from the human face. IEEE Intell. Robot. Syst. 1, 354–359 (1999)
Terashima, S., Aida, N.: Development of operation device for electric wheelchair by ocular movement mechanical & control engineering. In: Proceedings of the ... JSME Conference on Frontiers in Bioengineering, pp. 1348–2939 (2006)
Ino, T., Matsumoto, Y.: Development of intelligent wheelchair system with face and gaze based interface. In: Robot and Human Interactive Communication, pp. 262–267 (2001)
Katevas, N.I., Sgouros, N.M.: The autonomous mobile robot scenario: a sensor aided intelligent navigation system for powered wheelchairs. IEEE Robot. 4, 60–70 (2002)
Simpson, R.C., Levine, S.P., Bell, D.A., Jaros, L.A., Koren, Y., Borenstein, J.: NavChair: an assistive wheelchair navigation system with automatic adaptation. In: Mittal, V.O., Yanco, H.A., Aronis, J., Simpson, R. (eds.) Assistive Technology and Artificial Intelligence. LNCS, vol. 1458, pp. 235–255. Springer, Heidelberg (1998). doi:10.1007/BFb0055982
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Ishida, S., Takimoto, M., Kambayashi, Y. (2017). AR Based User Interface for Driving Electric Wheelchairs. In: Antona, M., Stephanidis, C. (eds) Universal Access in Human–Computer Interaction. Designing Novel Interactions. UAHCI 2017. Lecture Notes in Computer Science(), vol 10278. Springer, Cham. https://doi.org/10.1007/978-3-319-58703-5_11
Download citation
DOI: https://doi.org/10.1007/978-3-319-58703-5_11
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-58702-8
Online ISBN: 978-3-319-58703-5
eBook Packages: Computer ScienceComputer Science (R0)