M2TA - Mobile Mouse Touchscreen Accessible for Users with Motor Disabilities

Abstract

This paper addresses the accessibility challenges of people with motor impairments regarding their access to the computer. Our focus is a new mouse design, which in its traditional ergonomics may affect the interaction with a computer and, consequently, with the Web. We introduce the design and development of a mobile application, the M2TA, which transforms a touchscreen mobile device into a mouse controller. The mobile application provides more flexible/customizable interfaces, it is portable, and is cheaper. Two users with motor limitations, cerebral palsy, participated in the development process of the M2TA. They used mobile interfaces interacting with computer applications of their preference freely. We aimed to observe possible bugs and receive suggestions for the M2TA improvement. We also collected their satisfaction with the use of M2TA interfaces. Preliminary results are promising and indicate a good level of acceptance. Further studies are in progress to attest the M2TA potential, such as improving the quality of life of people with neuropsychomotor sequelae caused by TBI - Traumatic Brain Injury and Stroke - Stroke.

1 Introduction

People with motor impairments are excluded from social life in many situations (e.g., school, work, and even entertainment). In fact, many daily practices are not adapted for them. Fortunately, this reality has undergone considerable changes thanks to new public policies of development and the new paradigms of social inclusion. The development of accessibility software and tools (i.e., Assistive Technologies (ATs)) has also made it increasingly possible to provide communication and access to information for people with motor disabilities independently. People with motor disabilities are closer to computers and mobile devices thanks to the emergence of assistive software and adapted devices [6].

Unfortunately, web browsing on computers is still a challenging task for people with motor disabilities [1]. Keyboard and mouse are the most common input devices for computer access. In a mouse-based interface (i.e., GUI), cursor movements follow the users’ hand movement in two-dimensional space. When manipulating a mouse, users obtain visual and kinesthetic information of movement and position in computer screen interactions [2, 3]. Previous research has already identified obstacles faced by people with motor disabilities when interacting with graphical user interfaces (GUIs) using these devices [1]. Some people cannot use this device independently because of their sensory-motor limitations, acquired or congenital [4, 5].

Researchers and practitioners of TA have proposed various types of adapted devices (e.g., BJOY, HeadDev, Tobii PCEye, RCT-Barban, BigTrack Trackball, Orbitrack, etc.), which provide the same functions of traditional mouse data input. These solutions make it possible for people with motor impairments the access to the computer resources (e.g., Web browsing, text editor). But, frequently, these devices have high prices. Some of them do not have a good quality of finishing and design. They also suffer from the scarcity of distribution channels, maintenance, and repair places [12, 13].

One of the little known functions that smartphones and tablets have is to enable the device to be a desktop data entry component. An application running on the device is connected as a client to an application server computer. This allows the smartphone to send data as a wireless mouse, for example. There are a wide variety of applications¹ that can be used for this, but they are not necessarily interfaces that meet the direct needs of an individual with motor impairment [22].

Following this trend, our proposal aims at the design and development of layouts that simulate an accessible mouse on the touchscreen of mobile devices. This mobile application controls the computer’s cursor over wireless communication. The proposed solution, M2TA, is a more flexible and customizable tool than the adapted hardware. The M2TA has greater portability, less financial cost, and easy access. By offering several layouts, the M2TA also allows each user to choose the input interface best suited to their limitations.

People with motor disabilities and professional specialists in the area of rehabilitation have participated in the design and development process of M2TA. One of the main design challenges was the haptic feedback, such as that provided by controlling the standard mouse. For this, we use haptic devices - levers and other types of joysticks - over the smartphone touchscreens. Initial reports from users and experts indicate a good acceptance of M2TA. The diversity of mouse layouts is the M2TA more pleasured feature. Initial test results show its likely use in improving the quality of life of people with neuropsychomotor limitations [6].

2 Related Work

2.1 Adapted Input Devices

Input devices available on the market (e.g., mouse [7], trackball [8], joystick [9], light pen [10], touchscreen [11], etc.) differ in shape, size, mode, and control. The computer mouse requires motor skills such as the use of hands and fingers. In this context, people with motor impairments may face difficulties when using the mouse in its standard form. Some adaptations are applied based on the users’ functional abilities to facilitate the interaction with computers. For instance, accessible mouses offer alternative data input modalities based on pressing, traction, blowing, blinking, and contracting of users musculature [19]. The use of the foot, mouth, or actions by the displacement of the head, and movement of the gaze are examples of new input interactions.

Each of the above solutions addresses a different strategy for decreasing the limitation that physical or motor impairment imposes for the total or partial use of the mouse in its conventional form. Some examples of Assistive Technologies are listed below. They offer better possibilities of interaction and support the process of digital inclusion of people with motor disabilities.

• Joystick: A pointing device in which the movement of the cursor is controlled through stick which the user can move in all directions and reproduced on the computer screen. It has two or more buttons for performing the mouse activation functions. Examples: BJOY², Talking Joystick Mouse³ and Joystick - For Chin⁴.

• Mouse controlled by the movement of the head: a resource used by people who can not use the mouse and the keyboard with the upper or lower limbs, but can move the head with some control. It is currently possible to control the mouse cursor with the movement of the head captured by a standard webcam (for example, HeadMouse⁵, HeadDev⁶ and TrackerPro⁷).

• Mouse controlled by eye movement: suitable for users who can move their eyes and stare at specific points in the screen. No head movement is required to promote the cursor movement (e.g., Tobii PCEye⁸ and Mouse Eyepiece⁹).

• Mouse controlled by the lips movement and activated by blow or suction: it allows the user to operate the computer entirely through mouth. Small lips movements enable the cursor movement in the computer screen movements, while left and right clicks are done by a light blow or suction, or even by an external trigger. One of the best known mouse in this category is the Integra Mouse¹⁰ designed for people with high spinal cord injury (quadriplegia), progressive muscular dystrophy, neuromotor diseases and multiple sclerosis.

• Mouse controlled by the feet: the activation of functions and the cursor control is performed using the feet to facilitate the use, to increase the functionality and to reduce the adaptation time for users with severe motor limitations in the upper limbs (Example: Roller Mouse¹¹, Button Mouse¹² and RCT-Barban¹³).

• TrackBall: a device with a ball, of 7 cm in diameter on average that allows the cursor movement the screen requiring less motor control by the user. It has two large buttons with functions equivalent to the left and right keys of the conventional mouse with attractive looks and bright colors (e.g., Big Ball Mouse¹⁴ and BigTrack Trackball¹⁵).

• Touch sensitive: allows the user to accurately control the cursor speed and direction on the computer with just a soft touch. It requires the minimum effort and motor coordination to use the mouse. The touchpad mouse¹⁶ and Orbitrack¹⁷ are examples of this type of mouse. The last one has a control ring and does not require extensive movements of the hands and wrists but only the touch of the finger.

The assistive technologies listed in Fig. 1 are the most relevant to our study. However, many other products targeting users with motor impairments are available. Frequently, the choice of the most adequate approach depends on its economic cost. In general, Assistive Technology products are expensive due to their limited market. This inconvenient, in many cases, makes them unattainable for most of the users with disabilities.

Fig. 1.

Examples of adaptations: (a) RollerMouse, (b) BJoy, (c) Button Mouse, (d) Touchpad.

2.2 The Use of Smartphone as Input Devices

This subsection presents design and implementation research to turn smartphones into remote computer controllers, allowing users to use smartphones as mouse and keyboard by operating the computer remotely and wirelessly.

The advantages of using smartphones for this purpose are mainly in the ease of move because it is a small device and the possibility of use with only one hand. Also, they have a great potential for connectivity (Wi-Fi, Bluetooth and Internet access), good human-computer interface and with extensive popular use, being an excellent alternative as remote controllers for the computer.

The Accurate Air Mouse (AAMouse) [14] tracks device movement in real-time, allowing any mobile device with a microphone, such as a smartphone or a smartwatch, to serve as a mouse controlling an electronic device with speakers. In this way, low acoustic signals are emitted and the mobile device records and send to the emitter, which can estimate the position of the device based on the Doppler effect. Then the distance between the loudspeakers and the starting position of the device is calculated based on the sound frequency.

The SMTFController [15] describes how to turn smartphones into remote computer controllers. It can be used in classrooms and meeting/conference rooms for presentation and interactive discussion. It also allows users to use multiple smartphones to operate the same computer, facilitating group discussion and classroom interaction.

Some researchers present methods to control the mouse from the computer through real-time speech recognition using a smartphone [16]. The virtual mouse operation maps spoken commands using the device’s microphone to the mouse computer movement command. The purpose of this method is to control different digital devices by voice. The Tongue Drive System (TDS) [17] is a wireless wearable assistive technology that allows people with severe disabilities to control their computers, wheelchairs and electronic devices using the tongue movement. TDS translates the user’s language tracking to drive commands that will be read by a bluetooth module and then embedded in the smartphone. It can act as a mouse driver on other devices connected to the network with the solution.

There is also a method for using the finger movements to interact with the smartphone screen and to control the computer mouse. The application design resembles the trackball and uses scanning techniques to improve the entry speed and to reduce the user’s fatigue [18].

3 M2TA Solution

This research focuses on the development of one more mouse alternatives that allow computer access by people with motor disabilities. It is a portable touchscreen version of the conventional mouse. As seen in the related works section, researchers have already proposed various hardware solutions as an alternative to the traditional mouse. In this context, our research draws on these works by adopting metaphor input interfaces similar to those input devices (e.g., RollerMouse, Touchpad, BJOY, and Button Mouse).

In our research the mobile device functions as a computer input device. The smartphone or the tablet is the support for the execution of interfaces that simulate those input device layouts. Thus, each user can choose the type of mouse that is best suited to their needs and abilities [20]. Also, the user does not need to acquire new hardware devices, which often is difficult to maintain and expensive.

3.1 Architecture

The application architecture consists of a client (Android Application) that runs on the user device and a server (Java Application) that runs on a computer. Client and server communicate via wireless technology (i.e., Bluetooth or Wifi). Thus, the server interprets the commands sent by the Android application and generates events as standard mouse commands (Fig. 2).

Fig. 2.

M2TA main components.

3.2 Mobile Application

The application interface adapts itself according to the screen size of the device. The intention is to take advantage of the available space. The user can also switch to other input interfaces in the settings options menu. We developed four kinds of input layouts: M-Roller, Mjoy, M-Button, and MTouch. Figure 3 shows our proposal.

Fig. 3.

M2TA layout interfaces

MTouch detects the movements and gestures of the user’s finger on the mobile touchscreen. This Android application translates and transforms gestures to the cursor movement on the computer, mirroring the user’s action on the computer display. For instance, one-finger touch triggers the mouse click. Two-finger touch throws double-click on the computer. Three-finger touch is equivalent to a middle-click. Finally, a long touch on the mobile screen means a right mouse click on the computer.

For the mini-joystick layout (Mjoy), the application does the translation of the torque direction and force performed by the user. For this, we define a ratio of motion to the cursor (1–100 pixels per movement). This design has the advantage of being as intuitive and easy to learn as the touchpad.

The MRoller interface has only horizontal and vertical scrolling of the cursor. It also offers four distinct buttons with click functions (Table 1). Half-click and double-click buttons are the most helpful for the user with motor disabilities.

Table 1. M2TA feature and corresponded computer event.

The M-Button layout is a “scan” mouse. It has four buttons to perform the cursor movement - to the right, up, down, left. It also has four buttons with distinct click functions and a centralized button to configure the scan function. The scanning on this interface works by alternating between motion and function options, varying every 5 s. Once the user has chosen one of the two options, he can switch between the four buttons every 5 s. For example, when starting MButton, the user chooses the group of directional buttons he wants to interact. Then, the application waits for the user’s touch for five seconds. If the user touches anywhere on the screen, then the computer move the mouse cursor up. If not, the device waits another five seconds enabling the right arrow as the standard touch.

Users can also use these distinct M2TA layouts with their feet if they have sufficient motor control for the task. Our approach already reached the final prototyping process. Initial usability studies are in progress, some them are described in this paper.

The actions performed in the mobile application and sent to the server computer are: (1) directional cursor movements controlled by the virtual buttons or the joystick; and (2) mouse features represented by colored buttons (Table 1).

4 Initial Usability Evaluation

Four groups of specialists and two users with motor disabilities (in this case, cerebral palsy) participated in the development process of M2TA. They interacted freely with desktop applications using M2TA. Our goal was to seek for application’s errors and receive suggestions for M2TA improvement. Besides, we observed their satisfaction, comfort, and the resulting impacts in the computer access experience using M2TA solutions. The evaluation was exploratory and aimed at obtaining information of qualitative nature, gathering the participants’ perceptions.

4.1 Participants

Expert Users.

We evaluated the proposed technology with four groups, each one composed by two or more professionals. They are from well-established health and education institutions in South America, such as: Instituto Teletón Chile (Assistive Technology Sector - Santiago Unit), SARAH Network of Rehabilitation Hospitals (Bioengineering Sector - Brasília Unit - Brazil), Federal University of Ceará - Brazil (Physiotherapy Course), and Estácio of Ceará University Center - Brazil (Physiotherapy Course).

End Users.

We tested M2TA with two participants with cerebral palsy (1 male, 29 years old, graduated; and one female, 26 years old, artist). They volunteered to participate in the tests. Both users had motor disabilities and they considered smartphones and tablets as relevant devices for them to use. The inclusion criteria for participants with disabilities included a self-reported diagnosis of disability, which in both users affected the upper limbs movement function. Also, they reported difficulties in using the standard mouse. During the first interactions, we observed mainly the users’ motor limitations included a lower range of hand movement, difficulty in motor coordination and body tremor.

4.2 Materials

During the evaluation sessions, the testers ran the M2TA on their own devices (mobile device and computer). The objective was to maintain the users’ device preferences and to observe the M2TA performance within a real context.

4.3 Procedure

We conducted sessions involving the experts at their respective workplaces. Each session lasted up to 60 min and had four phases: (i) technology demonstration (10 min), (ii) application usage (10 min), (iii) practices of SSPT (Single Switching Performance Test) software¹⁸ (10 min), and feedback and discussion (30 min).

With motor impaired users, we applied a free exploration test. Initially, the test observer described M2TA and installed it on the users’ devices. After that, they explained to the user the test purposes (i.e., to verify satisfaction, insights, and suggestions for improving M2TA). Each user was invited to use our approach freely (e.g., access a text editor, browse the Web). Additionally, we left a tablet with them during one week to use M2TA in their houses. Finally, we conducted an unstructured interview to get their feedback and insights (Fig. 4).

Fig. 4.

Session with an user with motor disabilities

4.4 Results

Observations and Suggestions Reported by the Experts.

Experts stated that M2TA benefits from the “enchantment” factor of smartphones and tablets. These devices are far more seductive than non-digital hardware or other analogic assistive technology. Also, they confirmed that having layout options in the same software is very advantageous for this kind of users.

The experts considered the use of M2TA simple and without great cognitive efforts for their learning. They noticed users could touch on the desired feature directly. In fact, touchscreen devices are valuable to people with low muscle strength, since these users can interact with touchscreen devices without much effort. One of the experts noted that M2TA has some features similar to Augmentative and Alternative Communication (AAC)¹⁹ tools. Although the central idea of M2TA is to replace the traditional mouse, it also becomes useful to enhance communication of people with motor disabilities. In fact, these people can use the M2TA independently, coupled to the communication software installed on the computer.

Another highlight point is the low maintenance cost of M2TA when compared to existing hardware solutions on the market. In fact, after some time of use, electromechanical materials began to get heavier. They need lubrication, and their extremities started to show signs of corrosion, originating from hands/feet sweat and users’ saliva. In contrast, M2TA is only dependent on the smartphone platform, which is easier to replace and maintain.

As negative aspects, experts have reported the disadvantage of having two regions of user’ visual focus (i.e., a user has to pay attention both the M2TA interface and the computer screen). This can be challenging for some users. Although, some of them may learn how to use M2TA without looking at the mobile interface over time. Experts have also indicated a possible fall risk of the device. In fact, interaction with the mobile device may cause it to fall when it is placed on a non-fixed surface. Another point to be studied and improved is the button sensitivity. It is common the occurrence of accidental touches, which activate unwanted functions as the devices screens are very sensitive.

We have accepted some of the experts’ suggestions for implementing a new version of M2TA. For example, we standardised the colours of the various M2TA layouts to make it more comfortable to use. Another potential use is the integration of M2TA with games to improve the physical and cognitive skills of patients in rehabilitation. Another suggestion was to combine M2TA with other assistive technologies, for example, wrist weights. The goal is to help reducing the lack of motor coordination of the user. To do this, we will need to carry out a more in-depth study with patients and rehabilitation professionals.

End-User Feedback.

In general, the two users were able to complete their activities on the computer (e.g., Web browsing, Facebook access, Text editing by using Windows virtual keyboard, multimedia player control, click and drag games). In the beginning, users needed the help of third parties to explain the M2TA operation. After that users performed the tasks without problems.

Overall, users enjoyed the experience of using M2TA. They stated that the four entry layouts in the same application is a great innovation. They were able to experiment all input layouts, and, thus, choose the most appropriate support for controlling the mouse cursor on the computer. The users considered that M2TA reduces the problem of access and transportation of a assistive technology, device, once it is a mobile application. They also stated that it made it possible to use our solution with their feet or even a little further away from the computer due to wireless communication.

They emphasized that the size of the buttons is relatively large, which reduced errors resulting from the interaction. Mistakes occurred, eventually, when they moved the joystick or selected an unwanted button involuntarily (e.g., the scan button on the M-Button interface). The researchers observed that the gap size between interface elements did not impact users’ performance.

Regarding the MJoy interface, the two users commented on the lack of precision of the virtual joystick. They also mentioned the difficulties in controlling the joystick because, with the pressure exerted to manipulate it, the device moved over the screen. In contrast, they had no problems with the return of tactile sensation. As an improvement, one of the users suggested that the smartphones physical volume buttons could also adjust the mouse speed control.

4.5 Discussion

In general, the evaluators, experts and end users, felt confident about using the M2TA and its functions of pushing buttons and sliding the mouse cursor. They were able to perform such interactions on the mobile device touchscreens. We observed that physical and technical differences between mobile devices have impacts on the use of M2TA. We realized that users have become more aware of their possibilities and limitations in interacting with the computer. Also, the initial tests allowed us to establish the advantages and disadvantages of each of the proposed mouse layouts.

A future performance evaluation (e.g., time and accuracy) may point out to some distinctions among each M2TA layout. However, such data alone should not be critical to a user’s choice of mouse design. These measurements depend on the experience of evaluators and the initial performance of their motor skills. For continuous and long-term use, users with motor impairments must choose the more appropriated assistive technologies resource in conjunction with a team of experts.

The evaluators pointed out that the financial cost of M2TA is not a significant barrier to its adoption since the average price of the mobile device is similar or lower than those of several accessible mouses on the market. Besides, smartphones and tablets are general purpose devices, and users can benefit of other features when they are not on the computer.

As a relevant result, we realize how people with motor disabilities interact with touchscreens and how this can allow designers and engineers to improve the usability of this type of interface.

Concerning the threats to validity, our initial analysis cannot yet be generalized, since the number of users was limited. However, the results provided new ideas and suggestions that will allow the improvement of our approach. The study examined only the interaction of users during the prototyping process of M2TA. This study also did not measure the impact of additional and unwanted touches (i.e., a rate of unintentional errors). In this sense, we need to implement more specific assessments of usability and performance of M2TA.

Due to sampling limitations, the results do not provide a definitive interpretation of the advantages and disadvantages of each M2TA mouse layout. Also, we can not say that the M2TA outperforms other accessible mouses devices regarding their comfort and input data rate.

5 M2TA Improvements

After this initial analysis we sought to use other haptic devices coupled to the mobile device screens. We combine other models of joysticks with M2TA. A new feature adapts the mobile application interface according to the place the user connects the physical joystick on the screen. Figure 5 shows some of the new supported joysticks.

Fig. 5.

New supported joysticks

We began planning M2TA usability assessment sessions based on Fitts’s law [21] - commonly referred as the paradigm in input device ergonomics. We have developed a tracking application to facilitate the analysis of the interface versus mouse cursor path. This will allow to identify which mouse layouts get the best performance results for each evaluator. Figure 6 shows an example of the execution of the tracking software that we developed. The evaluator has to select in the software ten objects of different sizes that appear in distinct positions on the screen. We define the distribution of points on display based on the use of a Website.

Fig. 6.

Examples of track and logging developed for future evaluation sessions

Our tracking software represents a typical demand of those who interact with the computer, that is, select objects in different screen places by pointing and clicking. If the participant hits the target correctly, the software provides positive visual feedback. After that, it displays the next objective. If the click fails, the software also shows the next point to be clicked on the screen. Timestamp, spatial information of cursor movements (x, y coordinates in pixels), and the button actions are recorded in a textual log. In the end, the software generates an image with the cursor tracking lines traversed during the navigation (Fig. 6).

6 Final Considerations and Future Work

The preliminary results obtained in this study suggest a high acceptance by the users and experts in motor disabilities. The two users with motor disabilities indicated a good satisfaction with the M2TA use and they presented an initial acceptable performance for their routine adoption. They also have caveats and suggestions for improvements that are in progress. Within this context, the proposed system is promising and still has many features to improve.

Our first future work is to implement the improvements highlighted using artificial intelligence techniques to learn how to use the system. Also, we are planning a usability assessment with a significative number of participants. The goal is to measure the performances of each layout and captures the new feedbacks and suggestions. For that, we will use the tracking software showed in Sect. 5. The sessions will also use the SSPT (Single Switching Performance Test) software.

In the long run, we plan to use M2TA in the rehabilitation of people with neuropsychomotor sequelae caused by TBI - Brain Injury and Stroke - Stroke. The idea is to combine the M2TA with games that only use the mouse²⁰.