Keywords

1 Introduction

A computer mouse is an established input device for computer users. A conventional mouse, however, only allows simple actions such as clicking its buttons, rolling the wheel, and moving it. To enrich the input vocabulary of the mouse, many previous researches have tried to augment it with new sensing capabilities. As a different approach, we have augmented the mouse pad instead of the mouse.

In this paper, we present “Sinkpad”, an augmented mouse pad that has a malleable surface consisting of an elastic material and that augments mouse functionalities by allowing the user to sink the mouse into the pad and tilt it on the pad (Fig. 1; a demonstration of Sinkpad is in [9]). It allows the user to perform interesting actions: sink, tilt, and sink+move. In addition, the pad provides the user with haptic feedback as the mouse deforms its surface. Moreover, the user can also use Sinkpad as a conventional mouse pad, because the pad serves as a flat surface unless the user sinks the mouse into it.

Fig. 1.
figure 1

(a) Using Sinkpad as a conventional mouse pad, (b) sinking a mouse into the pad.

Full size image

We conducted user evaluations on two-dimensional (2D) pointing and gathered feedback. First, we conducted a preliminary experiment to investigate the 2D pointing accuracy. The results show that users can perform 2D pointing as accurately on Sinkpad as on a regular pad and that Sinkpad had no effect on the accuracy of 2D mouse pointing. Second, we demonstrated Sinkpad at an academic workshop. We observed that all users could easily use their mice on the pad except for accidental clicks. The results of the experiment and the demonstration show that Sinkpad is usable as a conventional mouse pad and that users can get used to the new actions quickly.

2 Related Work

Sinkpad augments conventional mouse functionalities by using a malleable mouse pad that consists of an elastic material. Here, we summarize the work on augmenting the conventional mouse functionality and the input surfaces consisting of elastic materials.

Much research has gone into augmenting mice with diverse features, such as extending its degrees of freedom [1, 7, 10, 14], supporting touch interactions [2, 15, 18], adding pressure input [3, 12], providing actuated inflation and deflation [8], and making the buttons adapt to the user’s fingers [13]. In contrast, we augmented the mouse pad so that users can use a conventional or augmented mouse on it.

There has been a lot of development of input surfaces made from elastic materials. GelForce [16] calculated the force vectors on a surface made of elastic material. PhotoelasticTouch [11] was a tabletop system using deformable transparent objects on its surface. deForm [5] was a 2.5D surface that combined a deformable surface with arbitrary physical objects and manual manipulations that can handle a wide variety of inputs. While these systems required large form factors because they used optical sensing, our system is small because we use pressure sensing.

There is other research on a malleable surfaces such as [6, 17]. However, none of them focus on exploring the possibilities of a malleable mouse pad.

3 Sinkpad

Sinkpad is a malleable mouse pad, whose surface is smooth and soft, allowing the user to sink the mouse and tilt it into the pad, in addition to using conventional actions as shown in Fig. 2a. This design allows the user to perform three different techniques: sink, tilt, and sink+move.

Fig. 2.
figure 2

(a) Conventional actions, (b) sink, (c) tilt, (d) sink+move.

Full size image

  • Sink Sink the mouse downward vertically into the pad (Fig. 2b).

  • Tilt Tilt the mouse by sinking one side into the pad (Fig. 2c).

  • Sink+Move Move the mouse after sinking (Fig. 2d).

This design enables the user to perceive the depth of the mouse or the angle of tilt because of the deformation of the pad and haptic feedback provided by the deformation. Moreover, we also expect that the sink+move action allows the user to perform precise pointing because the user has to move their mice slowly due to greater friction from the pad while performing sink+move.

4 System Configuration

We describe the implementation of our prototype system: its hardware and analysis software.

4.1 Hardware

The Sinkpad hardware consists of two parts: a pad that is made of an elastic material and a sensing module that senses the actions. Figure 3 shows the hardware setup of Sinkpad. We attached the sensing module to the bottom of the pad.

Fig. 3.
figure 3

Hardware.

Full size image

The pad is made of an elastic material 1 cm thick, which has been cut into a square measuring 18 cm \(\times \) 18 cm. The elastic material is Hitohada gel (Exseal Corporation, Asker-C 0). The gel is soft enough that the mouse sinks into it when the user pushes it. At the same time, the gel is stiff enough that it serves as a flat surface until the user sinks the mouse into it. Moreover, we covered the gel with spandex fabric, as shown in Fig. 4, which makes the surface flexible enough to allow the mouse to slide smoothly.

Fig. 4.
figure 4

The pad.

Full size image

The sensing module uses a pressure-based approach to detect the actions. Figure 5 shows the sensing module. The sensing module consists of 64 pressure sensors, four multiplexers, and four microcontrollers. Our current prototype uses an FSR402 as a pressure sensor and mbed as a microcontroller. These 64 pressure sensors are arranged in an 8 \(\times \) 8 matrix. Each set of 16 pressure sensors was connected to a microcontroller via a multiplexer. The four microcontrollers are connected to a PC and send data at 25 fps. The reason why we used four microcontrollers instead of one is to increase the frame rate.

Fig. 5.
figure 5

The sensing module.

Full size image

4.2 Analysis Software

The analysis software measures the centroid and average of pressures in each frame. The area of Sinkpad is divided into an \(8 \times 8\) matrix. The value from each pressure sensor, which is sent to a microcontroller, is considered to be the pressure of the corresponding area in the matrix. The centroid \((x_{g},y_{g})\) is expressed as follows:

$$\begin{aligned} x_{g}=\frac{ \sum _{i=1}^{8} x_{i} \sum _{j=1}^{8}m_{(i,j)}}{\sum _{i=1}^{8}\sum _{j=1}^{8} m_{(i,j)}}, y_{g}=\frac{ \sum _{i=1}^{8} y_{i} \sum _{j=1}^{8}m_{(i,j)}}{\sum _{i=1}^{8} \sum _{j=1}^{8}m_{(i,j)}} \end{aligned}$$

where \(m_{(i,j)}\) is the pressure of sensor (i, j), \(x_i\) is the x-coordinate of the i-th column where the sensor is placed, and \(y_i\) is the y-coordinate of the i-th row.

The depth to which the mouse sinks into the pad is calculated from the average pressure. The tilt angle is calculated from the variation between the current and previous locations of the centroid \((x_{g},y_{g})\) under the condition that the average pressure is less than a certain threshold.

5 Applications

We present three example applications to explore the possibilities of using the pad.

5.1 Bringing a Background Window to the Front Quickly

The user can bring a background window between overlapping windows to the front quickly by sinking the mouse into the pad, as shown in Fig. 6. When the user hovers the pointer over the overlapping windows and sinks the mouse into the pad, some windows under the pointer become translucent (the number of windows depends on how strongly the user sinks the mouse into the pad), allowing the user to examine the content of the windows underneath. When the user clicks a button while sinking the mouse into the pad, the window, which is at the front of the overlapping windows except the translucent ones, is brought to the front. At this time, the translucency ends.

5.2 Examining Hidden Windows Quickly

The user can quickly examine the windows that are hidden under the overlapping windows by tilting the mouse as shown in Fig. 7. When the pointer hovers over the overlapping windows and the user tilts, some background windows move in the direction of the tilt. The deeper the background window is, the more it moves. When the user finishes tilting the mouse, the windows return to the positions before the tilt. Thus, the user can quickly examine overlapping windows. When the user finishes tilting the mouse, the windows return to the positions before the tilt. Thus, the user can move background windows that are between overlapping windows in order to examine them quickly.

Fig. 6.
figure 6

Bringing a background window to the front quickly.

Full size image
Fig. 7.
figure 7

Examining hidden windows quickly.

Full size image

5.3 Changing C-D Ratio Dynamically and Magnifying the Area Around the Pointer

The user can change the C-D ratio dynamically and magnify the area around the pointer as shown in Fig. 8. When the user performs sink+move, the pointer moves with a small C-D ratio. At the same time, the area around the pointer is magnified so that it can be viewed in more detail. Thus, the user can selectively perform normal pointing or precise pointing.

Fig. 8.
figure 8

Changing the C-D ratio dynamically and magnifying the area around the pointer.

Full size image

6 Evaluation 1

To examine how a user uses a mouse on Sinkpad in a 2D graphical user interface, we conducted a target acquisition experiment based on the ISO9241-9 standard for pointing evaluations [4]. We compared three cases: a mouse alone, a 3D mouse, and a mouse on Sinkpad.

6.1 Participants

Eighteen participants (14 males and 4 females) ranging in age from 21 to 32 took part in the experiment as volunteers. All participants were mice users on desktop computers.

6.2 Apparatus

The experiment was conducted on a desktop computer running Windows 7, with an Intel Core i3 540 CPU, and 2 GB of RAM. The monitor was 20.1 in. Dell E207WFP, with a resolution of 1680 \(\times \) 1050 pixels with a viewable screen width and height of 27 cm and 44 cm, at a refresh rate of 60 Hz. The devices used in the experiment were:

  • A mouse (Dell MS111 3-Button Optical USB 2.0 Mouse).

  • A 3D mouse (3Dconnexion SpaceNavigator 3DX-700028).

  • A mouse on Sinkpad.

6.3 Procedure

Participants were presented with a randomized series of target rings with different indexes of difficulty (2.04–4.70), as determined by two amplitudes (300, 600 pixels) and three target widths (16, 32, 64 pixels). Each target ring had 16 circular targets arranged in a circular layout as shown in Fig. 9. We asked each participant to click on the illuminated target. Once the target was clicked, the opposite target would be illuminated. The first three selections were illustrated by lines (Fig. 9). All participants were instructed to select the targets as fast and accurately as possible.

6.4 Design

The experiment was a 3\(\times \)2\(\times \)3 within-subjects design. The factors and levels were as follows:

  • Device {mouse, 3D mouse, mouse on Sinkpad}

  • Amplitude {300 and 600 pixels (103, 206 mm)}

  • Width {16, 32, and 64 pixels (4, 8, 12 mm)}

Participants were randomly assigned to one of six groups (three participants per group). The order of devices differed for each group for counter-balancing. With 18 participants and 15 selections, the total number of trials in the experiment was 18 \(\times \) 15 \(\times \) 3 \(\times \) 2 \(\times \) 3 = 4860.

Fig. 9.
figure 9

Experimental task showing circular targets.

Full size image

6.5 Results and Analysis

Table 1 shows the mean time, standard deviation, and mean error rate for each device. A paired t-test revealed that the 3D mouse was significantly slower than the mouse alone (\(t_{17}\) = 12.279, p = .000) and mouse on Sinkpad (\(t_{17}\) = 12.078, p = .000). The 3D mouse also had a significantly higher error rate than the mouse alone (\(t_{17}\) = 4.341, p = .000) and mouse on Sinkpad (\(t_{17}\) = 4.025, p = .000). On the other hand the mouse alone and mouse on Sinkpad showed no significant differences in error rate (\(t_{17}\) = .327, p = .374) and mean time (\(t_{17}\) = .425, p = .338). These results suggest that Sinkpad does not affect the speed and accuracy in 2D mouse pointing.

Table 1. Mean time, standard deviation, and mean error rate.
Full size table

7 Evaluation 2

To investigate the usability of Sinkpad, we demonstrated a prototype at an academic workshop (20th Workshop on Interactive Systems and Software held in Japan). Approximately 50 participants used the system. We observed those participants and collected their feedback on the pad.

We found that users could easily sink their mice into the pad by using their palms. Some people commented that they could recognize the depth and angle of tilt from both the deformation of the pad and haptic feedback provided by the deformation, and they had to move their mice slowly because of friction when they performed the sink+move action. These comments suggest that users can use our techniques and point precisely.

We also found a problem: if there is a button on the mouse in the direction of tilting, some people clicked it accidentally. This is because the user has to push the mouse to tilt it: if the user’s fingers are in contact with the mouse buttons, a mouse click could be accidentally triggered. To solve this problem, after this demonstration, we changed the analysis software so that it would ignore clicks during tilting.

8 Conclusions and Future Work

We presented Sinkpad, a mouse pad that has a malleable surface consisting of an elastic material. The user can perform three different actions by sinking the mouse into Sinkpad: sink, tilt, sink+move. Moreover, the user can also use Sinkpad as a conventional mouse pad. We presented three practical applications and presented the results of an evaluation showing that users could use the mouse on the pad as if it were a regular pad and could perform the sink, tilt, sink+move actions with fine control.

In the future, we plan to improve the system by adding more techniques such as ones utilizing z-axis angular motion of the mouse on Sinkpad. To this end, we will try a different hardware implementation, for example, using an array of photoreflectors for more precise sensing. We also plan to investigate the use of a 3D mouse on Sinkpad. Finally, we are interested in conducting user studies to measure the performance of Sinkpad in more realistic situations (e.g., in 3D applications such as 3D CAD).