Keywords

1 Research Objective

Users commonly perform tasks simultaneously while working on a computer or tablet; for example, they might draw a picture on a piece of paper while looking at a given design displayed on the screen of a computer or tablet, or they might answer questions while consulting a reference book displayed on the screen. However, when users want, for instance, to change the reference materials or expand the figure on the screen, they may think it difficult to operate the device. That is, because input interfaces such as keyboard and mouse are outside their working space, they must stop and move their working hand in order to operate them. In light of this example, the goal of the present study is to devise a system by which the user can easily operate a computer or tablet while working on a desk.

2 Our Solution

2.1 Input Interface

Many types of input interfaces, like physical keys (keyboard and mouse), cameras, and microphones, are used to operate a computer or tablet. In this study, the input interface chosen is based on recognition of the working hand. This hand-based interface is intuitive and easy to use. Prior researches have used various devices to detect a hand on a desk. For instance, image processing of camera-captured images could detect a hand [1], and Microsoft Kinect v2 and Leap Motion were used for recognizing hand positions and gestures [3]. With Microsoft Kinect v2, it is only necessary to install a camera; however, difficulties due to room lighting, background, and color of the hand arise. Leap Motion placed on a desk recognizes the position gesture of the hand, but it intrudes into the work space and cannot recognize the hand if an obstacle is placed in between the hand and camera.

Given the issues described above, in this study, changing the desk itself into an input interface that can detect touch is proposed. In this way, the user can work without having to change the environment around the desk. When a computer or tablet is operated, the positional data of the hand touching the desk is used. It is therefore necessary to develop a device for acquiring this positional data and a software program for operating the computer or tablet via this device.

2.2 Desk-Touch-Interface

The flow of the proposed desk-touch-interface using a desk is described in Fig. 1.

Fig. 1.
figure 1

Desk-touch-interface system

Full size image

First, the user changes the desk into an input interface, and places a hand on the desk. Then, the developed device (which changes the desk into an input interface) detects the position of the hand and operates the screen of the computer.

3 Implementation Prototype for Detecting a Touch of a Hand

A prototype device for detecting the hand by using the office desk as a touch interface was constructed as shown in Fig. 2. It includes a touch sensor, an acrylic board and RS232C-interface IC. Once the device is installed on the back of the desk and is connected to the computer, it can transmit the positioning data of the hand touching the desk.

Fig. 2.
figure 2

Device for detecting the touch of a hand

Full size image

3.1 Structure of the Touch Sensor

The touch sensor is constructed of a resistor, a microcomputer, and aluminium materials. Its function is to detect a hand touching the aluminium material. An outline of the touch-sensor circuit is shown in Fig. 3. When a voltage is applied to the output pin of the microcomputer, the waveform of the voltage at the input pin of the microcomputer changes when a hand touches the aluminium material. An example of the waveform is shown in Fig. 4.

Fig. 3.
figure 3

Outline of touch-sensor circuit

Full size image
Fig. 4.
figure 4

Example waveform (Color figure online)

Full size image

If a hand touches the aluminium material, the speed of rise and fall of the input voltage is reduced by the human body’s capacitance, and a delay occurs. The delay is measured by the microcomputer, which can thus detect a touch of a hand. The waveform depends on the value of the resistance, the area of the aluminium material, and the distance between the hand and the aluminium material. Even if a hand does not directly touch the aluminium material, the sensor can detect that a hand is approaching the aluminium material by changing these parameters. To detect a hand on the desk, the device (including the touch sensor) is installed on the back of the desk.

3.2 Number of Counts as Delay Time

The delay time is measured by the microcomputer, which uses a variable for measuring delay time, namely, the number of counts between the output-voltage rise (or fall) and the input-voltage rise (or fall). As a sensor value, the number of counts is sent to the host computer. Since number of counts is used instead of measuring real time, the delay time gets longer so that number of counts increases. However, when measured, the number of counts output every time is not stable. The microcomputer therefore calculates the mean of the number of counts and outputs a stable value. A PIC16F1827 microchip was used to output the number of counts. It is connected to both ends of the resistance such as the circuit in Fig. 3. The voltage on input pin is high when input voltage is 2 V, and it is low when input voltage is 0.8 V. The microcomputer’s I/O pin is connected to an RS232C-interface IC (model ADM3202AN) to send the number of counts as a sensor value to the host computer.

3.3 Preliminary Experiment of the Touch Sensor

To evaluate the performance of the touch sensor, three experiments were carried out.

  • (a) Detection of touch on desk

The touch sensor was installed on the back of a desk, and the change in delay when the desk was touched investigated. The experimental conditions are listed as follows:

  • Aluminum material area: 50[mm] × 60[mm]

  • Resistance value: 1.0 [MΩ]

  • Desk material: Wood

  • Desk thickness: 30 [mm]

Mean number of counts when a hand touched the desk and when it did not touch the desk are listed in Table 1.

Table 1. Mean number of counts measured in experiment (a)
Full size table

The result shows that there is a difference between the values, indicating that a touch can be detected by the sensor.

  • (b) Changing the position of the hand on the desk

The change in the mean count given by a touch sensor when the hand position on the desk changes was investigated next. A touch sensor like that described in experiment (a) was used. Mean number of counts for the position of the hand are listed in Table 2.

Table 2. Mean number of counts in experiment (b)
Full size table

“D” is the distance from the center of the sensor to the hand. The results show that as the hand gets further from the sensor, the delay decreases at an almost constant rate. Although only one sensor was used in this experiment, using multiple touch sensors would make it possible to estimate the position of the hand on the desk.

  • (c) Estimating the position of the hand

Four sensors like those used in experiment (a) were installed 60 mm apart to form the touch positions as shown in Fig. 5. Each sensor is connected to an I/O pin of the microcomputer so that it measures the number of counts when a hand touches five points. That is, four points correspond to four sensors, and one point corresponds to the center of the rectangle created by the four sensors.

Fig. 5.
figure 5

Experiment(c)

Full size image

As shown by the results listed in Table 3, the four sensors obtained a different number of counts at each touch point. The values are similar when the same point was touched on different occasions. It is therefore concluded that the position of the hand can be estimated by using the values obtained from the four sensors. However, when the user does not touch the desk, each sensor outputs a different value because the number of counts obtained by each sensor is different (due to individual differences in the internal circuit of each pin).

Table 3. Mean number of counts in experiment (c)
Full size table

3.4 Normalized Sensor Data

Because the number of counts for each pin differs, the sensor data is normalized. That is, the difference between the number of counts when the desk is not touched and the maximum number of counts when it is touched was determined, and the difference is used to calculate the normalized number of counts, \( p_{n} \), for every sensor, given as

$$ p_{n} = \frac{{t_{n} - t_{l,n} }}{{t_{h,n} - t_{l,n} }} \times 100 $$
(1)

Where \( t_{l,n} \) is the number of counts of the nth sensor in case of no touch, \( t_{h,n} \) is the maximum number of counts of the nth sensor in case of touch on the nth sensor, and \( t_{n} \) is the number of the counts measured at that time (where n = 1, 2, 3, or 4). By using the normalized value, it is easy to estimate touch position and swipe direction.

3.5 Calibration

The number of counts corresponding to no touch and that corresponding to a touch must be calibrated whenever the device is started and whenever the user changes (because these values are different for each user). The device can be calibrated easily by simply touching the desk. The maximum count is reset whenever the user changes by simply placing a hand on the desk. In case the desk is changed, the user installs the device on the back of the desk and calibration is enabled.

3.6 Detecting a Touch

A threshold of the touch is defined before the touch position is estimated. A touch is taken as detected when the normalized counts exceed the threshold. Three states, namely, “not touch”, “tap”, “touch” are considered. As described later, these states are used to prevent mistouches. Hand-related information is obtained only in the “touch” state.

3.7 Estimating the Touch Position

The position of the hand is estimated by using the values provided by the four sensors. The coordinates of the position of the hand (hx, hy) are represented as

$$ hx = \frac{{x_{1} \times p_{1} + x_{2} \times p_{2} + x_{3} \times p_{3} + x_{4} \times p_{4} }}{{p_{1} + p_{2} + p_{3} + p_{4} }} $$
(2)
$$ hy = \frac{{y_{1} \times p_{1} + y_{2} \times p_{2} + y_{3} \times p_{3} + y_{4} \times p_{4} }}{{p_{1} + p_{2} + p_{3} + p_{4} }} $$
(3)

These coordinates were experimentally estimated by using the device. The experimental conditions are shown in Fig. 6. As described above, the device can acquire the positioning data of the hand, even when a piece of paper or a regular-size notebook are sandwiched between the desk and the hand.

Fig. 6.
figure 6

Conditions of the experiment

Full size image

4 Application of the Device

4.1 Outline of the Application

The device was evaluated in an application for operating images displayed on a screen while doing other work on the desk (e.g., drawing). The software used for this application was developed using Processing. A screenshot of the application in action is shown in Fig. 7; (A) indicates the folder containing the image, (B) indicates the main image, and (C) shows a preview of the current image and the previous/next images. Moving the hand on the desk changes the image or the folder. In this example application, when the hand is moved to the left, the main image changes into the previous image. When the hand is moved upwards, the folder is changed. A function to prevent malfunctions is activated by tapping the table twice to lock the screen. It is cancelled by tapping twice again.

Fig. 7.
figure 7

Screenshot of test application

Full size image

4.2 Detecting Swipe Direction

The application must be able to detect a swipe movement of the hand. The position data of the hand is used to find the touch point and the release point of the swipe. The swipe movement is then detected by using the difference between the two points.

4.3 Instructions for Operating Application

  1. 1.

    Install the device (by using masking tape) on the back of the desk where the user works. Then connect it to an AC adapter for supplying power

  2. 2.

    Launch the device and the software for changing the screen

  3. 3.

    Calibrate the device started by clicking the mouse on the screen or touching the screen

  4. 4.

    Change an image on the screen. The image in Fig. 7 is displayed when the user starts the software; the image can be changed by the hand moving right and left, whereas the folder can be changed by moving it up and down

  5. 5.

    Lock the screen by tapping the desk twice; cancel the lock by tapping twice again; this function prevents malfunction in case of touching by mistake

(Steps 4 and 5 are repeated.)

Observations.

The application allows the user to change the image and the folder simply by movements of the hand, even if a notebook is sandwiched between the hand and the desk; for instance, the image can be changed while drawing a picture in a notebook with one hand. Sometimes, the software wrongly detects a swipe movement. When no operation is performed, the device reacts to a hand or an arm. However, errors do not occur as long as the user makes no wide movements of the hand.

5 Related Work

Prior research on changing a desk into a touch interface has been reported. Wellner enabled the interaction operation on the desk by recognizing a hand on the desk by using a camera and projecting a picture on the desk with a projector [1]. Xiaojun et al. investigated the touch operation and the position of an appropriate window in order to support work using a desktop computer by installing a touch display in the desk [2]. Efanov et al. used a Microsoft Kinect v2 and Leap Motion for detecting the hand [3]. These studies use a camera, a touch display, and a hand-sensing device, but they are dependent on the environment of the room and the structure of the desk. Moreover, whenever the desk is changed, additional time is needed for reinstalling peripheral elements.

Several studies deal with detecting a hand and using it to interact with a desk. Grosse-Puppendahl et al. conducted research on ubiquitous computing with NFC using capacitance [4]. The receiver of the sensor is placed on the back of the desk, and several transmitters are placed on the desk for measuring capacitance. In this setting, a transmitter device is needed for measuring capacitance. The proposed hand-detection system, user oneself does not need to have a power source. Grosse-Puppendahl et al. also conducted research on recognizing the position of the hand and multi-touch with multi capacitance sensors [5]. This research is similar in part to the present work, although the proposed device makes use of single touch; it is also low cost, uses readily available electronic parts, and its small size allows it to fit in the palm of a hand.

6 Concluding Remarks

A device that enables touch interaction on a desk by using the human body’s capacitance was proposed. In addition, the device can detect touch position and swipe movement by using multi sensors. The device was used in an application for operating images displayed on a screen, even while doing other work on the desk. Although a basic application, namely, changing an image, was demonstrated, this application proved to be easy-to-achieve on a computer or tablet by using this device while working on a desk. In future work, the sensors in the device will be improved, various applications utilizing the device will be developed, and the applications will be experimentally evaluated. At this stage, only desks were used for interaction; in future work, using walls or doors for interaction will be considered.