Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

As tablet devices become more and more popular, various handwriting applications are developed and evaluated. When those handwriting applications are used in class or in meetings for note-taking, users iteratively change colors, stroke weights and other functions by means of menus. It is certain that the users want to take notes as if they are using real paper notebooks. Thus, they do not want to change their writing hand posture in order to open menus, and do not want to select items through several steps of touches or gestures. Such unnatural manipulations to open menus and select items are not adequate for note-taking applications.

To the extent at which users can select menus in a more natural way while taking notes is rather important for note-taking applications and their menu manipulations. Thus, we list natural note-taking criteria below.

  1. 1.

    The writing space should be as large as possible

  2. 2.

    Users can put their writing hand on a touch display when taking notes

  3. 3.

    Users do not have to change their writing hand posture to open menus

  4. 4.

    No stumbles while taking notes

Various research has been done with regard to menu design and menu manipulations. Most of the research focuses on rapidity and accuracy. Both rapidity and accuracy are important for menu manipulations, though they are not the only criteria of good note-taking applications.

From the view point of our criteria, we review menus with regard to general handwriting applications and menus of past researches. Then, we evaluate their adequacy in relation to natural note-taking. In terms of usability for note-taking, there are several drawbacks which we list below.

  1. 1.

    Occupying writing space

  2. 2.

    Opening menu manipulations

  3. 3.

    Hindering contents underneath menus

  4. 4.

    Being covered by manipulating hand

Most handwriting applications [2, 4, 9] routinely place their menus on either side of or at the top or bottom end of a display. Although those menus can be selected easily, they occupy writing space. This is drawback 1, and will be in conflict with our criteria 1. In addition, when there is a menu on a display at all times, users will be forced to keep their hand above the display, because, if they put their hand on the display, the hand accidentally touches the menu and selects items. This will be in conflict with our criteria 2.

In order to keep the writing space large, context menus (also called pop-up menus) [7, 8] will be a reasonable option for handwriting applications. Context menus appear near the manipulation point of a stylus pen through specific manipulations such as a long tap or double taps. Because they only appear while selecting items and will disappear after the selection, they do not occupy the writing space at all times.

Context menus also contain the drawback 2. Some handwriting applications already have embed context menus and those menus are mostly opened by the long tap manipulation; however, the activity of taking notes will stumble due to the long tap manipulation. Thus, it is inconsistent with our criteria 4. Other operations initiated through single touch interactions to open menus conflict with drawing manipulations.

Multi-touch interactions require complicated touch manipulations and thus, will disturb the posture of holding a stylus pen. This goes against our criteria 3.

When context menus appear close to the manipulating stylus pen, the menus tend to hinder contents underneath themselves. If something similar were to occur while taking notes in a real paper notebook, it would be like a small piece of memo paper suddenly appearing on the notes, and thus disturbing our recognition of the written contents. This is drawback 3.

There are several shapes of context menus. The general shape of menus are square and rectangular. Also, there are round shaped context menus such as Pie menu and Tracking menu [3, 5]. When selecting items, the manipulating stylus pen and hand tend to cover parts of those menus. This is drawback 4, and is inconsistent with our criteria 4.

Context menus solve the drawback of occupying writing space. We consider that context menus for note-taking applications still have drawbacks with regard to the positions in which they appear, and further, with regard to opening manipulations.

2 Approach

We adopt context menus as our menus of note-taking applications, and attempt to meet our criteria and overcome their drawbacks. To achieve our criteria 2, a function called “Palm rejection” [10] is needed. Palm rejection enables non-purposed palm touches to avoid accidental inking, which is mostly produced by unintended touches [1].

Furthermore, in order to meet our criteria 3 and 4, we make use of the unintended touches to open our menu.

Fig. 1.
figure 1

The flow of palm lift manipulation

Full size image

2.1 The Way of Opening Menus Naturally

In general, unintended touches are considered as unnecessary. Though we recognize all touches as variable. Most unintended touches are generated by a writing hand and those touches indicate where the writing hand is. We consider making use of this indication when opening menu manipulation.

In order to design our menu manipulation, we developed a simple handwriting application. The application is a web application and is developed by Javascript and HTML Canvas. When we use the web application on general multi-touch tablet devices such as iPad, the application detects multiple touch information.

For the purpose of distinction of intended touches and unintended touches, we embed a simple palm rejection function within our application. When there are multiple unintended touches, the mean value of the unintended touch-coordinates will be regarded as the center. Because unintended touches move rapidly with very short touch periods and are unstable, we include 20 bits of historical data in order to set a stable mean value.

Once the position of the manipulating hand becomes certain, there are no specific manipulations to open our menu, except lifting up the dominant hand from the touch display. Then, our context menu will pop-up at the recorded point. We named it “Palm lift manipulation (PLM)” and Fig. 1 shows the flow of the manipulation.

Most of other applications and researches embed long tap or multi-touch interaction to open their context menus. Whereas, long tap manipulation will be inconsistent with our criteria 4, and multi-touch interaction will be also inconsistent with our criteria 3: PLM on the other hand, does not have to do any specific manipulation to open the menu and, will be able to allow natural note-taking.

2.2 Representation of Our Menu

In this research, we adopt a characteristic arc shaped menu design [6] and call it “Arc menu”. When the menu is opened by PLM, the arc shaped design will contribute to the following points:

Because of the arc shaped design, when selecting items from the menu, users can recognize all items. This will help to overcome the drawback 4. If the menu was a different design such as Pie menu, users would need to move their hand more extensively than with the Arc menu to select items. The extra movement is inconsistent with our criteria 4. Figure 2 shows the comparison of the distance of the movement.

Fig. 2.
figure 2

The comparison of the distance of the movement

Full size image
Fig. 3.
figure 3

The comparison of the recognition of the writing place

Full size image

In addition, other context menus completely cover the underneath space and users cannot see the next writing place at all. Whereas, the middle of the Arc menu is empty, and thus, helps to recognize the place that will most probably receive writing. The slight difference may have a positive influence in achieving our criteria 4. Figure 3 shows the comparison of the recognition of the writing place.

3 Discussion

In this section we discuss our menu according to the following view points: adequacy of PLM and variability of the menu design.

3.1 Adequacy of Palm Lift Manipulation

We have adopted PLM in order to open menus naturally while taking notes. PLM overcomes drawbacks 1 and 2. With regard to general applications, users open menus through specific manipulations such as a long tap, double taps, or, multi-touch interactions. Such manipulations will add extra operations for users.

PLM has no specific operations to open the menu except lifting up a note-taking hand. When we are using a real pen and paper notebook, we often lift up our hand and choose other stationery from our pen case. Also while taking notes, we often lift up our hand to move to the next blank space. Therefore, we do not regard PLM as a specific manipulation.

PLM meets all our criteria 1, 2, 3 and 4. On the other hand, the disadvantage of PLM is that the menu does not appear when users do not put their hand on the display. This will happen when users take notes near the edge of the display. Under such circumstances, they are not able to put their hand on the device. To avert this disadvantage, we need to consider other supplemental options when users take notes near the edges.

We have arranged several positions with regard to opening the menu. The menu position around the writing stylus pen seams to be the fastest position for item selection. However, sometimes the writing contents will be hindered by the menu. In addition, the incompatibility between the position of the lifting palm and the position in which the menu may appear will cause confusion.

Although we have tried the opening position below the palm as well, a basic note-taking flow moves horizontally from left to right. Thus, the opening position below the palm will compel the user to move in zigzags which would conflict with our criteria 4. Under the palm on the other hand, does not lead to incompatibility, and further, does not disturb the note-taking flow. Thus, opening menus under the palm is a reasonable and adequate position.

3.2 Variability of the Menu Design

We have adopted the Arc menu, which is an arc shaped context menu. The Arc menu overcomes drawbacks 3 and 4.

The angle, size and number of items are heuristically decided. Thus, we should analyze each of them and bring out the best combination of them. We do not take into our account the difference of hand size. Children have smaller hands than adults, and those differences should be considered.

4 Conclusion

In this research we propose the novel manipulation technique named “Palm lift manipulation (PLM)” and applied the Arc menu. PLM makes use of unintended touch coordinates to set the opening position of the menu. PLM enables users to concentrate on what they are writing and lets users naturally open the menu. The Arc menu overcomes drawback 3 and 4, and thus it enables users to select items naturally. The combination of PLM and the Arc menu meet all our criteria and overcome all drawbacks.

However, there is an additional drawback, in which, the menu only appears when users are putting their manipulating hand on a display.

Since we have not had experiments, comparisons between PLM and other manipulation techniques will be needed. That will be the future topic of discussion.