Keywords

1 Introduction

With the development of VR technology, how to make full use of these mature resources, and bring real input devices into the VR to help people better design interaction in VR, whether in the full use of resources or in reduce the cost of learning, are very meaningful. In the field of visual feedback effects in digital painting, precise hand-eye coordination is needed in the painting process. However, in current process of human-computer interaction, many operating tools are separated from each other in their spatial distribution. Precise hand-eye coordination requires the most direct visual feedback on the operation. Experts usually rely on expensive direct input devices for their tasks. The most comfortable visual and operation spaces do not overlap. Operators can not see their hand operations from distant horizon, but have to lower their heads in order to achieve accurate hand-eye coordination [1]. This affects the efficiency and precise control of human-computer interaction and is also the cause of many modern diseases [2].

VR technology has been long used to study the design process [3], therefore, VR design software has always been a research hotspot [4]. With the rapid development of computer 3D scene processing capabilities and maturation of VR (Virtual Reality) technology, more and more attention will be paid to VR applications specifically designed for long-term interaction.

Full control of the user’s field of vision with head-mounted display technology helps us better understand the physical feedback requirements for visual movement. A large number of psychophysical studies have found that in the absence of visual feedback, our perception of body location is not accurate [5]. By manipulating feedback information such as touch and vision, one can change the body’s perception of touch [6], and even object that resembles the limb are considered part of the body. Many VR studies have attempted to manipulate the magnitude of steering of visual feedback [7, 8] by using illusion to guide users in virtual reality to avoid limited boundaries and obstacles in limited physical space and experience much greater than actual Space [9].

Researchers have been exploring the interactive VR visual feedback effect through a large number of ways. By testing different display methods, they studied the effect of painting accuracy and speed on interactive effects by displaying the angle of the brush, the size and angle of the interface. For example, most of the documents evaluated for pointing devices (usually referring to input devices other than keyboards, such as mice, handles, and touchscreens) are based on the Fitz-Law [10,11,12]. The assessment method of Fitz’s Law proposed that the click task completion time between different devices only becomes meaningful when the target size and distance are the same. By Fitz-Law, these data can be converted into performance indicators (bits/second) independent of experimental details. And in 1999 Accot and Zhai derived Steering Law for directional control in space constraints [10]. Steering Law is widely used in interactive interfaces research, especially the movement and selection of cursors in menu lists. However, the application of Steering Law in assessing the performance ability in the painting process is not universal. The focus of Steering Law’s evaluation of operations is on speed rather than accuracy, and as long as the operation does not go beyond the well-defined limits, the evaluation of the operation relies solely on its speed of completion. However, the emphasis on the accuracy of the tasks accomplished in painting does not have a clear requirement on speed. The error rate, time and output indicators based on the Fitz-Law are all evaluation indicators for the whole process, by comparing the difficulty levels and time usage. Mackenzie et al. proposed to evaluate the in-process click operation with six types of auxiliary measures [13]. Four of them determine the problems that may arise during the operation through specific boundaries or conditions. The Hausdorff distance, as a measure of the distance between curves or polygons [14], offers another way calculating deviations between curves. Shon et al. calculated the mean and maximum values of the shortest distance from the sample point on the drawn curve to the target curve in assessing the effect of force feedback on the mapping of the virtual physical surface in a VR environment [15]. The offset of the entire curve is included in the calculation range.

This article hopes to explore the dislocation of visual feedback on user accuracy and speed of operation through adjustment of the information of operational visual feedback, and to use these effects to design a better virtual reality work experience. Johnny et al. used Steering Law to study the effect of plotting on the steering control accuracy at different dimensions [10]. Although it is a unique experience to freely draw in three-dimensional space without limitation of a plane, it is not easy to draw an accurate three-dimensional curve in the unrestricted three-dimensional space. Many studies find it difficult for even professional designers to control their brushwork in three dimensions [16, 17]. Arora et al. compared the traditional mapping, the use of VR in the physical surface mapping, and simple three-dimensional mapping effect, and found that the lack of physical mapping surface in VR is the main reason for error. Compared with the traditional drawing directly on a 2-D plane, the use of VR rendering solid surface mapping will reduce the accuracy by 20%, while in the absence of planar support in the VR mapping will cause 53% loss of accuracy [18]. When using VR, because of the lack of physical reference surface in operation, and at the same time because users can only determine the location through other visual cues without directly seeing their hands and pens, more errors can happen. This is a strong driving force for integrating existing mapping methods into VR. Based on the above findings, this paper focuses on the effect on accuracy of drawing on solid surfaces by users in a VR environment with the influence of different control conditions, especially when the hand direction information is provided.

In view of the above issues, this dissertation focuses on the following two aspects:

  1. 1.

    Design and development of a VR test platform.

    The platform is based on a mature interactive device tablet and uses VR’s Unit 3D development system to build a virtual tablet in the VR. The virtual tablet is integrated into a virtual scene and completely corresponds to a real tablet. Any operation on the real tablet can be reflected in the virtual tablet in real time. This integrates the interaction between the real interactive device and the virtual digital tablet in the VR. In the most comfortable visual space and the comfortable two-handed operation space, the interactive purpose of the VR is achieved.

  2. 2.

    Using the test platform to study visual feedback in VR interactive painting process.

2 VR-Based Digital Tablet Interactive Platform Design and Development

2.1 Technical Steps

In this research, we take a mature interactive device tablet as an object and use VR’s Unit 3D development system to build a virtual tablet in VR. The virtual tablet is integrated in the virtual scene and fully corresponds with the real digital tablet. People All operations on the real digital tablet can be reflected in the virtual tablet in real time. This realizes the interaction between the real interactive device and the virtual digital tablet in the VR. In the most comfortable visual space and two-handed operation space, the interactive purpose of the VR is achieved (Fig. 1).

Fig. 1.
figure 1

Technical framework of interactive platform.

Full size image
  1. (1)

    Physical tablet positioning technology, real-time capture, track the location of the physical tablet, provide spatial positioning information for VR virtual tablet.

  2. (2)

    Real-time tablet brush status and position capture in real time, to provide information for VR virtual paintbrush.

  3. (3)

    Dynamic texture generation on the virtual tablet surface in the VR, real-time rendering of the interactive interface of the real tablet application. The technical solution adopts real-time screen capture technology to capture the process of drawing digital tablet (e.g. Photoshop) running on the computer, generate dynamic texture map, sent to Unity3D platform, integrate into VR virtual tablet surface;

  4. (4)

    Using Unity to provide support for HTC Vive development package SteamVR, to activate the VR helmet display;

  5. (5)

    In the VR scene of Unity3D platform, the dynamic integration of virtual scene, virtual paintbrush, virtual tablet and surface dynamic texture map is accomplished.

2.2 Realization of Digital Tablet Space Positioning

This study is to provide user of different drawing habit with the most natural experience of the VR environment. Therefore, the experimental platform will be built using the virtual desktop running on the host painting software interface projected to objects in the virtual reality environment (Fig. 2).

Fig. 2.
figure 2

Tablet and VR helmet relative positioning

Full size image

2.3 Get Virtual Tablet Brush Gesture Module

The actual effect of the operation of the virtual tablet and the pen gesture module is shown in Fig. 3. The right side of the figure is the running interface of the real-world tablet application (Photoshop), the left is virtual tablet and virtual pen runs in the VR scenario.

Fig. 3.
figure 3

Virtual tablet and pen gesture module running effect

Full size image

3 Interactive Visual Feedback Effect Experimental Design in VR

In the same operating display position and stylus display to complete the click and tracing tasks, and complete the questionnaire.

The tracing task requires the user to see the existing lines. The user performs the same tasks many times under different control conditions and compares the effect of different conditions on the test results.

Users in each experiment use the test software to complete three major tasks, click, line drawing and curve drawing.

When performing a click task, the drawing software generates a random point on the interface that requires the user to click at the point of creation with the stylus and the user is told to complete the click task as accurately as possible in the shortest amount of time. After the user completes the click, the software will record the information such as the distance between the target point and the generated point and the time spent on completing the task, and then generate the next point for the user to click.

Tracing tasks include straight line drawing and curve drawing. Users are required to draw lines along the random path generated by the software. The path is divided into two types: straight line and curved line. The input to the track has no direction. The user can start from either end without affecting Final deviation calculation.

Painting environment effect test process is shown in Figs. 4 and 5.

Fig. 4.
figure 4

Painting environment effect test

Full size image
Fig. 5.
figure 5

Software interface in the virtual artboard mapping

Full size image

3.1 Control Variables

  1. (1)

    Operating angle

In the experiment, the position of the physical tablet is placed either horizontally or vertically.

When placed horizontally, the user is in exactly the same position as the drawing on the desktop.

When placed vertically, the physical tablet is placed on the easel. The user can adjust the angle and height freely, but the overall angle between the tablet and horizontal level is greater than 60°.

  1. (2)

    Virtual digital tablet self-adjustment

User can choose to use the positioning handle to adjust the spatial angle and location of the test software interface in the VR so that the virtual tablet seen in the VR is separated from the tablet in real space by a spatial perspective and location. The users will choose the display they think is most comfortable. In this way, the influence of spatial misplacement on visual feedback operation can be studied.

  1. (3)

    Brush display

Three types of brush display were tested in the experiment:

A: Cursor only: only the cursor is displayed, the users do not know other information of the brush besides the plane position of the pen tip. The user can only proofread the direction by moving the cursor as a reference.

B: Full display: In addition to the cursor also shows a three-dimensional model of a brush and its actual location and angle in real time. Although the user cannot see their own hands, they can estimate the drawing direction based on the moving direction of the brush.

C: Hide the brush after drawing: User can see the brush before drawing, but the brush disappears after drawing. The user can better understand the direction of the drawing space through the direction of the brush before drawing, and can correct the direction of drawing according to its trace.

4 Data Analysis

4.1 Operating Angle

In both experiments, the results of the click, straight line and curve drawing task are shown in Fig. 6:

Fig. 6.
figure 6

Operating angle of the different types of tasks generated error distribution (Color figure online)

Full size image

In the test chart, the vertical ordinate represents the distribution of the average error recorded, divided into three groups according to the type of task, which are point, straight line and curve respectively. The average error values when the physical tablet is placed horizontally and vertically are represented by green and Blue, respectively. It seems that in the vertical placement position the error of the click operation increased, while the line drawing accuracy improved. The operation angle only makes a significant difference to the result of the curve drawing. The vertical operation significantly reduces the error caused by the curve drawing but has no noticeable impact on the precision of the click and drawing line. When tablet is placed horizontally, there is no significant effect of accuracy of the three tasks. That is, the operation angle can significantly improve the curve drawing accuracy, and has little effect on the precision of the line drawing and click tasks.

4.2 Questionnaire Feedback

In addition, the users were interviewed by questionnaire after the two tests were completed. And the subjective feelings of different operations and display positions were recorded, as shown in Fig. 7.

Fig. 7.
figure 7

Subjective questionnaire survey results for different angles of operation

Full size image

The user did not show a clear uniform preference for the operation and display area, except for the size of the display area. Oddly, there are only two cases that the display size was actually changed during the position switch. Based on fact that the actual feedback gives preference for the horizontal operation display size, it is suggested that there is a deviation of the user perception of the size of the two operating angles. Unfortunately, this fact was discovered after all experiments were completed, it remains unknown whether in the case of vertical work, user’s feeling is large or not.

4.3 Space Overlap

After testing the different angles of the tablet, the user can choose to continue the experiment at a comfortable angle (horizontal or vertical) in later experiments. And they also have the option to use the positioning handle to adjust the spatial angle and position of the test software interface in the VR. This will result in the separation of the spatial angle and the position of the virtual tablet in the VR and the actual tablet in the real space, as shown in Fig. 8. Obviously, the user chooses the display they think is the most comfortable one.

Fig. 8.
figure 8

Spatial overlap distribution of the average error in three tasks

Full size image

The experiment compares the average error between the overlap and the separation between the virtual tablet and the actual tablet. It is be found that although the separation of the operating plane and the virtual tablet distorts the tracing effect, the separated picture can help users to obtain a better field of view and improve the precision of the click task.

4.4 Scale of Virtual Digital Display

The display size is the size of the virtual tablet seen by the user in the VR. In the experiment, the virtual tablet is simply scaled up or down, the basic ratio of the software interface is not changed, and the drawing range of the user operation is changed correspondingly, that is, the actual movement of the task remains the same. The recording error is subjected to the actual operation error, not by the screen zooming Fig. 9.

Fig. 9.
figure 9

Shows the average error distribution for different display sizes and task types

Full size image

The error of the user’s drawing generally decreases with the reduction of the display area, which is consistent with the previous findings in this field. Larger brush strokes are more difficult to complete even though the actual range of activities in this experiment does not increase. Excluding the deviation of body-aware control of the larger size, it indicated that the size of the drawing curve is mainly due to visual feedback. By comparing the display size of the virtual tablet to the operating angle of the tablet, it showed that the effect of the screen size on the operation precision became significant only when the tablet is vertically operated.

4.5 Brush Display

In actual operation, the brush display increases the error. The users responded that brush blocked their sight, resulting in reduced operational accuracy and speed, as shown in Fig. 10. When comparing the drawing angle and brush display, there was an unexpected discovery. Brush display increases the error level of the drawing only when the tablet is placed vertically. Brush display in different working angles were very different compared with the vertical position, when the brush display was in horizontal position, it had little effect on the drawing accuracy. The accuracy of line drawing when operating in a vertical plane is affected more by the brush display, and the virtual brush has no significant effect on the tracing accuracy in a horizontal operation.

Fig. 10.
figure 10

The average error distribution in different brush display and task

Full size image

5 Conclusion

This paper attempts to set up a tablet-based VR interactive design and test platform for the purpose of realizing the “overlap” interactive effect of hand-eye interaction space from the perspective of visual feedback in the process of human-computer interaction design of product design. With the aid of unity 3D development platform, the real tablet and HD VR helmet were integrated in the VR system, and VR interactive visual feedback effect was tested and verified.

In the VR environment of the platform, the tablet display angle, the digital tablet and the virtual tablet spatial coincidence, the display interface, and painting tasks are tested, and influencing factors and visual feedback effects were analyzed and studied, the conclusions are summarized below:

  1. 1.

    Placing the tablet vertically can improve the accuracy of the curve drawing task.

  2. 2.

    When the digital tablet and the virtual tablet space are separated, the operator’s field of vision is improved and so are the click task speed and accuracy.

  3. 3.

    The digital tablet and the virtual tablet separation can cause space misplacement, will reduce the accuracy of direction control.

  4. 4.

    When the virtual brush is displayed, working in the vertical plane will significantly reduce the precision of line drawing tasks, but have no significant effect on the horizontal tablet.

These results are instructive for improving the visual feedback in VR-based human-computer interaction painting. Based on the tablet VR interactive design and testing platform described in this report, the interactive device can be integrated with the virtual digital tablet in the VR to realize mutual interactivity with the most comfortable visual space and the two-handed operating space.