Immersive human–computer interactive virtual environment using large-scale display system

Highlights

• This paper presents a novel approach to utilize a large-scale screen and HCI techniques for the users to experience the virtual reality (VR).

• The user instructions are learned by the combination of gesture recognition techniques (based on extended genetic algorithm) and motion estimation techniques (using fuzzy predictive control).

• A framework and flowchart is designed for the semantics of interactive HCI.

• Compared to traditional VR headsets and data gloves approaches, the proposing method is more effective, robust and revolutionary.

Abstract

Large-scale display system with immersive human–computer interaction (HCI) is an important solution for virtual reality (VR) systems. In contrast to the traditional human–computer interactive VR system that requires the user to wear heavy VR headsets for visualization and data gloves for HCI, the proposing method utilizes a large-scale display screen (with or without 3D glasses) to visualize the virtual environment and a bare-handed gesture recognition solution to receive user instructions. The entire framework is named as an immersive HCI system. Through a virtual 3D interactive rectangular parallelepipe, we establish the correspondence between the virtual scene and the control information. A bare-handed gesture recognition method is presented based on extended genetic algorithm. An arm motion estimation method is designed based on the fuzzy predictive control theory. Experimental results showed that the proposed method has lower error rates than most existing solutions with acceptable recognition frequency rates.

Introduction

The technology of large-scale display system using multi-projector system presents an important solution for human–computer interaction (HCI) [1], [2]. In addition, remote controls using bare hand gestures, body poses and limbs motions are more preferable for virtual reality (VR) experience, compared with wearing heavy VR devices, such as VR headsets and data gloves [3]. In this study, users will experience the virtual environment by roaming in front of a large-scale screen with multi-projector system. Instructions and orders can be received by bare-handed gesture recognition solutions [4].

The experimental setup of this study is illustrated in Fig. 1, which consists of a large-scale display, fifteen projectors, fifteen client PCs, two cameras, and one server. The user’s hand gesture and arm motion are captured by the two cameras. The hand gestures are translated to operating commands as grabbing, releasing, rotating etc. And the arm motions create navigate commands, such as move left, move right, move forward etc. By combining the operating commands and the navigation commands, we enable the users to experience the immersive HCI in the virtual environment.

Gesture recognition from a video camera is a challenging problem. First, the tightly coupled rotation, inclination and motion produce a large number of variables for computation. Second, hand region segmentation is difficult without professional devices, such as data gloves, long-sleeves shirt [5] and hand-held LED light pen [6]. Third, it remains difficult to track the head, body or limbs and combine the tracking information with bare-handed commands recognition techniques [3], [7]. In addition, integrating arm motion estimation into gesture recognition is one way to stabilize the motion tracking done by the cameras, resulting in a more robust HCI system.

In this study, we propose an immersive HCI VR framework based on computer vision techniques where the users are not required to wear extra sensors, clothing or equipment (only markers are available). The users can perform editing or roaming in a virtual 3D environment built by a automatic collaborated multi-projector system [8]. Comparing with the existing related works in the literature, we summarize the main contributions of this study as follows:

• A novel simplified skeletal hand model. A simplified skeletal model is introduced, which uses an ellipsoid palm with strip fingers to approximate the hand. Compared with the existing hand models, the skeletal model reduces the recognition errors caused by hand rotation and occlusion.

• A novel hand gesture recognition algorithm. A bare-handed gesture recognition algorithm is designed using extended genetic algorithm (GA). The extended GA naturally avoids local extremes, which increases the robustness of the gesture recognition algorithm. Results show that our method produces higher correct recognition rate comparing with existing methods.

• An novel arm motion estimation method. An arm motion estimation method based on a rectangular parallelepiped for virtual interaction (RPVI) and fuzzy predictive control (FPC) is proposed. Compared to the existing motion estimation methods, our method achieves more accurate arm motion recognition results.

• An immersive HCI VR framework. Combining all the techniques that we have used, the proposed HCI VR framework successfully accomplishes scene editing and walkthrough using bare-handed interactive commands.