Man and Machine Bidirectional Communication Interface Using Voice and Pointing Action

Abstract

In this research, we have constructed a dialog environment between human and computers, and proposed and verified the method of operating the information which unified the verbal information represented using the voice and the non-verbal information and a pointing action. Now, the interface environment of information equipments such as a computer does not change with the former one. For example, elderly people cannot use it very easily. In Japan, if the interconnection of various apparatus is carried out with the arrival of ubiquitous network society and new various functions are available in addition to the conventional functions, it is thought that products which need the complicated operations will be developed. It is expected that even people who can use the functions will feel it difficult to use new ones while growing older. Then, dialog environment which permits both a system and a user to use much easier operations is desired. In this research, the environment is constructed witch allows a user to communicate using a voice and gesture with the personification agent who is a salesman in a virtual store selling fountain pens. The specification using a pointing action and voice such as ‘Would you show me this pen?’ will make it possible foe user to specify the pen he wants to buy from fountain pens in a showcase. If the distance between his fingertip and the pen is distant, it may be difficult for the agent to infer which pen the user wants. This needs the further communication using voice and pointing action to make his intent clear.