Skip to main content
SpringerLink
Log in

Integration model of eye-gaze, voice and manual response in multimodal user interface

  • Published:
Journal of Computer Science and Technology Aims and scope Submit manuscript

Abstract

This paper reports the utility of eye-gaze, voice and manual response in the design of multimodal user interface. A device- and application-independent user interface model (VisualMan) of 3D object selection and manipulation was developed and validated in a prototype interface based on a 3D cube manipulation task. The multimodal inputs are integrated in the prototype interface based on the priority of modalities and interaction context. The implications of the model for virtual reality interface are discussed and a virtual environment using the multimodal user interface model is proposed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Apple Computer Inc. Apple Human Interface Guidelines: The Apple Desktop Interface. Addison-Wesley, Massachusetts, 1987.

    Google Scholar 

  2. Begault D R. 3D Sound for Virtual Reality and Multimedia. Academic Press, Boston, 1994.

    Google Scholar 

  3. Bolt R A. The Human Interface: Where People and Computers Meet Lifetime Learning Publications, Belmont, 1984.

    Google Scholar 

  4. Bruner J. Toward a Theory of Instruction. Harvard University Press, Cambridge, 1966.

    Google Scholar 

  5. Burdea G, Coiffet P. Virtual Reality Technology. John Wiley & Sons, New York, 1994.

    Google Scholar 

  6. Dannenberg R B, Blattner M M. Introduction: The trend toward multimedia interface. InMultimedia Interface Design, Blattner M M, Dannenber R B (eds.), Addison-Wesley, Massachusetts, 1992.

    Google Scholar 

  7. Dear D L. HyperCard: What is it?Byte, August 1988, 71–75.

  8. Encarnacao J, Foley J, Bryson S, Feiner S K, Gershon N. Research issues in perception and user interfaces.IEEE Computer Graphics and Applications, March 1994, 67–69.

  9. Foley J D. Interfaces for advanced computing.Scientific American, Oct. 1987, 97–90.

  10. Giagante M A. Virtual reality: Enabling technologies. InVirtual Reality Systems, Earnshaw R A, Gigante M A, Jones H (eds.), London: Academic Press, 1993.

    Google Scholar 

  11. Hauptmann A G, McAvinney P. Gestures with speech for graphic manipulation.International Journal of Man-Machine Studies, 1993, 38: 231–249.

    Article  Google Scholar 

  12. Hill W, Worblewski D, Mccandless T, Cohen R. Architectural qualities and principles for multimodal and multimedia interfaces. InMultimedia Interface Design, Blattner M M, Dannenberg R B (eds.), Massachusetts: Addison-Wesley, 1992.

    Google Scholar 

  13. Kalawsky R S. The Science of Virtual Reality and Virtual Environment. Wokingham: Addison-Wesley, 1993.

    Google Scholar 

  14. Jacob R J K. What you look at is what you get: Eye movement-based interaction techniques. InProc. ACM CHI’90 Human Factors in Computing Systems Conference, 1990, pp. 95–102.

  15. Latta J N, Oberg D J. A conceptual virtual model.IEEE Computer Graphics & Applications, Jan. 1994, 23–29.

  16. Microsoft Corporation. The Windows Interface: An Application Design Guide. Washington: Microsoft Press, 1992.

    Google Scholar 

  17. Pausch R, Grossweiler R. Application-independent object selection from inaccurate multimodal input. InMultimedia Interface Design, Blattner M M, Dannenberg R B (eds.), Massachusetts: Addison-Wesley, 1992.

    Google Scholar 

  18. Robertson P, Earnshaw R A, Thalmann D, Grave M, Gallop J, De Jong E M. Research issues in the foundations of visualization.IEEE Computer Graphics and Applications, March 1994, 73–76.

  19. Rudnicky A I, Hauptmann A G. Multimodal interaction in speech systems. InMultimedia Interface Design, Blattner M M, Dannenberg R B, Massachusetts: Addison-Wesley, 1992.

    Google Scholar 

  20. Wilsson M, Conway A. Enhanced interaction styles for user interfaces.IEEE Computer Graphics & Applications, March 1991, 79–90.

Download references

Author information

Authors and Affiliations

  1. State Key Laboratory of Human Factors, Hangzhou University, 310028, Hangzhou

    Wang Jian

Authors
  1. Wang Jian
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wang Jian.

Additional information

This research was supported by the grants from National Natural Science Foundation of China and 863 High-Tech Programme. Part of the paper was first presented at the IEEE International Conference on System, Man and Cybernetics in 1995 at Vancouver, British Columbia, Canada, and at the International Workshop on Virtual Reality and Scientific Visualization in Hangzhou, China, in April 1995.

Wang Jian received his Ph.D. degree in Engineering Psychology at Hangzhou University in 1990. He currently is a professor of psychology and the head of Human Factors Laboratory in Hangzhou University. His research interests include user interface design in VR environment, integration of eye-tracking in user interface design and cognitive process in human-computer interaction.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Wang, J. Integration model of eye-gaze, voice and manual response in multimodal user interface. J. of Comput. Sci. & Technol. 11, 512–518 (1996). https://doi.org/10.1007/BF02947219

Download citation

  • Received:

  • Revised:

  • Issue Date:

  • DOI: https://doi.org/10.1007/BF02947219

Keywords

Search

Navigation