ABSTRACT
Virtual environments (VEs) represent advanced, immersive, human-computer interaction (HCI) systems. Such interaction occurs through communication over several sensorial channels. Since the communication pathway between users and the simulation system groups several distinct channels, it is termed multimodal. The modalities used are primarily the visual, auditory, and haptic ones. The number, quality, and interaction between such modalities are key to the realism of the simulation and eventually to its usefulness. Thus, the need for increased immersion and interaction motivates system designers to explore the integration of additional modalities in VE systems and to take advantage of cross-modal effects. The downside, of course, is increased system complexity, cost,
17.1 Introduction ........................................................................................................................ 411 17.2 Multimodal Input from User to the Computer ................................................................... 413
17.2.1 Input Modalities .................................................................................................... 413 17.2.2 Multimodal Input Architecture ............................................................................. 414 17.2.3 Multimodal Input Integration and Synchronization ............................................. 416 17.2.4 Summary of Multimodal Input ............................................................................. 418
17.3 Machine-Human Communication ..................................................................................... 418 17.3.1 Multimodal Feedback Integration ......................................................................... 419
17.3.1.1 Visual-Auditory Feedback Interaction ................................................ 419 17.3.1.2 Visual-Haptic Feedback Interaction .................................................... 420 17.3.1.3 Haptic Channels Coupling ................................................................... 420
17.3.2 Sensorial Transposition ......................................................................................... 421 17.3.3 I/O Channel Coupling ........................................................................................... 423 17.3.4 Summary of Multimodal Feedback ...................................................................... 425
17.4 Multimodal VE Design....................................................................................................... 425 17.4.1 Advances in VE Design Using Multimodal Input ................................................ 425 17.4.2 Using Multiple Modalities in CVEs ...................................................................... 427
17.4.2.1 Effect of Multiple Modalities on Collaboration in CVEs .................... 428 17.4.2.2 Architectures for Multimodal CVEs .................................................... 428 17.4.2.3 Multimodal Interaction in Distributed Surgical Simulators ................ 429
17.4.3 Distributed Web-Based Multimodal VEs ............................................................. 430 17.5 Conclusions and Future Directions ..................................................................................... 430 Acknowledgments .......................................................................................................................... 431 References ...................................................................................................................................... 431
and possible integration/synchronization problems. In addition, cross-modal interaction can lead to perceptual illusions that must be considered by system developers.
