skip to main content
10.1145/1452392.1452394acmconferencesArticle/Chapter ViewAbstractPublication Pagesicmi-mlmiConference Proceedingsconference-collections
keynote

Audiovisual 3d rendering as a tool for multimodal interfaces

Published: 20 October 2008 Publication History

Abstract

In this talk, we will start with a short overview of 3D audiovisual rendering and its applicability to multimodal interfaces. In recent years, we have seen the generalization of 3D applications, ranging from computer games, which involve a high level of realism, to applications such as SecondLife, in which the visual and auditory quality of the 3D environment leaves much to be desired. In our introduction will attempt to examine the relationship between the audiovisual rendering of the environment and the interface. We will then review some of the audio-visual rendering algorithms we have developed in the last few years. We will discuss four main challenges we have addressed. The first is the development of realistic illumination and shadow algorithms which contribute greatly to the realism of 3D scenes, but could also be important for interfaces. The second involves the application of these illumination algorithms to augmented reality settings. The third concerns the development of perceptually-based techniques, and in particular using audio-visual cross-modal perception. The fourth challenge has been the development of approximate but "plausible", interactive solutions to more advanced rendering effects, both for graphics and audio. On the audio side, our review will include the introduction of clustering, masking and perceptual rendering for 3D spatialized audio and our recently developed solution for the treatment of contact sounds. On the graphics side, our discussion will include a quick overview of our illumination and shadow work, its application to augmented reality, our work on interactive rendering approximations and perceptually driven algorithms. For all these techniques we will discuss their relevance to multimodal interfaces, including our experience in a urban design case-study and attempt to relate them to recent interface research. We will close with a broad reflection on the potential for closer collaboration between 3D audiovisual rendering and multimodal interfaces.

References

[1]
Nicolas Bonneel, George Drettakis, Nicolas Tsingos, Isabelle Viaud-Delmon, and Doug James. Fast modal sounds with scalable frequency-domain synthesis. ACM Transactions on Graphics (SIGGRAPH Conference Proceedings), 27(3), August 2008.
[2]
Carsten Dachsbacher, Marc Stamminger, George Drettakis, and Frédo Durand. Implicit visibility and antiradiance for interactive global illumination. ACM Transactions on Graphics (SIGGRAPH Conference Proceedings), 26(3), August 2007.
[3]
Oliver Deussen, Carsten Colditz, Marc Stamminger, and George Drettakis. Interactive visualization of complex plant ecosystems. In Proceedings of the IEEE Visualization Conference. IEEE, October 2002.
[4]
George Drettakis, Nicolas Bonneel, Carsten Dachsbacher, Sylvain Lefebvre, Michael Schwarz, and Isabelle Viaud-Delmon. An interactive perceptual rendering pipeline using contrast and spatial masking. In Rendering Techniques (Proceedings of the Eurographics Symposium on Rendering). Eurographics, June 2007.
[5]
George Drettakis, Maria Roussou, Alex Reche, and Nicolas Tsingos. Design and evaluation of a real-world virtual environment for architecture and urban planning. Presence: Teleoperators & Virtual Environments, MIT Press, 2007.
[6]
Céline Loscos, George Drettakis, and Luc Robert. Interactive virtual relighting of real scenes. IEEE Transactions on Visualization and Computer Graphics, 6(3), July-September 2000.
[7]
Céline Loscos, Marie-Claude Frasson, George Drettakis, Bruce Walter, Xavier Granier, and Pierre Poulin. Interactive virtual relighting and remodeling of real scenes. In D. Lischinski and G. W. Larson, editors, Rendering Techniques (Proceedings of the Eurographics Workshop on Rendering), volume 10, pages 235--246, New York, NY, Jun 1999. Springer-Verlag/Wien.
[8]
Nicolas Tsingos, Emmanuel Gallo, and George Drettakis. Perceptual audio rendering of complex virtual environments. ACM Transactions on Graphics (SIGGRAPH Conference Proceedings), 23(3), July 2004.

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ICMI '08: Proceedings of the 10th international conference on Multimodal interfaces
October 2008
322 pages
ISBN:9781605581989
DOI:10.1145/1452392
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 October 2008

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. 3d audio
  2. computer graphics

Qualifiers

  • Keynote

Conference

ICMI '08
Sponsor:
ICMI '08: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERFACES
October 20 - 22, 2008
Crete, Chania, Greece

Acceptance Rates

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • 0
    Total Citations
  • 259
    Total Downloads
  • Downloads (Last 12 months)0
  • Downloads (Last 6 weeks)0
Reflects downloads up to 15 Feb 2025

Other Metrics

Citations

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media