Abstract
Music visualizations are nowadays included with virtually any media player. They usually rely on harmonic analysis of each sound channel, which automatically generate parameters for procedural image generation. However, only few music visualizations make use of 3d shapes. This paper proposes to use spectral mesh processing techniques, here manifold harmonics, to produce 3d stereo music visualization. The images are generated from 3d models by deforming an initial shape, mapping the sound frequencies to the mesh harmonics. A symmetry criterion is introduced to enhance the stereo effects on the deformed shape. A concise representation of the frequency mapping is proposed to allow for an animated gallery interface with genetic reproduction. Such galleries let the user quickly navigate between visual effects. Rendering such animated galleries in real time is a challenging task, since it requires computing and rendering the deformed shapes at a very high rate. This paper introduces a direct GPU implementation of manifold harmonics filters, which allows the displaying of the animated galleries.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Apple: PBORenderToVertexArray: render-to-vertex-array using FBO, PBO and VBO (2006). Developer.apple.com/mac/library/samplecode (2011)
Bellard, F.: FFmpeg (2004). www.ffmpeg.org
Bordignon, A., Sigaud, L., Tavares, G., Lopes, H., Lewiner, T., Morgado, W.: Arch generated shear bands in granular systems. Physica A 388(11), 2099–2108 (2009)
Breebaart, J., Faller, C.: Spatial Audio Processing. Wiley, New York (2007)
Clough, R.W., Penzien, J.: Dynamics of Structures. McGraw-Hill, New York (1975)
Comstock, H.: Radio adds third dimension. Popular Sci. pp. 104–106 (1953)
de Moura Pinto, F., Freitas, C.M.D.S.: Two-level interaction transfer function design combining boundary emphasis, manual specification and evolutive generation. In: Sibgrapi, pp. 281–288. IEEE Press, New York (2006)
Gardner, W.: 3D Audio Using Loudspeakers. Kluwer, Dordrecht (1998)
Hernandez, V., Roman, J., Vidal, V.: SLEPc: A scalable and flexible toolkit for the solution of eigenvalue problems. ACM Trans. Math. Softw. 31(3), 362 (2005)
Hiebert, G.: OpenAL programmer’s guide (2005). Connect.creativelabs.com/openal
Jenny, H.: Cymatics: A Study of Wave Phenomena & Vibration, 3rd edn. Macromedia (2001)
Kessenich, J.: The OpenGL Shading Language v 4.0 (2010). www.opengl.org/documentation/glsl
Kubelka, O.: Interactive music visualization. In: Central European Seminar on Computer Graphics (2000)
Lage, M., Lewiner, T., Lopes, H., Velho, L.: CHF: a scalable topological data structure for tetrahedral meshes. In: Sibgrapi, pp. 349–356. IEEE Press, New York (2005)
Lewiner, T., Vieira, T., Bordignon, A., Cabral, A., Marques, C., Paixão, J., Custódio, L., Lage, M., Andrade, M., Nascimento, R., de Botton, S., Pesco, S., Lopes, H., Mello, V., Peixoto, A., Martinez, D.: Tuning manifold harmonics filters. In: Sibgrapi, pp. 110–117. IEEE Press, New York (2010)
Lewiner, T., Vieira, T., Martínez, D., Peixoto, A., Mello, V., Velho, L.: Interactive 3D caricature from harmonic exaggeration. Comput. Graph. 35(3), 586–595 (2011)
Lévy, B., Zhang, H.R.: Spectral mesh processing. In: Siggraph Asia Course Note, pp. 1–47. ACM Press, New York (2009)
Liu, Y., Prabhakaran, B., Guo, X.: A robust spectral approach for blind watermarking of manifold surfaces. In: Multimedia and Security, pp. 43–52. ACM Press, New York (2008)
Marks, J., Andalman, B., Beardsley, P., Freeman, W., Gibson, S., Hodgins, J., Kang, T., Mirtich, B., Pfister, H., Ruml, W., et al.: Design galleries: a general approach to setting parameters for computer graphics and animation. In: Siggraph, p. 400. ACM Press, New York (1997)
O’Brien, J.F., Shen, C., Gatchalian, C.M.: Synthesizing sounds from rigid-body simulations. In: Symposium on Computer animation, pp. 175–181. ACM Press, New York (2002)
Ovsjanikov, M., Sun, J., Guibas, L.: Global intrinsic symmetries of shapes. In: SGP, pp. 1341–1348. Eurographics, Geneva (2008)
Patin, F.: Beat detection algorithms (2003). www.gamedev.net/reference/programming/features/beatdetection
Pentland, A., Williams, J.: Good vibrations: modal dynamics for graphics and animation. ACM Siggraph 23(3), 207–214 (1989)
Rong, G., Cao, Y., Guo, X.: Spectral mesh deformation. Vis. Comput. 24(7), 787–796 (2008)
Taubin, G.: A signal processing approach to fair surface design. In: Siggraph, pp. 351–358 (1995)
Vallet, B., Lévy, B.: Spectral geometry processing with manifold harmonics. Comput. Graph. Forum, 27, 251–260 (2008)
Vieira, T., Bordignon, A., Peixoto, A., Tavares, G., Lopes, H., Velho, L., Lewiner, T.: Learning good views through intelligent galleries. Comput. Graph. Forum 28(2), 717–726 (2009). (Eurographics Proceedings)
Wang, K., Luo, M., Bors, A., Denis, F.: Blind and robust mesh watermarking using manifold harmonics. In: ICIP, pp. 3657–3660. IEEE Press, New York (2009)
Wu, H.Y., Luo, T., Wang, L., Wang, X.L., Zha, H.: 3D shape retrieval by using manifold harmonics analysis with an augmentedly local feature representation. In: VRCAI, pp. 311–313. ACM Press, New York (2009)
Yinghui, C., Jing, W., Xiaohui, L.: Real-time deformation using modal analysis on graphics hardware. In: Graphite, pp. 173–176. ACM Press, New York (2006)
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Lewiner, T., Marques, C., Paixão, J. et al. Stereo music visualization through manifold harmonics. Vis Comput 27, 905–916 (2011). https://doi.org/10.1007/s00371-011-0617-4
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00371-011-0617-4