Skip to main content
Log in

Hearing with the eyes: modulating lyrics typography for music visualization

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

In human–computer interaction (HCI), typography was initially used for visual communication, which enhanced visual interest in graphic design. The investigation of how modulating visual elements (e.g., typography) to visualize sound (e.g., voice) has received substantial attention. Musical lyrics typography is a commonly used form of visual communication. However, the mapping of musical features to lyrics typography features is rarely diverse studied. And modulating lyrics typography from musical features by a certain model may not strongly arouse people’s perception of auditory and visual connections. In this paper, we first proposed several models of modulating typography from musical features. Then, we investigated which model can modulate lyrics typography well to visualize music. Finally, the experiment results show that the lyrics typography by the mapping of musical features (loudness, pitch, and duration) and typography feature (character size, baseline shift, and character width) can better arouse people’s perception of auditory and visual connection. And the lyrics typography modulated by a moderate mapping parameter can evoke people’s high visual aesthetic preference. We aspire for our work to offer a novel perspective on music visualization, assisting the hard-of-hearing people to experience musical content.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data availability

Data will be made available on reasonable request.

References

  1. Fadillah, C., Rahayu, R. R. A.: Sound Visualization Using Typography Composition Based GIF. In: 2019 International Conference on Sustainable Engineering and Creative Computing (ICSECC) pp. 309–314. IEEE. (2019)

  2. Lima, H.B., Santos, C.G.D., Meiguins, B.S.: A survey of music visualization techniques. ACM Comput Surv (CSUR) 54(7), 1–29 (2021)

    Article  Google Scholar 

  3. Castro, J. C. E., Martins, P., Boavida, A., and Machado, P.: «Máquina de Ouver»-from sound to type: finding the visual representation of speech by mapping sound features to typographic variables. In: Proceedings of the 9th International Conference on Digital and Interactive Arts pp. 1–8. (2019)

  4. de Lacerda Pataca, C., Costa, P.D.P.: Hidden bawls, whispers, and yelps: can text convey the sound of speech, beyond words? IEEE Trans. Affect. Comput. 14(1), 6–16 (2022)

    Article  Google Scholar 

  5. de Lacerda Pataca, C., Costa, P. D. P.: Speech modulated typography: towards an affective representation model. In: Proceedings of the 25th International Conference on Intelligent User Interfaces pp. 139–143. (2020)

  6. Parente, J., Martins, T., Bicker, J., Machado, P.: Speechtyper: From speech to typographic composition. In International Conference on Computational Intelligence in Music, Sound, Art and Design (Part of EvoStar) pp. 212–227. Cham: Springer International Publishing (2022)

  7. Matthias, W., Tim, S., and Angelo, S.: Voice driven type design. In: International Conference on Speech Technology and Human-Computer Dialogue, SpeD 2015, Bucharest, Romania, October 14–17, IEEE, pp. 1–9 (2015)

  8. de Lacerda Pataca, C., Watkins, M., Peiris, R., Lee, S., Huenerfauth, M.: Visualization of Speech Prosody and Emotion in Captions: Accessibility for Deaf and Hard-of-Hearing Users [n.d.]

  9. Jun, Y., Lee, H.: A sound brand identity design: The interplay between sound symbolism and typography on brand attitude and memory. J. Retail. Consum. Serv. 64(2022), 102724 (2022)

    Article  Google Scholar 

  10. Schlippe, T., Alessai, S., El-Taweel, G., Wölfel, M., Zaghouani, W.: Visualizing voice characteristics with type design in closed captions for arabic. In: 2020 International Conference on Cyberworlds (CW) pp. 196–203. IEEE. (2020)

  11. Kato, J., Nakano, T., Goto, M.: TextAlive: Integrated design environment for kinetic typography. In: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems pp. 3403-3412. (2015)

  12. Miniukovich, A., Scaltritti, M., Sulpizio, S., & De Angeli, A.: Guideline-based evaluation of web readability. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems pp. 1–12. (2019)

  13. Zhiquan, Y., Scott, E.H.: KTE2: an engine for kinetic typography. In: Proceedings of the 27th International Conference on Human Factors in Computing Systems, CHI 2009, Extended Abstracts Volume, Boston, MA, USA, April 4–9, 2009, Dan R. Olsen Jr., Richard B. Arthur, Ken Hinckley, Meredith Ringel Morris, Scott E. Hudson, and Saul Greenberg (Eds.). ACM, pp. 3413–3418 (2009)

  14. Lee, J., Hayashi, M., Nosaka, M.: The effect of a visual element on musical sensitivity. In: HCI International 2011–Posters’ Extended Abstracts: International Conference, HCI International 2011, Orlando, FL, USA, July 9-14, 2011, Proceedings, Part I 14 (pp 303–307). Springer Berlin Heidelberg. (2011)

  15. Fassnidge, C., Ball, D., Kazaz, Z., et al.: Hearing through your eyes: neural basis of audiovisual cross-activation, revealed by transcranial alternating current stimulation[J]. J. Cogn. Neurosci. 31(6), 922–935 (2019)

    Article  Google Scholar 

  16. Raden, A.Z.M., Qeis, M.I.: Song and typography: expressing the lyrics visually through lyrical typography[J]. Int. J. Sci. Technol. Res. 8, 61–64 (2019)

    Google Scholar 

  17. Lee, S. W., Essl, G.: Web-based temporal typography for musical expression and performance. In: NIME pp. 65–69 (2015)

  18. Mol, E.: ‘Trying to hear with the eyes’: slow looking and ontological difference in archaeological object analysis[J]. Nor. Archaeol. Rev. 54(1–2), 80–99 (2021)

    Article  Google Scholar 

  19. Eikelenboom, M., Wattel, R., De Vet, M.: Listening with your eyes: an accessible museum for deaf and hard-of-hearing visitors[J]. Int J Inclus Museum 12(3), 51 (2019)

    Article  Google Scholar 

  20. Guttman, S.E., Gilroy, L.A., Blake, R.: Hearing what the eyes see: Auditory encoding of visual temporal sequences[J]. Psychol. Sci. 16(3), 228–235 (2005)

    Article  Google Scholar 

  21. Pentimalli, B., Gobo, G.: “Hearing with the eyes” visual hearing in (a trio) music rehearsals[J]. Sens. Soc., 1–19 (2023)

  22. Katz, M.: Hearing through our eyes: musical archives and authentic performance[J]. Pop. Music Soc. 31(4), 511–527 (2008)

    Article  MathSciNet  Google Scholar 

  23. Rosenkvist, A., Eriksen, D.S., Koehlert, J., et al.: Hearing with eyes in virtual reality[C]. In: 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, pp. 1349–1350 (2019)

  24. MacDonald, J., Andersen, S., Bachmann, T.: Hearing by eye: how much spatial degradation can be tolerated?[J]. Perception 29(10), 1155–1168 (2000)

    Article  Google Scholar 

  25. Ludden, D.: Hearing with our eyes, seeing with our ears[J]. Psychology Today, 19 (2015)

  26. Agrawal, C., Peiris, R.L.: I see what you’re saying: A literature review of eye tracking research in communication of deaf or hard of hearing users[C] In: Proceedings of the 23rd International ACM SIGACCESS Conference on Computers and Accessibility. pp. 1–13 (2021)

  27. Toledo, J., Torres, J., Alonso, S., et al.: SLOAS: hearing with the eyes[C] In: Proceedings of the 4th WSEAS international conference on electronic, signal processing and control. 27 (2005)

  28. Deja, J.A., Dela Torre, A., Lee, H.J., et al.: Vitune: a visualizer tool to allow the deaf and hard of hearing to see music with their eyes[C]//extended abstracts of the. CHI Conf Human Factor Comput Syst. 2020, 1–8 (2020)

    Google Scholar 

  29. Chan, W.Y., Qu, H., Mak, W.H.: Visualizing the semantic structure in classical music works. IEEE Trans Visualizat Comput Graph 16(1), 161–173 (2009)

    Article  Google Scholar 

  30. Chew, E., François, A.R.: Interactive multi-scale visualizations of tonal evolution in MuSA. RT Opus 2. Comput Entertain (CIE) 3(4), 1–16 (2005)

    Google Scholar 

  31. Lima, H., Santos, C., Meiguins, B.: Visualizing the semantics of music. In 2019 23rd International Conference Information Visualisation (IV) pp. 352–357. IEEE. (2019)

  32. Ciuha, P., Klemenc, B., & Solina, F.: Visualization of concurrent tones in music with colours. In: Proceedings of the 18th ACM international conference on Multimedia pp. 1677–1680. (2010)

  33. Hiraga, R., Mizaki, R., Fujishiro, I.: Performance visualization: a new challenge to music through visualization. In: Proceedings of the tenth ACM international conference on Multimedia pp. 239–242. (2002)

  34. Malandrino, D., Pirozzi, D., Zaccagnino, G., Zaccagnino, R.: A color-based visualization approach to understand harmonic structures of musical compositions. In: 2015 19th International Conference on Information Visualisation pp. 56–61. IEEE. (2015)

  35. De Prisco, R., Malandrino, D., Pirozzi, D., Zaccagnino, G., Zaccagnino, R.: Understanding the structure of musical compositions: Is visualization an effective approach? Inf Visualizat 16(2), 139–152 (2017)

    Article  Google Scholar 

  36. Kim, J., Ananthanarayan, S., Yeh, T.: Seen music: ambient music data visualization for children with hearing impairments. In: Proceedings of the 14th International Conference on Interaction Design and Children pp. 426–429. (2015)

  37. Lehtiniemi, A., Holm, J.: Using animated mood pictures in music recommendation. In 2012 16th International Conference on Information Visualisation pp. 143–150. IEEE. (2012)

  38. Fonteles, J.H., Rodrigues, M.A.F., Basso, V.E.D.: Creating and evaluating a particle system for music visualization. J Visual Languages Comput 24(6), 472–482 (2013)

    Article  Google Scholar 

  39. Cantareira, G.D., Nonato, L.G., Paulovich, F.V.: Moshviz: A detail+ overview approach to visualize music elements. IEEE Trans Multimed 18(11), 2238–2246 (2016)

    Article  Google Scholar 

  40. Mitroo, J.B., Herman, N., Badler, N.I.: Movies from music: visualizing musical compositions. ACM SIGGRAPH Comput Graph 13(2), 218–225 (1979)

    Article  Google Scholar 

  41. Bergstrom, T., Karahalios, K., & Hart, J. C.: Isochords: visualizing structure in music. In: Proceedings of Graphics Interface 2007 pp. 297-304 (2007)

  42. Fujishiro, I., Haga, N., Nakayama, M.: SeeGroove: Supporting groove learning through visualization. In: 2015 International Conference on Cyberworlds (CW) pp. 189–192. IEEE. (2015)

  43. Machida, W., Itoh, T.: Lyricon: A visual music selection interface featuring multiple icons. In: 2011 15th International Conference on Information Visualisation pp. 145–150 IEEE. (2011)

  44. Gumulia, A., Puzon, B., Kosugi, N.: Music visualization: predicting the perceived speed of a composition--misual project--. In: Proceedings of the 19th ACM international conference on Multimedia pp. 949–952 (2011)

  45. Snydal, J., Hearst, M.: ImproViz: visual explorations of jazz improvisations. In: CHI'05 extended abstracts on Human factors in computing systems pp. 1805–1808 (2005)

  46. Hiraga, R., Watanabe, F., Fujishiro, I.: Music learning through visualization. In: Second International Conference on Web Delivering of Music, 2002. WEDELMUSIC 2002. Proceedings. pp. 101–108 IEEE (2002)

  47. Isaacson, E. J.: What You See Is What You Get: on Visualizing Music. In: ISMIR (pp. 389-395). (2005)

  48. Aoki, T., Chujo, R., Matsui, K., Choi, S., and Hautasaari, A.: Emoballoon-conveying emotional arousal in text chats with speech balloons. In: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems pp. 1-16 (2022)

  49. Mardirossian, A., Chew, E.: Visualizing Music: Tonal Progressions and Distributions. In: ISMIR pp. 189–194 (2007)

  50. Lim, S., Kim, S.: Implementation of Kinetic Typography by Motion Recognition Sensor. In: Jong Hyuk) Park, Yi Pan, Cheon-Shik Kim, and Yun Yang, (ed.) Future Information Technology, James J, pp. 503–507. Springer, Berlin Heidelberg, Berlin, Heidelberg (2014)

    Chapter  Google Scholar 

  51. Turgut, O.P.: Kinetic typography in movie title sequences. Procedia-Soc Behav Sci 51, 583–588 (2012)

    Article  Google Scholar 

  52. Özden, P.T.: Calligraphic forms in contemporary typographic design. Procedia—Social and Behavioral Sciences 122, 40–45. 2nd World Conference on Design, Arts and Education (DAE-2013). (2014)

  53. Fellows, K. S.: Typecast: the voice of typography (Doctoral dissertation, University of Iowa). (2009)

  54. Kadner, F., Keller, Y., Rothkopf, C.: Adaptifont: Increasing individuals’ reading speed with a generative font model and Bayesian optimization. In: Proceedings of the 2021 chi conference on human factors in computing systems pp. 1–11 (2021)

  55. Ekstrom, M.P.: Digital image processing techniques[M]. Academic Press, (2012)

  56. So, C.: Measuring aesthetic preferences of neural style transfer: more precision with the two-alternative-forced-choice task[J]. Int J Human-Comput Interact 39(4), 755–775 (2023)

    Article  Google Scholar 

  57. Koch, B. E.: Human emotion response to typographic design (Doctoral dissertation, University of Minnesota). (2011)

  58. Lim, S.: A study on the interactive expression of human emotions in typography. Int J Adv Cult Technol 10(1), 122–130 (2022)

    MathSciNet  Google Scholar 

  59. Maçãs, C., Palma, D., Rebelo, A.: Typem: A generative typeface that represents the emotion of the text. In: Proceedings of the 9th international conference on digital and interactive arts pp. 1–10 (2019)

Download references

Acknowledgements

This work is supported by the National Key R&D Program of China (2022YFB3303301), and the National Natural Science Foundation of China (No. 62107035 and No. 62006208).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Weitao You.

Ethics declarations

Conflict of interest

The authors have no competing interests to declare that are relevant to the content of this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Han, K., You, W., Shi, S. et al. Hearing with the eyes: modulating lyrics typography for music visualization. Vis Comput 40, 8345–8361 (2024). https://doi.org/10.1007/s00371-023-03239-5

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-023-03239-5

Keywords

Navigation