Skip to main content

Exploring Efficient-Tuned Learning Audio Representation Method from BriVL

  • Conference paper
  • First Online:
Neural Information Processing (ICONIP 2023)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1969))

Included in the following conference series:

  • 801 Accesses

Abstract

Recently, there has been an increase in the popularity of multimodal approaches in audio-related tasks, which involve using not only the audible modality but also textual or visual modalities in combination with sound. In this paper, we propose a robust audio representation learning method WavBriVL based on Bridging-Vision-and-Language (BriVL). It projects audio, image and text into a shared embedded space, so that multi-modal applications can be realized. We tested it on some downstream tasks and presented the images rearranged by our method and evaluated them qualitatively and quantitatively. The main purpose of this article is to: (1) Explore new correlation representations between audio and images; (2) Explore a new way to generate images using audio. The experimental results show that this method can effectively do a match on the audio image.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    https://github.com/descriptinc/lyrebird-wav2clip.

  2. 2.

    https://github.com/BAAI-WuDao/BriVL.

  3. 3.

    https://github.com/microsoft/unilm/tree/master/wavlm.

  4. 4.

    https://github.com/TUT-ARG/sed_eval.

  5. 5.

    https://github.com/leofanzeres/s2i.

  6. 6.

    https://github.com/CompVis/taming-transformers.

References

  1. Baevski, A., Zhou, Y., Mohamed, A., Auli, M.: wav2vec 2.0: a framework for self-supervised learning of speech representations. Adv. Neural Inf. Process. Syst. 33, 12449–12460 (2020)

    Google Scholar 

  2. Chen, H., Xie, W., Vedaldi, A., Zisserman, A.: Vggsound: a large-scale audio-visual dataset. In: ICASSP, pp. 721–725. IEEE (2020)

    Google Scholar 

  3. Chen, S., et al.: WavLM: large-scale self-supervised pre-training for full stack speech processing. IEEE J. Sel. Top. Sig. Process. 16(6), 1505–1518 (2022)

    Article  Google Scholar 

  4. Cramer, J., Wu, H.H., Salamon, J., Bello, J.P.: Look, listen, and learn more: Design choices for deep audio embeddings. In: ICASSP, pp. 3852–3856. IEEE (2019)

    Google Scholar 

  5. Cudeiro, D., Bolkart, T., Laidlaw, C., Ranjan, A., Black, M.J.: Capture, learning, and synthesis of 3d speaking styles. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10101–10111 (2019)

    Google Scholar 

  6. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)

  7. Drossos, K., Lipping, S., Virtanen, T.: Clotho: an audio captioning dataset. In: ICASSP, May 2020. https://arxiv.org/abs/1910.09387

  8. Esser, P., Rombach, R., Ommer, B.: Taming transformers for high-resolution image synthesis. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12873–12883 (2021)

    Google Scholar 

  9. Fanzeres, L.A., Nadeu, C.: Sound-to-imagination: unsupervised crossmodal translation using deep dense network architecture. arXiv preprint arXiv:2106.01266 (2021)

  10. Fei, N., et al.: Towards artificial general intelligence via a multimodal foundation model. Nat. Commun. 13(1), 1–13 (2022)

    Article  Google Scholar 

  11. Guzhov, A., Raue, F., Hees, J., Dengel, A.: Audioclip: extending clip to image, text and audio. arXiv preprint arXiv:2106.13043 (2021)

  12. He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.: Momentum contrast for unsupervised visual representation learning. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9726–9735 (2020). https://doi.org/10.1109/CVPR42600.2020.00975

  13. Ilharco, G., Zhang, Y., Baldridge, J.: Large-scale representation learning from visually grounded untranscribed speech. In: Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pp. 55–65. Association for Computational Linguistics, Hong Kong, China, November 2019. https://doi.org/10.18653/v1/K19-1006, https://aclanthology.org/K19-1006

  14. Jia, C., et al.: Scaling up visual and vision-language representation learning with noisy text supervision. In: International Conference on Machine Learning, pp. 4904–4916. PMLR (2021)

    Google Scholar 

  15. Kaplan, J., et al.: Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 (2020)

  16. Karras, T., Aila, T., Laine, S., Herva, A., Lehtinen, J.: Audio-driven facial animation by joint end-to-end learning of pose and emotion. ACM Trans. Graph. (TOG) 36(4), 1–12 (2017)

    Article  Google Scholar 

  17. Kazakos, E., Nagrani, A., Zisserman, A., Damen, D.: Slow-fast auditory streams for audio recognition. In: ICASSP, pp. 855–859 (2021). https://doi.org/10.1109/ICASSP39728.2021.9413376

  18. Pedersoli, F., Wiebe, D., Banitalebi, A., Zhang, Y., Yi, K.M.: Estimating visual information from audio through manifold learning. arXiv preprint arXiv:2208.02337 (2022)

  19. Piczak, K.J.: ESC: dataset for environmental sound classification. In: ACM Multimedia, p. 1015. ACM Press (2015). https://doi.org/10.1145/2733373.2806390, http://dl.acm.org/citation.cfm?doid=2733373.2806390

  20. Qiu, Y., Kataoka, H.: Image generation associated with music data. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 2510–2513 (2018)

    Google Scholar 

  21. Radford, A., et al.: Learning transferable visual models from natural language supervision. In: ICML (2021)

    Google Scholar 

  22. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10684–10695, June 2022

    Google Scholar 

  23. Salamon, J., Jacoby, C., Bello, J.P.: A dataset and taxonomy for urban sound research. In: ACM Multimedia, pp. 1041–1044. Orlando, FL, USA, Nov 2014

    Google Scholar 

  24. Sung-Bin, K., Senocak, A., Ha, H., Owens, A., Oh, T.H.: Sound to visual scene generation by audio-to-visual latent alignment (2023)

    Google Scholar 

  25. Tan, M., Le, Q.: EfficientNet: rethinking model scaling for convolutional neural networks. In: International Conference on Machine Learning, pp. 6105–6114. PMLR (2019)

    Google Scholar 

  26. Turpault, N., Serizel, R., Parag Shah, A., Salamon, J.: Sound event detection in domestic environments with weakly labeled data and soundscape synthesis. In: DCASE. New York City, United States, October 2019. https://hal.inria.fr/hal-02160855

  27. Wan, C.H., Chuang, S.P., Lee, H.Y.: Towards audio to scene image synthesis using generative adversarial network. In: ICASSP 2019–2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 496–500 (2019). https://doi.org/10.1109/ICASSP.2019.8682383

  28. Wu, H.H., Seetharaman, P., Kumar, K., Bello, J.P.: Wav2clip: learning robust audio representations from clip. In: ICASSP 2022–2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4563–4567. IEEE (2022)

    Google Scholar 

  29. Zhang, L., Agrawala, M.: Adding conditional control to text-to-image diffusion models (2023)

    Google Scholar 

  30. Zhao, P., Chen, Y., Zhao, L., Wu, G., Zhou, X.: Generating images from audio under semantic consistency. Neurocomputing 490, 93–103 (2022)

    Article  Google Scholar 

  31. Zhou, Y., Wang, Z., Fang, C., Bui, T., Berg, T.L.: Visual to sound: generating natural sound for videos in the wild. In: CVPR (2018)

    Google Scholar 

  32. Zhu, H., Luo, M.D., Wang, R., Zheng, A.H., He, R.: Deep audio-visual learning: A survey. Int. J. Autom. Comput. 18(3), 351–376 (2021)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Teik Toe Teoh .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Fang, S., Wu, Y., Gao, B., Cai, J., Teoh, T.T. (2024). Exploring Efficient-Tuned Learning Audio Representation Method from BriVL. In: Luo, B., Cheng, L., Wu, ZG., Li, H., Li, C. (eds) Neural Information Processing. ICONIP 2023. Communications in Computer and Information Science, vol 1969. Springer, Singapore. https://doi.org/10.1007/978-981-99-8184-7_4

Download citation

  • DOI: https://doi.org/10.1007/978-981-99-8184-7_4

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-99-8183-0

  • Online ISBN: 978-981-99-8184-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics