Abstract
Recently, there has been an increase in the popularity of multimodal approaches in audio-related tasks, which involve using not only the audible modality but also textual or visual modalities in combination with sound. In this paper, we propose a robust audio representation learning method WavBriVL based on Bridging-Vision-and-Language (BriVL). It projects audio, image and text into a shared embedded space, so that multi-modal applications can be realized. We tested it on some downstream tasks and presented the images rearranged by our method and evaluated them qualitatively and quantitatively. The main purpose of this article is to: (1) Explore new correlation representations between audio and images; (2) Explore a new way to generate images using audio. The experimental results show that this method can effectively do a match on the audio image.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Baevski, A., Zhou, Y., Mohamed, A., Auli, M.: wav2vec 2.0: a framework for self-supervised learning of speech representations. Adv. Neural Inf. Process. Syst. 33, 12449–12460 (2020)
Chen, H., Xie, W., Vedaldi, A., Zisserman, A.: Vggsound: a large-scale audio-visual dataset. In: ICASSP, pp. 721–725. IEEE (2020)
Chen, S., et al.: WavLM: large-scale self-supervised pre-training for full stack speech processing. IEEE J. Sel. Top. Sig. Process. 16(6), 1505–1518 (2022)
Cramer, J., Wu, H.H., Salamon, J., Bello, J.P.: Look, listen, and learn more: Design choices for deep audio embeddings. In: ICASSP, pp. 3852–3856. IEEE (2019)
Cudeiro, D., Bolkart, T., Laidlaw, C., Ranjan, A., Black, M.J.: Capture, learning, and synthesis of 3d speaking styles. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10101–10111 (2019)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
Drossos, K., Lipping, S., Virtanen, T.: Clotho: an audio captioning dataset. In: ICASSP, May 2020. https://arxiv.org/abs/1910.09387
Esser, P., Rombach, R., Ommer, B.: Taming transformers for high-resolution image synthesis. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12873–12883 (2021)
Fanzeres, L.A., Nadeu, C.: Sound-to-imagination: unsupervised crossmodal translation using deep dense network architecture. arXiv preprint arXiv:2106.01266 (2021)
Fei, N., et al.: Towards artificial general intelligence via a multimodal foundation model. Nat. Commun. 13(1), 1–13 (2022)
Guzhov, A., Raue, F., Hees, J., Dengel, A.: Audioclip: extending clip to image, text and audio. arXiv preprint arXiv:2106.13043 (2021)
He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.: Momentum contrast for unsupervised visual representation learning. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9726–9735 (2020). https://doi.org/10.1109/CVPR42600.2020.00975
Ilharco, G., Zhang, Y., Baldridge, J.: Large-scale representation learning from visually grounded untranscribed speech. In: Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pp. 55–65. Association for Computational Linguistics, Hong Kong, China, November 2019. https://doi.org/10.18653/v1/K19-1006, https://aclanthology.org/K19-1006
Jia, C., et al.: Scaling up visual and vision-language representation learning with noisy text supervision. In: International Conference on Machine Learning, pp. 4904–4916. PMLR (2021)
Kaplan, J., et al.: Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 (2020)
Karras, T., Aila, T., Laine, S., Herva, A., Lehtinen, J.: Audio-driven facial animation by joint end-to-end learning of pose and emotion. ACM Trans. Graph. (TOG) 36(4), 1–12 (2017)
Kazakos, E., Nagrani, A., Zisserman, A., Damen, D.: Slow-fast auditory streams for audio recognition. In: ICASSP, pp. 855–859 (2021). https://doi.org/10.1109/ICASSP39728.2021.9413376
Pedersoli, F., Wiebe, D., Banitalebi, A., Zhang, Y., Yi, K.M.: Estimating visual information from audio through manifold learning. arXiv preprint arXiv:2208.02337 (2022)
Piczak, K.J.: ESC: dataset for environmental sound classification. In: ACM Multimedia, p. 1015. ACM Press (2015). https://doi.org/10.1145/2733373.2806390, http://dl.acm.org/citation.cfm?doid=2733373.2806390
Qiu, Y., Kataoka, H.: Image generation associated with music data. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 2510–2513 (2018)
Radford, A., et al.: Learning transferable visual models from natural language supervision. In: ICML (2021)
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10684–10695, June 2022
Salamon, J., Jacoby, C., Bello, J.P.: A dataset and taxonomy for urban sound research. In: ACM Multimedia, pp. 1041–1044. Orlando, FL, USA, Nov 2014
Sung-Bin, K., Senocak, A., Ha, H., Owens, A., Oh, T.H.: Sound to visual scene generation by audio-to-visual latent alignment (2023)
Tan, M., Le, Q.: EfficientNet: rethinking model scaling for convolutional neural networks. In: International Conference on Machine Learning, pp. 6105–6114. PMLR (2019)
Turpault, N., Serizel, R., Parag Shah, A., Salamon, J.: Sound event detection in domestic environments with weakly labeled data and soundscape synthesis. In: DCASE. New York City, United States, October 2019. https://hal.inria.fr/hal-02160855
Wan, C.H., Chuang, S.P., Lee, H.Y.: Towards audio to scene image synthesis using generative adversarial network. In: ICASSP 2019–2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 496–500 (2019). https://doi.org/10.1109/ICASSP.2019.8682383
Wu, H.H., Seetharaman, P., Kumar, K., Bello, J.P.: Wav2clip: learning robust audio representations from clip. In: ICASSP 2022–2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4563–4567. IEEE (2022)
Zhang, L., Agrawala, M.: Adding conditional control to text-to-image diffusion models (2023)
Zhao, P., Chen, Y., Zhao, L., Wu, G., Zhou, X.: Generating images from audio under semantic consistency. Neurocomputing 490, 93–103 (2022)
Zhou, Y., Wang, Z., Fang, C., Bui, T., Berg, T.L.: Visual to sound: generating natural sound for videos in the wild. In: CVPR (2018)
Zhu, H., Luo, M.D., Wang, R., Zheng, A.H., He, R.: Deep audio-visual learning: A survey. Int. J. Autom. Comput. 18(3), 351–376 (2021)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Fang, S., Wu, Y., Gao, B., Cai, J., Teoh, T.T. (2024). Exploring Efficient-Tuned Learning Audio Representation Method from BriVL. In: Luo, B., Cheng, L., Wu, ZG., Li, H., Li, C. (eds) Neural Information Processing. ICONIP 2023. Communications in Computer and Information Science, vol 1969. Springer, Singapore. https://doi.org/10.1007/978-981-99-8184-7_4
Download citation
DOI: https://doi.org/10.1007/978-981-99-8184-7_4
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-8183-0
Online ISBN: 978-981-99-8184-7
eBook Packages: Computer ScienceComputer Science (R0)