Abstract
Event representation learning is a crucial prerequisite for many domains in event understanding, which aims to project events into dense vectors, where similar events are embedded close to each other while distinct events are separated. Most existing methods enhanced representation with external knowledge, which only based on unimodal and margin loss, ignoring the diversity of data and the impact of multiple samples. Therefore, we propose a novel Multimodal Event Contrastive Representation Learning (ECRL) framework. ECRL enhances event representations by making better use of image situational information. Specifically, we introduce multimodal contrastive learing and prototype-based clustering methods that allows us to avoid semantic-related samples in same batch being represent as negative, while considering multiple negatives. At the same time, we proposed action-guided-attention mechanism to capture core event participants in image. Extensive experiments on three benchmark datasets of event similarity task show that the ECRL outperforms other baseline.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., Joulin, A.: Unsupervised learning of visual features by contrasting cluster assignments (2020)
Cuturi, M.: Sinkhorn distances: lightspeed computation of optimal transportation distances. Adv. Neural. Inf. Process. Syst. 26, 2292â2300 (2013)
Ding, X., Liao, K., Liu, T., Li, Z., Duan, J.: Event representation learning enhanced with external commonsense knowledge. arXiv preprint arXiv:1909.05190 (2019)
Ding, X., Zhang, Y., Liu, T., Duan, J.: Deep learning for event-driven stock prediction. In: Twenty-Fourth International Joint Conference on Artificial Intelligence (2015)
Ding, X., Zhang, Y., Liu, T., Duan, J.: Knowledge-driven event embedding for stock prediction. In: Proceedings of Coling 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pp. 2133â2142 (2016)
Gao, J., Wang, W., Yu, C., Zhao, H., Ng, W., Xu, R.: Improving event representation via simultaneous weakly supervised contrastive learning and clustering. arXiv preprint arXiv:2203.07633 (2022)
Granroth-Wilding, M., Clark, S.: What happens next? Event prediction using a compositional neural network model. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 30 (2016)
Hong, X., Sayeed, A., Demberg, V.: Learning distributed event representations with a multi-task approach. In: Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics (2018)
Kartsaklis, D., Sadrzadeh, M.: A study of entanglement in a categorical framework of natural language. eprint arxiv (2014)
Lee, H., Hudson, D.A., Lee, K., Manning, C.D.: SLM: learning a discourse language representation with sentence unshuffling. arXiv preprint arXiv:2010.16249 (2020)
Li, M., et al.: Cross-media structured common space for multimedia event extraction. arXiv preprint arXiv:2005.02472 (2020)
Li, Z., Ding, X., Liu, T.: Constructing narrative event evolutionary graph for script event prediction. arXiv preprint arXiv:1805.05081 (2018)
Modi, A., Titov, I.: Learning semantic script knowledge with event embeddings. arXiv preprint arXiv:1312.5198 (2013)
Oord, A.v.d., Li, Y., Vinyals, O.: Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 (2018)
Pennington, J., Socher, R., Manning, C.: GloVe: global vectors for word representation. In: Conference on Empirical Methods in Natural Language Processing (2014)
Schmitz, M., Soderland, S., Bart, R., Etzioni, O., et al.: Open language learning for information extraction. In: Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (2012)
Vijayaraghavan, P., Roy, D.: Lifelong knowledge-enriched social event representation learning. In: Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume (2021)
Weber, N., Balasubramanian, N., Chambers, N.: Event representations with tensor-based compositions. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32 (2018)
Yatskar, M., Zettlemoyer, L., Farhadi, A.: Situation recognition: Visual semantic role labeling for image understanding. In: Computer Vision & Pattern Recognition, pp. 5534â5542 (2016)
Zhang, L., Zhou, D., He, Y., Yang, Z.: MERL: multimodal event representation learning in heterogeneous embedding spaces. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, pp. 14420â14427 (2021)
Acknowledgements
This work was supported by the Major Program of the National Natural Science Foundation of China (No. 61991410) and the Program of the Pujiang National Laboratory (No. P22KN00391).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
Âİ 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Liu, W., Wu, Q., Xie, S., Li, W. (2023). Event Contrastive Representation Learning Enhanced with Image Situational Information. In: Liu, F., Duan, N., Xu, Q., Hong, Y. (eds) Natural Language Processing and Chinese Computing. NLPCC 2023. Lecture Notes in Computer Science(), vol 14303. Springer, Cham. https://doi.org/10.1007/978-3-031-44696-2_27
Download citation
DOI: https://doi.org/10.1007/978-3-031-44696-2_27
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-44695-5
Online ISBN: 978-3-031-44696-2
eBook Packages: Computer ScienceComputer Science (R0)
