Abstract
Giving machines the ability to understand the intent of human actions is a basic goal of Natural Language Understanding. In the context of that, a task called the Multi-axis Event Processes Typing is proposed, which aims to comprehend the overall goal of an event sequence from the aspect of action and object. Existing works utilize fine-tuning to mine the semantic information of the event processes in the pre-trained language models and achieve good performance. Prompt tuning is effective in fully exploiting the capabilities of pre-trained language models. To mine more sufficient semantic information of the event process, it is crucial to utilize appropriate prompts to guide the pre-trained language models. Moreover, most existing prompt tuning methods use unified prompt encodings. Due to the complex correlations between events of event processes, it is hard to capture context-sensitive semantic information of the event processes. In this paper, we propose PTSTEP, an encoder-decoder based method with continuous prompts. Specifically, we propose a context-aware prompt encoder to obtain a more expressive continuous prompt. Parameters in the pre-trained language model are fixed. On the encoder, the continuous prompt guide the model to mine more semantic information of the event process. On the decoder, the context-aware continuous prompt guide the model to better understand the event processes. PTSTEP outperforms the state-of-the-art method by 0.82% and 3.74% respectively on action MRR and object MRR. The significant improvements prove the effectiveness of our method.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Berant, J., et al.: Modeling biological processes for reading comprehension. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1499–1510 (2014)
Brown, T., et al.: Language models are few-shot learners. Adv. Neural. Inf. Process. Syst. 33, 1877–1901 (2020)
Chaturvedi, S., Peng, H., Roth, D.: Story comprehension for predicting what happens next. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 1603–1614 (2017)
Chen, M., Zhang, H., Wang, H., Roth, D.: What are you trying to do? semantic typing of event processes. In: Fernández, R., Linzen, T. (eds.) Proceedings of the 24th Conference on Computational Natural Language Learning, CoNLL 2020, Online, November 19–20, 2020, pp. 531–542. Association for Computational Linguistics (2020). https://doi.org/10.18653/v1/2020.conll-1.43
Do, Q., Lu, W., Roth, D.: Joint inference for event timeline construction. In: Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pp. 677–687 (2012)
Hambardzumyan, K., Khachatrian, H., May, J.: Warp: Word-level adversarial reprogramming. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 4921–4933 (2021)
Kurby, C.A., Zacks, J.M.: Segmentation in the perception and memory of events. Trends Cogn. Sci. 12(2), 72–79 (2008)
Lester, B., Al-Rfou, R., Constant, N.: The power of scale for parameter-efficient prompt tuning. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3045–3059 (2021)
Lewis, M., et al.: BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Online (2020)
Li, X.L., Liang, P.: Prefix-tuning: Optimizing continuous prompts for generation. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, Online (2021)
Lin, S.T., Chambers, N., Durrett, G.: Conditional generation of temporally-ordered event sequences. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 7142–7157 (2021)
Liu, X., et al.: P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 61–68 (2022)
Liu, Y., et al.: Roberta: A robustly optimized BERT pretraining approach. CoRR abs/1907.11692 (2019)
Mostafazadeh, N., Roth, M., Louis, A., Chambers, N., Allen, J.: Lsdsem 2017 shared task: The story cloze test. In: Proceedings of the 2nd Workshop on Linking Models of Lexical, Sentential and Discourse-level Semantics, pp. 46–51 (2017)
Pepe, S., Barba, E., Blloshmi, R., Navigli, R.: Steps: Semantic typing of event processes with a sequence-to-sequence approach. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 36, pp. 11156–11164 (2022)
Petroni, F., et al.: Language models as knowledge bases? In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2463–2473 (2019)
Rashkin, H., Sap, M., Allaway, E., Smith, N.A., Choi, Y.: Event2mind: Commonsense inference on events, intents, and reactions. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 463–473 (2018)
Sap, M., et al.: Atomic: An atlas of machine commonsense for if-then reasoning. In: Proceedings of the AAAI Conference On Artificial Intelligence, vol. 33, pp. 3027–3035 (2019)
Schick, T., Schmid, H., Schütze, H.: Automatically identifying words that can serve as labels for few-shot text classification. In: Proceedings of the 28th International Conference on Computational Linguistics, pp. 5569–5578 (2020)
Shin, T., Razeghi, Y., Logan IV, R.L., Wallace, E., Singh, S.: Autoprompt: Eliciting knowledge from language models with automatically generated prompts. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4222–4235 (2020)
Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4–9, 2017, Long Beach, CA, USA, pp. 5998–6008 (2017)
Wang, Z., Zhang, H., Fang, T., Song, Y., Wong, G.Y., See, S.: Subeventwriter: Iterative sub-event sequence generation with coherence controller. In: Goldberg, Y., Kozareva, Z., Zhang, Y. (eds.) Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7–11, 2022, pp. 1590–1604. Association for Computational Linguistics (2022). https://aclanthology.org/2022.emnlp-main.103
Zacks, J.M., Tversky, B.: Event structure in perception and conception. Psychol. Bull. 127(1), 3 (2001)
Zhang, H., Chen, M., Wang, H., Song, Y., Roth, D.: Analogous process structure induction for sub-event sequence prediction. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1541–1550 (2020)
Zhang, T., Chen, M., Bui, A.A.T.: Diagnostic prediction with sequence-of-sets representation learning for clinical events. In: Michalowski, M., Moskovitch, R. (eds.) AIME 2020. LNCS (LNAI), vol. 12299, pp. 348–358. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59137-3_31
Zhong, Z., Friedman, D., Chen, D.: Factual probing is [mask]: Learning vs. learning to recall. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5017–5033 (2021)
Acknowledgements
This work was supported by the Strategic Priority Research Program of Chinese Academy of Sciencecs, Grant NO.XDC02040400.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Zhu, W., Xu, Y., Xu, H., Tang, M., Zhu, D. (2023). PTSTEP: Prompt Tuning for Semantic Typing of Event Processes. In: Iliadis, L., Papaleonidas, A., Angelov, P., Jayne, C. (eds) Artificial Neural Networks and Machine Learning – ICANN 2023. ICANN 2023. Lecture Notes in Computer Science, vol 14256. Springer, Cham. https://doi.org/10.1007/978-3-031-44213-1_45
Download citation
DOI: https://doi.org/10.1007/978-3-031-44213-1_45
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-44212-4
Online ISBN: 978-3-031-44213-1
eBook Packages: Computer ScienceComputer Science (R0)