Abstract
Dot-Product based attention mechanism is among recent attention mechanisms. It showed an outstanding performance with BERT. In this paper, we propose a dependency-parsing mask to reinforce the padding mask, at the multi-head attention units. Padding mask, is already used to filter padding positions. The proposed mask, aims to improve BERT attention filter. The conducted experiments, show that BERT performs better with the proposed mask.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Graves, A.: Long Short Term Memory. Springer, Cham (2012). https://doi.org/10.1007/978-3-642-24797-2-4
Sepp H., Jürgen S.: Long short-term memory. Neural Comput. 9(8), 1735–1780. PMID 9377276. S2CID 1915014 (1997). https://doi.org/10.1162/neco.1997.9.8.1735
Sak, H., Senior, A., Beaufays, F.: Long Short-Term Memory recurrent neural network architectures for large scale acoustic modeling (2014)
Vaswani, A., et al.: Attention Is All You Need (2017)
Luong, M.-T.: Effective approaches to attention-based neural machine translation (2015)
Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT pre-training of deep bidirectional transformers for language understanding (2018)
Honnibal, M., Montani, I.: spaCy 2 natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. (2017)
Open Sourcing BERT: State-of-the-Art Pre-training for Natural Language Processing. Google AI Blog. Accessed 27 Nov 2019
Clark, K., Khandelwal, U.M, Levy, O., Manning, C.: What Does BERT Look at? An Analysis of BERT’s attention. In: Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP (2019)
Peters, M., et al.: Deep contextualized word representations (2018)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Mechouma, T., Biskri, I., Meunier, J.G. (2022). Reinforcement of BERT with Dependency-Parsing Based Attention Mask. In: Bădică, C., Treur, J., Benslimane, D., Hnatkowska, B., Krótkiewicz, M. (eds) Advances in Computational Collective Intelligence. ICCCI 2022. Communications in Computer and Information Science, vol 1653. Springer, Cham. https://doi.org/10.1007/978-3-031-16210-7_9
Download citation
DOI: https://doi.org/10.1007/978-3-031-16210-7_9
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-16209-1
Online ISBN: 978-3-031-16210-7
eBook Packages: Computer ScienceComputer Science (R0)