Skip to main content
SpringerLink
Log in

Role of Punctuation in Semantic Mapping Between Brain and Transformer Models

  • Conference paper
  • First Online:
Machine Learning, Optimization, and Data Science (LOD 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13811))

Abstract

Modern neural networks specialised in natural language processing (NLP) are not implemented with any explicit rules regarding language. It has been hypothesised that they might learn something generic about language. Because of this property much research has been conducted on interpreting their inner representations. A novel approach has utilised an experimental procedure that uses human brain recordings to investigate if a mapping from brain to neural network representations can be learned. Since this novel approach has been introduced, more advanced models in NLP have been introduced. In this research we are using this novel approach to test four new NLP models to try and find the most brain aligned model. Moreover, in our effort to unravel important information on how the brain processes text semantically, we modify the text in the hope of getting a better mapping out of the models. We remove punctuation using four different scenarios to determine the effect of punctuation on semantic understanding by the human brain. Our results show that the RoBERTa model is most brain aligned. RoBERTa achieves a higher accuracy score on our evaluation than BERT. Our results also show for BERT that when punctuation was removed a higher accuracy was achieved and that as the context length increased the accuracy did not decrease as much as the original results that include punctuation.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    The data and the original code can be found at this link.

  2. 2.

    https://github.com/NeuraSearch/Brain-Transformer-Mapping-Punctuation.git.

References

  1. Acunzo, D.J., Low, D.M., Fairhall, S.L.: Deep neural networks reveal topic-level representations of sentences in medial prefrontal cortex, lateral anterior temporal lobe, precuneus, and angular gyrus. Neuroimage 251, 119005 (2022). https://doi.org/10.1016/j.neuroimage.2022.119005

    Article  Google Scholar 

  2. Beltagy, I., Lo, K., Cohan, A.: SciBERT: A Pretrained Language Model for Scientific Text. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3615–3620. Association for Computational Linguistics, Hong Kong, November 2019. https://doi.org/10.18653/v1/D19-1371

  3. Blank, I.A., Fedorenko, E.: Domain-general brain regions do not track linguistic input as closely as language-selective regions. J. Neurosci. Official J. Soc. Neurosci. 37(41), 9999–10011 (2017). https://doi.org/10.1523/JNEUROSCI.3642-16.2017

    Article  Google Scholar 

  4. Brennan, J., Nir, Y., Hasson, U., Malach, R., Heeger, D.J., Pylkkänen, L.: Syntactic structure building in the anterior temporal lobe during natural story listening. Brain Lang. 120(2), 163–173 (2012). https://doi.org/10.1016/j.bandl.2010.04.002

    Article  Google Scholar 

  5. Caucheteux, C., King, J.R.: Language processing in brains and deep neural networks: computational convergence and its limits. Technical report, BioRxiv, January 2021. https://doi.org/10.1101/2020.07.03.186288. Section: New Results Type: article

  6. Cer, D., et al.: Universal sentence encoder for English. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 169–174. Association for Computational Linguistics, Brussels, November 2018. https://doi.org/10.18653/v1/D18-2029

  7. Chen, Y., Gilroy, S., Maletti, A., May, J., Knight, K.: Recurrent neural networks as weighted language recognizers. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 2261–2271. Association for Computational Linguistics, New Orleans, June 2018. https://doi.org/10.18653/v1/N18-1205

  8. Chong, M., Specia, L., Mitkov, R.: Using natural language processing for automatic detection of plagiarism. In: Proceedings of the 4th International Plagiarism Conference (IPC-2010) (2010)

    Google Scholar 

  9. Clark, K., Luong, M.T., Le, Q.V., Manning, C.D.: ELECTRA: pre-training text encoders as discriminators rather than generators (2020)

    Google Scholar 

  10. Conneau, A., Kruszewski, G., Lample, G., Barrault, L., Baroni, M.: What you can cram into a single \$ &!#* vector: probing sentence embeddings for linguistic properties. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2126–2136. Association for Computational Linguistics, Melbourne, July 2018. https://doi.org/10.18653/v1/P18-1198

  11. Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q.V., Salakhutdinov, R.: Transformer-XL: attentive language models beyond a fixed-length context. arXiv:1901.02860 [cs, stat], June 2019. arXiv: 1901.02860

  12. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171–4186. Association for Computational Linguistics, Minneapolis, June 2019. https://doi.org/10.18653/v1/N19-1423

  13. Ek, A., Bernardy, J.P., Chatzikyriakidis, S.: How does punctuation affect neural models in natural language inference. In: Proceedings of the Probability and Meaning Conference (PaM 2020), pp. 109–116. Association for Computational Linguistics, Gothenburg, June 2020

    Google Scholar 

  14. Etaiwi, W., Naymat, G.: The impact of applying different preprocessing steps on review spam detection. Procedia Comput. Sci. 113, 273–279 (2017). https://doi.org/10.1016/j.procs.2017.08.368

    Article  Google Scholar 

  15. Frank, S.L., Otten, L.J., Galli, G., Vigliocco, G.: The ERP response to the amount of information conveyed by words in sentences. Brain Lang. 140, 1–11 (2015). https://doi.org/10.1016/j.bandl.2014.10.006

    Article  Google Scholar 

  16. Friederici, A.D.: The brain basis of language processing: from structure to function. Physiol. Rev. 91(4), 1357–1392 (2011). https://doi.org/10.1152/physrev.00006.2011

    Article  Google Scholar 

  17. Fyshe, A., Talukdar, P.P., Murphy, B., Mitchell, T.M.: Interpretable semantic vectors from a joint model of brain- and text- based meaning. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 489–499. Association for Computational Linguistics, Baltimore, June 2014. https://doi.org/10.3115/v1/P14-1046

  18. Hale, J., Dyer, C., Kuncoro, A., Brennan, J.R.: Finding syntax in human encephalography with beam search. arXiv:1806.04127, June 2018

  19. Hong, W., Ji, K., Liu, J., Wang, J., Chen, J., Chu, W.: GilBERT: generative vision-language pre-training for image-text retrieval. In: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1379–1388. SIGIR 2021, Association for Computing Machinery, New York, July 2021. https://doi.org/10.1145/3404835.3462838

  20. Howard, J., Ruder, S.: Universal language model fine-tuning for text classification. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 328–339. Association for Computational Linguistics, Melbourne, July 2018. https://doi.org/10.18653/v1/P18-1031

  21. Huth, A.G., de Heer, W.A., Griffiths, T.L., Theunissen, F.E., Gallant, J.L.: Natural speech reveals the semantic maps that tile human cerebral cortex. Nature 532(7600), 453–458 (2016). https://doi.org/10.1038/nature17637

    Article  Google Scholar 

  22. Jain, S., Huth, A.G.: Incorporating context into language encoding models for fMRI. Technical report, BioRxiv, November 2018. https://doi.org/10.1101/327601. Section: New Results Type: Article

  23. Jia, Q., Li, J., Zhang, Q., He, X., Zhu, J.: RMBERT: news recommendation via recurrent reasoning memory network over BERT. In: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1773–1777. SIGIR 2021, Association for Computing Machinery, New York, July 2021. https://doi.org/10.1145/3404835.3463234

  24. Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., Soricut, R.: ALBERT: a lite bert for self-supervised learning of language representations, April 2020

    Google Scholar 

  25. Lerner, Y., Honey, C.J., Silbert, L.J., Hasson, U.: Topographic mapping of a hierarchy of temporal receptive windows using a narrated story. J. Neurosci. 31(8), 2906–2915 (2011). https://doi.org/10.1523/JNEUROSCI.3684-10.2011. publisher: Society for Neuroscience Section: Articles

  26. Linzen, T., Dupoux, E., Goldberg, Y.: Assessing the ability of LSTMs to learn syntax-sensitive dependencies. arXiv:1611.01368, November 2016

  27. Moore, N.: What’s the point? The role of punctuation in realising information structure in written English. Funct. Linguist. 3(1), 1–23 (2016). https://doi.org/10.1186/s40554-016-0029-x

    Article  Google Scholar 

  28. Nishimoto, S., Vu, A.T., Naselaris, T., Benjamini, Y., Yu, B., Gallant, J.L.: Reconstructing visual experiences from brain activity evoked by natural movies. Curr. Biol. 21(19), 1641–1646 (2011). https://doi.org/10.1016/j.cub.2011.08.031

    Article  Google Scholar 

  29. Peng, H., Schwartz, R., Thomson, S., Smith, N.A.: Rational recurrences. arXiv:1808.09357, August 2018

  30. Peters, M.E., et al.: Deep contextualized word representations. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 2227–2237. Association for Computational Linguistics, New Orleans, June 2018. https://doi.org/10.18653/v1/N18-1202

  31. Reddy, A.J., Wehbe, L.: Can fMRI reveal the representation of syntactic structure in the brain? In: Advances in Neural Information Processing Systems, vol. 34, pp. 9843–9856. Curran Associates, Inc. (2021)

    Google Scholar 

  32. Rowling, J.K.: Harry Potter and the Philosopher’s Stone, vol. 1, 1st edn. Bloomsbury Publishing, London (1997)

    Google Scholar 

  33. Sanh, V., Debut, L., Chaumond, J., Wolf, T.: DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter

    Google Scholar 

  34. Shain, C., et al.: ‘Constituent length’ effects in fMRI do not provide evidence for abstract syntactic processing. Preprint, Neuroscience, November 2021. https://doi.org/10.1101/2021.11.12.467812

  35. Søgaard, A.: Evaluating word embeddings with fMRI and eye-tracking. In: Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP, pp. 116–121. Association for Computational Linguistics, Berlin, August 2016. https://doi.org/10.18653/v1/W16-2521

  36. Sudre, G., et al.: Tracking neural coding of perceptual and semantic features of concrete nouns. Neuroimage 62(1), 451–463 (2012). https://doi.org/10.1016/j.neuroimage.2012.04.048

    Article  Google Scholar 

  37. Toneva, M., Wehbe, L.: Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain). In: Advances in Neural Information Processing Systems, vol. 32. Curran Associates, Inc. (2019)

    Google Scholar 

  38. Wehbe, L., Murphy, B., Talukdar, P., Fyshe, A., Ramdas, A., Mitchell, T.: Simultaneously uncovering the patterns of brain regions involved in different story reading subprocesses. PLoS ONE 9(11), e112575 (2014). https://doi.org/10.1371/journal.pone.0112575

    Article  Google Scholar 

  39. Wehbe, L., Vaswani, A., Knight, K., Mitchell, T.: Aligning context-based statistical models of language with brain activity during reading. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 233–243. Association for Computational Linguistics, Doha, October 2014. https://doi.org/10.3115/v1/D14-1030

  40. Weiss, G., Goldberg, Y., Yahav, E.: On the practical computational power of finite precision RNNs for language recognition. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 740–745. Association for Computational Linguistics, Melbourne, July 2018. https://doi.org/10.18653/v1/P18-2117

  41. Zhu, X., Li, T., de Melo, G.: Exploring semantic properties of sentence embeddings. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 632–637. Association for Computational Linguistics, Melbourne, July 2018. https://doi.org/10.18653/v1/P18-2100

  42. Zhuang, L., Wayne, L., Ya, S., Jun, Z.: A robustly optimized BERT pre-training approach with post-training. In: Proceedings of the 20th Chinese National Conference on Computational Linguistics, pp. 1218–1227. Chinese Information Processing Society of China, Huhhot, August 2021

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. NeuraSearch Laboratory, University of Strathclyde, Glasgow, UK

    Zenon Lamprou & Yashar Moshfeghi

  2. University of Glasgow, Glasgow, UK

    Frank Pollick

Authors
  1. Zenon Lamprou
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Frank Pollick
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yashar Moshfeghi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yashar Moshfeghi .

Editor information

Editors and Affiliations

  1. University of Catania, Catania, Italy

    Giuseppe Nicosia

  2. University of Reading, Reading, UK

    Varun Ojha

  3. University of Oxford, Oxford, UK

    Emanuele La Malfa

  4. University of Cambridge, Cambridge, UK

    Gabriele La Malfa

  5. University of Florida, Gainesville, FL, USA

    Panos Pardalos

  6. Free University of Bozen-Bolzano, Bolzano, Italy

    Giuseppe Di Fatta

  7. University of Catania, Catania, Italy

    Giovanni Giuffrida

  8. Dana-Farber Cancer Institute, Boston, MA, USA

    Renato Umeton

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Lamprou, Z., Pollick, F., Moshfeghi, Y. (2023). Role of Punctuation in Semantic Mapping Between Brain and Transformer Models. In: Nicosia, G., et al. Machine Learning, Optimization, and Data Science. LOD 2022. Lecture Notes in Computer Science, vol 13811. Springer, Cham. https://doi.org/10.1007/978-3-031-25891-6_35

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-25891-6_35

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-25890-9

  • Online ISBN: 978-3-031-25891-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation