skip to main content
10.1145/3632971.3632973acmotherconferencesArticle/Chapter ViewAbstractPublication PagesjcraiConference Proceedingsconference-collections
research-article

A Study on Semantic Understanding of Large Language Models from the Perspective of Ambiguity Resolution

Published: 13 February 2024 Publication History

Abstract

Generative large language models (LLMs) have demonstrated outstanding performance in language understanding and generation tasks. Whether current LLMs can truly understand meaning or not is an open question. Semantic disambiguation is an important indicator to test machine language ability. To this end, three experiments were conducted to test the ambiguity resolution ability. In Experiment 1, the LLMs were asked to make a semantic discrimination on ambiguity phrases. The ChatGPT (May 3 Version) and ChatGPT Plus (May 3 Version) showed a random level performance of discrimination. To make the task easier for the LLMs, ambiguity sentences were presented with context in the Experiment 2. While the LLMs unified the context to fill the position for ambiguity word, the performance of semantic discrimination on these sentences remained at random level. In addition to test the performance of LMMs, prediction is one of the key ability of LLMs. The prediction by unifying the context is one of the key skill for LLMs. Therefore, the Experiment 3 further test the similarity between the human prediction mechanism and the strong generation capabilities of LLMs. Weak similarities were only found under the condition of local constraint features gradually accumulating, specifically, the cosine similarity of predicted sentence semantic vectors and ending word semantic vectors negatively correlated with close probability only. These series of experiments demonstrate that, it is far away to claim that LLMs truly understand human sentences.

References

[1]
Bang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., ... & Fung, P. (2023). A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.04023.
[2]
Wray, R. E., Kirk, J. R., & Laird, J. E. (2021). Language Models as a Knowledge Source for Cognitive Agents. arXiv preprint arXiv:2109.08270.
[3]
Chomsky, N., Roberts, I., & Watumull, J. (2023). Noam Chomsky: The False Promise of ChatGPT. The New York Times, 8.
[4]
Zipf, G. K. (2016). Human behavior and the principle of least effort: An introduction to human ecology. Ravenio Books.
[5]
Piantadosi, S. T., Tily, H., & Gibson, E. (2012). The communicative function of ambiguity in language. Cognition, 122(3), 280-291.
[6]
Xu Bo, Sun Maosong, & Jin Guangjin. (2003). Some important problems in Chinese information processing. Beijing: Science Press. (in chinese).
[7]
Liu, W., Xiang, M., & Ding, N. (2023). Adjective Scale Probe: Can Language Models Encode Formal Semantics Information?.
[8]
Cai, Z. G., Haslett, D. A., Duan, X., Wang, S., & Pickering, M. J. (2023). Does ChatGPT resemble humans in language use?. arXiv preprint arXiv:2303.08014.
[9]
Huang Borong, Liao Xudong. (2017). Modern Chinese. Higher Education Press. (in chinese).
[10]
Lv Shuxiang. (1984). An example of ambiguity. Chinese Language, 5, 321-328. (in chinese).
[11]
Wang Weicheng. (1988). The relationship between syntax, semantics and pragmatics from the perspective of ambiguity. Language Teaching and Research, (1), 32-41. (in chinese).
[12]
Hohwy, J. (2013). The predictive mind. OUP Oxford.
[13]
Machinery, C. (1950). Computing machinery and intelligence-AM Turing. Mind, 59(236), 433.
[14]
Rayner, K., & Duffy, S. A. (1986). Lexical complexity and fixation times in reading: Effects of word frequency, verb complexity, and lexical ambiguity. Memory & cognition, 14(3), 191-201.
[15]
Zhou Zhijin, Chen Yongming, & Yang Lixia. (2003). The process of disambiguation of Chinese homophone ambiguities and its inhibition mechanism. Acta Psychologica Sinica, 35(1), 1-8.
[16]
Meng Qingxin. (2021). A neurophysiological study of predictive processing based on intrasentence lexical production [Unpublished master's thesis]. Jiangsu Normal University.
[17]
Kutas, M., & Hillyard, S. A. (1984). Brain potentials during reading reflect word expectancy and semantic association. Nature, 307(5947), 161-163.
[18]
Wang, J., Zhang, Y., Zhang, L., Yang, P., Gao, X., Wu, Z., ... & Zhang, J. (2022). Fengshenbang 1.0: Being the foundation of chinese cognitive intelligence. arXiv preprint arXiv:2209.02970.
[19]
Goldstein, A., Zada, Z., Buchnik, E., Schain, M., Price, A., Aubrey, B., ... & Hasson, U. (2022). Shared computational principles for language processing in humans and deep language models. Nature neuroscience, 25(3), 369-380.
[20]
Federmeier, K. D., & Kutas, M. (2005). Aging in context: age‐related changes in context use during language comprehension. Psychophysiology, 42(2), 133-141.
[21]
Rabovsky, M., Hansen, S. S., & McClelland, J. L. (2018). Modelling the N400 brain potential as change in a probabilistic representation of meaning. Nature Human Behaviour, 2(9), 693-705.
[22]
Lawrence, J. (1993). Introduction to neural networks. California Scientific Software.

Index Terms

  1. A Study on Semantic Understanding of Large Language Models from the Perspective of Ambiguity Resolution
            Index terms have been assigned to the content through auto-classification.

            Recommendations

            Comments

            Information & Contributors

            Information

            Published In

            cover image ACM Other conferences
            JCRAI '23: Proceedings of the 2023 International Joint Conference on Robotics and Artificial Intelligence
            July 2023
            216 pages
            ISBN:9798400707704
            DOI:10.1145/3632971
            This work is licensed under a Creative Commons Attribution International 4.0 License.

            Publisher

            Association for Computing Machinery

            New York, NY, United States

            Publication History

            Published: 13 February 2024

            Check for updates

            Qualifiers

            • Research-article
            • Research
            • Refereed limited

            Funding Sources

            Conference

            JCRAI 2023

            Contributors

            Other Metrics

            Bibliometrics & Citations

            Bibliometrics

            Article Metrics

            • 0
              Total Citations
            • 77
              Total Downloads
            • Downloads (Last 12 months)77
            • Downloads (Last 6 weeks)7
            Reflects downloads up to 16 Feb 2025

            Other Metrics

            Citations

            View Options

            Login options

            View options

            PDF

            View or Download as a PDF file.

            PDF

            eReader

            View online with eReader.

            eReader

            HTML Format

            View this article in HTML Format.

            HTML Format

            Figures

            Tables

            Media

            Share

            Share

            Share this Publication link

            Share on social media