Anupiya Nugaliyadde
ABSTRACT
Word representations are created using analogy context-based statistics and lexical relations on words. Word representations are inputs for the learning models in Natural Language Understanding (NLU) tasks. However, to understand language, knowing only the context is not sufficient. Reading between the lines is a key component of NLU. Embedding deeper word relationships which are not represented in the context enhances the word representation. This paper presents a word embedding which combines an analogy, context-based statistics using Word2Vec, and deeper word relationships using Conceptnet, to create an expanded word representation. In order to fine-tune the word representation, Self-Organizing Map is used to optimize it. The proposed word representation is compared with semantic word representations using Simlex 999. Furthermore, the use of 3D visual representations has shown to be capable of representing the similarity and association between words. The proposed word representation shows a Spearman correlation score of 0.886 and provided the best results when compared to the current state-of-the-art methods, and exceed the human performance of 0.78.
- T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean, "Distributed representations of words and phrases and their compositionality," in Advances in neural information processing systems, 2013, pp. 3111--3119. Google Scholar
Digital Library
- G. Recski, E. Iklódi, K. Pajkossy, and A. Kornai, "Measuring semantic similarity of words using concept networks," in Proceedings of the 1st Workshop on Representation Learning for NLP, 2016, pp. 193--200.Google Scholar
- B. Dhingra, H. Liu, R. Salakhutdinov, and W. W. Cohen, "A comparative study of word embeddings for reading comprehension," arXiv preprint arXiv:1703.00993, 2017.Google Scholar
- R. Collobert and J. Weston, "A unified architecture for natural language processing: Deep neural networks with multitask learning," in Proceedings of the 25th international conference on Machine learning, 2008, pp. 160--167: ACM. Google ScholarDigital Library
- A. Kumar et al., "Ask me anything: Dynamic memory networks for natural language processing," in International Conference on Machine Learning, 2016, pp. 1378--1387. Google ScholarDigital Library
- A. Nugaliyadde, K. W. Wong, F. Sohel, and H. Xie, "Reinforced memory network for question answering," in International Conference on Neural Information Processing, 2017, pp. 482--490: Springer.Google Scholar
- D. Bahdanau, K. Cho, and Y. Bengio, "Neural machine translation by jointly learning to align and translate," arXiv preprint arXiv:1409.0473, 2014.Google Scholar
- T.-H. Wen, M. Gasic, N. Mrksic, P.-H. Su, D. Vandyke, and S. Young, "Semantically conditioned lstm-based natural language generation for spoken dialogue systems," arXiv preprint arXiv:1508.01745, 2015.Google Scholar
- X. Zhang and Y. LeCun, "Text understanding from scratch," arXiv preprint arXiv:1502.01710, 2015.Google Scholar
- J. P. Chiu and E. Nichols, "Named entity recognition with bidirectional LSTM-CNNs," arXiv preprint arXiv:1511.08308, 2015.Google Scholar
- T. Mikolov, K. Chen, G. Corrado, and J. Dean, "Efficient estimation of word representations in vector space," arXiv preprint arXiv:1301.3781, 2013.Google Scholar
- J. Pennington, R. Socher, and C. Manning, "Glove: Global vectors for word representation," in Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), 2014, pp. 1532--1543.Google Scholar
- C. Potts, D. Lassiter, R. Levy, and M. C. Frank, "Embedded implicatures as pragmatic inferences under compositional lexical uncertainty," Journal of Semantics, vol. 33, no. 4, pp. 755--802, 2016.Google Scholar
- A. Nugaliyadde, K. W. Wong, F. Sohel, and H. Xie, "Multi-level Search of a Knowledgebase for Semantic Parsing," in International Workshop on Multi-disciplinary Trends in Artificial Intelligence, 2017, pp. 44--53: Springer.Google Scholar
- H. Liu and P. Singh, "ConceptNet---a practical commonsense reasoning tool-kit," BT technology journal, vol. 22, no. 4, pp. 211--226, 2004. Google ScholarDigital Library
- F. Hill, R. Reichart, and A. Korhonen, "Simlex-999: Evaluating semantic models with (genuine) similarity estimation," Computational Linguistics, vol. 41, no. 4, pp. 665--695, 2015. Google ScholarDigital Library
- J. A. Bullinaria and J. P. Levy, "Extracting semantic representations from word co-occurrence statistics: A computational study," Behavior research methods, vol. 39, no. 3, pp. 510--526, 2007.Google Scholar
- R. Lebret and R. Collobert, "Word emdeddings through hellinger PCA," arXiv preprint arXiv:1312.5542, 2013.Google Scholar
- T. Mikolov, W.-t. Yih, and G. Zweig, "Linguistic regularities in continuous space word representations," in Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2013, pp. 746--751.Google Scholar
- Y. Goldberg and O. Levy, "word2vec Explained: deriving Mikolov et al.'s negative-sampling word-embedding method," arXiv preprint arXiv:1402.3722, 2014.Google Scholar
- R. Speer, J. Chin, and C. Havasi, "ConceptNet 5.5: An Open Multilingual Graph of General Knowledge," in AAAI, 2017, pp. 4444--4451. Google ScholarDigital Library
- R. Speer and J. Chin, "An ensemble method to produce high-quality word embeddings," arXiv preprint arXiv:1604.01692, 2016.Google Scholar
- G. N. Leech, Principles of pragmatics. Routledge, 2016.Google ScholarCross Ref
- T. Kohonen, "The self-organizing map," Neurocomputing, vol. 21, no. 1-3, pp. 1--6, 1998.Google Scholar
- L. Finkelstein et al., "Placing search in context: The concept revisited," in Proceedings of the 10th international conference on World Wide Web, 2001, pp. 406--414: ACM. Google ScholarDigital Library
- J. Weston et al., "Towards ai-complete question answering: A set of prerequisite toy tasks," arXiv preprint arXiv:1502.05698, 2015.Google Scholar
Index Terms
- Enhancing Semantic Word Representations by Embedding Deep Word Relationships
Recommendations
Adaptive Probabilistic Word Embedding
WWW '20: Proceedings of The Web Conference 2020Word embeddings have been widely used and proven to be effective in many natural language processing and text modeling tasks. It is obvious that one ambiguous word could have very different semantics in various contexts, which is called polysemy. Most ...
Improving Vietnamese WordNet using word embedding
NLPIR '19: Proceedings of the 2019 3rd International Conference on Natural Language Processing and Information RetrievalThis paper presents a simple but effective method to improve the quality of WordNet synsets and extract glosses for synsets. We translate the Princeton WordNet and other intermediate WordNets to a target language using a machine translator, then the ...
Incorporating Prior Knowledge into Word Embedding for Chinese Word Similarity Measurement
Word embedding-based methods have received increasing attention for their flexibility and effectiveness in many natural language-processing (NLP) tasks, including Word Similarity (WS). However, these approaches rely on high-quality corpus and neglect ...
Comments