Towards semantically linked multilingual corpus

Highlights

• We propose a framework for semantically linked multilingual corpus.

• We provide a solution to construct semantically linked multilingual corpus.

• Establishing metadata and content based semantic links in different granularities.

• Analyzing multilingual applications of semantically linked multilingual corpus.

• Study the cognitive theoretical basis of semantic associations and semantic links.

Abstract

Multilingual information processing gains more and more attention in recent years with the development of information globalization. Multilingual corpus is a key challenge for multilingual information extraction, analysis, management and service in a wide range of systems. This work addresses on the study and analysis of semantic associations among elements in a multilingual corpus. A solution is proposed in this paper to optimize the semantic organization of multilingual corpus by linking the corpus elements into a semantic link network. This enhances the text-basd applications of multilingual corpus such as corpus linguistics study, dictionary search, machine translation and cross-lingual information retrieval.

Introduction

The rapid development of information and communication technologies enables the information globalization. Multilingual information has been coming into our daily life and business. Users are often puzzled to face with multilingual information resources when surfing on the Web because most of them are only familiar with one or two natural languages. It is necessary to find effective ways to bridge the gap caused by different languages, and multilingual information processing has been gaining more and more attention in recent years.

Multilingual information processing includes organization, search, translation, management and analysis, where organization is the primary basis for other services. An intuitive idea for multilingual information organization is to weave them into a network with semantic associations. Semantic association is a concept from the cognitive science. When a concept is mentioned, other concepts occurring in the mind of human being are considered “having semantic association with” the mentioned concept. Semantic associations among resources have great influence on information search, and finding relevant multilingual information resources is the basis of the utilizing of multilingual information. Multilingual information resources and semantic associations among them formulate a complex network, which is the research object of intelligence analysis. So it is important to establish semantic associations among multilingual information resources for management, search and analysis.

Semantic associations exist in information systems ubiquitously, in which links are used to reflect the semantic associations among the multilingual information resources. For example, hyperlinks among the web pages imply the semantic associations among information resources represented by URI in the Web; semantic associations may be relationships in the specific domains such as refer and cite in the scientific literature. Some associations are hard to be represented by specific relationships, such as similar and relevant. Similar is a specific instance of relevant. If two things are similar, they must be also relevant; however, if two things are relevant, they may not be similar.

Multilingual information services are based on the result of multilingual information processing. Machine translation (MT) and cross-language information retrieval (CLIR) are two typical multilingual information services. They have the same ultimate aim – semantically linking multilingual information for users to easily access without language barriers. MT aims to translate text into other foreign natural languages, while CLIR aims to find the relevance between information resources in different natural languages. So users can use CLIR to find relevant information, and then use MT to translate the text for further reading and analysis.

Multilingual corpora are necessary for multilingual information processing systems such as MT and CLIR. Besides, multilingual corpora are important for contrastive linguistics (McEnery & Hardie, 2011). In linguistics, a corpus is a large and structured set of texts. Currently, corpora are electronically stored and processed, and they have been widely used in statistical analysis and hypothesis testing, checking occurrences or validating linguistic rules within a specific language territory. Corpora have different categories according to different perspectives:

• Monolingual corpus and multilingual corpus. A corpus containing texts in a single language is called monolingual corpus, while a corpus containing text data in multiple languages is called multilingual corpus. Multilingual corpora that have been specially formatted for side-by-side comparison are called aligned parallel corpora, which are heavily depended by the Statistical MT systems.

• Raw corpus and labeled corpus. Raw corpus contains the plain text without any manual annotation, while labeled corpus needs manual annotations on the collected texts. The manual annotation tasks mainly include word segmentation, part-of-speech tagging, syntactic annotation, semantic annotation and pragmatic/discourse annotation.

Web has been regarded as a large-scale multilingual corpus (Liu & Curran, 2006). Massive multilingual information is emerging quickly on the Web every time, social networks (He, Zha, & Li, 2013), cloud computing (Yee, Chia, Tsai, Tiong, & Kanagasabai, 2011) and it is expected to increase with the extension of the Web to the Web of Everything (Jara et al., 2014). However, there are no explicit semantic links or even no explicit hyperlinks among the multilingual information resources. Information resources are massive, but connections between them are sparse and the information resources formulate many information islands. This becomes an embarrassment of efficient access and use of multilingual information. Information resources are urgent to be semantically connected by establishing semantic links among multilingual information resources automatically in order to solve the described problem of information islands. Therefore, linking multilingual documents becomes one of the key challenges for the multilingual information effective sharing and use.

To establish semantic links among information resources, there are two feasible ways. One way is to build semantic links according to their attributes which are often contained in metadata of information resources, and the other way is to define semantic links according to the meaning implied in the textual descriptions. Metadata of information resources could be used to link multilingual documents based on the values of attributes ([Zhang et al., 2010], [Zhuge and Zhang, 2011]). An extensible semantic model is proposed for information management by organizing semantic data with event-linked network where events and links can be extracted from raw data in the Internet of Things (Sun et al., 2014; Sun and Jara, 2014 ). However, many multilingual documents on the Web have no complete metadata. Consequently, text analysis based on natural language processing becomes the alternative way to establishing the semantic links based on the full texts of documents. The processing objects of natural language processing include word, phrase, sentence and text (paragraph, section or document).

Establishing semantic links between multilingual information resources depends on the analysis on words, phrases and sentences in the descriptive text. All the information resources such as documents, images, audios and videos can be all described by text in natural languages. The establishment of attribute-based semantic associations has been discussed in the previous work (Zhang et al., 2013), so we focus on the content-based semantic associations among documents in this paper. To process the content of documents, natural language processing is a necessary technology, especially in establishing semantic links based on text analysis.

In this paper, a semantically linked multilingual corpus in science and technology has been designed for the following aims: (1) providing a multilingual corpus for natural language processing research such as machine translation in science and technology; (2) providing support for applications based on the semantically linked corpus such as multilingual information analysis and cross-lingual information retrieval. The main contributions include: (1) a framework for constructing semantically linked multilingual corpus based on the metadata and content of texts; (2) a solution to construct semantically linked multilingual corpus; and (3) the feasible multilingual applications based on the semantically linked multilingual corpus.