Deterministic Japanese Word Segmentation by Decision List Method

Abstract

In Japanese natural language processing, morphological analysis is a very important technique, and many methods for it have been proposed. The task of Japanese morphological analysis is essentially word segmentation. In this study, we propose a new method of Japanese word segmentation. Our method regards word segmentation as the classification problem and solves it by the decision list method. The advantage of our method is that it avoids the unknown word problem because it is a kind of character based method. Another advantage is that it is deterministic, and the time taken for deterministic analysis is proportional to the length of the sentence. Moreover, our approach can use various features to solve the classification problem, and various machine learning methods.