Analysing performance in a word prediction system with multiple prediction methods

Abstract

In this article, we investigate the performance of a hybrid prediction system with a phrase prediction utility in English word prediction from two viewpoints. From the application user’s point of view, measures of effort savings are important in word prediction. Global performance measures such as the average percentage of keystroke or character savings, however, hide rather than display the details of the functioning of the prediction system as a whole. In the present study, we analysed in detail the performance of a prediction system with a phrase prediction utility along with single word prediction. Our preliminary results with a corpus of 383 lexical bundles show that, from a technological viewpoint, the following three parameters affect the practical utility of the phrase prediction method in a hybrid prediction system: (1) cost of selecting an appropriate prediction mode for single word prediction and phrase prediction; (2) token frequency of phrases in the text predicted, and (3) coverage of the phrasal lexicon. We found that all three affect the phrase prediction performance in different proportions. When the percent of ambiguous search keys finding both phrases and single words is 20%, phrase frequency 35%, and coverage of the phrasal lexicon 98%, the character savings percentage for the whole text will be improved by 6% points under optimal conditions. The system is practically useful as long as an appropriate prediction mode can be selected automatically or the cost of disambiguation of a prediction mode is not too high.

Introduction

Like in many language technology applications in general (Dale et al., 2000, vii), the problem of word prediction boils down to finding a linguistically relevant mapping between the user input and system output. As user input, one or more word-initial characters, for instance, can be used in word completion, possibly matching a certain number of word tokens in the lexicon employed in the prediction system.

In practice, multiple prediction methods will have to be used in the same system to result in accurate enough predictions. Along with single word prediction, phrases consisting of two or more word tokens could be predicted in a hybrid prediction system with a phrase prediction utility. That kind of approach to word prediction is justified, because the token frequency of phrases appears to be sufficiently high in English texts in addition to being capable of modelling text structure as an alternation between single words and phrases (Erman and Warren, 2000). The German FASTY system developed by Matiasek et al. (2002) is an example of a prediction system with multiple prediction methods.

Regarding the quantification of word prediction performance, global measures of effort savings such as average keystroke or character savings percentages are common in present word prediction. A more detailed analysis of the functioning of different prediction techniques is, however, required from the point of view of technology. Apart from showing how various parameters of the prediction process proper actually affect the operation of a prediction system as a whole, an elaborated analysis of the performance of a novel prediction technique may reveal how that prediction method may be improved for its most optimal use.

A detailed analysis of the performance of a hybrid word prediction system with a phrase prediction utility alongside single word prediction is presented in the empirical part of this study, with a corpus of 383 lexical bundles and one text containing phrases of various kinds.

The following three parameters affect the performance of a hybrid prediction system with a phrase prediction utility: (1) cost of selecting an appropriate prediction mode for single word prediction and phrase prediction when it cannot be selected automatically based on the search key typed in; (2) phrase frequency in the predicted text, which must be sufficiently high to result in a significant performance improvement, say, about 5–10%; (3) coverage of the phrasal lexicon which must contain at least most of the phrases predicted. The hybrid prediction method presented here completes other prediction techniques such as a stipulated prediction method proposed by Shieber and Baker (2003), where a sequence of consonants are expanded into matching word tokens. Church and Thiesson (2005) suggest the use of wildcards in word prediction under degraded conditions.

The problem addressed in the experimental part of the study consists of determining the effect of the above three parameters, affecting the accuracy of a hybrid prediction system with a phrase prediction utility. To our knowledge, detailed analyses of parameters impacting on the prediction accuracy in that kind of prediction system are not yet reported in the literature, especially results regarding the cost of selecting an appropriate prediction technique.

The structure of the remaining part of the article is as follows: we first introduce word prediction in general terms, including basic prediction techniques currently used in commercial word prediction. Then we discuss measurement of word prediction performance from the point of view of the application user and technology, respectively. Section 4.2 attempts to show how the three parameters affect prediction accuracy and the operation of a hybrid prediction system as whole.

At the end of the article, we draw conclusions on the practical utility of phrase prediction as an additional prediction technique used together with single word prediction in a hybrid prediction system.