A procedure for the evaluation and improvement of an MT system by the end-user

Abstract

Evaluation of MT systems can be performed by developers or by end-users. In the following paper, we present an evaluation procedure (β-test) which involves both. It will be used to evaluate the ARIANE French to English system.

The quality of the MT system will be evaluated in terms of efficiency and up-grading capacity. Several thousands of pages will be translated by the system and revised by professional translators. This will give us the necessary figures to measure efficiency through cost saving. Problem sheets, filled in by the revisors, will be sent to the developers, who will use them to improve the MT system. Up-grading capacity will be evaluated with the help of these problem sheets and also by re-translating some parts of texts, at the end of the test, and comparing both versions.