Interdependence of expert categories and automated metrics applied to evaluate translation quality

Abstract

The article evaluates the quality of translation; we consider the applied and pragmatic aspects of such evaluation in the conditions of the current rapid increase in the number of texts to be translated. The article summarizes a plethora of assessment principles, each having its merits and drawbacks, and examines the correlation between the categories of adequacy and equivalence as the basic assessment parameters. We conclude that equivalence is oriented towards the result of translation, whereas adequacy indicates whether translation as process corresponds to the communicative situation given. Adequacy is thus viewed as a discursive feature. The article also looks into the possibility of applying these assessment criteria to automated evaluation of translation quality, and compares the specific computer metrics or, algorithms: Cosine Similarity, Word Error Rate, BLEU, ROUGE, NIST, METEOR. We analyze the potential for applying these metrics to man-made rather than machine translated. The n-gram metric for evaluation of translation with explicit ordering Meteor is the most efficient algorithm; however, it can only be applied advantageously to formal standardized texts. The article points out that literary translation can hardly be evaluated on the basis of n-gram correspondences due to it characteristic feature: ample use of non-equivalent units even at the sentence level. The purely mathematical procedures overviewed in the article cannot serve as linguistic metamodels of assessment unless supplemented with syntactic and semantic algorithms.