Instead of acquiring human ratings for all WMT submissions of the 3 language pairs, we will concentrate on a subset of the submissions and add online systems, as well as development systems of a production pipeline to generate a more challenging set of input systems.We strongly recommend to use the WMT2020 expert-based MQM labels as devset when developing your metric.We will conduct our own expert-based MQM human evaluation for selected language pairs (English to German, English to Russian and Chinese to English).As a consequence, we will integrate the following changes in this year's evaluation campaign: Furthermore, the same paper showed that automatic metrics based on pre-trained embeddings already outperform WMT human ratings on both language pairs. These findings call into question conclusions drawn on the basis of WMT human evaluation for high quality MT output. Recent work demonstrated that WMT DA has low correlation with expert-based human evaluations for WMT2020 English to German and Chinese to English. To create high quality datasets for developing and evaluating metrics.To test robustness of metrics when evaluating domains other than news data.To illustrate the suitability of an automatic evaluation metric as a surrogate for human evaluation.To achieve the strongest correlation with human judgement of translation quality over a diverse set of MT systems.The goals of the shared metrics task are: We invite submissions of reference-free metrics in addition to reference-based metrics. We will calculate the system-level, and segment-level correlations of your scores with human judgements. We are looking for automatic metric scores for translations at the system-level, and segment-level. We will provide you with MT system outputs along with source text and the human reference translations.
This shared task will examine automatic evaluation metrics for machine translation. Please enter yourself ASAP to this shared spreadsheet if you intend to submit to this year's metrics task. Update, 30 July 8:45 am UTC: additional system outputs added to newstest2021 en-de, en-ru and zh-en. Update, 27 July 12:15 pm UTC: en-de Challenge set source, ref and system outputs updated. (see below for the link and sumission details) System outputs are now available to download.
#MQM CALCULATOR DOWNLOAD#
Shared Task: Metrics Metrics Task Important Dates System outputs ready to download Metrics Task - EMNLP sixth Conference on Machine Translation
0 kommentar(er)
