Machine translation systems are usually evaluated on how similar their output is to a human reference translation. This results in a single score, which doesn't tell us much about the types of errors that a system makes. To this end, so-called test suites have been proposed. A typical test suite focuses on a particular aspect of translation, for instance translation of pronouns, translation of ambiguous words, or translation of subtle morphological distinctions.
For your thesis, you will first identify a challenging aspect of the translation between a particular language pair. Then, you will produce a test suite for this aspect, following the protocols proposed in earlier work. Finally, you will collect outputs from diverse machine translation systems and evaluate them with your test suite.
Pronoun test suites:
- https://aclanthology.org/W18-6435/
- https://aclanthology.org/W18-6307/
- https://aclanthology.org/2020.eamt-1.24/
Word sense disambiguation test suites:
- https://aclanthology.org/W18-6437/
- https://aclanthology.org/W19-5354/
- https://aclanthology.org/2020.lrec-1.452/
- https://aclanthology.org/2022.acl-long.298/
Morphology test suites: