Reviews - TC39 London: SMT output vs. NMT output

Update on TC39: SMT output vs. NMT output - what is new for translators on a hands-on practical level?

Second update on TC39. There is so much to tell about this conference, that I decided to pick out one useful topic for translators to start with and leave the full review for later.

SMT output vs. NMT output, what is new for translators on a hands-on practical level?

As a translator you have to be aware of the differences in output while working with output from SMT and NMT to avoid inadequate quality. Which error type is mainly made?

Neural is the magic word these days and several speakers addressed NMT as part of their presentations: Judith Klein in her opening talk "The best of 3 worlds: TM, SMT and NMT in Star Transit", Andrzej Zydron from XTM in his presentation "Beyond Neural MT", Emanuelle Esperança Rodier from the Université de Grenoble Alpes in her presentation "Evaluation of NMT and SMT systems: A study on uses and perceptions" (most of the below examples) and Alexander Waibel on his keynote address "A world without language barriers".

After the conference the proceedings normally get published on the conference website. So if you are interested in the detailed results of the research, go ahead and have a look. I will simplify the theoretical background here, shining light only on the practical use for translators.

Everybody is awed by NMT, so first of all some examples of how fluent the output of NMT can be in comparison to SMT:

  • Source: Se solicita la sustitución de todos los rodamientos del grupo traseiro.
  • SMT: Tausch aller Wälzlagergruppe beantraagt wird.
  • NMT: Der Austausch aller Lager der hinteren Gruppe wird angefordert.

 

  • Source: Anmerkungen oder Korrekturen sind keine eingegangen.
  • SMT: Any comments or correction are not have died.
  • NMT: Note or correction are not received.

Nice, isn't? NMT uses attention mechanisms. Attention is a useful feature in human brain to save computational resources. Not so in NMT systems. Unfortunately technology in that point has not yet reached human brain efficiency and there is a LOT of computing performance involved, which brings us to the limitations:

  1. NMT is good for units of less than 60 words.
  2. NMT is good for similar languages. EN>EN would have an LC factor of 1, EN>FR=0,8, EN>DU=0,75, EN>GER=0,6, EN>PL,RU,CS=0,45, EN>JA=0,2. Obviously for now NMT is not a good option for English into Japanese and works pretty good for English into French. It does not do a good job on morphologically rich languages into languages with primitive morphology (RU>EN).

Doing it "Neural" means using a fuzzy representation of knowledge. NMT systems try to capture the higher-level meaning of the text and are therefore rather able to generalize to new sentences than statistical systems.

Here some tricky typical error types translators have to be aware of.

  1. Unknown words

Be aware: sounds nice - is wrong!

NMT output has a considerably higher fluency, which makes it appear more human. As SMT becomes clumsy and uses source words when there are no results, NMT goes fuzzy and can replace or even add words, to make it look nice.

Examples:

  • Source: les adolescents japonais aiment les jeux vidéos
  • SMT: the adolescents japanese love electronic vidéos
  • NMT: japanese teenagers are interested in fashion

Note the word vidéos, that could not be translated and remained in SMT in the source language while NMT would replace it with an in this context often used word. A mistranslation when overlooked. You will not find source words in NMT output. These mistakes can be hard to spot.

  1. Extra words

Related to 1.

  • Source: avez-vous un menu ?
  • SMT: do you have a menu?
  • NMT: do you have a fixed menu?
  1. Missing an important content word

Be aware: Fluent, but totally wrong

  • Source: c' est le contrat d' achat de mes chèques de voyage.
  • SMT: it's the purchase agreement of my checks.
  • NMT: it's the seniority wage system.

Some mistakes you will not find in NMT output such as words taken over from source text. And some mistakes appear a lot less such as incorrect word forms or word order.

Translator´s action: Post-editing might take more or less the same time, but you need more awareness for hard to spot mistakes and read the source carefully to avoid mistranslations.

Contact Me