90% accuracy for translation software?

The USA Today research projects whose goal is to deliver, by 2010, software that can almost instantly translate Arabic and Mandarin Chinese with 90 to 95% accuracy. I’m no natural language researcher, but I’d be interested in knowing how they measure accuracy. I have seen some recent results on automated translation (French-English) and the results were alright some of the time, but even if you only screw up one word out of ten, is this good enough?

To anyone who has spent weeks crafting a ten pages paper: do you think you could get away by screwing up one word out of ten? Communication is a difficult task. It requires high accuracy most of the time.

Yes, I’m being hard. Automated translation is good and interesting work, but let us remain critical of the possibilities. While we lack strong AI, we cannot hope to replace human translators.

2 thoughts on “90% accuracy for translation software?”

  1. One popular way (IBM’s Bleu metric and the related NIST metric) relies on n-gram comparison of an MT-output sentence and a professionally-translated reference. There’s further pointers here: http://www.ics.mq.edu.au/~szwarts/MT-Evaluation.php

    Thanks. I surfed the Bleu paper. Correct me if I’m wrong, but there does not seem to have a unique meaning to “translate with accuracy of 90%”.

    do you think you could get away by screwing up one word out of ten?

    A qualified ‘yes’, the qualification is a big one, though: the goal isn’t to directly use the translations, but to make them available as an intermediate step in human translation.

    Yes. My wife is a translator and she already does this. I think most translators already benefit from the pretty good translation software out there. I have not read up on the productivity gains, but according to my wife, while the gains are real, they are not so high as one might think since a lot of the translator’s time is spent on “problem cases” that require research or deep thinking… and the software does not help with this. Though the Web does! I think most translators would trade automated translation for the Web any time. So maybe we could say that the most powerful translation aid is the Web.

    But are you sure everyone understands this? I have read somewhere (maybe slashdot) that American soldiers use translation software on the field to chat with civilians. This suggests that American soldiers don’t need interprets and they can just wander along with their software. Hmmm?

  2. I’m no natural language researcher, but I’d be interested in knowing how they measure accuracy.

    One popular way (IBM’s Bleu metric and the related NIST metric) relies on n-gram comparison of an MT-output sentence and a professionally-translated reference. There’s further pointers here: http://www.ics.mq.edu.au/~szwarts/MT-Evaluation.php

    do you think you could get away by screwing up one word out of ten?

    A qualified ‘yes’, the qualification is a big one, though: the goal isn’t to directly use the translations, but to make them available as an intermediate step in human translation. Good translators are few and far between, and machine translation stands a decent chance of either speeding up there work, or allowing less-fluent people to still produce a reasonable gloss of a document in a reasonable amount of time.

    Anecdotally, my French isn’t good enough to translate an academic article from scratch, but given a set of 90%-accurate machine translations I could probably make small corrections and choose between alternative translations quite effectively.

Leave a Reply

Your email address will not be published. Required fields are marked *