Papers

Peer-reviewed
2014

Crowdsourcing for Evaluating Machine Translation Quality.

LREC 2014 - NINTH INTERNATIONAL CONFERENCE ON LANGUAGE RESOURCES AND EVALUATION
  • Shinsuke Goto
  • ,
  • Donghui Lin
  • ,
  • Toru Ishida 0001

First page
3456
Last page
3463
Language
English
Publishing type
Research paper (international conference proceedings)
Publisher
EUROPEAN LANGUAGE RESOURCES ASSOC-ELRA

The recent popularity of machine translation has increased the demand for the evaluation of translations. However, the traditional evaluation approach, manual checking by a bilingual professional, is too expensive and too slow. In this study, we confirm the feasibility of crowdsourcing by analyzing the accuracy of crowdsourcing translation evaluations. We compare crowdsourcing scores to professional scores with regard to three metrics: translation-score, sentence-score, and system-score. A Chinese to English translation evaluation task was designed using around the NTCIR-9 PATENT parallel corpus with the goal being 5-range evaluations of adequacy and fluency. The experiment shows that the average score of crowdsource workers well matches professional evaluation results. The system-score comparison strongly indicates that crowdsourcing can be used to find the best translation system given the input of 10 source sentence.

Link information
DBLP
https://dblp.uni-trier.de/rec/conf/lrec/GotoLI14
Web of Science
https://gateway.webofknowledge.com/gateway/Gateway.cgi?GWVersion=2&SrcAuth=JSTA_CEL&SrcApp=J_Gate_JST&DestLinkType=FullRecord&KeyUT=WOS:000355611005012&DestApp=WOS_CPL
URL
http://www.lrec-conf.org/proceedings/lrec2014/summaries/756.html
URL
https://dblp.uni-trier.de/conf/lrec/2014
URL
https://dblp.uni-trier.de/db/conf/lrec/lrec2014.html#GotoLI14
ID information
  • DBLP ID : conf/lrec/GotoLI14
  • Web of Science ID : WOS:000355611005012

Export
BibTeX RIS