Evaluation Campaign

Registration and E-Mail List

In order to be able to participate in the evaluation please register here.

If you plan to participate in the evaluation campaign, please register to this e-mail list, in order to receive up-to-date information about developments in the campaign.

Permissible Training Data

Training of MT systems and language models for ASR is constrained to data supplied by the organizers or listed below. As for ASR acoustic modeling no training data are distributed. For German, participants are allowed to use any publicly available data recorded before July 17th 2012. For English the data has to be recorded before December 31st 2010.

Tracks

The IWSLT 2013 Evaluation Campaign will focus on the translation of TED Talks, a collection of public speeches on a variety of topics. The evaluation campaign will include the following tracks:

ASR track: automatic transcription of talks from audio to text

Languages: English and German
Input format: unsegmented SPHERE
Output format: CTM, no case, no punctuation, UTF8
English development data: dev2010, tst2010 and dev2012 Note: for the ASR track, only an uem file similar to dev2010.uem and dev2012.uem will be provided, i.e. without sentence segmentation! Therefore, automatic segmentation of the data is mandatory!
German development data: dev2012 Note: for the ASR track, only an uem file similar to dev2012.de-en.de.uem will be provided, i.e. without sentence segmentation! Therefore, automatic segmentation of the data is mandatory!
Evaluation Data:

SLT track: speech translation of talks from audio (or ASR output) to text

Input format: segmented SPHERE, or ASR output
Directions:

Official: English -> French, German -> English, English -> German
Optional: English-> Spanish, Portuguese (B), Italian, Chinese, Polish, Slovenian, Arabic, Persian

Output format: NIST XML format, true case with punctuation
ASR development data for English: dev2010, tst2010 and dev2012 Note: for the SLT track, only an uem file similar to dev2010-manualSegmentation.uem and dev2012-manualSegmentation.uem will be provided. The output will have to be in this segmentation
Automatic Transcripts of English Data: tst2010, dev2010 and dev2012.
Automatic Transcripts of German data: dev2012
MT Training and Development Data
Evaluation Data (reference ASR output and manual segmentation. For audio see ASR track):

MT track: text translation of talks for two language pairs plus eleven optional language pairs:

Input format: NIST XML format, true case with punctuation
Output format: NIST XML format, true case with punctuation
Directions:

Official: English -> French, German -> English, English -> German
Optional: English <-> Arabic, Spanish, Portuguese (B), Italian, Chinese, Polish, Persian, Slovenian, Turkish, Dutch, Romanian, Russian

Training and development data
Evaluation Data

Evaluation methods:

ASR track: word or character error rate
SLT/MT: BLEU, NIST, METEOR, TER (all directions)

Training of MT systems and language models for ASR is constrained to data supplied by the organizers. Supplied training, development, and test data will be available under the workshop’s webpage.

Data Permissible for MT model and ASR Language Model Training

Provided Data

Other Permissible Data

Parallel:

Monolingual:

LDC (German)

Miscellaneous:

Important Dates:

Workshop

February 2013: Call for Participation
Sept-Nov 2013: Early, late registration
Dec 5-6, 2013: Workshop

Scientific Papers

Sept 29, 2013: Paper Submission due
November 12, 2013: Review Feedback
November 18, 2013: Camera-ready paper due