This is a new version of the repository. Do let us know (lindat-help at ufal.mff.cuni.cz) if you encounter any issues.

WMT16 Tuning Shared Task Models (Czech-to-English)

Please use the following text to cite this item or export to a predefined format:
bibtexcmdi
Kamran, Amir; Jawaid, Bushra; Bojar, Ondřej and Stanojevic, Milos, 2016, WMT16 Tuning Shared Task Models (Czech-to-English), LINDAT/CLARIAH-CZ digital library at the Institute of Formal and Applied Linguistics (ÚFAL), http://hdl.handle.net/11372/LRT-1671.
Share
Authors
Item identifier
Project URL
Date issued
2016-03-21
Type
Language(s)
Description
The item contains models to tune for the WMT16 Tuning shared task for Czech-to-English. CzEng 1.6pre (http://ufal.mff.cuni.cz/czeng/czeng16pre) corpus is used for the training of the translation models. The data is tokenized (using Moses tokenizer), lowercased and sentences longer than 60 words and shorter than 4 words are removed before training. Alignment is done using fast_align (https://github.com/clab/fast_align) and the standard Moses pipeline is used for training. Two 5-gram language models are trained using KenLM: one only using the CzEng English data and the other is trained using all available English mono data for WMT except Common Crawl. Also included are two lexicalized bidirectional reordering models, word based and hierarchical, with msd conditioned on both source and target of processed CzEng.
Publisher
Acknowledgement
Subject(s)
Collections
LRT + Open Submissions Data & Tools
Show full item record
This item isPublicly Available
and licensed under:

Files in this item

  Download instructions for command line
Name
wmt16.czeng.blm.en.tgz
Size
7.79 GB
Format
application/x-gzip
Description
kenlm 5-gram language model (binarized) trained only on the English side of CzEng parallel data used
MD5
dd910814d89f3bb41261ead0a95930dc
Preview
  Download file
  File Preview
    • wmt16.czeng.blm.en12 GB
Name
cs2en_model.tgz
Size
36.56 GB
Format
application/x-gzip
Description
Contains Lexical Models (lex.e2f and lex.f2e), Phrase Table (phrase-table.gz), word based reordering model (reordering-table.wbe-msd-bidirectional-fe.gz), hierarchical reordering model (reordering-table.hier-msd-bidirectional-fe.gz) and the moses.ini file
MD5
9f97c40bab9bbc8844b362437ead3c71
Preview
  Download file
  File Preview
    • reordering-table.wbe-msd-bidirectional-fe.gz7 GB
    • moses.ini1 kB
    • lex.f2e668 MB
    • reordering-table.hier-msd-bidirectional-fe.gz7 GB
    • lex.e2f668 MB
    • phrase-table.gz21 GB
Name
wmt16.mono.blm.en.tgz
Size
60.04 GB
Format
application/x-gzip
Description
kenlm 5-gram language model (binarized) trained on all English mono data available for WMT except Common Crawl (see the Makefile for the details of mono data used)
MD5
a57e4fd4f43c05f826cda33cfe257eed
Preview
  Download file
  File Preview
    • wmt16.mono.blm.en98 GB
Name
Makefile
Size
16.96 KB
Format
application/octet-stream
Description
You can recreate the models using this Makefile
MD5
5f56434491ccb9591c35d8fe20fb8aa9
Preview
  Download file
  File Preview
Name
moses.ini
Size
1.29 KB
Format
application/octet-stream
Description
The moses.ini file for tuning
MD5
52551ca476c84dbaf9409ca3083b02c2
Preview
  Download file
  File Preview