Manual Re-evaluation of Translation Quality of WMT 2018 English-Czech systems

 
LINDAT / CLARIAH-CZ
  Authors
Popel, Martin ; Tomková, Markéta and Tomek, Jakub
  Item identifier
http://hdl.handle.net/11234/1-3209
 Date issued
2020-04-08
 Type
corpus, text
 Language(s)
Czech , English
 Description
This data set contains four types of manual annotation of translation quality, focusing on the comparison of human and machine translation quality (aka human-parity). The machine translation system used is English-Czech CUNI Transformer (CUBBITT). The annotations distinguish adequacy, fluency and overall quality. One of the types is Translation Turing test - detecting whether the annotators can distinguish human from machine translation. All the sentences are taken from the English-Czech test set newstest2018 (WMT2018 News translation shared task www.statmt.org/wmt18/translation-task.html), but only from the half with originally English sentences translated to Czech by a professional agency.
 Publisher
Charles University, Faculty of Mathematics and Physics, Institute of Formal and Applied Linguistics (UFAL)
 Acknowledgement

Ministerstvo školství, mládeže a tělovýchovy České republiky

Project code: LM2015071

Project name: LINDAT/CLARIN: Institut pro analýzu, zpracování a distribuci lingvistických dat

 Subject(s)
machine translation manual evaluation fluency adequacy Translation Turing test
 Collection(s)
LINDAT / CLARIAH-CZ Data & Tools
Show full item record
 

 Files in this item

This item is
Publicly Available
and licensed under:
Creative Commons - Attribution 4.0 International (CC BY 4.0)
Distributed under Creative Commons Attribution Required
Icon
Name
cubbitt-manual-evaluation.zip
Size
11.91 MB
Format
application/zip
Description
cubbitt-manual-evaluation
MD5
7c9ce67f90f3f0fc0d43ea4a1dc7404f
 Download file  Preview
 File Preview  
  • code
    • loadOneFileEvaluationContext_vsOtherSystems.m4 kB
    • libs
    • plotFig4_systemComparison.m7 kB
    • prepareContextEvaluationWorker_vsReference.m7 kB
    • plotSupFig_withoutContext.m2 kB
    • prepareContextEvaluationWorker_vsOtherSystems.m12 kB
    • plotFig6_TuringBarplot.m4 kB
    • evaluateTuring.m18 kB
    • readAnnotatedSentences.m2 kB
    • evaluateContext_vsOtherSystems.m4 kB
    • plotSupFig_documents.m1 kB
    • plotFig3_violinPlots.m5 kB
    • common
      • myGeneralSubplot.m2 kB
      • num2sepNumStr.m138 B
      • myGrpstats.m398 B
      • getPValueAsText.m646 B
      • mySaveAs.m990 B
      • getStarsFromPValueBasic.m698 B
      • createMaximisedFigure.m395 B
      • createDir.m129 B
      • myPrintPValue.m621 B
    • loadOneFileEvaluationContext_vsReference.m4 kB
    • plotSupFig_weightsAdequacyFluency.m4 kB
    • getPlottingConstants.m2 kB
    • plotFig6_TuringPieplot.m1 kB
    • evaluateContext_vsReference.m45 kB
    • evaluateWMT18.m9 kB
    • plotFig4_tags.m7 kB
    • plotFig3_WMT18.m3 kB
  • kappa
    • vsReference
      • run-kappas.sh444 B
      • results.txt890 B
      • prepare-quality.py784 B
    • Turing
      • run-kappas.sh786 B
      • compute-kappa.py1 kB
      • results.txt531 B
    • compute_agreement_scores.py12 kB
    • vsOtherSystems
      • run-kappas.sh213 B
      • results.txt335 B
      • prepare-improvements.py844 B
    • vsReferenceErrorTypes
      • run-kappas.sh173 B
      • results.txt1 kB
      • prepare-tags.py841 B
  • data
    • tsv
      • turing.tsv896 kB
      • quality-doc.tsv15 kB
      • improvements.tsv406 kB
      • tags.tsv269 kB
      • README-tsv.txt1 kB
      • quality-sentence.tsv613 kB
    • WMT18
      • Research-SrcDA-88_12_export_czech_only.csv240 kB
      • documents_categories.xlsx14 kB
      • newstest2018-encs-ref.cs.orig-en.txt337 kB
      • newstest2018-encs-ref.cs.orig-en.segmentID.txt6 kB
      • newstest2018-encs-ref.cs.txt614 kB
      • SrcDA_export_czech_only.csv840 kB
      • Research-SrcDA-88_12_export_czech_only
        • resultsWMT18.mat680 kB
      • translationsAll
        • newstest2018-encs-GoogleTranslate.txt329 kB
        • newstest2018-encs-SOURCE.txt325 kB
        • newstest2018-encs-REFERENCE.txt336 kB
        • newstest2018-encs-transformer.txt335 kB
      • translationsOrigEn
        • block_avg8_07d.txt193 kB
        • cuni_transformer.txt196 kB
        • onlineB.txt191 kB
        • Source.txt182 kB
        • mix_avg8_06d.txt193 kB
        • Reference.txt198 kB
        • uedin_nematus.txt192 kB
    • WMT17
      • context_annonymisedWMT17.xlsx405 kB
      • context_withInfo.xlsx404 kB
    • guidelines-vsReferenceErrorTypes.pdf56 kB
    • guidelines-vsReference.pdf47 kB
    • vsReferenceErrorTypes
      • filled
        • tags_eval_context_A01_TR.xlsx82 kB
        • tags_eval_context_A11_TR.xlsx105 kB
        • tags_eval_context_A08_O1.xlsx64 kB
        • tags_eval_context_A10_TR.xlsx115 kB
        • results
        • tags_eval_context_A03_OT.xlsx109 kB
        • listFiles.xlsx10 kB
        • tags_eval_context_A12_OT.xlsx99 kB
      • answers23 B
      • extract-tags.py2 kB
    • vsOtherSystems
      • extract-improvements.py2 kB
      • filled
        • eval5_context_A01_TR.xlsx54 kB
        • eval5_context_A10_TR.xlsx79 kB
        • eval5_context_A11_TR.xlsx74 kB
        • results
        • listFiles.xlsx9 kB
        • eval5_context_A05_TR.xlsx77 kB
        • eval5_context_A04_TR.xlsx61 kB
      • answers
        • withInfo5_context_A08.xlsx105 kB
        • withInfo5_context_A10.xlsx115 kB
        • withInfo5_context_A05.xlsx136 kB
        • withInfo5_context_A02.xlsx125 kB
        • withInfo5_context_A09.xlsx108 kB
        • withInfo5_context_A11.xlsx122 kB
        • withInfo5_context_A06.xlsx124 kB
        • withInfo5_context_A03.xlsx92 kB
        • withInfo5_context_A12.xlsx113 kB
        • withInfo5_context_A07.xlsx115 kB
        • withInfo5_context_A04.xlsx138 kB
        • withInfo5_context_A01.xlsx92 kB
    • guidelines-Turing.pdf46 kB
    • vsReference
      • extract-quality.py2 kB
      • filled
        • eval_context_A10_TR.xlsx74 kB
        • eval_context_A09_OT.xlsx69 kB
        • eval_context_A07_OT.xlsx75 kB
        • eval_context_A08_O1.xlsx54 kB
        • eval_context_A01_TR.xlsx84 kB
        • eval_context_A08_O2.xlsx61 kB
        • eval_context_A06_TT.xlsx62 kB
        • eval_context_A10_TT.xlsx73 kB
        • listFiles.xlsx10 kB
        • eval_context_A05_TR.xlsx61 kB
        • eval_context_A11_TR.xlsx69 kB
        • eval_context_A12_OT.xlsx77 kB
        • eval_context_A04_TR.xlsx67 kB
        • eval_context_A06_OT.xlsx57 kB
        • eval_context_A02_TR.xlsx59 kB
        • results
        • eval_context_A03_OT.xlsx71 kB
        • eval_context_A09_TT.xlsx66 kB
      • answers
        • withInfo_context_A09.xlsx101 kB
        • tableC.xlsx11 kB
        • withInfo_context_A02.xlsx76 kB
        • withInfo_context_A03.xlsx91 kB
        • withInfo_context_A10.xlsx98 kB
        • documentsFiles1.png36 kB
        • withInfo_context_A04.xlsx105 kB
        • withInfo_context_A11.xlsx77 kB
        • context_withInfo.xlsx474 kB
        • withInfo_context_A05.xlsx82 kB
        • withInfo_context_A12.xlsx144 kB
        • context_annonymised.xlsx370 kB
        • withInfo_context_A06.xlsx84 kB
        • withInfo_context_A07.xlsx112 kB
        • tableA.xlsx9 kB
        • withInfo_context_A08.xlsx65 kB
        • documentsFiles2.png31 kB
        • tableB.xlsx9 kB
        • withInfo_context_A01.xlsx63 kB
    • Turing
      • extract-turing.py1 kB
      • filled
        • Turing_OT_03_A1_AT.xlsx28 kB
        • Turing_OT_04_B2_AT.xlsx28 kB
        • Turing_TT_09_A1_AT.xlsx31 kB
        • Turing_MT_01_A1_AT.xlsx28 kB
        • Turing_MT_08_B1_AT.xlsx25 kB
        • Turing_TT_01_B1_AT.xlsx28 kB
        • Turing_MT_02_B2_AT.xlsx25 kB
        • Turing_TT_07_B1_AT.xlsx33 kB
        • Turing_OT_01_A2_AT.xlsx28 kB
        • Turing_MT_08_A2_AT.xlsx28 kB
        • Turing_TT_01_A2_AT.xlsx29 kB
        • Turing_TT_07_A2_AT.xlsx28 kB
        • Turing_MT_02_A1_AT.xlsx28 kB
        • Turing_TT_02_B1_AT.xlsx24 kB
        • Turing_TT_05_B2_AT.xlsx24 kB
        • Turing_MT_06_A1_AT.xlsx24 kB
        • Turing_MT_01_B1_AT.xlsx26 kB
        • Turing_TT_09_B1_AT.xlsx28 kB
        • Turing_OT_03_A2_AT.xlsx29 kB
        • Turing_OT_01_B2_AT.xlsx27 kB
        • Turing_MT_01_A2_AT.xlsx24 kB
        • Turing_OT_04_B1_AT.xlsx27 kB
        • Turing_TT_01_B2_AT.xlsx27 kB
        • Turing_MT_08_B2_AT.xlsx27 kB
        • results
          • TuringResults.mat5 kB
        • Turing_TT_07_B2_AT.xlsx28 kB
        • Turing_OT_04_A2_AT.xlsx30 kB
        • Turing_TT_01_A1_AT.xlsx29 kB
        • Turing_MT_08_A1_AT.xlsx23 kB
        • Turing_MT_02_A2_AT.xlsx28 kB
        • Turing_TT_07_A1_AT.xlsx28 kB
        • Turing_TT_05_B1_AT.xlsx23 kB
        • Turing_TT_05_A2_AT.xlsx28 kB
        • Turing_MT_01_B2_AT.xlsx24 kB
      • answers
        • Turing_11_A2_withBasicInfo.xlsx39 kB
        • Turing_10_A2_withBasicInfo.xlsx37 kB
        • Turing_13_A2_withBasicInfo.xlsx39 kB
        • Turing_12_A2_withBasicInfo.xlsx35 kB
        • Turing_14_A2_withBasicInfo.xlsx37 kB
        • Turing_15_A2_withBasicInfo.xlsx36 kB
        • Turing_11_A1_withBasicInfo.xlsx39 kB
        • Turing_10_A1_withBasicInfo.xlsx37 kB
        • Turing_12_A1_withBasicInfo.xlsx35 kB
        • Turing_14_A1_withBasicInfo.xlsx37 kB
        • Turing_13_A1_withBasicInfo.xlsx38 kB
        • Turing_15_A1_withBasicInfo.xlsx36 kB
        • Turing_01_B2_withBasicInfo.xlsx38 kB
        • Turing_02_B2_withBasicInfo.xlsx38 kB
        • Turing_04_B2_withBasicInfo.xlsx38 kB
        • Turing_03_B2_withBasicInfo.xlsx39 kB
        • Turing_06_B2_withBasicInfo.xlsx37 kB
        • Turing_08_B2_withBasicInfo.xlsx38 kB
        • Turing_05_B2_withBasicInfo.xlsx39 kB
        • Turing_07_B2_withBasicInfo.xlsx40 kB
        • Turing_09_B2_withBasicInfo.xlsx39 kB
        • Turing_02_B1_withBasicInfo.xlsx38 kB
        • Turing_01_B1_withBasicInfo.xlsx38 kB
        • Turing_04_B1_withBasicInfo.xlsx38 kB
        • Turing_03_B1_withBasicInfo.xlsx39 kB
        • Turing_05_B1_withBasicInfo.xlsx39 kB
        • Turing_07_B1_withBasicInfo.xlsx40 kB
        • Turing_06_B1_withBasicInfo.xlsx37 kB
        • Turing_09_B1_withBasicInfo.xlsx39 kB
        • Turing_08_B1_withBasicInfo.xlsx38 kB
        • Turing_11_B2_withBasicInfo.xlsx39 kB
        • Turing_10_B2_withBasicInfo.xlsx37 kB
        • Turing_12_B2_withBasicInfo.xlsx35 kB
        • Turing_14_B2_withBasicInfo.xlsx37 kB
        • Turing_13_B2_withBasicInfo.xlsx38 kB
        • Turing_15_B2_withBasicInfo.xlsx36 kB
        • Turing_10_B1_withBasicInfo.xlsx37 kB
        • Turing_12_B1_withBasicInfo.xlsx35 kB
        • Turing_11_B1_withBasicInfo.xlsx39 kB
        • Turing_14_B1_withBasicInfo.xlsx38 kB
        • Turing_13_B1_withBasicInfo.xlsx38 kB
        • Turing_15_B1_withBasicInfo.xlsx36 kB
        • Turing_01_A2_withBasicInfo.xlsx38 kB
        • Turing_03_A2_withBasicInfo.xlsx39 kB
        • Turing_02_A2_withBasicInfo.xlsx38 kB
        • Turing_04_A2_withBasicInfo.xlsx38 kB
        • Turing_06_A2_withBasicInfo.xlsx37 kB
        • Turing_05_A2_withBasicInfo.xlsx39 kB
        • Turing_08_A2_withBasicInfo.xlsx38 kB
        • Turing_07_A2_withBasicInfo.xlsx40 kB
        • Turing_09_A2_withBasicInfo.xlsx39 kB
        • Turing_02_A1_withBasicInfo.xlsx38 kB
        • Turing_01_A1_withBasicInfo.xlsx38 kB
        • Turing_04_A1_withBasicInfo.xlsx38 kB
        • Turing_06_A1_withBasicInfo.xlsx37 kB
        • Turing_03_A1_withBasicInfo.xlsx39 kB
        • Turing_05_A1_withBasicInfo.xlsx39 kB
        • Turing_07_A1_withBasicInfo.xlsx40 kB
        • Turing_09_A1_withBasicInfo.xlsx39 kB
        • Turing_08_A1_withBasicInfo.xlsx38 kB
    • guidelines-vsOtherSystems.pdf47 kB
    • prepareTuring.m6 kB
    • README.txt14 kB
    • prepareContextEvaluation_vsOtherSystems.m9 kB
    • Figures.m22 kB
    • prepareContextEvaluation_vsReference.m15 kB