| dc.contributor.author | Popel, Martin |
| dc.contributor.author | Tomková, Markéta |
| dc.contributor.author | Tomek, Jakub |
| dc.date.accessioned | 2020-04-03T12:54:17Z |
| dc.date.available | 2020-04-03T12:54:17Z |
| dc.date.issued | 2020-04-08 |
| dc.identifier.uri | http://hdl.handle.net/11234/1-3209 |
| dc.description | This data set contains four types of manual annotation of translation quality, focusing on the comparison of human and machine translation quality (aka human-parity). The machine translation system used is English-Czech CUNI Transformer (CUBBITT). The annotations distinguish adequacy, fluency and overall quality. One of the types is Translation Turing test - detecting whether the annotators can distinguish human from machine translation. All the sentences are taken from the English-Czech test set newstest2018 (WMT2018 News translation shared task www.statmt.org/wmt18/translation-task.html), but only from the half with originally English sentences translated to Czech by a professional agency. |
| dc.language.iso | ces |
| dc.language.iso | eng |
| dc.publisher | Charles University, Faculty of Mathematics and Physics, Institute of Formal and Applied Linguistics (UFAL) |
| dc.rights | Creative Commons - Attribution 4.0 International (CC BY 4.0) |
| dc.rights.uri | http://creativecommons.org/licenses/by/4.0/ |
| dc.subject | machine translation |
| dc.subject | manual evaluation |
| dc.subject | fluency |
| dc.subject | adequacy |
| dc.subject | Translation Turing test |
| dc.title | Manual Re-evaluation of Translation Quality of WMT 2018 English-Czech systems |
| dc.type | corpus |
| metashare.ResourceInfo#ContentInfo.mediaType | text |
| dc.rights.label | PUB |
| has.files | yes |
| branding | LINDAT / CLARIAH-CZ |
| contact.person | Martin Popel popel@ufal.mff.cuni.cz Charles University, Faculty of Mathematics and Physics, Institute of Formal and Applied Linguistics (UFAL) |
| sponsor | Ministerstvo školství, mládeže a tělovýchovy České republiky LM2015071 LINDAT/CLARIN: Institut pro analýzu, zpracování a distribuci lingvistických dat nationalFunds |
| files.size | 12483334 |
| files.count | 1 |
Files in this item
This item is
Creative Commons - Attribution 4.0 International (CC BY 4.0)
Publicly Available
and licensed under:Creative Commons - Attribution 4.0 International (CC BY 4.0)
- Name
- cubbitt-manual-evaluation.zip
- Size
- 11.91 MB
- Format
- application/zip
- Description
- cubbitt-manual-evaluation
- MD5
- 7c9ce67f90f3f0fc0d43ea4a1dc7404f
- code
- loadOneFileEvaluationContext_vsOtherSystems.m4 kB
- libs
- fdr_bh
- license.txt1 kB
- fdr_bh.m8 kB
- Violinplot-Matlab-master
- violinplot.m3 kB
- README.md1 kB
- example.png88 kB
- ViolinGit.m13 kB
- dnafinder-Cohen-a2b974e
- LICENSE34 kB
- README.md3 kB
- kappa.m6 kB
- labelRepel
- labelRepelSimple.m8 kB
- labelRepelOld.m5 kB
- labelRepel.m14 kB
- fdr_bh
- plotFig4_systemComparison.m7 kB
- prepareContextEvaluationWorker_vsReference.m7 kB
- plotSupFig_withoutContext.m2 kB
- prepareContextEvaluationWorker_vsOtherSystems.m12 kB
- plotFig6_TuringBarplot.m4 kB
- evaluateTuring.m18 kB
- readAnnotatedSentences.m2 kB
- evaluateContext_vsOtherSystems.m4 kB
- plotSupFig_documents.m1 kB
- plotFig3_violinPlots.m5 kB
- common
- myGeneralSubplot.m2 kB
- num2sepNumStr.m138 B
- myGrpstats.m398 B
- getPValueAsText.m646 B
- mySaveAs.m990 B
- getStarsFromPValueBasic.m698 B
- createMaximisedFigure.m395 B
- createDir.m129 B
- myPrintPValue.m621 B
- loadOneFileEvaluationContext_vsReference.m4 kB
- plotSupFig_weightsAdequacyFluency.m4 kB
- getPlottingConstants.m2 kB
- plotFig6_TuringPieplot.m1 kB
- evaluateContext_vsReference.m45 kB
- evaluateWMT18.m9 kB
- plotFig4_tags.m7 kB
- plotFig3_WMT18.m3 kB
- kappa
- vsReference
- run-kappas.sh444 B
- results.txt890 B
- prepare-quality.py784 B
- Turing
- run-kappas.sh786 B
- compute-kappa.py1 kB
- results.txt531 B
- compute_agreement_scores.py12 kB
- vsOtherSystems
- run-kappas.sh213 B
- results.txt335 B
- prepare-improvements.py844 B
- vsReferenceErrorTypes
- run-kappas.sh173 B
- results.txt1 kB
- prepare-tags.py841 B
- vsReference
- data
- tsv
- turing.tsv896 kB
- quality-doc.tsv15 kB
- improvements.tsv406 kB
- tags.tsv269 kB
- README-tsv.txt1 kB
- quality-sentence.tsv613 kB
- WMT18
- Research-SrcDA-88_12_export_czech_only.csv240 kB
- documents_categories.xlsx14 kB
- newstest2018-encs-ref.cs.orig-en.txt337 kB
- newstest2018-encs-ref.cs.orig-en.segmentID.txt6 kB
- newstest2018-encs-ref.cs.txt614 kB
- SrcDA_export_czech_only.csv840 kB
- Research-SrcDA-88_12_export_czech_only
- resultsWMT18.mat680 kB
- translationsAll
- newstest2018-encs-GoogleTranslate.txt329 kB
- newstest2018-encs-SOURCE.txt325 kB
- newstest2018-encs-REFERENCE.txt336 kB
- newstest2018-encs-transformer.txt335 kB
- translationsOrigEn
- block_avg8_07d.txt193 kB
- cuni_transformer.txt196 kB
- onlineB.txt191 kB
- Source.txt182 kB
- mix_avg8_06d.txt193 kB
- Reference.txt198 kB
- uedin_nematus.txt192 kB
- WMT17
- context_annonymisedWMT17.xlsx405 kB
- context_withInfo.xlsx404 kB
- guidelines-vsReferenceErrorTypes.pdf56 kB
- guidelines-vsReference.pdf47 kB
- vsReferenceErrorTypes
- filled
- tags_eval_context_A01_TR.xlsx82 kB
- tags_eval_context_A11_TR.xlsx105 kB
- tags_eval_context_A08_O1.xlsx64 kB
- tags_eval_context_A10_TR.xlsx115 kB
- results
- TogetherPassedSpam
- contextQuality.mat121 kB
- annotators
- TogetherPassedSpam
- tags_eval_context_A03_OT.xlsx109 kB
- listFiles.xlsx10 kB
- tags_eval_context_A12_OT.xlsx99 kB
- answers23 B
- extract-tags.py2 kB
- filled
- vsOtherSystems
- extract-improvements.py2 kB
- filled
- eval5_context_A01_TR.xlsx54 kB
- eval5_context_A10_TR.xlsx79 kB
- eval5_context_A11_TR.xlsx74 kB
- results
- TogetherPassedSpam
- contextImprovements.mat125 kB
- annotators
- TogetherPassedSpam
- listFiles.xlsx9 kB
- eval5_context_A05_TR.xlsx77 kB
- eval5_context_A04_TR.xlsx61 kB
- answers
- withInfo5_context_A08.xlsx105 kB
- withInfo5_context_A10.xlsx115 kB
- withInfo5_context_A05.xlsx136 kB
- withInfo5_context_A02.xlsx125 kB
- withInfo5_context_A09.xlsx108 kB
- withInfo5_context_A11.xlsx122 kB
- withInfo5_context_A06.xlsx124 kB
- withInfo5_context_A03.xlsx92 kB
- withInfo5_context_A12.xlsx113 kB
- withInfo5_context_A07.xlsx115 kB
- withInfo5_context_A04.xlsx138 kB
- withInfo5_context_A01.xlsx92 kB
- guidelines-Turing.pdf46 kB
- vsReference
- extract-quality.py2 kB
- filled
- eval_context_A10_TR.xlsx74 kB
- eval_context_A09_OT.xlsx69 kB
- eval_context_A07_OT.xlsx75 kB
- eval_context_A08_O1.xlsx54 kB
- eval_context_A01_TR.xlsx84 kB
- eval_context_A08_O2.xlsx61 kB
- eval_context_A06_TT.xlsx62 kB
- eval_context_A10_TT.xlsx73 kB
- listFiles.xlsx10 kB
- eval_context_A05_TR.xlsx61 kB
- eval_context_A11_TR.xlsx69 kB
- eval_context_A12_OT.xlsx77 kB
- eval_context_A04_TR.xlsx67 kB
- eval_context_A06_OT.xlsx57 kB
- eval_context_A02_TR.xlsx59 kB
- results
- TogetherPassedSpam
- contextQuality.mat167 kB
- annotators
- Translators
- contextQuality.mat87 kB
- annotators
- Translatologs
- contextQuality.mat55 kB
- annotators
- Other
- contextQuality.mat83 kB
- annotators
- TogetherPassedSpam
- eval_context_A03_OT.xlsx71 kB
- eval_context_A09_TT.xlsx66 kB
- answers
- withInfo_context_A09.xlsx101 kB
- tableC.xlsx11 kB
- withInfo_context_A02.xlsx76 kB
- withInfo_context_A03.xlsx91 kB
- withInfo_context_A10.xlsx98 kB
- documentsFiles1.png36 kB
- withInfo_context_A04.xlsx105 kB
- withInfo_context_A11.xlsx77 kB
- context_withInfo.xlsx474 kB
- withInfo_context_A05.xlsx82 kB
- withInfo_context_A12.xlsx144 kB
- context_annonymised.xlsx370 kB
- withInfo_context_A06.xlsx84 kB
- withInfo_context_A07.xlsx112 kB
- tableA.xlsx9 kB
- withInfo_context_A08.xlsx65 kB
- documentsFiles2.png31 kB
- tableB.xlsx9 kB
- withInfo_context_A01.xlsx63 kB
- Turing
- extract-turing.py1 kB
- filled
- Turing_OT_03_A1_AT.xlsx28 kB
- Turing_OT_04_B2_AT.xlsx28 kB
- Turing_TT_09_A1_AT.xlsx31 kB
- Turing_MT_01_A1_AT.xlsx28 kB
- Turing_MT_08_B1_AT.xlsx25 kB
- Turing_TT_01_B1_AT.xlsx28 kB
- Turing_MT_02_B2_AT.xlsx25 kB
- Turing_TT_07_B1_AT.xlsx33 kB
- Turing_OT_01_A2_AT.xlsx28 kB
- Turing_MT_08_A2_AT.xlsx28 kB
- Turing_TT_01_A2_AT.xlsx29 kB
- Turing_TT_07_A2_AT.xlsx28 kB
- Turing_MT_02_A1_AT.xlsx28 kB
- Turing_TT_02_B1_AT.xlsx24 kB
- Turing_TT_05_B2_AT.xlsx24 kB
- Turing_MT_06_A1_AT.xlsx24 kB
- Turing_MT_01_B1_AT.xlsx26 kB
- Turing_TT_09_B1_AT.xlsx28 kB
- Turing_OT_03_A2_AT.xlsx29 kB
- Turing_OT_01_B2_AT.xlsx27 kB
- Turing_MT_01_A2_AT.xlsx24 kB
- Turing_OT_04_B1_AT.xlsx27 kB
- Turing_TT_01_B2_AT.xlsx27 kB
- Turing_MT_08_B2_AT.xlsx27 kB
- results
- TuringResults.mat5 kB
- Turing_TT_07_B2_AT.xlsx28 kB
- Turing_OT_04_A2_AT.xlsx30 kB
- Turing_TT_01_A1_AT.xlsx29 kB
- Turing_MT_08_A1_AT.xlsx23 kB
- Turing_MT_02_A2_AT.xlsx28 kB
- Turing_TT_07_A1_AT.xlsx28 kB
- Turing_TT_05_B1_AT.xlsx23 kB
- Turing_TT_05_A2_AT.xlsx28 kB
- Turing_MT_01_B2_AT.xlsx24 kB
- answers
- Turing_11_A2_withBasicInfo.xlsx39 kB
- Turing_10_A2_withBasicInfo.xlsx37 kB
- Turing_13_A2_withBasicInfo.xlsx39 kB
- Turing_12_A2_withBasicInfo.xlsx35 kB
- Turing_14_A2_withBasicInfo.xlsx37 kB
- Turing_15_A2_withBasicInfo.xlsx36 kB
- Turing_11_A1_withBasicInfo.xlsx39 kB
- Turing_10_A1_withBasicInfo.xlsx37 kB
- Turing_12_A1_withBasicInfo.xlsx35 kB
- Turing_14_A1_withBasicInfo.xlsx37 kB
- Turing_13_A1_withBasicInfo.xlsx38 kB
- Turing_15_A1_withBasicInfo.xlsx36 kB
- Turing_01_B2_withBasicInfo.xlsx38 kB
- Turing_02_B2_withBasicInfo.xlsx38 kB
- Turing_04_B2_withBasicInfo.xlsx38 kB
- Turing_03_B2_withBasicInfo.xlsx39 kB
- Turing_06_B2_withBasicInfo.xlsx37 kB
- Turing_08_B2_withBasicInfo.xlsx38 kB
- Turing_05_B2_withBasicInfo.xlsx39 kB
- Turing_07_B2_withBasicInfo.xlsx40 kB
- Turing_09_B2_withBasicInfo.xlsx39 kB
- Turing_02_B1_withBasicInfo.xlsx38 kB
- Turing_01_B1_withBasicInfo.xlsx38 kB
- Turing_04_B1_withBasicInfo.xlsx38 kB
- Turing_03_B1_withBasicInfo.xlsx39 kB
- Turing_05_B1_withBasicInfo.xlsx39 kB
- Turing_07_B1_withBasicInfo.xlsx40 kB
- Turing_06_B1_withBasicInfo.xlsx37 kB
- Turing_09_B1_withBasicInfo.xlsx39 kB
- Turing_08_B1_withBasicInfo.xlsx38 kB
- Turing_11_B2_withBasicInfo.xlsx39 kB
- Turing_10_B2_withBasicInfo.xlsx37 kB
- Turing_12_B2_withBasicInfo.xlsx35 kB
- Turing_14_B2_withBasicInfo.xlsx37 kB
- Turing_13_B2_withBasicInfo.xlsx38 kB
- Turing_15_B2_withBasicInfo.xlsx36 kB
- Turing_10_B1_withBasicInfo.xlsx37 kB
- Turing_12_B1_withBasicInfo.xlsx35 kB
- Turing_11_B1_withBasicInfo.xlsx39 kB
- Turing_14_B1_withBasicInfo.xlsx38 kB
- Turing_13_B1_withBasicInfo.xlsx38 kB
- Turing_15_B1_withBasicInfo.xlsx36 kB
- Turing_01_A2_withBasicInfo.xlsx38 kB
- Turing_03_A2_withBasicInfo.xlsx39 kB
- Turing_02_A2_withBasicInfo.xlsx38 kB
- Turing_04_A2_withBasicInfo.xlsx38 kB
- Turing_06_A2_withBasicInfo.xlsx37 kB
- Turing_05_A2_withBasicInfo.xlsx39 kB
- Turing_08_A2_withBasicInfo.xlsx38 kB
- Turing_07_A2_withBasicInfo.xlsx40 kB
- Turing_09_A2_withBasicInfo.xlsx39 kB
- Turing_02_A1_withBasicInfo.xlsx38 kB
- Turing_01_A1_withBasicInfo.xlsx38 kB
- Turing_04_A1_withBasicInfo.xlsx38 kB
- Turing_06_A1_withBasicInfo.xlsx37 kB
- Turing_03_A1_withBasicInfo.xlsx39 kB
- Turing_05_A1_withBasicInfo.xlsx39 kB
- Turing_07_A1_withBasicInfo.xlsx40 kB
- Turing_09_A1_withBasicInfo.xlsx39 kB
- Turing_08_A1_withBasicInfo.xlsx38 kB
- guidelines-vsOtherSystems.pdf47 kB
- tsv
- prepareTuring.m6 kB
- README.txt14 kB
- prepareContextEvaluation_vsOtherSystems.m9 kB
- Figures.m22 kB
- prepareContextEvaluation_vsReference.m15 kB