Corpus for training and evaluating diacritics restoration systems

  Authors

Náplava, Jakub ; Straka, Milan ; Hajič, Jan and Straňák, Pavel

  Item identifier

http://hdl.handle.net/11234/1-2607

 Date issued

2018-01-31

 Type

corpus, text

 Size

48 entries

 Language(s)

Croatian , Czech , French , Hungarian , Irish , Latvian , Polish , Romanian , Slovak , Spanish , Turkish , Vietnamese

 Description

Corpus of texts in 12 languages. For each language, we provide one training, one development and one testing set acquired from Wikipedia articles. Moreover, each language dataset contains (substantially larger) training set collected from (general) Web texts. All sets, except for Wikipedia and Web training sets that can contain similar sentences, are disjoint. Data are segmented into sentences which are further word tokenized. All data in the corpus contain diacritics. To strip diacritics from them, use Python script diacritization_stripping.py contained within attached stripping_diacritics.zip. This script has two modes. We generally recommend using method called uninames, which for some languages behaves better. The code for training recurrent neural-network based model for diacritics restoration is located at https://github.com/arahusky/diacritics_restoration.

 Publisher

Charles University, Faculty of Mathematics and Physics, Institute of Formal and Applied Linguistics (UFAL)

 Acknowledgement

Ministerstvo školství, mládeže a tělovýchovy České republiky

Project code: LM2015071

Project name: LINDAT/CLARIN: Institut pro analýzu, zpracování a distribuci lingvistických dat

 Subject(s)

diacritical marks generation natural language correction

 Collection(s)

LINDAT / CLARIAH-CZ Data & Tools

Show full item record