Corpus for training and evaluating diacritics restoration systems

  Autoři

Náplava, Jakub ; Straka, Milan ; Hajič, Jan and Straňák, Pavel

  Identifikátor

http://hdl.handle.net/11234/1-2607

 Datum vydání

2018-01-31

 Typ

corpus, text

 Velikost

48 entries

 Jazyky

Croatian , Czech , French , Hungarian , Irish , Latvian , Polish , Romanian , Slovak , Spanish , Turkish , Vietnamese

 Popis

Corpus of texts in 12 languages. For each language, we provide one training, one development and one testing set acquired from Wikipedia articles. Moreover, each language dataset contains (substantially larger) training set collected from (general) Web texts. All sets, except for Wikipedia and Web training sets that can contain similar sentences, are disjoint. Data are segmented into sentences which are further word tokenized. All data in the corpus contain diacritics. To strip diacritics from them, use Python script diacritization_stripping.py contained within attached stripping_diacritics.zip. This script has two modes. We generally recommend using method called uninames, which for some languages behaves better. The code for training recurrent neural-network based model for diacritics restoration is located at https://github.com/arahusky/diacritics_restoration.

 Nakladatel

Charles University, Faculty of Mathematics and Physics, Institute of Formal and Applied Linguistics (UFAL)

 Sponzoři

Ministerstvo školství, mládeže a tělovýchovy České republiky

Kód projektu: LM2015071

Jméno projektu: LINDAT/CLARIN: Institut pro analýzu, zpracování a distribuci lingvistických dat

 Klíčová slova

diacritical marks generation natural language correction

 Kolekce

LINDAT / CLARIAH-CZ Data & Tools

Zobrazit celý záznam