Plaintext Wikipedia dump 2018

  Autoři

Rosa, Rudolf

  Identifikátor

http://hdl.handle.net/11234/1-2735

 Datum vydání

2018-02-25

 Typ

corpus, text

 Velikost

21 gb, 297 files, 8503209631 words

 Jazyky

Abkhazian , Achinese , Adyghe , Afrikaans , Akan , Albanian , Amharic , Arabic , Aragonese , Armenian , Arpitan , Assamese , Asturian , Atikamekw , Avaric , Aymara , Azerbaijani , Bambara , Banjar , Bashkir , Basque , Bavarian , Belarusian , Bengali , Bishnupriya , Bislama , Bosnian , Breton , Buginese , Bulgarian , Burmese , Catalan , Cebuano , Central Bicolano , CentralKhmer , CentralKurdish , Chamorro , Chechen , Cherokee , Cheyenne , Chinese , ChurchSlavic , Chuvash , Cornish , Corsican , Cree , Creek , CrimeanTatar , Croatian , Czech , Danish , Dhivehi , Dimli (individual language) , Dinka , Dotyali , Dutch , Dzongkha , EasternMari , Egyptian Arabic , English , Erzya , Esperanto , Estonian , Ewe , Extremaduran , Faroese , Fijian , Fiji Hindi , Finnish , French , Friulian , Fulah , Gagauz , Galician , GanChinese , Ganda , Georgian , German , Gilaki , GoanKonkani , Gothic , Guarani , Gujarati , Haitian , HakkaChinese , Hausa , Hawaiian , Hebrew , Herero , Hindi , HiriMotu , Hungarian , Icelandic , Ido , Igbo , Iloko , Indonesian , Interlingua (International Auxiliary Language Association) , Interlingue , Inuktitut , Inupiaq , Irish , Italian , Jamaican Creole English , Japanese , Javanese , Kabardian , Kabiyè , Kabyle , Kalaallisut , Kalmyk , Kannada , Kanuri , Karachay-Balkar , Kara-Kalpak , Kashmiri , Kashubian , Kazakh , Kikuyu , Kinyarwanda , Kirghiz , Kölsch , Komi , Komi-Permyak , Kongo , Korean , Kurdish , Ladino , Lak , Lao , Latgalian , Latin , Latvian , Lezghian , Ligurian , Limburgan , Lingala , LiteraryChinese , Lithuanian , Livvi , Lojban , Lombard , LowerSorbian , LowGerman , Luxembourgish , Macedonian , Macedo-Romanian , Maithili , Malagasy , Malay (macrolanguage) , Malayalam , Maltese , Manx , Maori , Marathi , Marshallese , Mazanderani , Minangkabau , Min Dong Chinese , Mingrelian , MinNan Chinese , Mirandese , Modern Greek (1453-) , Moksha , Mongolian , Nauru , Navajo , Ndonga , Neapolitan , Nepali , Newari , NorthernFrisian , NorthernLuri , Northern Sami , Norwegian , Norwegian Nynorsk , Novial , Nyanja , Occitan(post 1500) , Official Aramaic (700-300 BCE) , Old English (ca. 450-1100) , Oriya , Oromo , Ossetian , Pali , Pampanga , Pangasinan , Panjabi , Papiamento , Pedi , PennsylvaniaGerman , Persian , Pfaelzisch , Picard , Piemontese , Pitcairn-Norfolk , Polish , Pontic , Portuguese , Pushto , Quechua , Romanian , Romansh , Rundi , Russia Buriat , Russian , Rusyn , Samoan , Samogitian , Sango , Sanskrit , Sardinian , Saterfriesisch , Scots , ScottishGaelic , Serbian , Serbo-Croatian , Shona , Sicilian , Silesian , Sindhi , Sinhala , Slovak , Slovenian , Somali , SouthAzerbaijani , SouthernSotho , Spanish , SrananTongo , Sundanese , Swahili(macrolanguage) , Swati , Swedish , Tagalog , Tahitian , Tajik , Tama (Colombia) , Tamil , Tatar , Telugu , Tetum , Thai , Tibetan , Tigrinya , TokPisin , Tonga(Tonga Islands) , ToskAlbanian , Tsonga , Tswana , Tulu , Tumbuka , Turkish , Turkmen , Tuvinian , Twi , Udmurt , Uighur , Ukrainian , UpperSorbian , Urdu , Uzbek , Venda , Venetian , Veps , Vietnamese , Vlaams , Vlax Romani , Volapük , Võro , Walloon , Waray (Philippines) , Welsh , WesternFrisian , WesternMari , Western Panjabi , Wolof , WuChinese , Xhosa , Yakut , Yiddish , Yoruba , YueChinese , Zeeuws , Zhuang , Zulu

 Popis

Wikipedia plain text data obtained from Wikipedia dumps with WikiExtractor in February 2018. The data come from all Wikipedias for which dumps could be downloaded at [https://dumps.wikimedia.org/]. This amounts to 297 Wikipedias, usually corresponding to individual languages and identified by their ISO codes. Several special Wikipedias are included, most notably "simple" (Simple English Wikipedia) and "incubator" (tiny hatching Wikipedias in various languages). For a list of all the Wikipedias, see [https://meta.wikimedia.org/wiki/List_of_Wikipedias]. The script which can be used to get new version of the data is included, but note that Wikipedia limits the download speed for downloading a lot of the dumps, so it takes a few days to download all of them (but one or a few can be downloaded fast). Also, the format of the dumps changes time to time, so the script will probably eventually stop working one day. The WikiExtractor tool [http://medialab.di.unipi.it/wiki/Wikipedia_Extractor] used to extract text from the Wikipedia dumps is not mine, I only modified it slightly to produce plaintext outputs [https://github.com/ptakopysk/wikiextractor].

 Nakladatel

Charles University, Faculty of Mathematics and Physics, Institute of Formal and Applied Linguistics (UFAL)

 Sponzoři

Ministerstvo školství, mládeže a tělovýchovy České republiky

Kód projektu: CZ.02.1.01/0.0/0.0/16_013/0001781

Jméno projektu: LINDAT/CLARIN - Výzkumná infrastruktura pro jazykové technologie - rozšíření repozitáře a výpočetní kapacity

 Klíčová slova

Wikipedia text corpora monolingual corpus

 Kolekce

LINDAT / CLARIAH-CZ Data & Tools

Zobrazit celý záznam