Plaintext Wikipedia dump 2018

  Authors

Rosa, Rudolf

  Item identifier

http://hdl.handle.net/11234/1-2735

 Date issued

2018-02-25

 Type

corpus, text

 Size

21 gb, 297 files, 8503209631 words

 Language(s)

Abkhazian , Achinese , Adyghe , Afrikaans , Akan , Albanian , Amharic , Arabic , Aragonese , Armenian , Arpitan , Assamese , Asturian , Atikamekw , Avaric , Aymara , Azerbaijani , Bambara , Banjar , Bashkir , Basque , Bavarian , Belarusian , Bengali , Bishnupriya , Bislama , Bosnian , Breton , Buginese , Bulgarian , Burmese , Catalan , Cebuano , Central Bicolano , CentralKhmer , CentralKurdish , Chamorro , Chechen , Cherokee , Cheyenne , Chinese , ChurchSlavic , Chuvash , Cornish , Corsican , Cree , Creek , CrimeanTatar , Croatian , Czech , Danish , Dhivehi , Dimli (individual language) , Dinka , Dotyali , Dutch , Dzongkha , EasternMari , Egyptian Arabic , English , Erzya , Esperanto , Estonian , Ewe , Extremaduran , Faroese , Fijian , Fiji Hindi , Finnish , French , Friulian , Fulah , Gagauz , Galician , GanChinese , Ganda , Georgian , German , Gilaki , GoanKonkani , Gothic , Guarani , Gujarati , Haitian , HakkaChinese , Hausa , Hawaiian , Hebrew , Herero , Hindi , HiriMotu , Hungarian , Icelandic , Ido , Igbo , Iloko , Indonesian , Interlingua (International Auxiliary Language Association) , Interlingue , Inuktitut , Inupiaq , Irish , Italian , Jamaican Creole English , Japanese , Javanese , Kabardian , Kabiyè , Kabyle , Kalaallisut , Kalmyk , Kannada , Kanuri , Karachay-Balkar , Kara-Kalpak , Kashmiri , Kashubian , Kazakh , Kikuyu , Kinyarwanda , Kirghiz , Kölsch , Komi , Komi-Permyak , Kongo , Korean , Kurdish , Ladino , Lak , Lao , Latgalian , Latin , Latvian , Lezghian , Ligurian , Limburgan , Lingala , LiteraryChinese , Lithuanian , Livvi , Lojban , Lombard , LowerSorbian , LowGerman , Luxembourgish , Macedonian , Macedo-Romanian , Maithili , Malagasy , Malay (macrolanguage) , Malayalam , Maltese , Manx , Maori , Marathi , Marshallese , Mazanderani , Minangkabau , Min Dong Chinese , Mingrelian , MinNan Chinese , Mirandese , Modern Greek (1453-) , Moksha , Mongolian , Nauru , Navajo , Ndonga , Neapolitan , Nepali , Newari , NorthernFrisian , NorthernLuri , Northern Sami , Norwegian , Norwegian Nynorsk , Novial , Nyanja , Occitan(post 1500) , Official Aramaic (700-300 BCE) , Old English (ca. 450-1100) , Oriya , Oromo , Ossetian , Pali , Pampanga , Pangasinan , Panjabi , Papiamento , Pedi , PennsylvaniaGerman , Persian , Pfaelzisch , Picard , Piemontese , Pitcairn-Norfolk , Polish , Pontic , Portuguese , Pushto , Quechua , Romanian , Romansh , Rundi , Russia Buriat , Russian , Rusyn , Samoan , Samogitian , Sango , Sanskrit , Sardinian , Saterfriesisch , Scots , ScottishGaelic , Serbian , Serbo-Croatian , Shona , Sicilian , Silesian , Sindhi , Sinhala , Slovak , Slovenian , Somali , SouthAzerbaijani , SouthernSotho , Spanish , SrananTongo , Sundanese , Swahili(macrolanguage) , Swati , Swedish , Tagalog , Tahitian , Tajik , Tama (Colombia) , Tamil , Tatar , Telugu , Tetum , Thai , Tibetan , Tigrinya , TokPisin , Tonga(Tonga Islands) , ToskAlbanian , Tsonga , Tswana , Tulu , Tumbuka , Turkish , Turkmen , Tuvinian , Twi , Udmurt , Uighur , Ukrainian , UpperSorbian , Urdu , Uzbek , Venda , Venetian , Veps , Vietnamese , Vlaams , Vlax Romani , Volapük , Võro , Walloon , Waray (Philippines) , Welsh , WesternFrisian , WesternMari , Western Panjabi , Wolof , WuChinese , Xhosa , Yakut , Yiddish , Yoruba , YueChinese , Zeeuws , Zhuang , Zulu

 Description

Wikipedia plain text data obtained from Wikipedia dumps with WikiExtractor in February 2018. The data come from all Wikipedias for which dumps could be downloaded at [https://dumps.wikimedia.org/]. This amounts to 297 Wikipedias, usually corresponding to individual languages and identified by their ISO codes. Several special Wikipedias are included, most notably "simple" (Simple English Wikipedia) and "incubator" (tiny hatching Wikipedias in various languages). For a list of all the Wikipedias, see [https://meta.wikimedia.org/wiki/List_of_Wikipedias]. The script which can be used to get new version of the data is included, but note that Wikipedia limits the download speed for downloading a lot of the dumps, so it takes a few days to download all of them (but one or a few can be downloaded fast). Also, the format of the dumps changes time to time, so the script will probably eventually stop working one day. The WikiExtractor tool [http://medialab.di.unipi.it/wiki/Wikipedia_Extractor] used to extract text from the Wikipedia dumps is not mine, I only modified it slightly to produce plaintext outputs [https://github.com/ptakopysk/wikiextractor].

 Publisher

Charles University, Faculty of Mathematics and Physics, Institute of Formal and Applied Linguistics (UFAL)

 Acknowledgement

Ministerstvo školství, mládeže a tělovýchovy České republiky

Project code: CZ.02.1.01/0.0/0.0/16_013/0001781

Project name: LINDAT/CLARIN - Výzkumná infrastruktura pro jazykové technologie - rozšíření repozitáře a výpočetní kapacity

 Subject(s)

Wikipedia text corpora monolingual corpus

 Collection(s)

LINDAT / CLARIAH-CZ Data & Tools

Show full item record