onion
Please use the following text to cite this item or export to a predefined format:
Pomikálek, Jan, 2011,
onion, LINDAT/CLARIAH-CZ digital library at the Institute of Formal and Applied Linguistics (ÚFAL),
http://hdl.handle.net/11858/00-097C-0000-000D-F67B-7.
Authors
Item identifier
Project URL
Demo URL
Date issued
2011
Type
Size
17 kb
Language(s)
Description
onion (ONe Instance ONly) is a tool for removing duplicate parts from large collections of texts. The tool has been implemented in Python, licensed under New BSD License and made an open source software (available for download including the source code at http://code.google.com/p/onion/). It is being successfuly used for cleaning large textual corpora at Natural language processing centre at Faculty of informatics, Masaryk university Brno and it's industry partners. The research leading to this piece of software was published in author's Ph.D. thesis "Removing Boilerplate and Duplicate Content from Web Corpora". The deduplication algorithm is based on comparing n-grams of words of text. The author's algorithm has been shown to be more suitable for textual corpora deduplication than competing algorithms (Broder, Charikar): in addition to detection of identical or very similar (95 %) duplicates, it is able to detect even partially similar duplicates (50 %) still achieving great performace (further described in author's Ph.D. thesis). The unique deduplication capabilities and scalability of the algorithm were been demonstrated while building corpora of American Spanish, Arabic, Czech, French, Japanese, Russian, Tajik, and six Turkic languages consisting --- several TB of text documents were deduplicated resulting in corpora of 70 billions tokens altogether.
Publisher
Acknowledgement
PRESEMT
Project code:Error: Cannot load project code
Project name:Error: Cannot load project name
Lexical Computing Ltd.
Project code:Error: Cannot load project code
Project name:Error: Cannot load project name
Collections
Files in this item
- Name
- onion-1.1.tar.gz
- Size
- 16.73 KB
- Format
- application/x-gzip
- Description
- Onion (ONe Instance ONly) is a tool for removing duplicate parts from large collections of texts
- MD5
- 8d97874f25abc72015e8ef63fca62f64
- onion-1.1
- onion-1.1