BulTreeBank Tokenizer

 
LRT + Open Submissions
  Authors
Simov, Kiril and Simov, Kiril
  Item identifier
http://hdl.handle.net/11372/LRT-1240
 Project URL
http://www.bultreebank.org/clark/index.html
 Date issued
2014-07-30
 Type
toolService
 Description
The tokenizer is covering all languages that use Latin1, Laitn2, Latin3 and Cyrillic tables of Unicode. Can be extended to cover other tables in Unicode if necessary. The implementation is as a cascaded regular grammar in CLaRK. It recognizes over 60 token categories. It is easy to be adapted to new token categories.
 Publisher
Linguistic Modeling Department, IPP, Bulgarian Academy of Sciences
 Collection(s)
LRT + Open Submissions Data & Tools
Show full item record