BulTreeBank Tokenizer

  Authors

Simov, Kiril ; Simov, Kiril

  Item identifier

http://hdl.handle.net/11372/LRT-1240

 Project URL

http://www.bultreebank.org/clark/index.html

 Date issued

2014-07-30

 Type

toolService

 Description

The tokenizer is covering all languages that use Latin1, Laitn2, Latin3 and Cyrillic tables of Unicode. Can be extended to cover other tables in Unicode if necessary. The implementation is as a cascaded regular grammar in CLaRK. It recognizes over 60 token categories. It is easy to be adapted to new token categories.

 Publisher

Linguistic Modeling Department, IPP, Bulgarian Academy of Sciences

 Collection(s)

LRT + Open Submissions Data & Tools

Show full item record