Korektor is a statistical spellchecker and (occasional) grammar checker released under 2-Clause BSD license and versioned using Semantic Versioning.
Korektor started with Michal Richter's diploma thesis Advanced Czech Spellchecker, but it is being developed further. There are two versions: a command line utility (tested on Linux, Windows and OS X) and a REST service with publicly available API and HTML front end.
The original OS X SpellServer providing System Service integrating Korektor with native OS X GUI applications is no longer developed, but do not hesitate to contact us if you are interested in it.
Copyright 2015 by Institute of Formal and Applied Linguistics, Faculty of Mathematics and Physics, Charles University in Prague, Czech Republic.
Korektor Spellchecker is a browser plugin which allows using the Korektor Spellchecker for most editable input fields. The plugin allows either directly correcting the content, or showing a dialog with suggested corrections.
Note that the dialog with suggestions is injected directly into the original page, so there can be various problems on untested sites. However, the plugin seem to work fine on many sites.
Although the sources of Korektor service and this plugin are available under BSD-3-Clause license, please respect the CC BY-SA-NC licence of the spellchecking models.
The plugin is available for the following browers:
Online demo is available as one of LINDAT/CLARIN services.
Web service is also available as one of LINDAT/CLARIN services.
Korektor releases are available on GitHub, either as a pre-compiled binary package, or source code packages only.
To use Korektor, a spellchecker model is needed. The language models are available from LINDAT/CLARIN infrastructure and described further in the Korektor User's Manual. Currently the following language models are available:
The original Michal Richter's version can be downloaded here.
Korektor is an open-source project and is freely available for non-commercial purposes. The library is distributed under 2-Clause BSD license and the associated models and data under CC BY-NC-SA, although for some models the original data used to create the model may impose additional licensing conditions.
If you use this tool for scientific work, please give credit to us by referencing Korektor website and Richter et al. 2012.
Korektor releases are available on GitHub, either as a pre-compiled binary package, or source code only. The binary package contains Linux, Windows and OS X binaries.
To use Korektor, a language model is needed. Here is a list of available language models.
If you want to compile Korektor manually, sources are available on on GitHub, both in the pre-compiled binary package releases and in the repository itself.
G++ 4.7 or newer, clang 3.2 or newer, Visual C++ 2015 or newer
make: on Windows, you can use mingw32-make
To compile Korektor, run make in the src directory.
Make targets and options:
exe: compile the binaries (default)
lib: compile the static library
BITS=32 or BITS=64: compile for specified 32-bit or 64-bit architecture instead of the default one
mode=RELEASE: create release build which statically links the C++ runtime and uses LTO
mode=DEBUG: create debug build
mode=PROFILE: create profile build
Platform can be selected using one of the following options:
PLATFORM=linux, PLATFORM=linux-gcc: gcc compiler on Linux operating system, default on Linux
PLATFORM=linux-clang: clang compiler on Linux, must be selected manually
PLATFORM=osx, PLATFORM=osx-clang: clang compiler on OS X, default on OS X; BITS=32+64 enables multiarch build
PLATFORM=win, PLATFORM=win-gcc: gcc compiler on Windows (TDM-GCC is well tested), default on Windows
PLATFORM=win-vs: Visual C++ 2015 compiler on Windows, must be selected manually; note that the
cl.exe compiler must be already present in PATH and corresponding BITS=32 or BITS=64
must be specified
Either POSIX shell or Windows CMD can be used as shell, it is detected automatically.
Korektor uses C++ BuilTem system, please refer to its manual if interested in all supported options.
Korektor is a statistical spellchecker and (occasional) grammar checker.
Like any supervised machine learning tool, Korektor needs a trained linguistic model.
We now describe the available language models, and then the command line tools
korektor and tokenizer.
Czech models are distributed under the CC BY-NC-SA licence. The Czech morphology used by the model is based on MorfFlex CZ Czech models work in Korektor version 2 or later.
Czech models are versioned according to the release date in format
YYMMDD, where YY, MM and DD are two-digit representation of
year, month and day, respectively. The latest version is 130202.
The latest version 130202 of the Czech Korektor models can be downloaded from LINDAT/CLARIN repository.
This work has been using language resources developed and/or stored and/or distributed by the LINDAT/CLARIN project of the Ministry of Education of the Czech Republic (project LM2010013).
The latest Czech models were created by Michal Richter as part of his Master thesis and are described in (Richter et al. 2012).
The MorfFlex CZ dictionary was created by Jan Hajič and Jaroslava Hlaváčová.
The Czech model contains the following variants:
korektok-czech-130202/diacritics_h2mor.confkorektok-czech-130202/spellchecking_h2mor.confkorektok-czech-130202/spellchecking_h2mor_2edits.confspellchecking_h2mor.conf model, but it may be considerably slower.
The korektor binary is used to run the Korektor. The only required argument
is the model configuration which should be used for correcting. The input is read
from standard input, it should be in UTF-8 encoding and it can be either
already tokenized and segmented, segmented only, or it can be a plain text.
The output is written to standard output and it is in UTF-8 encoding.
The full command syntax of korektor is
korektor [options] model_configuration
Options: --input=untokenized|untokenized_lines|segmented|vertical|horizontal
--output=original|xml|vertical|horizontal
--corrections=maximum_number_of_corrections
--viterbi_order=viterbi_decoding_order
--viterbi_beam_size=maximum_viterbi_beam_size
--viterbi_stage_pruning=maximum_viterbi_stage_cost_increment
--context_free
--version
--help
The input format is specified using the --input option. Currently supported
input formats are:
untokenized (default): the input is a plain text, which is segmented and
tokenized automatically. Note that sentences can span over multiple lines,
but an empty lines always terminate a sentence.
untokenized_lines: very similar to untokenized, the only difference
is that sentences cannot span over multiple lines, so every newline is
a sentence terminator.
segmented: the input is assumed to be segmented using newlines, but it is
tokenized automatically.
vertical: the input is tokenized and segmented in vertical format, every
line is considered a word, with empty line denoting end of sentence.
horizontal: the input is tokenized and segmented in horizontal format,
every line is a sentence, with words separated by spaces.
The maximum number of corrections that Korektor should return for every word is
specified using the --corrections option, and defaults to one.
Note that some output formats cannot handle multiple corrections, because they can only replace the original word by a corrected one.
The output format is specified using the --output option. Currently
supported output formats are:
original (default when number of corrections is 1): the original words
are replaced by the corrected ones, all other characters including spaces are
preserved. Note that this output format cannot handle multiple corrections
per word.
xml (default when number of corrections is greater than 1): the original
input is encoded as XML and the suggested corrections are marked
using the following XML elements:
spelling
element with the suggested corrections listed in the suggestions attribute
ordered by correction probability with the most probable one first
grammar
element is used instead of spelling
Hoši jely k babicce.The output in
xml output format with at most three corrections is
Hoši <grammar suggestions="jeli jely jel">jely</grammar> k <spelling suggestions="babičce babice babince">babicce</spelling>.
vertical: each word is printed on a separate line, with empty line
denoting end of sentence. If there are any suggested corrections for a word,
they are printed on the same line as the original words using several tab
separated columns:
S or G, where S denotes
a spelling correction and G denotes a grammar correction
Hoši jely k babicce.The output in
vertical output format with at most three corrections with
explicitly marked tab characters is
Hoši jely<---tab---->G<--tab-->jeli<---tab---->jely<--tab--->jel k babicce<--tab-->S<--tab-->babičce<--tab-->babice<--tab-->babince .
horizontal: the original words are replaced by the corrected ones. Each
sentence is printed on separate line and all words are space separated. Note
that this output format cannot handle multiple corrections per word.
Context free corrections can be generated by supplying the --context_free
option. In that case each word is considered separately and sentences boundaries
are ignored. This mode produces much worse results and should be used only when
no context is really available.
The decoding Viterbi algorithm can be tweaked using the following options:
--viterbi_order: Use specific Viterbi decoding order instead of the default
one. Use 1 for fastest execution, but worst accuracy. Setting this to higher
value than maximum model order minus one has no effect.
--viterbi_beam_size: Limit Viterbi beam size to specified constant.
Use smaller value for faster execution, but worse accuracy.
--viterbi_stage_pruning: Limit maximum cost increment in one Viterbi stage.
Use smaller value for faster execution, but worse accuracy.
The tokenizer binary is used to run the tokenizer. The input is read
from standard input, it should be in UTF-8 encoding and it can be either
already tokenized and segmented, segmented only, or it can be a plain text.
The output is written to standard output and it is in UTF-8 encoding.
The full command syntax of tokenizer is
korektor [options] model_configuration
Options: --input=untokenized|untokenized_lines|segmented|vertical|horizontal
--output=vertical|horizontal
--version
--help
The input format is specified using the --input option. Currently supported
input formats are:
untokenized (default): the input is a plain text, which is segmented and
tokenized automatically. Note that sentences can span over multiple lines,
but an empty lines always terminate a sentence.
untokenized_lines: very similar to untokenized, the only difference
is that sentences cannot span over multiple lines, so every newline is
a sentence terminator.
segmented: the input is assumed to be segmented using newlines, but it is
tokenized automatically.
vertical: the input is tokenized and segmented in vertical format, every
line is considered a word, with empty line denoting end of sentence.
horizontal: the input is tokenized and segmented in horizontal format,
every line is a sentence, with words separated by spaces.
The output format is specified using the --output option. Currently
supported output formats are:
vertical: each word is printed on a separate line, with empty line
denoting end of sentence.
horizontal: each sentence is printed on separate line and all words are
space separated.
The REST server can be run using the korektor_server binary.
The binary uses MicroRestD as a REST
server implementation and provides
Korektor REST API.
The full command syntax of korektor_server is
korektor_server [options] port (model_name weblicht_language model_file acknowledgements)* Options: --daemon
The korektor_server can run either in foreground or in background (when
--daemon is used). The specified model files are loaded during start and
kept in memory all the time. This behaviour might change in future to load the
models on demand.
In order to create a new spellchecker model for Korektor, several models must be created and a configuration file describing these models must be provided.
Korektor uses flexible morphology system which associates several
morphological factors to every word, with the word itself being considered
as a first one. Usually the factors are form, lemma and tag, but
arbitrary factors may be used. Note that currently there is a hard limit of
four factors for efficiency (you can change FactorList::MAX_FACTORS if you
want more).
For each morphological factor a language model is needed.
The last required model is an error model describing costs of various spelling errors.
To create a morphology model, a morphology lexicon input file must be provided
and processed by the create_morphology binary.
The morphology lexicon is an UTF-8 encoded file in the following format:
|
| (there must be the same number of tokens as
on the first line) and the second column is a count number of this morphology
entry.
Example:
form|lemma|tag dog|dog|NN 68 likes|like|VB 220 ...
The create_morphology should be run as follows:
create_morphology in_morphology_lexicon out_bin_morphology out_bin_vocabulary out_test_file
The language model for each morphological factor should be created by an external tool such as SRILM or KenLM and stored in ARPA format.
To create a binary representation of such model in ARPA format, the
create_lm_binary tool should be used as follows:
create_lm_binary in_arpa_model in_bin_morphology in_bin_vocabulary factor_name lm_order out_bin_lm
To create an error model, a textual error model description must be provided
and processed by the create_error_model binary.
The textual error model description is in UTF-8 format and contains one error
model item (edit operation) per line. Each item contain three tab separated
columns signature, edit distance and cost:
signature: Describes the edit operation in the following format:
s_ab: substitution of letter a for letter b
i_abc: insertion of letter a between letters b and c
d_ab: deletion of letter a following after letter b
swap_ab: swap of letters ab to ba
case: change of letter casing (lowercase to uppercase and vice verse)
substitutions: default substitution operation used when no s_.. rule apply
insertions: default insertion operation used when no i_... rule apply
deletions: default deletion operation used when no d_.. rule apply
swaps: default swap operation used when no swap_.. rule apply
edit distance: Integral edit distance of this operation. This distance
is used during the similar words lookup which is limited by maximum edit
distance. The sensible default is 1, but it can be useful to use 0 for
example when removing/adding diacritical mark only.
cost: The logarithm of the probability of the edit operation.
The first five lines must contain operations case, substitutions,
insertions, deletions and swaps, in this order.
Example (with textually marked tab characters):
case<------tab------->0<---tab--->2.6 substitutions<--tab-->1<---tab--->3.8 insertions<---tab---->1<---tab--->4.4 deletions<----tab---->1<---tab--->3.5 swaps<------tab------>1<---tab--->4.1 s_qw<-------tab------>1<---tab--->3.7 s_ui<-------tab------>1<---tab--->2.3 s_yi<-------tab------>1<---tab--->2.1 s_aá<-------tab------>0<---tab--->1.7 i_iuo<------tab------>1<---tab--->4.8 ...
The create_error_model binary should be run as follows:
create_morphology --binarize in_txt_error_model out_bin_error_model
The configuration specifies morphology model, language models and error model to use. In addition, it specifies similar word searching strategy and it can enable a diagnostics mode.
When a file is specified in the configuration file, its name is considered to be relative to the directory containing the configuration file.
The configuration file is line oriented and each line should adhere to one of the following formats:
# is ignored
morpholex=bin_morphology_file: use the specified morphology model
lm-bin_lm_file-model_order-model_weight: use the specified language
model. The corresponding factor is stored in the language model itself.
Model weight is a floating point multiplicative factor used when all factor
language model probabilities are summed together.
errormodel=bin_error_model_file: use the specified error model file
search-casing_treatment-max_edit_distance-max_cost: defines method for finding
possible corrections. Multiple methods can be specified in the configuration
file. The methods are tried in the order of their appearance in the
configuration file until one produces nonempty set of possible corrections.
Each search method has the following options:
casing_treatment: there are three possible method names:
case_sensitive: the casing of the original word is honored when
looking up possible suggestions in the morphology model
ignore_case: the casing of the original word is ignored when
looking up possible suggestions in the morphology model, and the casing
defined in the morphology model is used instead of the original one
ignore_case_keep_orig: the casing of the original word is ignored
when looking up possible suggestions in the morphology model, but the
generated suggestions have the same casing as the original word
max_edit_distance: the maximum edit distance of possible corrections
max_cost: the maximum cost of possible corrections
diagnostics=bin_vocabulary_file: use diagnostics mode which dumps a lot of
information during spellchecking. A vocabulary file created during
morphology model creation is needed to print out
the morphological factors.
The lines can be in arbitrary order, only the relative ordering of search-
lines is utilized when finding possible corrections.
As an example consider the configuration file spellchecking_h2mor.txt of
korektor-czech-130202 model:
# Binary file containing morphology and lexicon. morpholex=data/morphology_h2mor_freq2.bin # Error model binary. errormodel=data/error_model_train0.bin # Language models in the following format: # lm-[filename]-[order]-[weight] lm-data/form_lm_h2mor.bin-3-0.40 lm-data/lemma_lm_h2mor.bin-3-0.1 lm-data/tag_lm_h2mor.bin-3-0.50 # Search options, each item specify a distinct search rounds. # Searches are triggered in the order specified in this file, # whenever one of the search rounds find at least one possibility, # the consecutive search rounds are not triggered. # The format is the following: # search-[casing_treatment]-[max_edit_distance]-[max_cost] search-case_sensitive-1-6 search-ignore_case_keep_orig-1-6 search-ignore_case_keep_orig-2-9 # The diagnostics mode can be activated by uncommenting the following line. #diagnostics=data/morphology_h2mor_freq2_vocab.bin
Current Authors:
Original Author:
This work has been using language resources developed and/or stored and/or distributed by the LINDAT/CLARIN project of the Ministry of Education of the Czech Republic (project LM2010013).
Acknowledgements for individual language models are listed in Korektor User's Manual.
@InProceedings{richter12,
booktitle = {Proceedings of the 24th International Conference on Computational Linguistics (Coling 2012)},
title = {Korektor--A System for Contextual Spell-checking and Diacritics Completion},
editor = {Martin Kay and Christian Boitet},
author = {Michal Richter and Pavel Stra{\v{n}}{\'{a}}k and Alexandr Rosen},
year = {2012},
publisher = {Coling 2012 Organizing Committee},
organization = {{IIT} Bombay},
address = {Mumbai, India},
venue = {{IIT} Bombay, {VMCC}},
pages = {1--12}
}
If you prefer to reference Korektor by a permanent identifier (PID),
you can use http://hdl.handle.net/11234/1-1469.