1. Introduction

Korektor is a statistical spellchecker and (occasional) grammar checker released under 2-Clause BSD license and versioned using Semantic Versioning.

Korektor started with Michal Richter's diploma thesis Advanced Czech Spellchecker, but it is being developed further. There are two versions: a command line utility (tested on Linux, Windows and OS X) and a REST service with publicly available API and HTML front end.

The original OS X SpellServer providing System Service integrating Korektor with native OS X GUI applications is no longer developed, but do not hesitate to contact us if you are interested in it.

Copyright 2015 by Institute of Formal and Applied Linguistics, Faculty of Mathematics and Physics, Charles University in Prague, Czech Republic.

2. Online

2.1. Korektor Spellchecker Browser Plugin

Korektor Spellchecker is a browser plugin which allows using the Korektor Spellchecker for most editable input fields. The plugin allows either directly correcting the content, or showing a dialog with suggested corrections.

Note that the dialog with suggestions is injected directly into the original page, so there can be various problems on untested sites. However, the plugin seem to work fine on many sites.

Although the sources of Korektor service and this plugin are available under BSD-3-Clause license, please respect the CC BY-SA-NC licence of the spellchecking models.

The plugin is available for the following browers:

• Chrome: available as Korektor Spellchecker plugin in the Google Web Store
• Firefox: available as Korektor Spellchecker plugin in the AMO (Mozilla Add-ons repository)

2.2. Korektor Online Demo

Online demo is available as one of LINDAT/CLARIN services.

2.3. Korektor Web Service

Web service is also available as one of LINDAT/CLARIN services.

3. Release

3.1. Download

Korektor releases are available on GitHub, either as a pre-compiled binary package, or source code packages only.

• Latest release
• All releases, Changelog

3.1.1. Spellchecker Models

To use Korektor, a spellchecker model is needed. The language models are available from LINDAT/CLARIN infrastructure and described further in the Korektor User's Manual. Currently the following language models are available:

• Czech: korektor-czech-130202 (documentation)

3.1.2. Original Michal Richter's version

The original Michal Richter's version can be downloaded here.

3.2. License

Korektor is an open-source project and is freely available for non-commercial purposes. The library is distributed under 2-Clause BSD license and the associated models and data under CC BY-NC-SA, although for some models the original data used to create the model may impose additional licensing conditions.

If you use this tool for scientific work, please give credit to us by referencing Korektor website and Richter et al. 2012.

4. Installation

Korektor releases are available on GitHub, either as a pre-compiled binary package, or source code only. The binary package contains Linux, Windows and OS X binaries.

To use Korektor, a language model is needed. Here is a list of available language models.

If you want to compile Korektor manually, sources are available on on GitHub, both in the pre-compiled binary package releases and in the repository itself.

4.1. Requirements

• G++ 4.7 or newer, clang 3.2 or newer, Visual C++ 2015 or newer
• make: on Windows, you can use mingw32-make

4.2. Compilation

To compile Korektor, run make in the src directory.

Make targets and options:

• exe: compile the binaries (default)
• lib: compile the static library
• BITS=32 or BITS=64: compile for specified 32-bit or 64-bit architecture instead of the default one
• mode=RELEASE: create release build which statically links the C++ runtime and uses LTO
• mode=DEBUG: create debug build
• mode=PROFILE: create profile build

4.2.1. Platforms

Platform can be selected using one of the following options:

• PLATFORM=linux, PLATFORM=linux-gcc: gcc compiler on Linux operating system, default on Linux
• PLATFORM=linux-clang: clang compiler on Linux, must be selected manually
• PLATFORM=osx, PLATFORM=osx-clang: clang compiler on OS X, default on OS X; BITS=32+64 enables multiarch build
• PLATFORM=win, PLATFORM=win-gcc: gcc compiler on Windows (TDM-GCC is well tested), default on Windows
• PLATFORM=win-vs: Visual C++ 2015 compiler on Windows, must be selected manually; note that the cl.exe compiler must be already present in PATH and corresponding BITS=32 or BITS=64 must be specified

Either POSIX shell or Windows CMD can be used as shell, it is detected automatically.

4.2.2. Further Details

Korektor uses C++ BuilTem system, please refer to its manual if interested in all supported options.

5. User's Manual

Korektor is a statistical spellchecker and (occasional) grammar checker. Like any supervised machine learning tool, Korektor needs a trained linguistic model. We now describe the available language models, and then the command line tools korektor and tokenizer.

5.1. Czech Korektor Models

Czech models are distributed under the CC BY-NC-SA licence. The Czech morphology used by the model is based on MorfFlex CZ Czech models work in Korektor version 2 or later.

Czech models are versioned according to the release date in format YYMMDD, where YY, MM and DD are two-digit representation of year, month and day, respectively. The latest version is 130202.

5.1.1. Download

The latest version 130202 of the Czech Korektor models can be downloaded from LINDAT/CLARIN repository.

5.1.2. Acknowledgements

This work has been using language resources developed and/or stored and/or distributed by the LINDAT/CLARIN project of the Ministry of Education of the Czech Republic (project LM2010013).

The latest Czech models were created by Michal Richter as part of his Master thesis and are described in (Richter et al. 2012).

The MorfFlex CZ dictionary was created by Jan Hajič and Jaroslava Hlaváčová.

5.1.2.1. Publications

• (Richter et al. 2012) Richter Michal, Straňák Pavel and Rosen Alexandr. Korektor – A System for Contextual Spell-checking and Diacritics Completion In Proceedings of the 24th International Conference on Computational Linguistics (Coling 2012), pages 1-12, Mumbai, India, 2012.

5.1.3. Czech Model Variants

The Czech model contains the following variants:

korektok-czech-130202/diacritics_h2mor.conf

Spellchecker model which only _adds_ diacritical marks. Note that the diacritical marks are not removed by the model, so you have to strip them manually if you want to ignore them.

korektok-czech-130202/spellchecking_h2mor.conf

Spellchecker model which considers corrections with edit distance at most once. You should use this model for generic spellchecking.

korektok-czech-130202/spellchecking_h2mor_2edits.conf

Spellchecker model which considers corrections with edit distance at most two. This model can be useful if the required corrections are not found by the spellchecking_h2mor.conf model, but it may be considerably slower.

5.2. Running the Korektor

The korektor binary is used to run the Korektor. The only required argument is the model configuration which should be used for correcting. The input is read from standard input, it should be in UTF-8 encoding and it can be either already tokenized and segmented, segmented only, or it can be a plain text. The output is written to standard output and it is in UTF-8 encoding.

The full command syntax of korektor is

korektor [options] model_configuration
Options: --input=untokenized|untokenized_lines|segmented|vertical|horizontal
         --output=original|xml|vertical|horizontal
         --corrections=maximum_number_of_corrections
         --viterbi_order=viterbi_decoding_order
         --viterbi_beam_size=maximum_viterbi_beam_size
         --viterbi_stage_pruning=maximum_viterbi_stage_cost_increment
         --context_free
         --version
         --help

5.2.1. Input Formats

The input format is specified using the --input option. Currently supported input formats are:

• untokenized (default): the input is a plain text, which is segmented and tokenized automatically. Note that sentences can span over multiple lines, but an empty lines always terminate a sentence.
• untokenized_lines: very similar to untokenized, the only difference is that sentences cannot span over multiple lines, so every newline is a sentence terminator.
• segmented: the input is assumed to be segmented using newlines, but it is tokenized automatically.
• vertical: the input is tokenized and segmented in vertical format, every line is considered a word, with empty line denoting end of sentence.
• horizontal: the input is tokenized and segmented in horizontal format, every line is a sentence, with words separated by spaces.

5.2.2. Number of Corrections

The maximum number of corrections that Korektor should return for every word is specified using the --corrections option, and defaults to one.

Note that some output formats cannot handle multiple corrections, because they can only replace the original word by a corrected one.

5.2.3. Output Formats

The output format is specified using the --output option. Currently supported output formats are:

• original (default when number of corrections is 1): the original words are replaced by the corrected ones, all other characters including spaces are preserved. Note that this output format cannot handle multiple corrections per word.
• xml (default when number of corrections is greater than 1): the original input is encoded as XML and the suggested corrections are marked using the following XML elements:
  • spelling corrections for a word w are marked using the spelling element with the suggested corrections listed in the suggestions attribute ordered by correction probability with the most probable one first
  • grammar corrections are marked as spelling corrections, but the grammar element is used instead of spelling
  To illustrate, consider the input

    Hoši jely k babicce.
  

  The output in xml output format with at most three corrections is

    Hoši <grammar suggestions="jeli jely jel">jely</grammar> k <spelling suggestions="babičce babice babince">babicce</spelling>.
  

• vertical: each word is printed on a separate line, with empty line denoting end of sentence. If there are any suggested corrections for a word, they are printed on the same line as the original words using several tab separated columns:
  • the first column contain the original word
  • the second column contain either letter S or G, where S denotes a spelling correction and G denotes a grammar correction
  • the rest of the columns are the suggested corrections ordered by correction probability with the most probable one first
  To illustrate, consider the input

    Hoši jely k babicce.
  

  The output in vertical output format with at most three corrections with explicitly marked tab characters is

    Hoši
    jely<---tab---->G<--tab-->jeli<---tab---->jely<--tab--->jel
    k
    babicce<--tab-->S<--tab-->babičce<--tab-->babice<--tab-->babince
    .
  

• horizontal: the original words are replaced by the corrected ones. Each sentence is printed on separate line and all words are space separated. Note that this output format cannot handle multiple corrections per word.

5.2.4. Context Free Corrections

Context free corrections can be generated by supplying the --context_free option. In that case each word is considered separately and sentences boundaries are ignored. This mode produces much worse results and should be used only when no context is really available.

5.2.5. Viterbi Decoding Options

The decoding Viterbi algorithm can be tweaked using the following options:

• --viterbi_order: Use specific Viterbi decoding order instead of the default one. Use 1 for fastest execution, but worst accuracy. Setting this to higher value than maximum model order minus one has no effect.
• --viterbi_beam_size: Limit Viterbi beam size to specified constant. Use smaller value for faster execution, but worse accuracy.
• --viterbi_stage_pruning: Limit maximum cost increment in one Viterbi stage. Use smaller value for faster execution, but worse accuracy.

5.3. Running the tokenizer

The tokenizer binary is used to run the tokenizer. The input is read from standard input, it should be in UTF-8 encoding and it can be either already tokenized and segmented, segmented only, or it can be a plain text. The output is written to standard output and it is in UTF-8 encoding.

The full command syntax of tokenizer is

korektor [options] model_configuration
Options: --input=untokenized|untokenized_lines|segmented|vertical|horizontal
         --output=vertical|horizontal
         --version
         --help

5.3.1. Input Formats

The input format is specified using the --input option. Currently supported input formats are:

• untokenized (default): the input is a plain text, which is segmented and tokenized automatically. Note that sentences can span over multiple lines, but an empty lines always terminate a sentence.
• untokenized_lines: very similar to untokenized, the only difference is that sentences cannot span over multiple lines, so every newline is a sentence terminator.
• segmented: the input is assumed to be segmented using newlines, but it is tokenized automatically.
• vertical: the input is tokenized and segmented in vertical format, every line is considered a word, with empty line denoting end of sentence.
• horizontal: the input is tokenized and segmented in horizontal format, every line is a sentence, with words separated by spaces.

5.3.2. Output Formats

The output format is specified using the --output option. Currently supported output formats are:

• vertical: each word is printed on a separate line, with empty line denoting end of sentence.
• horizontal: each sentence is printed on separate line and all words are space separated.

5.4. Running the REST Server

The REST server can be run using the korektor_server binary. The binary uses MicroRestD as a REST server implementation and provides Korektor REST API.

The full command syntax of korektor_server is

korektor_server [options] port (model_name weblicht_language model_file acknowledgements)*
Options: --daemon

The korektor_server can run either in foreground or in background (when --daemon is used). The specified model files are loaded during start and kept in memory all the time. This behaviour might change in future to load the models on demand.

6. Model Creation

In order to create a new spellchecker model for Korektor, several models must be created and a configuration file describing these models must be provided.

Korektor uses flexible morphology system which associates several morphological factors to every word, with the word itself being considered as a first one. Usually the factors are form, lemma and tag, but arbitrary factors may be used. Note that currently there is a hard limit of four factors for efficiency (you can change FactorList::MAX_FACTORS if you want more).

For each morphological factor a language model is needed.

The last required model is an error model describing costs of various spelling errors.

6.1. Creating a Morphology Model

To create a morphology model, a morphology lexicon input file must be provided and processed by the create_morphology binary.

6.1.1. Morphology Lexicon Input Format

The morphology lexicon is an UTF-8 encoded file in the following format:

• the first line contains names of the factors delimited by |
• following lines contain two space separated columns, the first column is a word factor strings delimited by | (there must be the same number of tokens as on the first line) and the second column is a count number of this morphology entry.

Example:

form|lemma|tag
dog|dog|NN 68
likes|like|VB 220
...

6.1.2. Running create_morphology

The create_morphology should be run as follows:

create_morphology in_morphology_lexicon out_bin_morphology out_bin_vocabulary out_test_file

6.2. Creating a Language Model for Each Morphological Factor

The language model for each morphological factor should be created by an external tool such as SRILM or KenLM and stored in ARPA format.

To create a binary representation of such model in ARPA format, the create_lm_binary tool should be used as follows:

create_lm_binary in_arpa_model in_bin_morphology in_bin_vocabulary factor_name lm_order out_bin_lm

6.3. Creating an Error Model

To create an error model, a textual error model description must be provided and processed by the create_error_model binary.

6.3.1. Error Model Input Format

The textual error model description is in UTF-8 format and contains one error model item (edit operation) per line. Each item contain three tab separated columns signature, edit distance and cost:

• signature: Describes the edit operation in the following format:
  • s_ab: substitution of letter a for letter b
  • i_abc: insertion of letter a between letters b and c
  • d_ab: deletion of letter a following after letter b
  • swap_ab: swap of letters ab to ba
  • case: change of letter casing (lowercase to uppercase and vice verse)
  • substitutions: default substitution operation used when no s_.. rule apply
  • insertions: default insertion operation used when no i_... rule apply
  • deletions: default deletion operation used when no d_.. rule apply
  • swaps: default swap operation used when no swap_.. rule apply

• edit distance: Integral edit distance of this operation. This distance is used during the similar words lookup which is limited by maximum edit distance. The sensible default is 1, but it can be useful to use 0 for example when removing/adding diacritical mark only.

• cost: The logarithm of the probability of the edit operation.

The first five lines must contain operations case, substitutions, insertions, deletions and swaps, in this order.

Example (with textually marked tab characters):

case<------tab------->0<---tab--->2.6
substitutions<--tab-->1<---tab--->3.8
insertions<---tab---->1<---tab--->4.4
deletions<----tab---->1<---tab--->3.5
swaps<------tab------>1<---tab--->4.1
s_qw<-------tab------>1<---tab--->3.7
s_ui<-------tab------>1<---tab--->2.3
s_yi<-------tab------>1<---tab--->2.1
s_aá<-------tab------>0<---tab--->1.7
i_iuo<------tab------>1<---tab--->4.8
...

6.3.2. Running create_error_model

The create_error_model binary should be run as follows:

create_morphology --binarize in_txt_error_model out_bin_error_model

6.4. Configuration File

The configuration specifies morphology model, language models and error model to use. In addition, it specifies similar word searching strategy and it can enable a diagnostics mode.

When a file is specified in the configuration file, its name is considered to be relative to the directory containing the configuration file.

The configuration file is line oriented and each line should adhere to one of the following formats:

• an empty line or a line starting with # is ignored
• morpholex=bin_morphology_file: use the specified morphology model
• lm-bin_lm_file-model_order-model_weight: use the specified language model. The corresponding factor is stored in the language model itself. Model weight is a floating point multiplicative factor used when all factor language model probabilities are summed together.
• errormodel=bin_error_model_file: use the specified error model file
• search-casing_treatment-max_edit_distance-max_cost: defines method for finding possible corrections. Multiple methods can be specified in the configuration file. The methods are tried in the order of their appearance in the configuration file until one produces nonempty set of possible corrections. Each search method has the following options:
  • casing_treatment: there are three possible method names:
    • case_sensitive: the casing of the original word is honored when looking up possible suggestions in the morphology model
    • ignore_case: the casing of the original word is ignored when looking up possible suggestions in the morphology model, and the casing defined in the morphology model is used instead of the original one
    • ignore_case_keep_orig: the casing of the original word is ignored when looking up possible suggestions in the morphology model, but the generated suggestions have the same casing as the original word
  • max_edit_distance: the maximum edit distance of possible corrections
  • max_cost: the maximum cost of possible corrections
• diagnostics=bin_vocabulary_file: use diagnostics mode which dumps a lot of information during spellchecking. A vocabulary file created during morphology model creation is needed to print out the morphological factors.

The lines can be in arbitrary order, only the relative ordering of search- lines is utilized when finding possible corrections.

As an example consider the configuration file spellchecking_h2mor.txt of korektor-czech-130202 model:

# Binary file containing morphology and lexicon.
morpholex=data/morphology_h2mor_freq2.bin

# Error model binary.
errormodel=data/error_model_train0.bin

# Language models in the following format:
# lm-[filename]-[order]-[weight]
lm-data/form_lm_h2mor.bin-3-0.40
lm-data/lemma_lm_h2mor.bin-3-0.1
lm-data/tag_lm_h2mor.bin-3-0.50

# Search options, each item specify a distinct search rounds.
# Searches are triggered in the order specified in this file,
# whenever one of the search rounds find at least one possibility,
# the consecutive search rounds are not triggered.
# The format is the following:
# search-[casing_treatment]-[max_edit_distance]-[max_cost]
search-case_sensitive-1-6
search-ignore_case_keep_orig-1-6
search-ignore_case_keep_orig-2-9

# The diagnostics mode can be activated by uncommenting the following line.
#diagnostics=data/morphology_h2mor_freq2_vocab.bin

7. Contact

Current Authors:

• Milan Straka, straka@ufal.mff.cuni.cz
• Pavel Straňák, stranak@ufal.mff.cuni.cz
• Loganathan Ramasamy, ramasamy@ufal.mff.cuni.cz

Original Author:

• Michal Richter, michalisek@gmail.com

Korektor website.

Korektor LINDAT/CLARIN entry.

8. Acknowledgements

This work has been using language resources developed and/or stored and/or distributed by the LINDAT/CLARIN project of the Ministry of Education of the Czech Republic (project LM2010013).

Acknowledgements for individual language models are listed in Korektor User's Manual.

8.1. Publications

• (Richter et al. 2012) Richter Michal, Straňák Pavel and Rosen Alexandr. Korektor – A System for Contextual Spell-checking and Diacritics Completion In Proceedings of the 24th International Conference on Computational Linguistics (Coling 2012), pages 1-12, Mumbai, India, 2012.

8.2. Bibtex for Referencing

@InProceedings{richter12,
  booktitle    = {Proceedings of the 24th International Conference on Computational Linguistics (Coling 2012)},
  title        = {Korektor--A System for Contextual Spell-checking and Diacritics Completion},
  editor       = {Martin Kay and Christian Boitet},
  author       = {Michal Richter and Pavel Stra{\v{n}}{\'{a}}k and Alexandr Rosen},
  year         = {2012},
  publisher    = {Coling 2012 Organizing Committee},
  organization = {{IIT} Bombay},
  address      = {Mumbai, India},
  venue        = {{IIT} Bombay, {VMCC}},
  pages        = {1--12}
}

8.3. Permanent Identifier

If you prefer to reference Korektor by a permanent identifier (PID), you can use http://hdl.handle.net/11234/1-1469.