1. Introduction
  2. Online
  3. Release
  4. Installation
  5. User's Manual
  6. Model Creation
  7. Contact
  8. Acknowledgements

1. Introduction

Korektor is a statistical spellchecker and (occasional) grammar checker released under 2-Clause BSD license and versioned using Semantic Versioning.

Korektor started with Michal Richter's diploma thesis Advanced Czech Spellchecker, but it is being developed further. There are two versions: a command line utility (tested on Linux, Windows and OS X) and a REST service with publicly available API and HTML front end.

The original OS X SpellServer providing System Service integrating Korektor with native OS X GUI applications is no longer developed, but do not hesitate to contact us if you are interested in it.

Copyright 2015 by Institute of Formal and Applied Linguistics, Faculty of Mathematics and Physics, Charles University in Prague, Czech Republic.

2. Online

2.1. Korektor Spellchecker Browser Plugin

Korektor Spellchecker is a browser plugin which allows using the Korektor Spellchecker for most editable input fields. The plugin allows either directly correcting the content, or showing a dialog with suggested corrections.

Note that the dialog with suggestions is injected directly into the original page, so there can be various problems on untested sites. However, the plugin seem to work fine on many sites.

Although the sources of Korektor service and this plugin are available under BSD-3-Clause license, please respect the CC BY-SA-NC licence of the spellchecking models.

The plugin is available for the following browers:

2.2. Korektor Online Demo

Online demo is available as one of LINDAT/CLARIN services.

2.3. Korektor Web Service

Web service is also available as one of LINDAT/CLARIN services.

3. Release

3.1. Download

Korektor releases are available on GitHub, either as a pre-compiled binary package, or source code packages only.

3.1.1. Spellchecker Models

To use Korektor, a spellchecker model is needed. The language models are available from LINDAT/CLARIN infrastructure and described further in the Korektor User's Manual. Currently the following language models are available:

3.1.2. Original Michal Richter's version

The original Michal Richter's version can be downloaded here.

3.2. License

Korektor is an open-source project and is freely available for non-commercial purposes. The library is distributed under 2-Clause BSD license and the associated models and data under CC BY-NC-SA, although for some models the original data used to create the model may impose additional licensing conditions.

If you use this tool for scientific work, please give credit to us by referencing Korektor website and Richter et al. 2012.

4. Installation

Korektor releases are available on GitHub, either as a pre-compiled binary package, or source code only. The binary package contains Linux, Windows and OS X binaries.

To use Korektor, a language model is needed. Here is a list of available language models.

If you want to compile Korektor manually, sources are available on on GitHub, both in the pre-compiled binary package releases and in the repository itself.

4.1. Requirements

4.2. Compilation

To compile Korektor, run make in the src directory.

Make targets and options:

4.2.1. Platforms

Platform can be selected using one of the following options:

Either POSIX shell or Windows CMD can be used as shell, it is detected automatically.

4.2.2. Further Details

Korektor uses C++ BuilTem system, please refer to its manual if interested in all supported options.

5. User's Manual

Korektor is a statistical spellchecker and (occasional) grammar checker. Like any supervised machine learning tool, Korektor needs a trained linguistic model. We now describe the available language models, and then the command line tools korektor and tokenizer.

5.1. Czech Korektor Models

Czech models are distributed under the CC BY-NC-SA licence. The Czech morphology used by the model is based on MorfFlex CZ Czech models work in Korektor version 2 or later.

Czech models are versioned according to the release date in format YYMMDD, where YY, MM and DD are two-digit representation of year, month and day, respectively. The latest version is 130202.

5.1.1. Download

The latest version 130202 of the Czech Korektor models can be downloaded from LINDAT/CLARIN repository.

5.1.2. Acknowledgements

This work has been using language resources developed and/or stored and/or distributed by the LINDAT/CLARIN project of the Ministry of Education of the Czech Republic (project LM2010013).

The latest Czech models were created by Michal Richter as part of his Master thesis and are described in (Richter et al. 2012).

The MorfFlex CZ dictionary was created by Jan Hajič and Jaroslava Hlaváčová.

5.1.2.1. Publications

5.1.3. Czech Model Variants

The Czech model contains the following variants:

korektok-czech-130202/diacritics_h2mor.conf
Spellchecker model which only _adds_ diacritical marks. Note that the diacritical marks are not removed by the model, so you have to strip them manually if you want to ignore them.

korektok-czech-130202/spellchecking_h2mor.conf
Spellchecker model which considers corrections with edit distance at most once. You should use this model for generic spellchecking.

korektok-czech-130202/spellchecking_h2mor_2edits.conf
Spellchecker model which considers corrections with edit distance at most two. This model can be useful if the required corrections are not found by the spellchecking_h2mor.conf model, but it may be considerably slower.

5.2. Running the Korektor

The korektor binary is used to run the Korektor. The only required argument is the model configuration which should be used for correcting. The input is read from standard input, it should be in UTF-8 encoding and it can be either already tokenized and segmented, segmented only, or it can be a plain text. The output is written to standard output and it is in UTF-8 encoding.

The full command syntax of korektor is

korektor [options] model_configuration
Options: --input=untokenized|untokenized_lines|segmented|vertical|horizontal
         --output=original|xml|vertical|horizontal
         --corrections=maximum_number_of_corrections
         --viterbi_order=viterbi_decoding_order
         --viterbi_beam_size=maximum_viterbi_beam_size
         --viterbi_stage_pruning=maximum_viterbi_stage_cost_increment
         --context_free
         --version
         --help

5.2.1. Input Formats

The input format is specified using the --input option. Currently supported input formats are:

5.2.2. Number of Corrections

The maximum number of corrections that Korektor should return for every word is specified using the --corrections option, and defaults to one.

Note that some output formats cannot handle multiple corrections, because they can only replace the original word by a corrected one.

5.2.3. Output Formats

The output format is specified using the --output option. Currently supported output formats are:

5.2.4. Context Free Corrections

Context free corrections can be generated by supplying the --context_free option. In that case each word is considered separately and sentences boundaries are ignored. This mode produces much worse results and should be used only when no context is really available.

5.2.5. Viterbi Decoding Options

The decoding Viterbi algorithm can be tweaked using the following options:

5.3. Running the tokenizer

The tokenizer binary is used to run the tokenizer. The input is read from standard input, it should be in UTF-8 encoding and it can be either already tokenized and segmented, segmented only, or it can be a plain text. The output is written to standard output and it is in UTF-8 encoding.

The full command syntax of tokenizer is

korektor [options] model_configuration
Options: --input=untokenized|untokenized_lines|segmented|vertical|horizontal
         --output=vertical|horizontal
         --version
         --help

5.3.1. Input Formats

The input format is specified using the --input option. Currently supported input formats are:

5.3.2. Output Formats

The output format is specified using the --output option. Currently supported output formats are:

5.4. Running the REST Server

The REST server can be run using the korektor_server binary. The binary uses MicroRestD as a REST server implementation and provides Korektor REST API.

The full command syntax of korektor_server is

korektor_server [options] port (model_name weblicht_language model_file acknowledgements)*
Options: --daemon

The korektor_server can run either in foreground or in background (when --daemon is used). The specified model files are loaded during start and kept in memory all the time. This behaviour might change in future to load the models on demand.

6. Model Creation

In order to create a new spellchecker model for Korektor, several models must be created and a configuration file describing these models must be provided.

Korektor uses flexible morphology system which associates several morphological factors to every word, with the word itself being considered as a first one. Usually the factors are form, lemma and tag, but arbitrary factors may be used. Note that currently there is a hard limit of four factors for efficiency (you can change FactorList::MAX_FACTORS if you want more).

For each morphological factor a language model is needed.

The last required model is an error model describing costs of various spelling errors.

6.1. Creating a Morphology Model

To create a morphology model, a morphology lexicon input file must be provided and processed by the create_morphology binary.

6.1.1. Morphology Lexicon Input Format

The morphology lexicon is an UTF-8 encoded file in the following format:

Example:

form|lemma|tag
dog|dog|NN 68
likes|like|VB 220
...

6.1.2. Running create_morphology

The create_morphology should be run as follows:

create_morphology in_morphology_lexicon out_bin_morphology out_bin_vocabulary out_test_file

6.2. Creating a Language Model for Each Morphological Factor

The language model for each morphological factor should be created by an external tool such as SRILM or KenLM and stored in ARPA format.

To create a binary representation of such model in ARPA format, the create_lm_binary tool should be used as follows:

create_lm_binary in_arpa_model in_bin_morphology in_bin_vocabulary factor_name lm_order out_bin_lm

6.3. Creating an Error Model

To create an error model, a textual error model description must be provided and processed by the create_error_model binary.

6.3.1. Error Model Input Format

The textual error model description is in UTF-8 format and contains one error model item (edit operation) per line. Each item contain three tab separated columns signature, edit distance and cost:

The first five lines must contain operations case, substitutions, insertions, deletions and swaps, in this order.

Example (with textually marked tab characters):

case<------tab------->0<---tab--->2.6
substitutions<--tab-->1<---tab--->3.8
insertions<---tab---->1<---tab--->4.4
deletions<----tab---->1<---tab--->3.5
swaps<------tab------>1<---tab--->4.1
s_qw<-------tab------>1<---tab--->3.7
s_ui<-------tab------>1<---tab--->2.3
s_yi<-------tab------>1<---tab--->2.1
s_aá<-------tab------>0<---tab--->1.7
i_iuo<------tab------>1<---tab--->4.8
...

6.3.2. Running create_error_model

The create_error_model binary should be run as follows:

create_morphology --binarize in_txt_error_model out_bin_error_model

6.4. Configuration File

The configuration specifies morphology model, language models and error model to use. In addition, it specifies similar word searching strategy and it can enable a diagnostics mode.

When a file is specified in the configuration file, its name is considered to be relative to the directory containing the configuration file.

The configuration file is line oriented and each line should adhere to one of the following formats:

The lines can be in arbitrary order, only the relative ordering of search- lines is utilized when finding possible corrections.

As an example consider the configuration file spellchecking_h2mor.txt of korektor-czech-130202 model:

# Binary file containing morphology and lexicon.
morpholex=data/morphology_h2mor_freq2.bin

# Error model binary.
errormodel=data/error_model_train0.bin

# Language models in the following format:
# lm-[filename]-[order]-[weight]
lm-data/form_lm_h2mor.bin-3-0.40
lm-data/lemma_lm_h2mor.bin-3-0.1
lm-data/tag_lm_h2mor.bin-3-0.50

# Search options, each item specify a distinct search rounds.
# Searches are triggered in the order specified in this file,
# whenever one of the search rounds find at least one possibility,
# the consecutive search rounds are not triggered.
# The format is the following:
# search-[casing_treatment]-[max_edit_distance]-[max_cost]
search-case_sensitive-1-6
search-ignore_case_keep_orig-1-6
search-ignore_case_keep_orig-2-9

# The diagnostics mode can be activated by uncommenting the following line.
#diagnostics=data/morphology_h2mor_freq2_vocab.bin

7. Contact

Current Authors:

Original Author:

Korektor website.

Korektor LINDAT/CLARIN entry.

8. Acknowledgements

This work has been using language resources developed and/or stored and/or distributed by the LINDAT/CLARIN project of the Ministry of Education of the Czech Republic (project LM2010013).

Acknowledgements for individual language models are listed in Korektor User's Manual.

8.1. Publications

8.2. Bibtex for Referencing

@InProceedings{richter12,
  booktitle    = {Proceedings of the 24th International Conference on Computational Linguistics (Coling 2012)},
  title        = {Korektor--A System for Contextual Spell-checking and Diacritics Completion},
  editor       = {Martin Kay and Christian Boitet},
  author       = {Michal Richter and Pavel Stra{\v{n}}{\'{a}}k and Alexandr Rosen},
  year         = {2012},
  publisher    = {Coling 2012 Organizing Committee},
  organization = {{IIT} Bombay},
  address      = {Mumbai, India},
  venue        = {{IIT} Bombay, {VMCC}},
  pages        = {1--12}
}

8.3. Permanent Identifier

If you prefer to reference Korektor by a permanent identifier (PID), you can use http://hdl.handle.net/11234/1-1469.