HindMonoCorp 0.5

HindEnCorp 0.5 and HindMonoCorp 0.5 File Formats
================================================

This file describes the file formats of the Hindi-English and Hindi-only
corpora released in 2014 under the names HindEnCorp 0.5 and HindMonoCorp 0.5.

More details about the preparation of the corpora can be found in the paper:

  Ondřej Bojar, Vojtěch Diatka, Pavel Rychlý, Pavel Straňák, Aleš Tamchyna
  and Dan Zeman. HindEnCorp - Hindi-English and Hindi-only Corpus for
  Machine Translation. In Proc. of LREC 2014. Reykjavik, Iceland. ISBN
  978-2-9517408-8-4. ELRA. 2014.

or on the corpora web page:
  http://ufal.mff.cuni.cz/hindencorp

Please cite this paper if you make any use of the corpora. BibTeX citation
format below.


Common Properties
-----------------

All the files are plain text:

- compressed with gzip
- encoded in UTF-8
- with unix line breaks (LF)
- with tab-delimited columns

The monolingual and parallel corpora have different columns.

The actual corpus text is stored in one (monolingual corpus) or two (parallel
corpus) of the columns.


Plaintext vs. Export File Format
--------------------------------

Both the monolingual and the parallel corpus come in a simple plain text format
and in a tokenized, tagged and lemmatized format.

The plaintext format preserves the original tokenization (as much as possible
given the diverse sources included in our corpus).

The 'export' format is tokenized and represents each token as a '|'-delimited
triple of: the word form, the lemma, and part-of-speech tag. If there was the
character '|' (this character is also used instead of the proper Devanagari
Danda in some sources), we escape it as '&pipe;'.

There is exactly the same number of lines in the plaintext and export file
formats.


HindEnCorp Columns
------------------

The files hindencorp05.plaintext.gz and hindencorp05.export.gz each contain the
parallel corpus and differ only in the processing of the corpus texts. The
files have these columns:

- sou . . .