SumeCzech-NER

Title:

SumeCzech-NER

Creator:

Marek, Petr and Müller, Štěpán

Publisher:

Czech Technical University in Prague

Identifier:

http://hdl.handle.net/11234/1-3505

Subject:

SumeCzech, named entity recognition, named entitity corpus, and summarization

Type:

text and corpus

Description:

SumeCzech-NER SumeCzech-NER contains named entity annotations of SumeCzech 1.0 (Straka et al. 2018, SumeCzech: Large Czech News-Based Summarization Dataset). Format The dataset is split into four files. Files are in jsonl format. There is one JSON object on each line of the file. The most important fields of JSON objects are: - dataset: train, dev, test, oodtest - ne_abstract: list of named entity annotations of article's abstract - ne_headline: list of named entity annotations of article's headline - ne_text: list of name entity annotations of article's text - url: article's URL that can be used to match article across SumeCzech and SumeCzech-NER Annotations We used SpaCy's NER model trained on CoNLL-based extended CNEC 2.0. The model achieved a 78.45 F-Score on the dataset's testing set. The annotations are in IOB2 format. The entity types are: Numbers in addresses, Geographical names, Institutions, Media names, Artifact names, Personal names, and Time expressions. Tokenization We used the following Python code for tokenization: from typing import List from nltk.tokenize import word_tokenize def tokenize(text: str) -> List[str]: for mark in ('.', ',', '?', '!', '-', '–', '/'): text = text.replace(mark, f' {mark} ') tokens = word_tokenize(text) return tokens

Language:

Czech

Rights:

Mozilla Public License 2.0
http://opensource.org/licenses/MPL-2.0
PUB

Harvested from:

LINDAT/CLARIAH-CZ repository

Metadata only:

false

Date:

2021-01-30