InterCorp Release 11
Name | Czech – core | Czech – collections | other – core | other – collections | |
---|---|---|---|---|---|
Positions | Number of tokens | 132,508,429 | 115,574,528 | 340,554,768 | 1,550,923,096 |
Number of word forms | 106,898,538 | 88,872,779 | 283,075,338 | 1,225,361,750 | |
Structural attributes | Number of documents | 1,564 | 28 | 3,494 | 261 |
Number of texts | 1,507 | 111,672 | 3,232 | 1,841,341 | |
Number of sentences | 9,193,433 | 13,556,382 | 21,000,997 | 142,734,659 | |
Further information | reference | YES | |||
representative | NO | ||||
publication date | 2018 | ||||
foreign languages | 39 | ||||
tagged languages | 26 | ||||
lemmatized languages | 25 |
Access to the texts
After registration the corpus can be searched using a web interface. The registration is valid for all ICNC corpora with public access. If you already have a user name and password for the Czech part of the Czech National Corpus, you do not need to register for the parallel corpus.
InterCorp can be accessed via a standard web browser from KonText, the integrated search interface of the Czech National Corpus. A tutorial is available in Czech and a brief summary also in English.
After signing a non-profit licence agreement, texts from InterCorp can also be acquired as bilingual files including shuffled pairs of sentences. Please contact Martin Vavřín if you are interested.
New release of InterCorp is usually published once per year. With each new release, its size, possibly also the number of languages and the extent and quality of annotation may grow. Previous versions remain available (starting with release 6).
References
If you publish results based on InterCorp we would appreciate a link to the project site www.korpus.cz/intercorp. In your scientific publications please cite the following paper:
Čermák, F., Rosen, A. (2012). The case of InterCorp, a multilingual parallel corpus. International Journal of Corpus Linguistics. Vol. 13, no. 3, p. 411–427 (bibtex, electronic edition at ingentaConnect, preprint version).
For more references see the repository of bibliographical items based on the CNC. All references to work based on InterCorp are welcome. See here for details.
When citing a specific part of InterCorp please use the reference displayed in KonText in the corpus description, e.g. as:
Rosen, A., Vavřín, M., Zasina, A. J. (2018). The InterCorp Corpus – Czech1), version 11 of 19 October 2018. Institute of the Czech National Corpus, Charles University, Prague 2018. Available on-line: http://www.korpus.cz
Texts in the corpus
The core of InterCorp consists mostly of fiction, manually aligned. Intercorp offers also a selection of fully automatically processed texts, so-called collections. The choice in the present release includes:
- Political commentaries published by Project Syndicate and VoxEurop (formerly PressEurop)
- A package of legal texts of the European Union form the Acquis Communautaire corpus
- Proceedings of the European Parliament dated 2007–2011 from the Europarl corpus
- Film subtitles from the Open Subtitles database
- Translations of the Bible
These texts have been aligned automatically: search results may include a higher number of misaligned segments. Morevore, the collections do not retain all texts from the original resource. This includes texts that have no Czech counterpart. Some texts from the Acquis Communautaire and Europarl corpora have been partially corrected or omitted – as a result, they may differ in form or size if compared with the original source. A similar selection was applied to the Open Subtitles database, where – as an additional reduction – only a single translation was selected per title and language. On the other hand, some metadata items missing in the original resource but detectable from context or other sources have been added.
Each text has a Czech counterpart. As a result, Czech is the pivot language: for every text there is a single Czech version (original or translation), aligned with one or more foreign-language versions. The total size of the available part of InterCorp in release 11 from October 2018 is 283 mil. words in the aligned foreign language texts in the core part and 1,225 mil. words in the collections (see Version history). The share of the core and the collections in the corpus can be seen in the following charts. The charts show the volumes in millions of words.
Corpus size in thousands of words
Language | Core | Syndicate | Presseurop | Acquis | Europarl | Subtitles | Bible | Total | |
---|---|---|---|---|---|---|---|---|---|
ar | Arabic | 34 | 0 | 0 | 0 | 0 | 0 | 0 | 34 |
be | Belarusian | 4,426 | 0 | 0 | 0 | 0 | 0 | 0 | 4,426 |
bg | Bulgarian | 6,780 | 0 | 0 | 13,577 | 9,083 | 0 | 0 | 29,441 |
ca | Catalan | 5,596 | 0 | 0 | 0 | 0 | 0 | 736 | 6,332 |
da | Danish | 5,595 | 0 | 0 | 20,313 | 13,916 | 14,429 | 657 | 54,910 |
de | German | 34,915 | 4,457 | 2,483 | 20,610 | 13,088 | 8,392 | 724 | 84,669 |
el | Greek | 0 | 0 | 0 | 23,853 | 15,404 | 23,709 | 0 | 62,966 |
en | English | 27,968 | 4,604 | 2,670 | 22,902 | 15,576 | 52,105 | 730 | 126,555 |
es | Spanish | 23,349 | 5,322 | 2,859 | 26,262 | 16,249 | 36,650 | 0 | 110,691 |
et | Estonian | 0 | 0 | 0 | 14,896 | 10,899 | 10,298 | 0 | 36,093 |
fi | Finnish | 4,585 | 0 | 0 | 15,489 | 10,175 | 15,098 | 544 | 45,890 |
fr | French | 17,213 | 5,391 | 3,046 | 26,200 | 17,179 | 25,986 | 764 | 95,779 |
he | Hebrew | 0 | 0 | 0 | 0 | 0 | 16,221 | 0 | 16,221 |
hi | Hindu | 409 | 0 | 0 | 0 | 0 | 0 | 0 | 409 |
hr | Croatian | 20,147 | 0 | 0 | 0 | 0 | 19,048 | 571 | 39,767 |
hu | Hungarian | 5 783 | 0 | 0 | 17 852 | 12 198 | 21 115 | 0 | 56 948 |
is | Icelandic | 0 | 0 | 0 | 0 | 0 | 1,581 | 0 | 1,581 |
it | Italian | 11,400 | 1,141 | 2,747 | 23,771 | 15,494 | 14,700 | 684 | 69,937 |
ja | Japanese | 1,198 | 0 | 0 | 0 | 0 | 477 | 0 | 1,675 |
lt | Lithuanian | 287 | 0 | 0 | 17,316 | 11,213 | 558 | 471 | 29,844 |
lv | Latvian | 2,523 | 0 | 0 | 17,522 | 11,682 | 280 | 0 | 32,008 |
mk | Macedonian | 6,508 | 0 | 0 | 0 | 0 | 1,877 | 0 | 8,385 |
ms | Malay | 0 | 0 | 0 | 0 | 0 | 3,521 | 0 | 3,521 |
mt | Maltese | 0 | 0 | 0 | 13,953 | 0 | 0 | 0 | 13,953 |
nl | Dutch | 13,689 | 711 | 2,953 | 23,416 | 15,558 | 29,373 | 717 | 86,416 |
no | Norwegian | 6,675 | 0 | 0 | 0 | 0 | 0 | 721 | 7,397 |
pl | Polish | 24,292 | 0 | 2,378 | 19,594 | 12,811 | 26,572 | 583 | 86,230 |
pt | Portuguese | 4,032 | 520 | 3,000 | 27,301 | 16,485 | 43,392 | 760 | 95,489 |
rn | Romani | 14 | 0 | 0 | 0 | 0 | 0 | 0 | 14 |
ro | Romanian | 3,888 | 0 | 2,738 | 8,092 | 9,446 | 34,128 | 0 | 58,292 |
ru | Russian | 7,062 | 3,768 | 0 | 0 | 0 | 6,887 | 565 | 18,282 |
sk | Slovak | 8,545 | 0 | 0 | 18,401 | 12,734 | 5,134 | 561 | 45,376 |
sl | Slovenian | 3,534 | 0 | 0 | 18,485 | 12,241 | 17,023 | 0 | 51,282 |
sq | Albanian | 0 | 0 | 0 | 0 | 0 | 2,003 | 0 | 2,003 |
sr | Serbian | 10,661 | 0 | 0 | 0 | 0 | 20,727 | 0 | 31,388 |
sv | Swedish | 12,396 | 0 | 0 | 19,609 | 13,840 | 14,694 | 638 | 61,178 |
tr | Turkish | 0 | 0 | 0 | 0 | 0 | 21,190 | 0 | 21,190 |
uk | Ukrainian | 9,571 | 0 | 0 | 0 | 0 | 245 | 596 | 10,411 |
vi | Vietnamese | 0 | 0 | 0 | 0 | 0 | 1,474 | 0 | 1,474 |
Subtotal | 283,075 | 30,044 | 27,189 | 428,621 | 278,178 | 539,250 | 11,593 | 1,676 293 | |
cs | Czech | 106,899 | 4,124 | 2,310 | 19,085 | 12,188 | 50,604 | 562 | 195,771 |
TOTAL | 389,974 | 30,073 | 27,184 | 428,482 | 277,458 | 539,489 | 11,585 | 1,704,208 |
N.B.: Each Czech text is counted only once, even though it may have more than one foreign counterpart.
Morphosyntactic annotation
Texts in the following languages have received some morphosyntactic annotation. The format and often even the meaning of categories encoded in the morphosyntactic tags differs in most languages. Thus for each tagged language we provide a link to the tagset description. After selecting CQL as the query type, the tagset description is available also from the KonText search interface.
*) The corpus includes tags in a condensed form, e.g. V:Sg:Nom:Act:PrfPrc:Pos corresponds to [POS=V] [NUM=SG] [CASE=NOM] [VOICE=ACT] [PCP=PRFPRC] [CMP=POS]. Similarly, Pron:Pers:Sg:Ade:Up corresponds to [POS=PRON] [SUBCAT:PERS] [NUM:SG] [CASE=ADE] [CASECHANGE=UP].
**) Within a single morphological tag a colon rather than period is used as a separator of the individual categories, e.g. ADJA:Pos:Nom:Sg:Fem.
***) Tags in the corpus do not always correspond to those listed in the detailed description. Some morphological categories are omitted in the corpus tags, e.g. pronouns are always tagged only as “P-”. All tags, as used in ther corpus, are listed in the brief description.
****) The tag is in the UD (Universal Dependencies) format, components of the tag are separated by a vertical bar (|), e.g. the form школы in genitive singular is tagged as: NOUN|Animacy=Inan|Case=Gen|Gender=Fem|Number=Sing
. The query can be specified in the same way as for other languages, treating the tag as a string, i.e.\ [tag="NOUN.*Case=Gen\|Gender=Fem.*"]
or the tag components can be specified separately: [tag="Case=Gen" & tag="NOUN" & tag="Gender=Fem"]
(the order of categories is not significant). The result is identical in either case.
Tag formats specified in tagset descriptions differ from those actually used in the corpus also in some other languages. Please check the tag format before making a tag query if you are not sure. In a page displaying results open the View/Corpus-specific settings… menu to check the tag option in the Positional attributes box and choose the for each token option in the Viewing options box.
Queries including contracted forms into tagged or lemmatized texts may fail. This includes forms such as can't or I'm, which are split by the tagger into two parts (ca+n't and I+'m) with corresponding lemmas and tags. Similarly with Polish forms byłam or gdybyś (była+m and gdyby+ś). Tokenization may even introduce errors: gdzie ś za Wisłą. In this context, gdzieś is not a contraction. A query intended to find the whole contracted form should be typed in as a Phrase, with the split parts separated by a space. Only the individual parts of the contracted form are assigned a tag and a lemma.
Morphological tags including characters with a special meaning in regular expressions, e.g. “$” in the English tag “wp$”, must be preceded in queries by a backslash: tag=“wp\$”.
Structural attributes
Structure | Attribute | Description | Values |
---|---|---|---|
doc | doc.id | document identifier | author's_last_name-shortened_title / _ACQUIS / _EUROPARL / _PRESSEUROP_year / _SUBTITLES / _SYNDICATE_year / _OT / _NT |
text | text.id | text identifier | author's_last_name-shortened_title:0 / _ACQUIS:number / _EUROPARL:number / _PRESSEUROP:number / _SUBTITLES:number / _SYNDICATE_year:name / _OT:book / _NT:book |
text.author | author | last name, first name | |
text.title | full title | text | |
text.lang | language | ar / be / bg / ca / cs / da / de / el / en / es / et / fi / fr / he / hi / hr / hu / is / it / ja / lt / lv / mk / ms / mt / nb / nl / no / pl / pt / rn / ro / ru / sk / sl / sq / sr / sv / sy / tr / uk / vi / zh | |
text.version | version | number | |
text.group | core/collection | Core / Acquis / Europarl / PressEurop / Subtitles / Syndicate / Bible | |
text.publisher | publisher | text | |
text.pubplace | publication place | text | |
text.pubDateYear | publication year | number | |
text.pubDateMonth | publication month | number | |
text.origyear | original creation year | number | |
text.isbn | ISBN | number | |
text.txtype | text type | discussions - transcripts / drama / fiction / journalism - commentaries / journalism - news / legal texts / nonfiction / other / poetry / subtitles / religious | |
text.comment | comment | text | |
text.original | original version? | Yes / No | |
text.srclang | language of the original | ar / as / az / be / bg / bl / bn / bo / bs / bt / ca / cr / cs / ct / cz / da / de / dk / eb / el / en / es / et / eu / fa / fi / fr / ga / gr / he / hi / hr / hu / hy / id / ie / is / it / ja / ka / ko / ku / lt / lv / mk / mn / ms / mt / my / ni / nl / no / pl / po / ps / pt / rm / rn / ro / ru / se / sk / sl / sq / sr / sv / ta / th / ti / tl / tr / tu / uk / un / ur / vi / zh | |
text.translator | translator | last name, first name | |
text.transsex | translator's gender | F / M | |
text.authsex | author's gender | F / M | |
text.transcomment | translation comment | text | |
text.collectiontitle | collection title | text | |
text.volume | volume number | number | |
text.pages | number of pages | number | |
text.lang_var | language variety | de-AT / de-CH / de-DE / en-AU / en-CA / en-GB / en-UM / en-US / es-ES / es-MX / es-PE / fr-BE / fr-FR / it-CH / it-IT / nl-BE / nl-NL / pt-BR / pt-PT / sr-Latn-RS / sy-Cyrl-RS | |
text.wordcount | number of words | number | |
div | div.id | division identifier (Bible) | _NT / _OT:chapter |
div.type | division type | chapter | |
p | p.id | paragraph identifier | doc:text:div:par |
s | s.id | sentence identifier | doc:text:div:par:sent |
hi | hi.rend | typeface | italic / bold / bold italic |
lb | lb.id | verse identifier (Bible) | book:chapter:verse |
Acknowledgements
We are grateful for the possibility to use the following texts and software:
Texts:
- The latest (13th corrected) issue of the Czech Ecumenical Translation of the Bible could be included to the corpus thanks to the Czech Biblical Society, especially Petr Fryš.
- Fiction in many Slavic and some other languages from ASPAC – Amsterdam Slavic Parallel Aligned Corpus – with special thanks to Adrian Barentsen
- Political commentaries in a number of languages from the site Project Syndicate
- Newspaper texts in a number of languages from the Presseurop/VoxEurop server
- Legal texts in EU languages from the JRC-ACQUIS corpus
- Proceedings of the European Parliament from the EuroParl corpus
- Slovak-Czech concordances from the Slovak National Corpus
- Short stories in a number of languages My 1989 from Goethe Institut
- A number of texts in the Czech-Lithuanian section of the corpus and Jiří Levý's The Art of Translation in more languages – with special thanks to Patrick Corness
- George Orwell's novel 1984 in a number of languages from the Multext-East corpus
- Ukrainian and Polish texts from the PolUkr corpus
- Film subtitles from the database Open Subtitles
Pre-processing
- Parallel text editor InterText by Pavel Vondřička
- Aligner Hunalign
- Sentence splitter for Czech by Pavel Květoň
- Sentence splitter for Norwegian by Jarle Ebeling and Pavel Vondřička
- Sentence splitter Punkt for all other languages from Natural Language Toolkit
Taggers/lemmatizers:
- TreeTagger for Bulgarian, Dutch, English, Estonian (thanks to Helmut Schmid), French, Italian, Portuguese (thanks to Pablo Gamallo), Russian and Spanish
- HunPOS for Hungarian and other languages
- Tagger for Slovak (thanks to Radovan Garabík)
- Tagger for Norwegian (thanks to Pavel Vondřička)
- totale for Slovene (thanks to Tomaž Erjavec)
- RFTagger for German
- OMorFi+HunPOS for Finnish (thanks to Filip Ginter)
- Stagger and IceStagger for Swedish and Icelandic (thanks to Robert Östling)
- RelDI tagger for Croatian and Serbian (thanks to Nikola Ljubešić)
- LVTagger for Latvian (thanks to Pēteris Paikens and Michal Škrabal)
- UD Pipe for Belarusian and Ukrainian (thanks to Bohdan Moskalevskyi)