| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| en:cnk:intercorp:historie [2026/09/21 00:59] – [Release 16ud] rosen | en:cnk:intercorp:historie [2026/09/30 21:14] (current) – [Release 17ud] rosen |
|---|
| ====== InterCorp: Version history ====== | ======= InterCorp: Version history ======= |
| |
| ==== Version 17ud ==== | ===== Release 17ud ===== |
| |
| Released on 9 September 2026 (pilot version) | * Released on 9 September 2026 (pilot version) |
| | * [[https://wiki.korpus.cz/doku.php/en:cnk:intercorp:verze17ud|Information about the corpus]] |
| |
| == Data: == | == Data: == |
| |
| * **New books** in the core: 40 new Czech titles (1852 in total), 214 new titles across all languages (6669 in total); for an overview, see the [[https://jakobson.korpus.cz/~rosen/INTERCORP/ICv17ud/intercorp_core_inventory_prototype_v7.html|InterCorp Core Guide]]; see also the [[https://jakobson.korpus.cz/~rosen/INTERCORP/ICv17ud/intercorp_hall_of_fame_v2_updated.html|Hall of Fame]] listing everyone who contributed to the expansion of the core in version 17ud | * **New books** in the Core: 40 new Czech titles (1852 in total), 214 new titles across all languages (6669 in total) |
| * New language: **Kashubian** (code ''cb'', currently 2 books in the core) | * The new [[https://jakobson.korpus.cz/~rosen/INTERCORP/ICv17ud/intercorp-tools-static/index.html|InterCorp Tools]] serve as a **guide to InterCorp**, providing a wide range of information about the corpus and replacing the tables and charts included in the documentation of previous versions; they also include a Hall of Fame listing everyone who contributed to the preparation of new texts for the InterCorp Core release 17ud |
| * New text collection: **[[https://www.ted.com|TED Talks]]** -- 1383 Czech talk transcripts, i.e. 2.4 million positions; 39,374 transcripts across all languages, i.e. 28.9 million positions | * New language: **Kashubian** (code ''cb'', currently 2 books in the Core) |
| | * New text collection: **[[https://www.ted.com|TED Talks]]** -- 1383 talk transcripts in Czech, i.e. 2.4 million positions; 39,374 transcripts across all languages, i.e. 28.9 million positions |
| * Additions to the **Project Syndicate** collection from 2021--2025 | * Additions to the **Project Syndicate** collection from 2021--2025 |
| * The **Bible** is now also available in Spanish and Upper Sorbian | * The **Bible** is now also available in Spanish and Upper Sorbian |
| == Annotation: == | == Annotation: == |
| |
| * This is the first version available exclusively with linguistic annotation based on Universal Dependencies | * This is the first release available exclusively with linguistic annotation based on **Universal Dependencies** |
| * In Slavic and Romance languages, multiword verb forms are now annotated as well; see the positional attribute ''cw_feats'' – **content-word features** ([[pojmy:content-word features|cw_feats]]) | * In Slavic and Romance languages, multiword verb forms are now also annotated using the positional attribute ''cw_feats'' – **content-word features** ([[pojmy:content-word features|cw_feats]]) |
| * In the core, **authors** are now also identified by a name that is consistent across all languages (text attribute ''rf_author'') | * In the Core, **authors** are now also identified by a name that is consistent across languages (attribute ''text.rf_author'') |
| * The positional attributes that can be used to navigate syntactic structure have been extended to include references to the **word form of the head**: ''p_word'' (parent word form) and ''e_word'' (effective parent word form) | * The positional attributes useful for navigating syntactic structure have been extended with references to the **word form of the head**: ''p_word'' (parent word form) and ''e_word'' (effective parent word form) |
| * **Newer language models** (ud-2.17) were used for linguistic annotation; only Chinese will be annotated differently in the final version of 17ud (but the resulting annotation will be compatible with Universal Dependencies) | * **Literature for children and young adults**, previously identified as ''text.txtype=children's lit.'', is now identified as ''text.audience=jun''; for other texts, this attribute has the value ''gen''; the ''text.txtype'' attribute can thus be used to distinguish text types (''fiction'', ''poetry'', ''drama'', ''non-fiction'') |
| * New **segmentation and alignment tools** ([[https://github.com/thompsonb/vecalign|Vecalign]]) were used to prepare the TED Talks collection | * **Newer language models** from [[https://lindat.mff.cuni.cz/services/udpipe/|UDPipe]] (ud-2.17) were used for linguistic annotation; Chinese alone is annotated using [[https://trankit.readthedocs.io/|Trankit]] for better results (thanks to [[https://cz.linkedin.com/in/benjamin-reeves-|Benjamin David Reeves]] and [[https://ufal.mff.cuni.cz/pavel-stranak|Pavel Straňák]]) |
| | * New **segmentation and alignment tools** were used to prepare the TED Talks collection ([[https://github.com/thompsonb/vecalign|Vecalign]]) |
| |
| == Search interface: == | == Search interface: == |
| |
| * In Advanced Query, the value of the positional attribute ''deprel'' (the syntactic function of a word) can now be entered in the same way as values of the ''upos'' and ''feats'' attributes, i.e. by selecting it from the **Insert tag** menu | * The value of the positional attribute ''deprel'' (the syntactic function of a word) can be specified in Advanced Query in the same way as the values of ''upos'' and ''feats'', by selecting it from the **Insert tag** menu |
| * Clicking any word in one language in a concordance highlights its most probable **translation equivalent** in the second and any additional language (this feature also works in older versions of InterCorp) | * Clicking any word in the concordance in one language highlights its most likely **translation equivalent** in the other language or languages (this feature also works with earlier versions of InterCorp) |
| |
| == Bug fixes: == | == Bug fixes: == |
| |
| * The **author's name** was not given consistently for some texts within the same language (text attribute ''author'') | * **Author names** were not given consistently for some texts within the same language (attribute ''text.author'') |
| * In languages written in **Cyrillic**, characters graphically identical to Latin characters were sometimes incorrectly encoded as Latin; we have managed to eliminate most of these errors | * In languages written in **Cyrillic**, characters visually identical to Latin characters were sometimes incorrectly encoded as Latin; we have managed to eliminate most of these errors |
| | * A number of texts were not identified as **literature for children and young adults**; this has been corrected in connection with moving this information to the new ''text.audience'' attribute; the number of texts identified as such has increased from 5 to 122 |
| | * Due to minor corrections in the way **syntactic complexity and lexical diversity** are aggregated for individual texts, text types and languages, the aggregate values of these measures may in some cases differ slightly from the previous version even when the corpus content has not changed |
| |
| | == Known issues: == |
| |
| | * In **French**, ''Tense=Past'' and ''Tense=Pres'' are no longer specified in the ''feats'' attribute for participles (''VerbForm=Part''), so past and present participles cannot be distinguished in the InterCorp annotation. This is due to a change in the [[https://universaldependencies.org/treebanks/fr_gsd/index.html|French data]] used to train the [[https://lindat.mff.cuni.cz/services/udpipe/|UDPipe]] language model. The maintainers of the GSD treebank moved this information from the ''FEATS'' field to the ''MISC'' field, which is not included in the model. (For the data format, see [[https://universaldependencies.org/format.html|CoNLL-U]]; for the reasons behind the change, see the discussion [[https://github.com/UniversalDependencies/UD_French-GSD/issues/30|UD_French-GSD: Tense removed from FEATS]].) |
| ===== Release 16ud ===== | ===== Release 16ud ===== |
| |
