This is an old revision of the document!
Table of Contents
ARF (average reduced frequency)
ARF1) is one of the many adjusted frequencies of a word form in a corpus. Adjusted frequencies adjust the simple frequency (number of occurrences) of a given word or phenomenon in the corpus to the degree of the uniformity of how its occurrences are distributed, taking into account dispersion. ARF helps prevent a scenario where frequency lists attach significance to words which are contained only in one text but with many individual occurrences, whereas in the rest of the corpus (and the language) they are much less common.
Reduced frequency and ARF
Let us assume that we have found in the corpus two words with the same occurrence rate. The first of these two words is found only in one single document, whereas the other is more or less evenly distributed throughout the entire corpus. In all probability, the second word will be more common than the first, but the occurrence rate will not tell us that. This is why we introduce the so-called reduced frequency.
Its definition is as follows: We use the letter f to label the frequency of a given word in the corpus. We divide the positions in the entire corpus into f sections of equal size. If the total number of words in the corpus should be divisible by f, the sections would be the same size; in the opposite case they may differ in one position. A reduced frequency is then the number of sections in which the given word occurs at least once.
The first word from our example will have a reduced frequency of either 1 (if all of its occurrences fall under on section) or 2 (if the boundary between two sections should happen to be in the middle of a cluster of occurrences. The second word will have a much higher value for reduced frequency. In extremely unlikely cases the reduced frequency could theoretically be the same as the frequency, which would happen should every occurrence of a given word fall under one single section. This very rarely happens in reality, especially as far as words with higher frequencies are concerned.
The average reduced frequency (ARF) is then derived from the reduced frequency in the sense that it takes into account all possible compilations of the corpus (the order of the texts in it). It is calculated as an average of the reduced frequency from all possible compilations of the corpus.
ARF calculations
The value of ARF is given by
$$ARF = \frac{1}{v} \sum_{i=1}^{f} \min (d_{i}, v)$$
where $f$ is the frequency of the given expression in a corpus of the size $N$, $d_{i}$ are the distances between the individual occurrences of this expression in the corpus (the number of words lying between them) and $v$ is the average distance between its occurrences and is given by $v = \frac{N}{f}$.
ARF values
Protože N je dělitelné f pouze výjimečně, nabývá ARF typicky neceločíselných hodnot, což je pro upravené frekvence běžné. Hodnota ARF pro daný výraz je korekcí jeho frekvence založenou na rozložení jeho výskytů v korpusu: čím je rozložení rovnoměrnější, tím více se hodnota ARF blíží frekvenci a naopak; pro výrazy, jejichž výskyty jsou v korpusu soustředěny do jediného shluku, se hodnota ARF blíží jedné bez ohledu na frekvenci.
Maximální hodnota ARF je tedy rovna frekvenci (je-li $d_{i} = v$ pro všechna $i$, tj. jsou-li vzdálenosti mezi všemi výskyty daného výrazu shodné), její nejmenší hodnota je rovna jedné.
Hodnota ARF se pro frekventovaná slova s rovnoměrným rozložením výskytů pohybuje okolo třetiny jejich frekvence (specificky však jen pro frekvenci větší než 50 000), pro odborné termíny vyskytující se pouze v několika dokumentech ale může být i mnohonásobně (10-krát až 100-krát) menší než frekvence. ARF je ve srovnání s frekvencí mnohem méně náchylná na (ne)zařazení konkrétních textů do korpusu, a lépe tedy odpovídá intuitivně chápané běžnosti slov.
ARF je v českém prostředí známá díky její implementaci v někdejším korpusovém manažeru Manatee/Bonito (dnes v rozhraní KonText), obstála také ve srovnání s ostatními běžně používanými upravenými frekvencemi a disperzními mírami.2) Mimoto se ARF prakticky osvědčila jako hlavní kritérium pro stanovení běžnosti slov při sestavování obou nejnovějších frekvenčních slovníků češtiny.
— M. Křen, V. Cvrček
