Many association measures do not measure association (but frequency), and what to do about that
(2022) ICAME43 — Location: Anglia Ruskin University, Cambridge, UK (27.July.2022)
Files
No attached file found for this publication.
Details
- Authors
- Abstract
- The two most widely-used corpus statistics by far are probably frequencies (of occurrence and of co-occurrence) and association measures, whose purpose usually is to quantify the degree to which two 'things' like to co-occur, especially collocations for the co-occurrence of words and colligations/collostructions for the co-occurrence of words with syntactic constructions. As is well known, dozens of measures have been proposed – Pecina (2009) alone reviews more than 80 – but the frequencies of uses of association measures are just as Zipfian-distributed as the words or constructions to which they are applied: a small handful of measures (arguably, the loglikelihood ratio, pFisher-Yates exact, MI and derivatives such as MI2 and MI3, t, and Dice) covers probably the vast majority of applications. In this paper we will demonstrate that most of these association measures actually have a validity problem when one makes the pretty common-sense assumption that a valid measure M of X is one that measures X and not much else (so that M can in fact be interpreted in terms of X rather than, maybe some quantity of something else). But, as we demonstrate, most of these association measures do not measure what they are purported to measure (association) – instead, they repackage frequency information and capture very little of association. Thus, in this paper, we will do four things. First, we will demonstrate on the basis of several hypothetical collostruction results and two keyness results that the maybe most widely used association measure, the log-likelihood ratio, reacts more to frequency than to association; then we will show that the same is true in actual data (Adj-N collocations of fast and three other speed adjectives in the British National Corpus). Results from generalized additive models show that the loglikelihood ratio values for the collocations are (i) extremely predictable from the co-occurrence frequency alone (R2>0.94) and are (ii) hardly correlated at all with potential gold standard measure that *only* measures association (R2<0.06). Second, for the same data, we will also show that the loglikelihood ratio can be nearly predicted perfectly from an interaction of frequency of association, but with frequency playing a *much* stronger role: association can only affect the loglikelihood ratio with greater than average frequencies. That high per definitionem correlation between frequency and the loglikelihood ratio of course also means that our favorite measure of association does not permit the user to identify low-frequency-but-high-association collocations … Third, we will summarily discuss other association measures to show, for instance, that (i) the t-score fares even words in terms of its validity, that (ii) MI fares much better, but that (iii) MI2 and MI3 are very problematic in how they take a good measure (MI) and systematically make it worse, and that (iv) Dice and log Dice differ strongly in terms of validity (the latter being much better than the former). Finally, we will outline a three-step procedure of how one can define a true measure of association, one that is by design independent of co-occurrence frequency. We then apply this measure to the speed adjectives and show that is indeed uncorrelated with frequency (R2with cooccurrence frequency < 0.01) but that it is very well correlated with what we should from now on be using as a gold standard measure of association, the log odds ratio. We conclude with some comments on the use of cut-off points for significant/interesting associations and a pointer to how the logic underlying this frequency-less association measure can be used for other corpus statistics.
- Affiliations
Citations
Gries, S. Th., & Paquot, M. (2022). Many association measures do not measure association (but frequency), and what to do about that. ICAME43, Anglia Ruskin University, Cambridge, UK. https://hdl.handle.net/2078.5/106327