Different Tastes of Entities: Investigating Human Label Variation in Named Entity Annotations
(2024) Workshop on Understanding Implicit and Underspecified Language — Vol. 3, p. 73-81 (2024)
Files
2024.unimplicit-1.7.pdf
Open Access - Adobe PDF
- 397.23 KB
Details
- Authors
- Abstract
- Named Entity Recognition (NER) is a key information extraction task with a long-standing tradition. While recent studies address and aim to correct annotation errors via re-labeling efforts , little is known about the sources of human label variation, such as text ambiguity, annotation error, or guideline divergence. This is especially the case for high-quality datasets and beyond English CoNLL03. This paper studies disagreements in expert-annotated named entity datasets for three languages: English, Danish, and Bavarian. We show that text ambiguity and artificial guideline changes are dominant factors for diverse annotations among high-quality revisions. We survey student annotations on a subset of difficult entities and substantiate the feasibility and necessity of manifold annotations for understanding named entity ambiguities from a distributional perspective.
- Affiliations
Citations
Peng, S., Sun, Z., Loftus, S., & Plank, B. (2024). Different Tastes of Entities: Investigating Human Label Variation in Named Entity Annotations. Workshop on Understanding Implicit and Underspecified Language, 3, 73-81. https://doi.org/10.18653/v1/2024.unimplicit-1.7 (Original work published 2024)