Identifying inherent disagreement in natural language inference

Zhang, Xinliang Frederick;de Marneffe, Marie-Catherine
(2021) Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies

Files

2021naacl-main390.pdf
  • Open Access
  • Adobe PDF
  • 326.1 KB
Download

Details

Authors
Abstract
Natural language inference (NLI) is the task of determining whether a piece of text is entailed, contradicted by or unrelated to another piece of text. In this paper, we investigate how to tease systematic inferences (i.e., items for which people agree on the NLI label) apart from disagreement items (i.e., items which lead to different annotations), which most prior work has overlooked. To distinguish systematic inferences from disagreement items, we propose Artificial Annotators (AAs) to simulate the uncertainty in the annotation process by capturing the modes in annotations. Results on the CommitmentBank, a corpus of naturally occurring discourses in English, confirm that our approach performs statistically significantly better than all baselines. We further show that AAs learn linguistic patterns and context-dependent reasoning.
Affiliations

Citations

  • APA
  • Chicago
  • FWB

Zhang, X. F., & de Marneffe, M.-C. (2021). Identifying inherent disagreement in natural language inference. Proceedings of the 2021 Annual Conference of the North American Chapter of the Association for Computational Linguistics. Published. Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. https://doi.org/10.18653/v1/2021.naacl-main.390 (Original work published 2021)