Do You Know That Florence Is Packed with Visitors? Evaluating State-of-the-art Models of Speaker Commitment
(2019) Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics — Location: Florence, Italy (July.2019)
Files
P19-1412.pdf
Open Access - Adobe PDF
- 759.47 KB
Details
- Authors
- Abstract
- When a speaker, Mary, asks Do you know that Florence is packed with visitors?, we take her to believe that Florence is packed with visitors, but not if she asks Do you think that Florence is packed with visitors? Inferring speaker commitment (aka event factuality) is crucial for information extraction and question answering. Here we explore the hypothesis that linguistic deficits drive the error patterns of speaker commitment models by analyzing the linguistic correlates of model errors on a challenging naturalistic dataset. We evaluate two state-of-the-art speaker commitment models on the CommitmentBank, an English dataset of naturally occurring discourses. The CommitmentBank is annotated with speaker commitment towards the content of the complement (Florence is packed with visitors in our example) of clause-embedding verbs (know, think) under four entailment-canceling environments. We found that a linguisticallyinformed model outperforms a LSTM-based one, suggesting that linguistic knowledge is needed to capture such challenging naturalistic data. A breakdown of items by linguistic features reveals asymmetrical error patterns: while the models achieve good performance on some classes (e.g., negation), they fail to generalize to the diverse linguistic constructions (e.g., conditionals) in natural language, highlighting directions for improvement
- Affiliations
Citations
Jiang, N., & de Marneffe, M.-C. (2019). Do You Know That Florence Is Packed with Visitors? Evaluating State-of-the-art Models of Speaker Commitment. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL 2019). Published. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy. https://doi.org/10.18653/v1/p19-1412 (Original work published 2019)