Breaking NLP: Using Morphosyntax, Semantics, Pragmatics and World Knowledge to Fool Sentiment Analysis Systems

Mahler, Taylor;Cheung, Willy;Elsner, Micha;King, David;White, Michael;et.al.
(2017) Proceedings of the First Workshop on Building Linguistically Generalizable NLP Systems — Location: Copenhagen, Denmark (September.2017)

Files

W17-5405v2.pdf
  • Open Access
  • Adobe PDF
  • 336.01 KB
Download

Details

Authors
Show more
Abstract
This paper describes our “breaker” submission to the 2017 EMNLP “Build It Break It” shared task on sentiment analysis. In order to cause the “builder” systems to make incorrect predictions, we edited items in the blind test data according to linguistically interpretable strategies that allow us to assess the ease with which the builder systems learn various components of linguistic structure. On the whole, our submitted pairs break all systems at a high rate (72.6%), indicating that sentiment analysis as an NLP task may still have a lot of ground to cover. Of the breaker strategies that we consider, we find our semantic and pragmatic manipulations to pose the most substantial difficulties for the builder systems.
Affiliations

Citations

  • APA
  • Chicago
  • FWB

Mahler, T., Cheung, W., Elsner, M., King, D., de Marneffe, M.-C., Shain, C., Stevens-Guille, S., & White, M. (2017). Breaking NLP: Using Morphosyntax, Semantics, Pragmatics and World Knowledge to Fool Sentiment Analysis Systems. Proceedings of the Workshop on Building Linguistically Generalizable NLP. Published. Proceedings of the First Workshop on Building Linguistically Generalizable NLP Systems, Copenhagen, Denmark. https://doi.org/10.18653/v1/w17-5405 (Original work published 2017)