Fine-tuning is not (always) overfitting artifacts
(2023) 31st European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning — Location: Bruges, Belgium (4.October.2023)
Files
Bogaert2023_ES2023-152.pdf
Open Access - Adobe PDF
- 1.87 MB
Details
- Authors
Bogaert, Jérémie 
De Bodt, Cyril
- Abstract
- Since their release, transformers, and in particular fine-tuned transformers are widely used for text related classification tasks. However, only a few studies try to understand how fine-tuning actually works and existing alternatives, such as feature-based transformers, are often overlooked. In this work, we study a French transformer model, CamemBERT, to compare the fine-tuned and feature-based approaches in terms of their performances, interpretability and embedding space. We observe that while fine-tuning has a limited impact on performances in our case study, it significantly affects the intepretability (by better isolating words that are intuitively connected to the classification task) and embedding space (by summarizing the majority of the relevant information into a fewer dimensions) of the results. We conclude by highlighting open questions regarding the generalization potential of fine-tuned embeddings.
- Affiliations
Citations
Bogaert, J., Jean, E., De Bodt, C., & Standaert, F.-X. (2023). Fine-tuning is not (always) overfitting artifacts. ESANN proceedings, 1(1), 1-6. https://doi.org/10.14428/esann/2023.ES2023-152 (Original work published 2023)