BERTinchamps: Cost-Effective In-House Training of Language Models in French
(2024) Italian Journal of Computational Linguistics (IJCOL) — Vol. 10, n° 2, p. 131-146 (2024)
Files
IJCOLfinalversion.pdf
Author preprint
Open Access - Adobe PDF
- 783.45 KB
Details
- Authors
- Abstract
- (en) Many in-house applications are envisioned for Language Models (LMs) across various fields. In the medical domain, LMs could automate tasks such as summarizing the health condition of a patient and codifying electronic health records. They also hold potential in the legal field and in journalism. While training LMs directly inside an institution is desirable for leveraging local data and addressing data privacy concerns, this process demands a costly and complex computational infrastructure. This paper explores the recent Cramming approach as a cost-effective way to locally train medium-sized LMs, in one day and using one graphics processing unit (GPU). We show that the Cramming approach that was originally designed for the English language can be transposed to French, that the resulting models can be fine-tuned to domain- specific tasks in the French language, and that pre-training by including in-house data increases the performance of the models for journalism data. This research opens the path to the creation of medium-sized LMs that are tailored to the specific needs of institutions that handle sensitive textual data in another language than English.
- Affiliations
Citations
Fierens, A., & Jodogne, S. (2024). BERTinchamps: Cost-Effective In-House Training of Language Models in French. Italian Journal of Computational Linguistics (IJCOL), 10(2), 131-146. https://doi.org/10.17454/IJCOL102.07 (Original work published 2024)