Jargon: A Suite of Language Models and Evaluation Tasks for French Specialized Domains

Segonne, Vincent;Mannion, Aidant;Alonzo Canul, Laura Cristina;Audibert, Alexandre;Schwab, Didier;et.al.
(2024) The 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024) — Location: Torino, Italia (20.May.2024)

Files

LREC-COLING2024.pdf
  • Open Access
  • Adobe PDF
  • 294.02 KB
Download

Details

Authors
  • Segonne, Vincent
    Author
  • Mannion, Aidant
    Author
  • Alonzo Canul, Laura Cristina
    Author
  • Audibert, Alexandre
    Author
  • Norré, MagaliUCLouvain
    Author
  • François, Thomasorcid-logoUCLouvain
    Author
  • Schwab, Didier
    Author
Show more
Abstract
Pretrained Language Models (PLMs) are the de facto backbone of most state-of-the-art NLP systems. In this paper, we introduce a family of domain-specific pretrained PLMs for French, focusing on three important domains: transcribed speech, medicine, and law. We use a transformer architecture based on efficient methods (LinFormer) to maximise their utility, since these domains often involve processing long documents. We evaluate and compare our models to state-of-the-art models on a diverse set of tasks and datasets, some of which are introduced in this paper. We gather the datasets into a new French-language evaluation benchmark for these three domains. We also compare various training configurations: continued pretraining, pretraining from scratch, as well as single- and multi-domain pretraining. Extensive domain-specific experiments show that it is possible to attain competitive downstream performance even when pre-training with the approximative LinFormer attention mechanism. For full reproducibility, we release the models and pretraining data, as well as contributed datasets.
Affiliations

Citations

  • APA
  • Chicago
  • FWB

Segonne, V., Mannion, A., Alonzo Canul, L. C., Audibert, A., Liu, X., Macaire, C., Pupier, A., Zhou, Y., Aguiar, M., Herron, F., Norré, M., Amini, M.-R., Bouillon, P., Eshkol-Taravella, I., Esperança-Rodier, E., François, T., Goeuriot, L., Goulian, J., Lafourcade, M., et al. (2024). Jargon: A Suite of Language Models and Evaluation Tasks for French Specialized Domains. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, 9463-9476. https://hdl.handle.net/2078.5/214178 (Original work published 2024)