From eyes to language : a multimodal approach to explore visual-language processing beyond typical visual orthography
(2025)
Files
Van_Audenhaege_Alice-Dissertation.pdf
Restricted Access - Adobe PDF
- 38.27 MB
Details
- Authors
- Supervisors
- Abstract
- Human language has evolved into a wide range of communication systems to meet varying sensory, cultural, and environmental demands. In everyday life, linguistic information can be perceived through multiple channels, including not only vocal sounds but also facial movements, gestures and written or even tactile symbols. However, the linguistic computations for processing alternative communication forms such as lipreading, sign language or tactile symbols have been poorly investigated compared to those for written or spoken language. This thesis explores various language modalities and their interaction, using neuroimaging and behavioural methods across different populations, including typically-developing adults, hearing individuals proficient in sign language, adults with dyslexia and braille pre-readers. In the first chapter, we investigate the role of visual category-selective regions in the ventral occipito-temporal cortex during sign language processing, using multivariate fMRI analyses. Results show differential involvement of key regions such as the anterior Visual Word Form Area (VWFA), and hand-, face- and body-selective areas in signers versus non- signers when processing of linguistic (signs) and non-linguistic gestures. The second chapter examines phonological representations from visual and auditory speech. Through an fMRI study, we show that the VWFA and the left superior temporal sulcus (STS) process phonological information from both speech sounds and lip movements, but with distinct multisensory encoding mechanisms. Beyond VWFA and left STS, we show that the right STS and sensorimotor regions also represent phonemes using a joint neural code for vision and audition. In the third chapter, we assess the relationship between visual speech processing and reading ability in typically-developing and dyslexic adults. We observe no clear association between lipreading and reading, suggesting that these two audio-visual skills rely on partially different cognitive mechanisms. The final chapter focuses on blind (or visually impaired) braille pre-readers, examining how auditory language experience influences reading development without visual input. We show that oral language predictors are unaffected by visual deprivation, while some aspects of emergent literacy are less developed at the start of reading acquisition. This indicates that oral language alone is insufficient for fully grasping print's communicative function and its link with spoken language. In conclusion, this work highlights the presence of linguistic representations in the visual cortex and the interaction between language modalities at the behavioural level.
- Affiliations
UCLouvain SSH/IPSY - Psychological Sciences Research Institute UCLouvain SSS/IONS - Institute of NeuroScience
Citations
Van Audenhaege, A. (2025). From eyes to language : a multimodal approach to explore visual-language processing beyond typical visual orthography. https://hdl.handle.net/2078.5/240938