The influence of language in higher-level visual cortex

Symposium: Friday, May 16, 2025, 8:00 – 10:00 am, Talk Room 1

Schedule of Events | Search Abstracts | Symposia | Talk Sessions | Poster Sessions

Organizers: Oscar Woolnough¹, Alex L White²; ¹UTHealth Houston, ²Barnard College, Columbia University
Presenters: Alex L White, Alexia Dalski, Marisa Nordt, Marina Bedny, Oscar Woolnough, Emily X Meschke

Vision and language are often considered to be separate cognitive systems, studied by separate research teams. But they interact when we describe verbally the things we see and when we read written text. The development of these critical skills induces changes to brain organization, such that the visual and language systems influence one another. However, it has long been controversial whether apparently linguistic activity in the ventral visual stream, such as when reading, is purely the result of top-down feedback from the language network. To the contrary, recent work has demonstrated the critical nature of local language processing within visually-responsive areas of ventral temporal cortex. Language has multifaceted influences on visual cortex, from short-term modulations of activity shaped by ongoing linguistic processing, to long-term influences on functional organization across development. As children learn to read, they adapt their visual system to efficiently process written words, a cultural invention the visual system has not directly evolved to handle. They also need to be able to learn the names of and semantic relationships between objects in their environment. As we age, this ability to retrieve object names can degrade, resulting in anomias. In this symposium we will highlight the nature of the complex, bidirectional interplay between visual and linguistic processing. Specifically, there are three main themes: 1) How do the demands of linguistic tasks modulate visual processing? Alex White will begin with a review of top-down effects during visual word recognition and task-dependent functional connectivity between the ‘visual word form area’ and language areas. Alexia Dalski will describe how linguistic vs perceptual tasks change the nature of how visual cortex processes both words and emojis in adults and children. 2) How does learning language affect the development of visual cortex? Marisa Nordt will demonstrate how childhood language learning longitudinally shapes the development of category-selective visual cortex and how this reorganization predicts word and face recognition performance. Marina Bedny will describe how visual cortex repurposes to subserve language for speech and braille processing in blind readers and for sign language in Deaf signers. 3) How do semantics shape the organization of visual cortex? Oscar Woolnough will describe how mapping semantic and lexical processing in the ventral visual stream can predict stimulation-induced language disruptions and avoid post-surgical reading and naming deficits. Emily Meschke will report new discoveries about multimodal semantic representations that can be evoked by visual or linguistic input. Bringing together advances in neuroimaging, lesional, and computational methods, the speakers will outline a contemporary view on the nature of the interconnectivity between language and vision.

Talk 1

Interactions between vision and language when reading words

Alex L White¹, Vassiki S Chauhan¹; ¹Barnard College, Columbia University

Reading depends on a brain region known as the “visual word form area” (VWFA) in the left ventral occipitotemporal cortex. Its function is controversial. Some researchers emphasize its bottom-up visual selectivity for words (as compared to other visual stimuli), while others attribute its activity largely to feedback from the spoken language network. I will review recent fMRI studies that vary both task demands and visual stimulus properties to investigate the nature of top-down influences in the VWFA. The data show that the VWFA is uniquely modulated by a cognitive signal that is specific to voluntary linguistic processing. This signal differs from a more generic attentional effect because it enhances the response to some stimuli and suppresses others. Moreover, functional connectivity analyses suggest a source: communication between the VWFA and a left frontal language area increases when the participant is trying to read words. These results support a hybrid model: the VWFA is inherently selective for familiar orthography, but its linguistic processing is not fully automatic. Rather, the VWFA falls under control of the language network when the task demands it.

Talk 2

A preference for word forms precedes the preference for linguistic processing in the OTS-words subregions during development

Alexia Dalski¹, Antonia Schulz¹, Marie Klaes¹, Max Pirsch¹, Maria Meinhardt¹, Agon Ukaj¹, Laura Faßbender², Gudrun Schwarzer², Kalanit Grill-Spector³, Mareike Grotheer¹; ¹Philipps-Universität Marburg, ²Justus-Liebig-Universität Giessen, ³Stanford University

The visual word form areas in the occipito-temporal sulcus (mOTS-words and pOTS-words) are crucial for reading, and their responses are driven by both the visual features of text and linguistic task demands. We investigated the impact of text features and linguistic processing on OTS-words responses in adults (study 1, N=15) and longitudinally in children before and after first grade (study 2, N=15): Experiment 1 compared responses to text with other visual categories (faces, objects, limbs and houses) to localize OTS-words subregions. Experiment 2 compared responses during a linguistic and a color task performed either only on emojis (children) or on emoji and text stimuli (adults). In adults, both OTS-words subregions were identified by their text selectivity in experiment 1. In experiment 2, mOTS-words showed a preference for linguistic processing and a task-stimulus interaction, preferring emojis during the linguistic task. pOTS-words showed an overall preference for emojis. In children, OTS-words subregions showed no preference for text (limbs were the preferred category) or linguistic processing before schooling. After first grade, in experiment 1, a preference for text was observed in the OTS-words subregions, allowing for their reliable identification within individuals. In contrast to adults, the sub-regions did however not show a preference for linguistic processing in experiment 2 after first grade, suggesting that text selectivity may emerge earlier in development than the linguistic task preference. This indicates that while regions involved in visual word form processing emerge rapidly with literacy training, their functional specialization for linguistic processing requires additional experience.

Talk 3

Longitudinal development of category representations in high-level visual cortex during childhood

Marisa Nordt^1,2, Jesse Gomez³, Vaidehi S Natu¹, Alex A Rezai¹, Dawn Finzi¹, Holly Kular¹, Kalanit Grill-Spector¹; ¹Stanford University, ²RWTH Aachen, ³Princeton University

Human ventral temporal cortex (VTC) contains category-selective regions that respond preferentially to ecologically relevant categories such as faces, bodies, and words. How do these regions develop during childhood? Here, I will present our longitudinal work examining this development using both univariate and multivariate measures. In a first project, we used functional magnetic resonance imaging to measure longitudinal development of category-selectivity in school-age children. We discovered that, from childhood to the teens, face- and word-selective regions in VTC expand, but limb-selective regions shrink and lose their preference for limbs. Critically, as a child develops, increases in word-selectivity are linked to decreases in limb selectivity, revealing that limb-selectivity in VTC is repurposed into word-selectivity. These results provide evidence for cortical recycling during childhood development. In a second project, we addressed the question how distributed category representations develop during childhood and if this development relates to behavioral changes in recognition. We longitudinally measured the development of distributed responses across VTC to 10 categories in school-age children over several years. Our results reveal both strengthening and weakening of category representations with age, which was mainly driven by changes across category-selective voxels. Representations became particularly more distinct for words in the left hemisphere and for faces bilaterally. Critically, distinctiveness for words and faces across category-selective voxels in left and right lateral VTC, respectively, predicted children’s word and face recognition performance. These results suggest that the development of distributed representations in VTC has behavioral ramifications and advance our understanding of prolonged cortical development during childhood.

Talk 4

The interaction of innate constraints and experience at the language/vision interface

Marina Bedny¹, Elizabeth J Saccone¹, Mengyu Tian^1,2, Marcin Szwed⁴, Piotr Tomaszewski³, Maria Zimmermann^1,3,4; ¹Johns Hopkins University, ²Beijing Normal University, ³University of Warsaw, ⁴Jagiellonian University

Although most humans learn their first language via speech, visual communication is part of our evolutionary heritage and one of our earliest ways of connecting to other people in infancy (e.g., looking at faces). Studies with people who have distinctive sensory experiences (i.e., people born deaf or blind) reveal how intrinsic connections between the visual and language systems enable a broad range of adaptive behaviors. The lateral ventral occipito-temporal cortex (vOTC) sits at the junctions of the visual and language systems and in sighted people develops specialization for visual print (i.e., visual word form area.) We find that in people born blind, the lateral vOTC shows enhanced responses to spoken and written (tactile braille) language. Language responses peak in the location of the so called ‘fusiform face area’ and extend throughout the lateral vOTC and into early visual circuits. Deaf speakers of visuo-manual sign languages robustly recruit the lateral vOTC during language comprehension. The lateral vOTC becomes synchronized across Deaf individuals when viewing Polish Sign Language stories and shows higher synchrony for stories and sentences than lists of unconnected words. During story comprehension, the vOTC shows functional connectivity with the fronto-temporal language network. Together these data reveal the intrinsic connectivity between the lateral vOTC and language systems as well as the capacity of this connectivity to adapt to varying behavioral needs of the individual.

Talk 5

Dissociating visual, semantic, and lexical processing in human ventral temporal cortex

Oscar Woolnough¹, Kathryn Snyder¹, Meredith McCarty¹, Elliot Murphy¹, Nitin Tandon¹; ¹UTHealth Houston

Language-dominant ventral occipitotemporal cortex (vOTC) is crucial for multiple visual language tasks, including reading and naming. Lesions in vOTC can result in selective impairment of either reading or naming, resulting in alexia or anomia. Across multiple tasks, activity in vOTC is sensitive to visual, semantic, and lexical factors, but how do these factors interact within vOTC and with broader brain networks? And which of these factors are predictive of post-lesion deficits? I will first review evidence from our large population intracranial recordings during reading and object naming. Sensitivity to multiple lexical and semantic factors, including word frequency and concreteness, originates earliest within mid-fusiform cortex before rapidly spreading backward to earlier visual cortex, and out to frontal and parietal cortex. This reveals complex spatiotemporal interactions, both within vOTC and with the broader semantic and language networks. I will also present causal evidence, from direct cortical stimulation and post-resection deficits, for the existence of a dissociable, unimodal visual word form area and a basal temporal language area, alongside multimodal cortex underlying both reading and naming. We demonstrate an anterior-posterior split within vOTC, with stimulation of more posterior sites eliciting perceptible visual phenomena, while more anterior sites disrupt comprehension without overt visual distortions. The anteromedial vOTC has the highest probability of producing naming disruption, while posterolateral regions result in greater reading-specific disruption. Together, these results suggest the existence of multiple causally distinct but interactive functional regions within vOTC underlying visual language.

Talk 6

Visual-semantic representations within the distributed conceptual network of the human brain

Emily X Meschke¹, Jack L Gallant¹; ¹University of California at Berkeley

Most vision studies focus primarily on the visual modality, but much of our ability to reason and communicate about the world is based on lexical-semantic and conceptual information that is derived from vision, from other modalities, and from our prior experience. A prior study from our lab showed that information from visual modality-specific networks enters the multi-modal conceptual network through a distributed set of category-specific channels arranged along the anterior border of the visual cortex (Popham et al., 2021). However, little is known about how visual information is represented in the distributed conceptual network itself. To investigate this issue we used fMRI and voxelwise encoding models to compare visual-semantic representations measured during movie watching to lexical-semantic representations measured during narrative comprehension in the same participants. Comparison of the fit encoding models across experiments revealed a set of multi-modal patches located within the larger distributed conceptual network. These results provide further insights about how visual information is used to form our conceptual understanding of the world.