Carina Wong¹, Sage Aronson¹, Michelle Greene¹; ¹Barnard College

Reading and visual categorization are both considered automatic cognitive processes, but their relative time courses are unknown. Pictures and their respective category word labels can each provide category information. Still, we do not yet know whether these interact to facilitate or interfere with processing speed. We used a paradigm inspired by the Stroop effect to investigate the extent and time course of the interaction between scenes and category labels that are either congruent or incongruent. We recorded 128-channel EEG while participants viewed stimuli in each of four conditions: words alone (scene category names printed over a phase-randomized scene), pictures alone, and word-picture composite stimuli in which both cues corresponded to the same scene category (congruent, e.g., the word “church” on top of an image of a church), and both cues corresponded to different categories (incongruent, e.g., the word “church” on top of an image of a park). The participants were tasked with categorizing the target word or picture as “indoor” or “outdoor”. We used a linear support vector machine to assess the time course of scene categorization using whole-brain patterns and five-fold cross-validation. As expected, we found earlier and more robust decoding for pictures than words alone. Interestingly, although pictures alone and congruent composites had similar decoding onsets, we observed stronger and more sustained decoding accuracy for the composites, suggesting that congruent scene category words can affect the robustness of early scene category representations. Finally, we noticed little difference in decoding accuracy for congruent versus incongruent composites, suggesting that although congruent words can facilitate scene representations, pictorial content alone is sufficient to create a scene representation. Altogether, these results suggest that there is facilitation in neural processing when two cues contribute to corresponding to the same information, but little to no interference when the cues conflict.

Acknowledgements: NSF CAREER 2240815 to MRG