Emalie McMahon^1,2, Elizabeth Jiwon Im³, Michael F. Bonner¹, Leyla Isik¹; ¹Johns Hopkins University, ²Massachusetts Institute of Technology, ³Stanford University

Recent work has argued that in addition to the dorsal and ventral visual streams, there is a third visual stream projecting laterally from early visual cortex to the superior temporal sulcus that is specialized for dynamic social content. A key characteristic of the dorsal and ventral streams is hierarchical computations. Here, we investigate whether the lateral visual stream also has hierarchical representations of social actions by combining EEG and fMRI data, which allows us to investigate the direction of information flow through lateral regions of the brain. Separate participants viewed the same videos while neural responses were recorded with EEG and fMRI. We first performed an EEG decoding analysis and found that the relative latency of decoding visual and social features from EEG progresses from low-level to abstract features, consistent with hierarchical processing. Next, we predicted the average response in functional ROIs from fMRI from the EEG signal across time. We find a short latency in predicting low-level regions and a longer latency in predicting all other regions, suggesting any early-late dissociation in information flow across these regions. Finally, using a novel EEG-fMRI encoding procedure, we track the time course of social action features across lateral stream ROIs. We find that, similar to the whole brain analysis, mid-level features come online in the lateral stream before high-level features of social actions. However, social action information is represented in mid- and high-level regions around the same time, suggesting the lateral pathway may not be organized in a strict feedforward hierarchy. Together these results provide novel insights into the neural processes that support dynamic, social vision.

Acknowledgements: This work was funded by NIMH R01MH132826 awarded to L.I.