Manasi Malik¹ (), Shari Liu¹, Tianmin Shu¹, Leyla Isik¹; ¹Johns Hopkins University

The ability to interpret social information from visual scenes is critical to human cognition, yet the neural computations underlying this ability remain poorly characterized. We present a novel fMRI dataset using procedurally generated stimuli to investigate these computations. We collected fMRI data from thirty participants as they watched animated videos from the PHASE dataset, depicting two agents interacting in ways that resemble real-life social behaviors. Participants rated the agents’ relationships as "friendly," "neutral," or "adversarial." Participants also completed standard localizer tasks to identify brain regions associated with theory-of-mind, social interaction perception, and physical reasoning. Additionally, we collected individual social ratings for each video, Autism Spectrum Questionnaire, and demographic data. Our dataset offers two significant advantages. First, it provides a unique opportunity to compare neural data with computational models. Prior work has identified two theoretically distinct models that uniquely explain human social scene understanding - a bottom-up graph neural network based on visual information and a generative inverse planning model grounded in mental state inference. However, generative inverse planning models have rarely been compared to neural representations, largely because existing datasets lack stimuli designed with physical simulators. Our dataset addresses this limitation by using stimuli derived from a physical simulator, thus allowing for generative models to be built to work with them. This pairing enables unique comparisons between neural representations and both neural network-based and generative inverse planning models of social scene recognition. Second, the procedural generation of stimuli provides ground truth information about visual features (e.g., agent size, trajectories) and higher-level physical and social knowledge (e.g., agent goals, strength). This allows for systematic exploration of the role these features play in social scene understanding in the brain. This richly annotated fMRI dataset collected using procedurally designed stimuli, will advance our understanding of the neural basis of human social scene understanding.

Acknowledgements: This work was funded by NIMH R01MH132826 awarded to L.I.