Mark Lescroart¹; ¹University of Nevada, Reno

Many brain regions respond selectively to particular categories of images, but why and how these response properties arise remains unclear. Here, we investigate whether selectivity for scenes (as measured by BOLD fMRI) is related to dimensions of scene structure in natural visual experience. To address this question, we used data from the Natural Scenes Dataset (NSD), which contains 7T BOLD fMRI responses to thousands of natural images, and the Visual Experience Dataset, which contains hundreds of hours of first-person video of natural behaviors. We developed an image-computable model that formalizes 3D scene structure as the distribution of surface normals and distances in each image or video frame. We used this model to compute 3D structural features in thousands of clips from VED and to every image from NSD. We then estimated selectivity for these features in the brain. We used regularized regression to estimate weights that reflect the importance of each feature in each voxel in the NSD fMRI data. To find dimensions of tuning for 3D structural features, we ran principal components analysis (PCA) on the estimated regression weights for all voxels in scene-selective areas. To find dimensions of 3D structure in visual experience, we ran PCA on the 3D structural feature values in the VED clips. Intriguingly, we found qualitative and quantitative matches among the resulting dimensions: the second dimension of tuning in the brain was reliably correlated with the first experiential dimension, and both of these dimensions reflected a distinction between open, navigable scenes and close-up scenes. Furthermore, voxels in scene-selective areas showed reliable pattens of projections onto these dimensions, with clusters of voxels responding similarly to relatively open or closed scenes. These results suggest that dimensions of visual experience may provide local organizing principles for feature representations in the cortex.