Jenny Bosten¹, Chris Racey¹, Kendrick Kay², Ian Pennock¹; ¹University of Sussex, ²University of Minnesota

Color is an integral part of visual perception, yet it is often omitted from encoding models. Similarly, the retinal image deviates substantially from the RGB images typically used to train encoding models, yet the effect of retinal optics on the stimuli is often ignored. We built a Gabor wavelet pyramid-based encoding model that includes biologically plausible color and retinal optics generated using ISETBio (Cottaris et al., 2019, Journal of Vision). We fit this model using fractional ridge regression (Rokem & Kay, 2020, GigaScience), and predicted occipital BOLD responses to the color-calibrated natural scene stimuli in the 7T fMRI Natural Scenes Dataset (Allen et al., 2022, Nature Neuroscience). We created a luminance (L+M)-only model without retinal optics as a baseline model. For V1 voxels, adding color information from the ‘cardinal’ retinogeniculate color channels (S/(L+M) and L/(L+M)) increased average prediction accuracy (Pearson's r) by about 20%. Including optical factors using ISETBio to estimate retinal images increased average prediction accuracy a further 5%. Our results show that incorporating color and the retinal image into Gabor wavelet-based encoding models improves prediction of BOLD responses to natural scene stimuli. We conclude that these well-established biologically relevant features of low-level visual signals are important, and can be successfully incorporated in encoding models.

Acknowledgements: The work was funded by ERC grant 949242 COLOURCODE to JMB