Steven J. Luck¹, John E. Kiat¹; ¹University of California, Davis

Visual working memory is essential for navigating and interacting with the visual environment, so it is important to understand how complex natural scenes are represented in working memory. Natural scenes are characterized by complex contours, continuously varying feature gradients, and spatial relationships, but most recent research on visual working memory has focused on simplified arrays of discrete artificial objects, favoring experimental control over ecological relevance. Similarly, quantitative models of visual working memory have focused on pre-parsed discrete objects that vary along a small number of simple dimensions such as color and orientation, and it is not clear how these models could be updated to represent complex, photograph-like scenes. To address this limitation of current models, we have developed a “population vector model” of visual working memory that represents a complex scene as a noisy vector of activation values across a population of neurons in visual cortex. It uses CORnet—a convolutional neural network designed to mimic the properties of the ventral object recognition pathway—to estimate the population representation of a given scene at different levels of abstraction, from V1 through inferotemporal cortex (IT). We tested this model in neurotypical young adults using behavioral change detection experiments and using a modified 1-back design to collect EEG data. We found that the model predicted behavioral accuracy and response times extremely well, accounting for over 75% of the variance across scenes. The model also predicted differences in the scalp patterns of EEG activity across scenes during the delay period. In addition, we found that the abstract IT-like representations of the model accounted for much more unique variance in behavior and brain activity than the spatially detailed V1-like representations. Although far from complete, this model provides a path toward understanding how complex naturalistic scenes are stored in working memory.

Acknowledgements: This work was made possible by NEI grant R01EY033329