Amy Nguyen¹, Vivian Gao, Sage Aronson, Michelle Greene; ¹Barnard College

Scene understanding is considered rapid and effortless, yet its processing mechanisms are currently unknown. Scenes differ in content, and it may be the case that some scenes are understood more efficiently than others. However, no framework exists to predict why some images may be processed more efficiently. We adopt an information theoretic approach, measuring the relative amounts of visual and semantic information in photographs and assessing the processing efficiency via time-resolved EEG decoding. We amassed a novel dataset of over 67,000 RAW photographs from over 260 scene categories. RAW images were compressed to PNG, and visual information scores were computed as the difference in resulting file sizes. We reasoned that more compressible images had less relative visual information. To assess semantic information, participants provided descriptions of each scene. We calculated five metrics from natural language processing and, using principal component analysis, extracted the first component as a semantic information measure. Unexpectedly, visual and semantic information scores were not strongly correlated (r=0.05). For our ERP experiments, we selected the 20 images with the highest and lowest scores in visual and semantic information. Each of the 40 images was presented for 500 ms and repeated 30 times while participants performed an orthogonal task to maintain attention. We used time-resolved, whole-brain decoding with a linear support vector machine whereby image identity was predicted with five-fold cross-validation. For both experiments, decoding accuracy was higher for low information images. Moreover, the latency of maximum decoding accuracy was earlier for low information images. These results indicate that scene processing is delayed for high-information images, and this is true for both visual and semantic information. This suggests that the visual system has information capacity limitations for both visual and semantic information.

Acknowledgements: Supported by CAREER 2240815 to MRG