Jefferson Ortega¹, David Whitney¹; ¹University of California, Berkeley

Context shapes everything in vision. To recognize an object’s lightness, shape, color, or motion, we rely on background information. This reliance is even more critical when interpreting emotions. Facial expressions are inherently ambiguous, often requiring contextual cues for accurate understanding (Barrett et al. 2011; Chen and Whitney 2019). But does using context involve a distinct visual skill? Do some individuals intuitively seek and use contextual clues more effectively than others? We explored this question through an eye-tracking study, examining how individuals use background context to recognize emotions in real time. Participants continuously reported the valence and arousal of an actor in a film under two conditions: a context-only condition (where the actor was blurred out, leaving only the background scene visible) and a ground-truth condition (where the entire scene was visible, including the actor). When observers are prevented from seeing the actor’s face (context-only condition), they need to search the background scene context for valuable information. We find that some observers are better at this than others and these skilled observers reveal where, in the background scene context, the informative information is found that facilitates emotion recognition. Surprisingly, we found that the observers who were best able to continuously report emotion in the ground truth condition (where everything was visible) were those who gazed at the same (non-face) locations in the background as the most accurate observers in the context-only condition. The results reveal that specific spatiotemporal moments in the background scene context are highly informative for emotion recognition, but not all observers understand or are able to use this information. This suggests that certain individuals possess a unique, implicit visual skill: the ability to recognize and extract critical contextual information for emotion recognition. Our results highlight significant individual differences in how we integrate context to decode emotions.

Acknowledgements: Supported in part by the National Institute of Health (grant no. R01CA236793) to D.W and (grant no. 1F99NS141343) to J.O.