Understanding the interplay between reward and attention, and its effects on visual perception and action

Friday, May 7, 3:30 – 5:30 pm
Royal Ballroom 4-5

Organizers: Vidhya Navalpakkam, Caltech; Leonardo Chelazzi, University of Verona, Medical School, Italy and  Jan Theeuwes, Vrije Universiteit, the Netherlands

Presenters: Leonardo Chelazzi (Department of Neurological and Visual Sciences, University of Verona – Medical School, Italy), Clayton Hickey (Department of Cognitive Psychology, Vrije Universiteit Amsterdam, The Netherlands), Vidhya Navalpakkam (Division of Biology, Caltech, Pasadena), Miguel Eckstein (Department of Psychology, University of California, Santa Barbara), Pieter R. Roelfsema (Dept. Vision & Cognition, Netherlands Institute for Neuroscience, Amsterdam), Jacqueline Gottlieb (Dept. of Neuroscience and Psychiatry, Columbia University, New York)

Symposium Description

Adaptive behavior requires that we deploy attention to behaviorally relevant objects in our  visual environment. The mechanisms of selective visual attention and how it affects visual perception have been a topic of extensive research in the last few decades. In comparison, little is known about the role of reward incentives and how they affect attention and visual perception. Generally, we choose actions that in prior experience have resulted in a rewarding outcome, a principle that has been formalized in reward learning theory. Recent developments in vision research suggest that selective attention may be guided by similar economic principles. This symposium will provide a forum for researchers examining the interplay between reward and attention, and their effects on visual perception and action, to present their work and discuss their developing ideas.

The goal of this symposium will be to help bridge the existing gap between the fields of vision that focused on attention, and decision-making that focused on reward, to better understand the combined roles of reward and attention on visual perception and action. Experts from different faculties including psychology, neuroscience and computational modeling will present novel findings on reward and attention, and outline challenges and future directions, that we hope will lead to a cohesive theory. The first three talks will focus on behavior and modeling. Leo Chelazzi will speak about how attentional deployment may be biased by the reward outcomes of past attentional episodes, such as the gains and losses associated with attending to objects in the past. Vidhya Navalpakkam will speak about how reward information may bias saliency computations to influence overt attention and choice in a visual search task. Miguel Eckstein will show how human eye movement strategies are influenced by reward, and how they compare with an ideal reward searcher. The last three talks will focus on neurophysiological evidence for interactions between reward and attention. Clayton Hickey will provide behavioral and EEG evidence for direct, non-volitional role of reward-related reinforcement learning in human attentional control. Pieter Roelfsema will present neural evidence on the remarkable correspondence between effects of reward and attention on competition between multiple stimuli, as early as in V1, suggesting a unification of theories of reward expectancy and attention. Finally, Jacqueline Gottlieb will present neural evidence from LIP on how reward-expectation shapes attention, and compare it with studies on how reward-expectation shapes decision-making.

We expect the symposium to be relevant to a wide audience with interests in psychology, neuroscience, and modeling of attention, reward, perception or decision-making.

Abstracts

Gains and losses adaptively adjust attentional deployment towards specific objects

Leonardo Chelazzi, Department of Neurological and Visual Sciences, University of Verona – Medical School, Italy; Andrea Perlato, Department of Neurological and Visual Sciences, University of Verona – Medical School, Italy; Chiara Della Libera, Department of Neurological and Visual Sciences, University of Verona – Medical School, Italy

The ability to select and ignore specific objects improves considerably due to prior experience (attentional learning). However, such learning, in order to be adaptive, should depend on the more-or-less favourable outcomes of past attentional episodes. We have systematically explored this possibility by delivering monetary rewards to human observers performing attention-demanding tasks. In all experiments, participants were told that high and low rewards indexed optimal and sub-optimal performance, respectively, though reward amount was entirely pre-determined. Firstly, we demonstrated that rewards adjust the immediate consequences of actively ignoring a distracter, known as negative priming. Specifically, we found that negative priming is only obtained following high rewards, indicating that lingering inhibition is abolished by poor outcomes. Subsequently, we assessed whether rewards can also adjust attentional biases in the distant future. Here, observers were trained with a paradigm where, on each trial, they selected a target while ignoring a distracter, followed by differential reward. Importantly, the probability of a high vs. low reward varied for different objects. Participants were then tested days later in the absence of reward. We found that now the observers’ ability to select and ignore specific objects strongly depended on the probability of high vs. low reward associated to a given object during training and also critically on whether the imbalance had been applied when the object was shown as target or distracter during training. These observations show that an observer’s attentional biases towards specific objects strongly reflect the more-or-less favourable outcomes of past attentional processing of the same objects.

Understanding how reward and saliency affect overt attention and decisions

Vidhya Navalpakkam, Division of Biology, Caltech; Christof Koch, Division of Engineering, Applied Science and Biology, Caltech; Antonio Rangel, Division of Humanities and Social Sciences, Caltech; Pietro Perona, Division of Engineering and Applied Science, Caltech

The ability to rapidly choose among multiple valuable targets embedded in a complex perceptual environment is key to survival in many animal species. Targets may differ both in their reward value as well as in their low-level perceptual properties (e.g., visual saliency). Previous studies investigated separately the impact of either value on decisions, or saliency on attention, thus it is not known how the brain combines these two variables to influence attention and decision-making. In this talk, I will describe how we addressed this question with three experiments in which human subjects attempted to maximize their monetary earnings by rapidly choosing items from a brief display. Each display contained several worthless items (distractors) as well as two targets, whose value and saliency were varied systematically. The resulting behavioral data was compared to the predictions of three computational models which assume that: (1) subjects seek the most valuable item in the display, (2) subjects seek the most easily detectable item (e.g., highest saliency), (3) subjects behave as an ideal Bayesian observer who combines both factors to maximize expected reward within each trial. We find that, regardless of the motor response used to express the choices, decisions are influenced by both value and feature-contrast in a way that is consistent with the ideal Bayesian observer. Thus, individuals are able to engage in optimal reward harvesting while seeking multiple relevant targets amidst clutter. I will describe ongoing studies on whether attention, like decisions, may also be influenced by value and saliency to optimize reward harvesting.

Optimizing eye movements in search for rewards

Miguel Eckstein, Department of Psychology, University of California, Santa Barbara; Wade Schoonveld, Department of Psychology, University of California, Santa Barbara; Sheng Zhang, Department of Psychology, University of California, Santa Barbara

There is a growing literature investigating how rewards influence the planning of saccadic eye movements and the activity of underlying neural mechanisms (for a review see, Trommershauser et al., 2009). Most of these studies reward correct eye movements towards a target at a given location (e.g., Liston and Stone, 2008). Yet, in every day life, rewards are not directly linked to eye movements but rather to a correct perceptual decision and follow-up action.  The role of eye movements is to explore the visual scene and maximize the gathering of information for a subsequent perceptual decision.  In this context, we investigate how varying the rewards across locations assigned to correct perceptual decisions in a search task influences the planning of human eye movements. We extend the ideal Bayesian searcher (Najemnik & Geisler, 2005) by explicitly including reward structure to: 1) determine the (optimal) fixation sequences that maximize total reward gains; 2) predict the theoretical increase in gains from taking into account reward structure in planning eye movements during search.  We show that humans strategize their eye movements to collect more reward.  The pattern of human fixations shares many of the properties with the fixations of the ideal reward searcher. Human increases in total gains from using information about the reward structure are also comparable to the benefits in gains of the ideal searcher.  Finally, we use theoretical simulations to show that the observed discrepancies between the fixations of humans and the ideal reward searcher do not have major impact in the total collected rewards. Together, the results increase our understanding of how rewards influence optimal and human saccade planning in ecologically valid tasks such as visual search.

Incentive salience in human visual attention

Clayton Hickey, Department of Cognitive Psychology, Vrije Universiteit Amsterdam; Leonardo Chelazzi, Department of Neurological and Visual Sciences, University of Verona – Medical School; Jan Theeuwes, Department of Cognitive Psychology, Vrije Universiteit Amsterdam

Reward-related midbrain dopamine guides animal behavior, creating automatic approach towards objects associated with reward and avoidance from objects unlikely to be beneficial. Using measures of behavior and brain electricity we show that the dopamine system implements a similar principle in the deployment of covert attention in humans. Participants attend to an object associated with monetary reward and ignore an object associated with sub-optimal outcome, and do so even when they know this will result in bad task performance. The strength of reward’s impact on attention is predicted by the neural response to reward feedback in anterior cingulate cortex, a brain area known to be a part of the dopamine reinforcement circuit. These results demonstrate a direct, non-volitional role for reinforcement learning in human attentional control.

Reward expectancy biases selective attention in the primary visual cortex

Pieter R. Roelfsema, Dept. Vision & Cognition, Netherlands Institute for Neuroscience, Amsterdam; Chris van der Togt, Dept. Vision & Cognition, Netherlands Institute for Neuroscience, Amsterdam; Cyriel  Pennartz, Dept. Vision & Cognition, Netherlands Institute for Neuroscience, Amsterdam; Liviu Stănişor, Dept. Vision & Cognition, Netherlands Institute for Neuroscience, Amsterdam

Rewards and reward expectations influence neuronal activity in many brain regions as stimuli associated with a higher reward tend to give rise to stronger neuronal responses than stimuli associated with lower rewards. It is difficult to dissociate these reward effects from the effects of attention, as attention also modulates neuronal activity in many of the same structures (Maunsell, 2004). Here we investigated the relation between rewards and attention by recording neuronal activity in the primary visual cortex (area V1), an area usually not believed to play a crucial role in reward processing, in a curve-tracing task with varying rewards. We report a new effect of reward magnitude in area V1 where highly rewarding stimuli cause more neuronal activity than unrewarding stimuli, but only if there are multiple stimuli in the display. Our results demonstrate a remarkable correspondence between reward and attention effects. First, rewards bias the competition between simultaneously presented stimuli as is also true for selective attention. Second, the latency of the reward effect is similar to the latency of attentional modulation (Roelfsema, 2006). Third, neurons modulated by rewards are also modulated by attention. These results inspire a unification of theories about reward expectation and selective attention.

How reward shapes attention and the search for information

Jacqueline Gottlieb, Dept. of Neuroscience and Psychiatry, Columbia University; Christopher Peck, Dept. of Neuroscience and Psychiatry, Columbia University; Dave Jangraw, Dept. of Neuroscience and Psychiatry, Columbia University

In the neurophysiological literature with non-human primates, much effort has been devoted to understanding how reward expectation shapes decision making, that is, the selection of a specific course of action. On the other hand, we know nearly nothing about how reward shapes attention, the selection of a source of information. And yet, understanding how organisms value information is critical for predicting how they will allocate attention in a particular task. In addition, it is critical for understanding active learning and exploration, behaviors that are fundamentally driven by the need to discover new information that may prove valuable for future tasks.
To begin addressing this question we examined how neurons located in the parietal cortex, which encode the momentary locus of attention, are influenced by the reward valence of visual stimuli. We found that reward predictors bias attention in valence-specific manner. Cues predicting reward produced a sustained excitatory bias and attracted attention toward their location. Cues predicting no reward produced a sustained inhibitory bias and repulsed attention from their location. These biases were persisted and even grew with training, even though they came in conflict with the operant requirement of the task, thus lowering the animal’s task performance.  This pattern diverges markedly from the assumption of reinforcement learning (that training improves performance and overcomes maladaptive biases, and suggests that the effects of reward on attention may differ markedly from the effects on decision making. I will discuss these findings and their implications for reward and reward-based learning in cortical systems of attention.