Surface material perception
Friday, May 9, 2008, 3:30 – 5:30 pm Royal Palm 6-8
Organizer: Roland W Fleming (Max Planck Institute for Biological Cybernetics, T�bingen, Germany)
Presenters: Roland W. Fleming (Max Planck Institute for Biological Cybernetics, T�bingen, Germany), Melvyn A. Goodale (The University of Western Ontario), Isamu Motoyoshi (NTT Communication Science Laboratories), Daniel Kersten (University of Minnesota), Laurence T Maloney (New York University), Edward H Adelson (MIT)
Symposium Description
When we look at an everyday object we gain information about its location and shape and also about the material it is made of. The apparent color of an orange signals whether it is ripe; its apparent gloss and mesoscale texture inform us whether it is fresh. All of these judgments are visual judgments about the physical chemistry of surfaces, their material properties. In the past few years, researchers have begun to study the visual assessment of surface material properties, notably gloss and mesoscale texture (�roughness�). Their research has been facilitated by advances in computer graphics, statistical methodology, and experimental methods and also by a growing realization that the visual system is best studied using stimuli that approximate the environment we live in. This symposium concerns recent research in material perception presented by six researchers in computer science, neuroscience and visual perception.
The successive mappings from surface property to retinal image to neural state to material judgments are evidently complex. Coming to understand how each step leads to the next is a fascinating series of challenges that crosses disciplines. An initial challenge is to work out how changes in surface material properties are mirrored in changes in retinal information, to identify the cues that could potentially signal a surface material property such as gloss or roughness.
A second challenge is to determine which cues are actually used by the visual system in assessing material properties. Of particular interest are recent claims that very simple image statistics contain considerable information relevant to assessing surface material properties. A third challenge concerns the neural encoding of surface properties and what we can learn from neuroimaging, a fourth, how variations in one surface material property affect perception of a second.
A final � and fundamental — challenge is to work out how the organism learns to use visual estimates of material properties to guide everyday actions — to decide which oranges to eat and which to avoid.
The symposium is likely to be of interest to a very wide range or researchers in computer vision, visual neuroscience and visual perception, especially perception of color. lightness and texture.
Abstracts
Perception of materials that transmit light
Roland W. Fleming, Max Planck Institute for Biological Cybernetics, T�bingen, Germany
Many materials that we commonly encounter, such as ice, marmalade and wax, transmit some proportion of incident light. Broadly, these can be separated into transparent and translucent materials. Transparent materials (e.g. gemstones, water) are dominated by specular reflection and refraction, leading to a characteristic glistening, pellucid appearance. Translucent materials (e.g. marble, cheese) exhibit sub-surface light scattering, in which light bleeds diffusely through the object creating a distinctive soft or glowing appearance. Importantly, both types of material are poorly approximated by Metelli�s episcotister or other models of thin neutral density filters that have shaped our understanding of transparency to date. I will present various psychophysical and theoretical studies that we have performed using physically based computer simulations of light transport through solid transmissive objects. One important observations is that these materials do not exhibit many image features traditionally thought to be central to transparency perception (e.g. X-junctions). However, they compensate with a host of novel cues, which I will describe. I will discuss the perceptual scales of refractive index and translucency and report systematic failures of constancy across changes in illumination, 3D shape and context. I will discuss conditions under which various low-level image statistics succeed and fail to predict material appearance. I will also discuss the difficulties posed by transmissive materials for the estimation of 3D shape. Under many conditions, human vision appears to use simple image heuristics rather than correctly inverting the physics. I will show how this can be exploited to create illusions of material appearance.
How we see stuff: fMRI and behavioural studies of visual routes to the material properties of objects
Melvyn A. Goodale
Almost all studies of visual object recognition have focused on the geometric structure of objects rather than their material properties (as revealed by surface-based visual cues such as colour and texture). But recognizing the material from which an object is made can assist in its identification � and can also help specify the forces required to pick up that object. In two recent fMRI studies (Cant & Goodale, 2007; Cant et al., submitted), we demonstrated that the processing of object form engages more lateral regions of the ventral stream such as area LO whereas the processing of an object�s surface properties engages more medial regions in the ventral stream, particularly areas in the lingual, fusiform, and parahippocampal cortex. These neuroimaging data are consistent with observations in neurological patients with visual form agnosia (who can still perceive colour and visual texture) and patients with cerebral achromatopsia (who can still perceive form). The former often have lesions in area LO and the latter in more medial ventral-stream areas. In a behavioural study with healthy observers (Cant et al., in press), we showed that participants were able to ignore form while making surface-property classifications, and to ignore surface properties while making form classifications � even though we could demonstrate mutual interference between different form cues. Taken together, these findings suggest that the perception of the material properties depends on medial occipito-temporal areas that are anatomically and functionally distinct from more lateral occipital areas involved in the perception of object shape.
Histogram skewness and glossiness perception
Isamu Motoyoshi
Human can effortlessly judge the glossiness of natural surfaces with complex mesostructure. The visual system may utilize simple statistics of the image to achieve this ability (Motoyoshi, Sharan, Nishida & Adelson, 2007a; Motoyoshi, Nishizawa & Uchikawa, 2007b). We have shown that the perceived glossiness of various surfaces is highly correlated with the skewness (3rd-order moment) of the luminance histogram, and that this image property can be easily computed by the known early visual mechanisms. Our ‘skewness aftereffect’ demonstrated the existence of such skewness detectors and their link to the perceived glossiness. However, simple skewness detectors are not very sensitive to image spatial structures. They might not be able to distinguish a glossy surface from, say, a matte surface covered with white dusts while humans can do. These unsolved issues and questions will be discussed together with our latest psychophysical data. Our glossiness study suggests that the perception of material properties may be generally based on simple ‘pictorial cues’ in the 2D image, rather than on complex inverse optics computations. This hypothesis is supported by the finding that simple image manipulation techniques can dramatically alter the apparent surface qualities including translucency and metallicity (Motoyoshi, Nishida & Adelson, 2005).
Object lightness and shininess
Daniel Kersten
Under everyday viewing conditions, observers can determine material properties at a glance–such as whether an object has light or dark pigmentation, or whether it is shiny or matte. How do we do this? The first problem–lightness perception–has a long history in perception research, yet many puzzles remain, such as the nature of the neural mechanisms for representing and combining contextual information. The second–“shininess”–has a shorter history, and seems to pose even stiffer challenges to our understanding of how vision arrives at determinations of material properties. I will describe results from two approaches to these two problems. For the first problem, I will describe neuroimaging results showing that cortical MR activity in retinotopic areas, including V1, is correlated with context-dependent lightness variations, even when local luminance remains constant. Further, responses to these lightness variations, measured with a dynamic version of the Craik-O’Brien illusion, are resistant to a distracting attentional task. For the second problem, I will describe an analysis of natural constraints that determine human perception of shininess given surface curvature, and given object motion. One set of demonstrations show that apparent shininess is a function of how statistical patterns of natural illumination interact with surface curvature. A second set of demonstrations illustrates how the visual system is sensitive to the way that specularities slide across a surface.
Multiple surface material properties, multiple visual cues
Laurence T. Maloney
Previous research on visual perception of surface material has typically focused on single material properties and single visual cues, with no consideration of possible interactions. I�ll first describe recent work in which we examined how multiple visual cues contribute to visual perception of a single material property, the roughness of 3D rendered surfaces, viewed binocularly. We found that the visual system made substantial use of visual cues that were in fact useless in estimating roughness under the conditions of our experiments. I�ll discuss what the existence of pseudo-cues implies about surface material perception. In a separate experiment, we used a conjoint measurement design to determine how observers represent perceived 3D texture (�bumpiness�) and specularity (�glossiness�) and modeled how each of these two surface material properties affects perception of the other. Observers made judgments of �bumpiness� and �glossiness� of surfaces that varied in both surface texture and specularity. We found that a simple additive model captures visual perception of texture and specularity and their interactions. We quantify how changes in each surface material property affect judgments of the other. Conjoint measurement is potentially a powerful tool for analyzing surface material perception in realistic environments.
What is material perception good for?
Edward H. Adelson
What are the essential ways in which vision helps us interface with the physical world? What is the special role of material perception? One way to approach this question is: 1. Marry a vision scientist. 2. Have children with her. 3. Take videos of your children interacting with the world. 4. Study these videos, taking note of the essential tasks children must master. 5. Make your colleagues watch these videos. For some tasks (e.g., learning the alphabet or recognizing giraffes) material perception is relatively unimportant, but for others (e.g., eating, walking, getting dressed, playing outside, taking a bath) it is critical. The mastery of materials — the way they look, feel, and respond to manipulation — is one of the main tasks of childhood. Why, then, is so little known about material perception, as compared to, say, object recognition? One of the issues seems to be that material perception is embedded in procedural knowledge (knowing how to do), whereas object recognition is embedded in declarative knowledge (knowing how to describe). This suggests that material perception should be approached from multiple modalities including vision, touch, and motor control. It suggests that the brain might contain mechanisms devoted to the joint visual/haptic analysis of stiffness, slipperiness, roughness, and the like. In pursuit of this program, we have recently been showing our home videos to colleagues in other fields.