25 Years of Seeing ‘Stuff’—Advances and Challenges in Material Perception
Symposium: Friday, May 16, 2025, 10:30 am – 12:30 pm, Talk Room 2Schedule of Events | Search Abstracts | Symposia | Talk Sessions | Poster Sessions
Organizers: Vivian C. Paulun1, Roland Fleming2; 1Massachusetts Institute of Technology, 2Justus Liebig University Giessen and Center for Mind, Brain and Behavior (CMBB), Universities of Marburg, Giessen and TU Darmstadt
Presenters: Edward Adelson, Roland Fleming, Vivian Paulun, Wenyan Bi, Bei Xiao, Maarten Wijntjes
Vision is more than just “know[ing] what is where by looking” (Marr, 1982). A central aspect of visual perception is working out the physical properties of surfaces and objects. To interact effectively with the world, we need to recognize whether something is made of wood, marble, chocolate or silk and whether it is soft, slippery, brittle or elastic. From delicate porcelain plates to sticky honey, from polished oak wood to fluffy merino wool—our world is filled with an incredible diversity of materials. Materials are ubiquitous in our world so material perception is deeply intertwined with all levels of visual processing from color and texture to shape and motion, from scenes and intuitive physics to visually-guided actions. In the past 25 years we have seen substantial progress in our understanding of material perception and shed light on the underlying cues and computations. Novel insights and experimental results have fueled theoretical debates that go far beyond material perception, spanning the role of image statistics and heuristics as well as analysis-by-synthesis and mental physics simulation in visual perception. The aim of this symposium is to review the most exciting advances in the field, identify common themes as well as discrepancies and highlight the most pressing challenges in material perception research today. The symposium will include six talks from speakers at various career stages, with complementary backgrounds and scientific approaches. The presentations will showcase work investigating material perception using a breadth of methodologies spanning psychophysics, neuroimaging, computational modeling and AI, computer graphics and art history. Edward Adelson will kick off the symposium by setting material perception in context, introducing fundamental questions about how we perceive the material world and drawing important parallels between human and computer vision. Roland Fleming will provide a broad overview of new directions in material perception research enabled by technological developments and theoretical advances. Next, Vivian Paulun will focus in on the perception of dynamic materials, by asking how the visual system draws rich inferences about mechanical properties like viscosity, elasticity and from the way materials flow, bounce or deform. Wenyan Bi will describe how generative AI models can be used to investigate the cues underlying judgments about mechanical properties like cloth stiffness. Bei Xiao will discuss the (mis-)alignment of visual and semantic representations in material perception. Finally, Maarten Wijntjes will showcase what we can learn about material perception from studying the art of material depiction. Each talk will be 20 minutes long, including 3 minutes for Q&A. There is little overlap of this symposium with the regular VSS program or any symposium in the past 15 years. Unlike other sessions at the conference, this symposium is dedicated solely to materials, and it will allow speakers to zoom out to the bigger picture questions and theoretical frameworks. We expect this topic to be of interest to VSS members from a multitude of subfields, such as the perception of color, shape, motion, objects, events, relations, and scenes, intuitive physics, perception and action, and visual neuroscience.
Talk 1
On Seeing Stuff
Edward Adelson1; 1Massachusetts Institute of Technology
It is gratifying to know that this paper has had a lasting impact. It contains no experimental results and no theorems, and therefore was not published in a peer reviewed journal. The paper is an invitation to share in two key activities, which are looking and thinking. The paper expresses the hope that it might be possible to understand material perception by starting with a small number of simple concepts and measurements. Is this still viable? The success of deep learning suggests that visual perception involves representations and computations of extraordinarily high dimension, and that we may be fooling ourselves when we imagine that we can describe perception in terms of simple principles. Nonetheless, we have no choice but to press forward and extract as much understanding as we can. We have to keep looking and thinking and experimenting and introspecting.
Talk 2
Progress on Seeing Stuff
Roland Fleming1; 1Justus Liebig University Giessen and Center for Mind, Brain and Behavior (CMBB), Universities of Marburg, Giessen and TU Darmstadt
Over the last 25 years we have seen massive progress in our understanding of material perception. When we started out, computer graphics was just about reaching physical realism and the range of material properties we could feasibly investigate was extremely limited. Our conceptual frameworks were legacies of lightness and colour constancy research. In the meantime, the scope of scientific questions our field is addressing has exploded to encompass topics spanning from the role of low- and mid-level cues in the estimation of diverse optical and mechanical properties, to the relationship of material perception to high-level concepts, learning, language, action and multimodal representations. Nowadays, material perception research even poses challenges for widely-held tenets of vision science, like inverse optics. I will summarise the massive changes we have witnessed in material perception research and highlight the impact of recent and emerging developments like crowdsourcing, interactive computer graphics, motion capture, deep learning and generative AI. In the process I will describe some major theoretical advances in our understanding of material perception and the consequences these have for how we should think about vision science more broadly.
Talk 3
Bouncing, Bending, Bubbling: Seeing Dynamic Stuff
Vivian Paulun1; 1Massachusetts Institute of Technology
We can draw rich inferences from observing how materials flow, deform, and interact, from how gooey honey slowly drips off a spoon, powdery sand trickles between someone’s fingers or how jelly bouncily wobbles back and forth on a plate. Visually inferring materials and their properties from dynamic interactions yields powerful perceptual effects despite unique computational challenges. Computationally, the task is ill-posed: The observable behavior of any material depends on a multitude of factors, its physical properties and the forces applied. The motion of a liquid in a bowl depends on its viscosity as well as the movement of the whisk. How a box sinks into a couch cushion depends on the properties of both, the cushion and the box. Because the observed behavior can vary infinitely, this task cannot be solved using simple pattern recognition—an immense challenge to state-of-the-art AI. Yet, humans robustly use dynamic information to visually determine the kind of ‘Stuff’ in front of them, e.g., liquid, solid, or jelly, and estimate its mechanical properties, e.g., elasticity or viscosity. When brought into conflict with texture cues, dynamic material information clearly dominates our percept. Despite its inherent ambiguity, the brain can infer materials from dynamics with minimal information, e.g. indirectly from how other objects move in response to an interaction. I will characterize such perceptual effects and draw conclusions about the cues and computations underlying dynamic material perception. Furthermore, I will show evidence for distinct neural representation of dynamic ‘Stuff’ in both the ventral and dorsal visual pathways.
Talk 4
Intuitive physics underlies material perception: Computational, psychophysical, and neural evidence
Wenyan Bi1, Ilker Yildirim1; 1Department of Psychology, Yale University
From the wrinkles and folds a soft object makes, how do we see, not just these changes in geometry, but also physical material properties, including their mass and stiffness? A common view states that the brain relies on high-level image and motion statistics that differentiate the degrees of these physical properties (e.g., discriminating a soft cloth from a stiff cloth). I’ll counter this view with an alternative framework, in which the brain inverts an internalized, physics-based generative model to arrive at the scene-level causes underlying visual inputs. In this account, material perception is cast as posterior inference of physical properties under a generative model of “soft body dynamics” (a game-engine style description of how non-rigid materials move and react to external forces) and simple graphics to project these scenes to sensory measurements. I’ll present a computational model that implements this framework and evaluate it in psychophysical and neural experiments. First, I’ll show that the physics-based model explains both the successes and failures in material perception across multiple match-to-sample tasks. It outperforms a performant DNN model that solves material perception by acquiring high-level image and motion statistics for discriminating the physical properties. Next, I'll evaluate these models in a new fMRI experiment. I'll present evidence for a double dissociation, where a set of higher-order frontoparietal regions aligns with the physics-based model and an occipitotemporal region aligns with DNN. Together, these findings suggest that visual material perception transcends image statistics to also involve intuitive physics—formalized as probabilistic simulations of soft-body dynamics.
Talk 5
Probing visual and semantic representations in material perception using psychophysics and deep learning
Bei Xiao1, Chenxi Liao1, Masataka Sawayama2; 1Department of Computer Science, American University, 2University of Tokyo
The look and feel of materials are an integral part of our daily experience. Seeing and understanding materials allows us to interact with materials in diverse tasks. In the past decades, significant progress has been made on visual inference of material properties. Relatively little is known about the semantic representation of materials. Much of this knowledge is represented symbolically in language, which allows us to articulate material qualities and appearances. The immense diversity, complexity, and versatility of materials present challenges in verbalization. In this talk, I will discuss our recent progress in material perception using psychophysics and deep learning methods. First, I will introduce a deep generative framework to synthesize realistic and diverse material appearances and learn an interpretable latent space that can capture perceptually relevant visual information. I will discuss how the learned latent space can be used to probe human perception of translucency. In the second part, I will discuss how we extend our framework to elucidate the relationship between language and vision in material perception in both familiar and unfamiliar ambiguous materials. By comparing the representations derived across modalities, our results reveal the alignment and misalignment of vision-language connection and underscore the importance of leveraging the vision and semantic features to reveal behavioral relevant features in material perception. I will close by presenting our latest results on modeling the relationship between material categorization and material discrimination, and the future directions of integrating our approach to discover neural correlates of material perception and related cognitive tasks.
Talk 6
25 Years of Seeing Stuff? 2500 Years of Depicting Stuff!
Maarten Wijntjes1; 1Delft University of Technology
About 2500 years ago, “[Parrhasius] entered into a pictorial contest with Zeuxis, who represented some grapes, painted so naturally that the birds flew towards the spot where the picture was exhibited. Parrhasius, on the other hand, exhibited a curtain, drawn with such singular truthfulness, that Zeuxis, […] haughtily demanded that the curtain should be drawn aside to let the picture be seen.” With an important role for optical (grapes) and mechanical (curtain) material properties, Pliny the Elder accounts the birth of material depiction. The distinction between optical and mechanical properties is not the only parallel to be drawn with “On seeing stuff”. Adelson (2001) argues that to understand material perception, one needs to understand how images are made. In other words, to understand vision, one needs to understand depiction. Whereas in vision the dichotomy between stuff and things dominates, in depiction the more relevant dichotomy is that between stuff and space. These two ‘formal elements’ of depiction have interesting histories related the invention of new media (stuff) and new projection techniques (space). This contribution will discuss de vision and depiction of material properties through time and medium. I will discuss the handling of highlights, demonstrate a ‘distant viewing’ approach to the history of material depiction and compare the medium of engravings with oil paint.