That Sounds Right: Hearing Objects Helps Us Recognize Them More Quickly

When we imagine a typical scene—for example, a kitchen—we tend to recall not just what a place looks like, but the sounds we associate with it. These audio associations can help us recognize objects more quickly, wrote Jamal R. Williams (University of California, San Diego) and Viola S. Störmer (Dartmouth College) in Psychological Science, such that we may identify a photo of a tea kettle more quickly when we hear the sound of water boiling or an “audio scene” of someone working in their kitchen. 

“Despite the intuitive feeling that our visual experience is coherent and comprehensive, the world is full of ambiguous and indeterminate information,” wrote Williams and Störmer. “Our study reveals the importance of audiovisual interactions to support meaningful perceptual experiences in naturalistic settings.” 

In the first of four experiments, Williams and Störmer tasked 20 undergraduate students with identifying obfuscated images of various objects while listening to congruent or incongruent sounds. In one trial, for example, participants might view an obscured image of a car that faded into view while listening to the congruent sound of car driving by or the incongruent sound of a tea kettle boiling. Participants were instructed to press the space bar as soon as they were confident they could identify the image, at which point they were asked to identify which of two images had appeared in the task: the car or a random unrelated object like a cash register. 

The researchers found that participants were able to identify objects more quickly and just as accurately while listening to the sound made by that object versus an unrelated sound. Similarly, in their second study of 30 students, Williams and Störmer found that participants also identified images of conceptually related but silent objects more quickly while listening to audio scenes commonly associated with that object—for example, identifying a park bench while listening to sounds from a park. 

“This, for the first time, demonstrates that, even when the information is indirectly related to possibly hundreds of objects and not directly related to a single object, people can still leverage this information to facilitate their sensory processing,” Williams said in an interview with APS. 

Related content: Hearing is Believing: Sounds Can Alter Our Visual Perception

In their third experiment, Williams and Störmer put 30 students to the test with a slightly more difficult version of the same task: This time, instead of being asked to choose between two unrelated objects during the identification stage of the task (a car or a cash register), participants were presented with two objects from the same category (the car that appeared while the sound was played or a different car). As in the previous studies, participants were able to identify the correct object more quickly when a congruent sound was played while it faded into view. In their fourth study, Williams and Störmer found that a group of 30 students were able to distinguish between similar objects more quickly when the image aligned with the audio scenes played for them during the task. 

This suggests that sounds can help us make fine-grained discriminations between objects, not just categorical ones, Williams said in an interview. 

“Our perceptual system excels at navigating the complexities of the world by integrating information from different senses,” Williams and Störmer wrote. “Our results suggest that perception integrates contextual information at various levels of processing and can leverage general, gist-like information across sensory modalities to facilitate visual object perception.” 

Participants’ ability to quickly integrate novel audiovisual information suggests that this process occurs relatively automatically, without volitional intent, but more research is needed to determine the extent to which automatic processes contribute to object recognition, the researchers concluded. 

“These audiovisual interactions between naturalistic sounds and visual objects may be driven by a lifetime of experience and the computational architecture that gives rise to them is an automatic feature of a probabilistic mind,” Williams and Störmer wrote. 

Related content we think you’ll enjoy

Feedback on this article? Email [email protected] or login to comment.

Reference 

Williams, J. R., Störmer, V. S. (in press) Cutting through the noise: Auditory scenes and their effects on visual object processing. Psychological Science.