Minds, Technology, and Society Seminar Series:
Monday, March 2, 2020
Professor, Research Scientist, Veterans Affairs
Loma Linda University, School of Medicine
“Temporal Dynamics of Audiovisual Speech Perception”
Abstract: In this talk, I will discuss several experiments investigating the contributions of two types of visual speech cues to audiovisual speech perception: configural cues (relative arrangement of visible articulators) and kinematic cues (motion of visible articulators). One experiment will focus on audiovisual integration of sublexical speech stimuli using the McGurk effect. A second set of experiments will focus on the audiovisual advantage for speech recognition in noise at the sentence level. In describing the contribution of kinematic cues, particular attention will be paid to the temporal correlation between the inter-lip distance function (roughly a measure of lip and jaw motion over time) and the acoustic speech envelope. In brief, both configural and kinematic cues influence audiovisual speech perception at the sublexical and sentence levels, but their relative influence depends on several factors including the temporal context and signal-to-noise ratio. For the speech-in-noise experiments, the effect on behavioral performance of (high vs. low) temporal correlation between the inter-lip distance function and the acoustic speech envelope is overpredicted by a model that estimates intelligibility as the ability to reconstruct the original speech envelope from a weighted combination of these signals. If time permits, I will conclude the talk by presenting data from stroke patients showing how audiovisual-speech timing at the word level can be disrupted by damage to cortical speech networks.
Speaker Bio: Jonathan Venezia is a Research Scientist at the VA Loma Linda Healthcare System and a Research Assistant Professor at the Loma Linda University School of Medicine in the Department of Otolaryngology. He completed his Ph.D. in Cognitive Neuroscience at the University of California, Irvine in 2014. His research involves a combination of psychophysics and neuroimaging to characterize high-level representations of auditory and audiovisual speech with a focus on reverse correlation techniques (e.g., receptive field mapping). At the VA since 2016, Dr. Venezia applies these techniques to study how representations of speech and other multisensory signals change as a result of hearing loss, stroke, and traumatic brain injury. His work is funded by the Department of Veterans Affairs, Department of Defense, and American Speech-Language-Hearing Foundation.