Frame rate as a QoS parameter and its influence on speech perception

Abstract.

The preservation of QoS for multimedia traffic through a data network is a difficult problem. We focus our attention on video frame rate and study its influence on speech perception.

When sound and picture are discrepant (e.g., acoustic ‘ba’ combined with visual ‘ga’), subjects perceive a different sound (such as ‘da’). This phenomenon is known as the McGurk effect. In this paper, the influence of degraded video frame rate on speech perception is studied.

It is shown that, when frame rate decreases, correct hearing is improved for discrepant stimuli and is degraded for congruent (voice and picture are the same) stimuli. Furthermore, we studied the case where lip closure was always captured by the synchronization of sampling time and lip position. In this case, frame rate has little effect on mishearing for congruent stimuli. For discrepant stimuli, mishearing is decreased with degraded frame rate. These results indicate that the stiff motion of lips resulting from low frame rate cannot give enough labial information for speech perception. In addition, the effect of delaying the picture to correct for low frame rate was studied. The results, however, were not as definitive as expected, because of compound effects related to the synchronization of sound and picture. Finally, we inspected the still pictures of normal Japanese speech and determined a lower limit of frame rate from the view point of assisting hearing.