Abstract
Lipreading, also known as visual speech recognition (VSR), refers to recognizing people’s speech content only through the sequence of lip movements. Benefit from the development of deep learning, research on lipreading has made great progress in recent years. At present, 3D CNN and 2D CNN are mixed to extract spatio-temporal features in the front-end network of most lipreading models. In this paper, to make 2D convolution have the ability of 3D convolution in the feature extraction stage without increasing model calculation, we combined TSM with several channel attention modules and conduct ablation studies to validate their effectiveness. Then, we inserted the TSM-SE module that proposed into the front-end network so that 2D convolution has the ability to extract fine-grained spatio-temporal features. On the other hand, we solved the potential impact of the time shift module on the dependence between channels. We verified the effectiveness of the proposed method on LRW and LRW-1000 which are challenging large-scale word-level lipreading datasets and reached a new state-of-the-art.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Zhou, Z., Zhao, G., Hong, X., et al.: A review of recent advances in visual speech decoding. Image Vis. Comput. 32(9), 590–605 (2014)
Zhou, Z., Zhao, G., Pietikäinen, M.: Towards a practical lipreading system. In: CVPR 2011, 137–144. IEEE (2011)
Zhou, Z., Hong, X., Zhao, G., et al.: A compact representation of visual speech data using latent variables. IEEE Trans. Pattern Anal. Mach. Intell. 36(1), 1 (2013)
Pei, Y., Kim, T.K., Zha, H.: Unsupervised random forest manifold alignment for lipreading. In: Proceedings of the IEEE International Conference on Computer Vision, 129–136 (2013)
Potamianos, G., Neti, C., Iyengar, G., et al.: A cascade visual front end for speaker independent automatic speechreading. Int. J. Speech Technol. 4(3), 193–208 (2001)
Chung, J.S., Zisserman, A.: Lip reading in the wild. In: Lai, S.-H., Lepetit, V., Nishino, K., Sato, Y. (eds.) ACCV 2016. LNCS, vol. 10112, pp. 87–103. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-54184-6_6
Yang, S., Zhang, Y., Feng, D., et al.: LRW-1000: a naturally-distributed large-scale benchmark for lip reading in the wild. arXiv e-prints arXiv:1810.06990 (2018)
Stafylakis, T., Tzimiropoulos, G.: Combining residual networks with LSTMs for lipreading. arXiv preprint arXiv:1703.04105 (2017)
Weng, X., Kitani, K.: Learning spatio-temporal features with two-stream deep 3D CNNS for lipreading. arXiv preprint arXiv:1905.02540 (2019)
Feng, D., Yang, S., Shan, S., et al.: Learn an effective lip reading model without pains. arXiv preprint arXiv:2011.07557 (2020)
Lin, J., Gan, C., Han, S.: Tsm: Temporal shift module for efficient video understanding. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7083–7093 (2019)
Martinez, B., Ma, P., Petridis, S., et al.: Lipreading using temporal convolutional networks. In: ICASSP 2020–2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 6319–6323. IEEE (2020)
Zhang, Y., Yang, S., Xiao, J., et al.: Can we read speech beyond the lips? rethinking roi selection for deep visual speech recognition. arXiv preprint arXiv:2003.03206 (2020)
Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7132–7141 (2018)
Stafylakis, T., Khan, M.H., Tzimiropoulos, G.: Pushing the boundaries of audiovisual word recognition using residual networks and LSTMs. Comput. Vis. Image Underst. 176, 22–32 (2018)
Woo, S., Park, J., Lee, J.-Y., Kweon, I.S.: Cbam: convolutional block attention module. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11211, pp. 3–19. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01234-2_1
Hou, Q., Zhou, D., Feng, J.: Coordinate attention for efficient mobile network design. arXiv preprint arXiv:2103.02907 (2021)
Wang, C.: Multi-grained spatio-temporal modeling for lip-reading. arXiv preprint arXiv:1908.1161 (2019)
Acknowledgments
This work was supported by the National Natural Science Foundation of China under Grant (No. 62061045, 61862061, 61563052). The Funds for Creative Groups of Higher Educational Research Plan in Xinjiang Uyghur Autonomous, China under Grant (No. XJEDU2017T002), and 2018th Scientific Research Initiate Program of Doctors of Xinjiang University under Grant No. 24470.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Li, H., Mamut, M., Yadikar, N., Zhu, Y., Ubul, K. (2021). Channel Enhanced Temporal-Shift Module for Efficient Lipreading. In: Feng, J., Zhang, J., Liu, M., Fang, Y. (eds) Biometric Recognition. CCBR 2021. Lecture Notes in Computer Science(), vol 12878. Springer, Cham. https://doi.org/10.1007/978-3-030-86608-2_52
Download citation
DOI: https://doi.org/10.1007/978-3-030-86608-2_52
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-86607-5
Online ISBN: 978-3-030-86608-2
eBook Packages: Computer ScienceComputer Science (R0)