Skip to main content
SpringerLink
Log in

TSG-net: a residual-based informing network for 3D Gaze estimation

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

The appearance-based method for gaze estimation has great potential to work well under various conditions, but current learning-based methods ignore inferior eye images in datasets caused by poor eye region locating, occlusions and abnormal head poses. These images badly impact the accuracy of estimation. In the study, inspired by binocular vision characteristics, we propose two cooperative sub-networks, True Gaze Consistency Network (TG-Net) and Single Gaze Inconsistency Network(SG-Net) which composes TSG-Net. TG-Net and SG-Net cooperate through a residual paradigm and Informing module. More specially, TG-Net explicitly extracts the consistency of paired eyes and weights high-level features from two paired-eye images utilizing SE-Block and an artificial gaze direction, named True Gaze. SG-Net outputs residual momentums based on True Gaze for better estimation of paired eyes. Experimental results on three benchmark datasets demonstrate that the proposed method performs competitively against the existed representative CNN-Based methods. TSG-Net improves on the state-of-the-art by 22% on MPIIGaze and shows more advantages in additional analysis.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Notes

  1. According to [25], the overlapping range of paired eye is approximately 120 degrees, the angle error between the True Gaze constructed in (1) and the Single Gaze is a maximum of 60 degrees.

References

  1. Cazzato D, Leo M, Distante C, Voos H (2020) When i look into your eyes: a survey on computer vision contributions for human gaze estimation and tracking. Sensors 20(13):3739

    Article  Google Scholar 

  2. Chen H, Wang Y, Xu C, Shi B, Xu C, Tian Q, Xu C (2020) Addernet: do we really need multiplications in deep learning? In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 1468–1477

  3. Cheng Y, Lu F, Zhang X (2018) Appearance-based gaze estimation via evaluation-guided asymmetric regression. In: ECCV, pp 100–115

  4. Chennamma H, Yuan X (2013) A survey on eye-gaze tracking techniques. arXiv:13126410

  5. Fischer T, Jin Chang H, Demiris Y (2018) Rt-gene: real-time eye gaze estimation in natural environments. In: ECCV, pp 334–352

  6. Hennessey C, Noureddin B, Lawrence P (2006) A single camera eye-gaze tracking system with free head motion. In: ACM symposium on eye tracking research & applications. ACM, pp 87–94

  7. Howard A, Zhmoginov A, Chen LC, Sandler M, Zhu M (2018) Inverted residuals and linear bottlenecks: mobile networks for classification, detection and segmentation

  8. Hu J, Shen L, Sun G (2018) Squeeze-and-excitation networks. In: CVPR, pp 7132–7141

  9. Kaur H, Manduchi R (2020) Eyegan: gaze-preserving, mask-mediated eye image synthesis. In: The IEEE winter conference on applications of computer vision, pp 310–319

  10. Kim J, Lee JK, Lee KM (2016) Accurate image super-resolution using very deep convolutional networks. In: CVPR

  11. Krafka K, Khosla A, Kellnhofer P, Kannan H, Bhandarkar S, Matusik W, Torralba A (2016) Eye tracking for everyone. In: CVPR, pp 2176–2184

  12. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: NeurIPS, pp 1097–1105

  13. LeCun Y, Bottou L, Bengio Y, Haffner P et al (1998) Gradient-based learning applied to document recognition. Proceedings of the IEEE 86(11):2278–2324

    Article  Google Scholar 

  14. Lu F, Sugano Y, Okabe T, Sato Y (2014) Adaptive linear regression for appearance-based gaze estimation. IEEE TPAMI 36(10):2033–2046

    Article  Google Scholar 

  15. Mora KAF, Monay F, Odobez JM (2014) Eyediap: a database for the development and evaluation of gaze estimation algorithms from rgb and rgb-d cameras. In: ACM symposium on eye tracking research & applications, pp 255–258

  16. Morimoto CH, Mimica MR (2005) Eye gaze tracking techniques for interactive applications. Computer Vision and Image Understanding 98(1):4–24

    Article  Google Scholar 

  17. Park S, Spurr A, Hilliges O (2018) Deep pictorial gaze estimation. In: ECCV, pp 721–738

  18. Park S, Mello SD, Molchanov P, Iqbal U, Hilliges O, Kautz J (2019) Few-shot adaptive gaze estimation. In: Proceedings of the IEEE international conference on computer vision, pp 9368–9377

  19. Ranjan R, De Mello S, Kautz J (2018) Light-weight head pose invariant gaze tracking. In: CVPRW

  20. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv:14091556

  21. Sugano Y, Matsushita Y, Sato Y (2014) Learning-by-synthesis for appearance-based 3d gaze estimation. In: CVPR, pp 1821–1828

  22. Tan KH, Kriegman DJ, Ahuja N (2002) Appearance-based eye gaze estimation. In: WACV. IEEE, pp 191–195

  23. Valenti R, Sebe N, Gevers T (2011) Combining head pose and eye location information for gaze estimation. IEEE TIP 21(2):802–815

    MathSciNet  MATH  Google Scholar 

  24. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I (2017) Attention is all you need. In: NeurIPS, pp 5998–6008

  25. Walsh T (2010) Visual fields: examination and interpretation, vol 3. Oxford University Press

    Book  Google Scholar 

  26. Wilson AC, Roelofs R, Stern M, Srebro N, Recht B (2018) The marginal value of adaptive gradient methods in machine learning, pp 4148–4158

  27. Witzner HD, Qiang J (2010) In the eye of the beholder: a survey of models for eyes and gaze. IEEE TPAMI 32(3):478–500

    Article  Google Scholar 

  28. Yamazoe H, Utsumi A, Yonezawa T, Abe S (2008) Remote gaze estimation with a single camera based on facial-feature tracking without special calibration actions. In: ACM symposium on Eye tracking research & applications. ACM, pp 245–250

  29. Yoo DH, Chung MJ (2005) A novel non-intrusive eye gaze estimation using cross-ratio under large head motion. CVIU 98(1):25–51

    Google Scholar 

  30. Zhang X, Sugano Y, Fritz M, Bulling A (2015) Appearance-based gaze estimation in the wild. In: CVPR, pp 4511–4520

  31. Zhang X, Sugano Y, Bulling A (2017a) Everyday eye contact detection using unsupervised gaze target discovery. In: UIST

  32. Zhang X, Sugano Y, Fritz M, Bulling A (2017) Mpiigaze: real-world dataset and deep appearance-based gaze estimation. IEEE TPAMI 41(1):162–175

    Article  Google Scholar 

  33. Zhu Z, Ji Q (2005) Eye gaze tracking under natural head movements. CVPR, IEEE 1:918–923

    Google Scholar 

  34. Zhu Z, Ji Q, Bennett KP (2006) Nonlinear eye gaze mapping function estimation via support vector regression. ICPR, IEEE 1:1132–1135

    Google Scholar 

Download references

Acknowledgements

This work was supported in part by National Science Fund of China no.61871170; The Basic Research Program of KY2017210A001; Key Laboratory of Brain Machine Collaborative Intelligence of Zhejiang Province.

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, Hangzhou Dianzi University, Hangzhou, China

    Jianjun Li, Jialuo Fei & Shichao Cheng

  2. Key Laboratory of Brain Machine Collaborative Intelligence of Zhejiang Province, Zhejiang, China

    Jianjun Li, Jialuo Fei & Shichao Cheng

  3. Key Laboratory of Data Link Technology of CETC, Xi’An, China

    Zheng Tang & Guobao Hui

Authors
  1. Jianjun Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jialuo Fei
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shichao Cheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zheng Tang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Guobao Hui
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jianjun Li.

Ethics declarations

Conflicts of interest

The authors declare that they have no conflict of interest.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, J., Fei, J., Cheng, S. et al. TSG-net: a residual-based informing network for 3D Gaze estimation. Multimed Tools Appl 81, 3647–3662 (2022). https://doi.org/10.1007/s11042-021-11666-6

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-021-11666-6

Keywords

Search

Navigation