Skip to main content
SpringerLink
Log in

Spatialgaze: towards spatial gaze tracking for extended reality

  • Regular Paper
  • Published:
CCF Transactions on Pervasive Computing and Interaction Aims and scope Submit manuscript

Abstract

With the rise of Metaverse, Extended Reality (XR) and its enabling techniques have received increasing attention. Spatial gaze tracking is one of these techniques that enables capturing a user’s visual attention, so as to support immersive 3D experience and interaction. Due to the limitations in the employed visual models and algorithms, the existing proposals of gaze tracking can only provide planar gaze tracking or approximate spatial gaze tracking. A critical problem behind is that so far there isn’t an accurate and efficient approach for XR devices to sense the spatial gaze, that are modeled based on the vergence of the binocular visual axes. To address this problem, this paper proposes SpatialGaze, a spatial gaze tracking approach based on the realistic parallax-contingent visual model. SpatialGaze contains a tailored design for XR devices, which is accurate, lightweight, and practical for use. Our implementation and evaluation demonstrate that SpatialGaze achieves an average error of 0.52\(^\circ\) in direction tracking and an average error of 75.52 cm in depth perception. Compared to the baseline approach, SpatialGaze reduces the direction and depth errors by up to 52.62 and 75.15%, respectively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19

Similar content being viewed by others

References

  • Ahuja, K., Shah, D., Pareddy, S., Xhakaj, F., Ogan, A., Agarwal, Y., Harrison, C.: Classroom digital twins with instrumentation-free gaze tracking. In: Proceedings of the 2021 ACM Conference on Human Factors in Computing Systems (CHI). ACM, Yokohama, Japan (2021)

  • Bekerman, I., Gottlieb, P., Vaiman, M.: Variations in eyeball diameters of the healthy adults. J. Ophthalmol. 2014, 1–5 (2014)

    Article  Google Scholar 

  • Bell, R.J.T.: An elementary treatise on coordinate geometry of three dimensions. Macmillan (1923)

  • Bermejo, C., Chatzopoulos, D., Hui, P.: EyeShopper: estimating shoppers’ gaze using CCTV cameras. In: Proceedings of the 28th ACM International Conference on Multimedia (MM). ACM, Seattle, WA, USA (2020)

  • Bingham, G.P.: Optical flow from eye movement with head immobilized: “Ocular occlusion’’ beyond the nose. Vis. Res. 33(5–6), 777–789 (1993)

    Article  Google Scholar 

  • Buswell, G.T.: The relationship between eye-perception and voice-response in reading. J. Educ. Psychol. 12(4), 217–227 (1921)

    Article  Google Scholar 

  • Cao, J., Lin, C., Liu, Y., Li, Z.: Gaze tracking on any surface with your phone. In: Proceedings of the 20th ACM Conference on Embedded Network Sensor Systems (SenSys). ACM, Boston, MA, USA (2022)

  • Carter, B.T., Luke, S.G.: Best practices in eye tracking research. Int. J. Psychophysiol. 155, 49–62 (2020)

    Article  Google Scholar 

  • Chen, Y.-W., Kubo, K.: A robust eye detection and tracking technique using Gabor filters. In: Proceedings of the 3rd IEEE International Conference on Intelligent Information Hiding and Multimedia Signal Processing (IIH-MSP). IEEE, Kaohsiung, Taiwan (2007)

  • Chen, J., Tong, Y., Gray, W., Ji, Q.: A robust 3D eye gaze tracking system using noise reduction. In: Proceedings of the 2008 ACM Symposium on Eye Tracking Research and Applications (ETRA). ACM, Savannah, Georgia, USA (2008)

  • Chi, J., Liu, J., Wang, F., Chi, Y., Hou, Z.-G.: 3-d gaze-estimation method using a multi-camera-multi-light-source system. IEEE Trans. Instrum. Meas. 69(12), 9695–9708 (2020)

    Article  Google Scholar 

  • Creed, C., Frutos-Pascual, M., Williams, I.: Multimodal gaze interaction for creative design. In: Proceedings of the 2020 ACM Conference on Human Factors in Computing Systems (CHI). ACM, Honolulu, HI, USA (2020)

  • Deng, H., Zhu, W.: Monocular free-head 3D gaze tracking with deep learning and geometry constraints. In: Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV). IEEE, Venice, Italy (2017)

  • Dunn, D.: Required accuracy of gaze tracking for varifocal displays. In: Proceedings of the 26th IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, Osaka, Japan (2019)

  • Eckert, M., Volmerg, J.S., Friedrich, C.M.: Augmented reality in medicine: systematic and bibliographic review. JMIR Mhealth Uhealth 7(4), 10967 (2019)

    Article  Google Scholar 

  • Fuhl, W., Kasneci, G., Kasneci, E.: TEyeD: Over 20 million real-world eye images with pupil, eyelid, and iris 2D and 3D segmentations, 2D and 3D landmarks, 3D eyeball, gaze vector, and eye movement types. In: Proceedings of the 2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). IEEE, Bari, Italy (2021)

  • Gattullo, M., Scurati, G.W., Fiorentino, M., Uva, A.E., Ferrise, F., Bordegoni, M.: Towards augmented reality manuals for industry 4.0: a methodology. Robot. Comput. Integr. Manuf. 56, 276–286 (2019)

    Article  Google Scholar 

  • Guestrin, E.D., Eizenman, M.: General theory of remote gaze estimation using the pupil center and corneal reflections. IEEE Trans. Biomed. Eng. 53(6), 1124–1133 (2006)

    Article  Google Scholar 

  • He, D., Benhabib, B.: Solving the orientation-duality problem for a circular feature in motion. IEEE Trans. Syst. Man Cybern. 28(4), 506–515 (1998)

    Article  Google Scholar 

  • Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., Adam, H.: Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv (2017)

  • Huey, E.B.: Psychology and Pedagogy of Reading. MIT Press, Cambridge (1968)

    Google Scholar 

  • Huynh, S., Balan, R.K., Ko, J.: iMon: appearance-based gaze tracking system on mobile devices. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 5(4), 1–26 (2021)

    Article  Google Scholar 

  • Itoh, Y., Langlotz, T., Sutton, J., Plopski, A.: Towards indistinguishable augmented reality. ACM Comput. Surv. 54(6), 1–36 (2021)

    Article  Google Scholar 

  • Jianfeng, L., Shigang, L.: Eye-model-based gaze estimation by RGB-D camera. In: Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 592–596. IEEE, Columbus, OH, USA (2014)

  • Kacete, A., Séguier, R., Collobert, M., Royan, J.: Head pose free 3D gaze estimation using RGB-D camera. In: Proceedings of the 8th International Conference on Graphic and Image Processing (ICGIP), vol. 10225, pp. 357–363. SPIE, Tokyo, Japan (2016)

  • Kocejko, T., Bujnowski, A., Wtorek, J.: Eye mouse for disabled. In: Proceedings of the 2008 IEEE Conference on Human System Interactions (HSI). IEEE, Krakow, Poland (2008)

  • Kothari, R.S., Chaudhary, A.K., Bailey, R.J., Pelz, J.B., Diaz, G.J.: EllSeg: an ellipse segmentation framework for robust gaze tracking. IEEE Trans. Vis. Comput. Graph. 27(5), 2757–2767 (2021)

    Article  Google Scholar 

  • Krafka, K., Khosla, A., Kellnhofer, P., Kannan, H., Bhandarkar, S., Matusik, W., Torralba, A.: Eye tracking for everyone. In: Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Las Vegas, NV, USA (2016)

  • Kumar, C., Hedeshy, R., MacKenzie, I.S., Staab, S.: TAGSwipe: touch assisted gaze swipe for text entry. In: Proceedings of the 2020 ACM Conference on Human Factors in Computing Systems (CHI). ACM, Honolulu, HI, USA (2020)

  • Kumar, A., Braud, T., Lee, L.H., Hui, P.: Theophany: multimodal speech augmentation in instantaneous privacy channels. In: Proceedings of the 29th ACM International Conference on Multimedia (MM). ACM, Chengdu, China (2021)

  • Kumar, A., Lee, L.-H., Chauhan, J., Su, X., Hoque, M.A., Pirttikangas, S., Tarkoma, S., Hui, P.: PassWalk: Spatial authentication leveraging lateral shift and gaze on mobile headsets. In: Proceedings of the 30th ACM International Conference on Multimedia (MM). ACM, Lisbon, Portugal (2022)

  • Lai, C.-C., Shih, S.-W., Tsai, H.-R., Hung, Y.-P.: 3-d gaze tracking using pupil contour features. In: Proceedings of the 22nd IEEE International Conference on Pattern Recognition (ICPR), pp. 1162–1166. IEEE, Stockholm, Sweden (2014a)

  • Lai, C.-C., Shih, S.-W., Hung, Y.-P.: Hybrid method for 3-d gaze tracking using glint and contour features. IEEE Trans. Circuits Syst. Video Technol. 25(1), 24–37 (2014b)

    Article  Google Scholar 

  • Lander, C., löchtefeld, M., Krüger, A.: hEYEbrid: a hybrid approach for mobile calibration-free gaze estimation. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1(4), 1–29 (2018)

    Article  Google Scholar 

  • Lee, J.W., Cho, C.W., Shin, K.Y., Lee, E.C., Park, K.R.: 3D gaze tracking method using purkinje images on eye optical model and pupil. Opt. Lasers Eng. 50(5), 736–751 (2012)

    Article  Google Scholar 

  • Lee, L.-H., Zhou, P., Braud, T., Hui, P.: What is the metaverse? An immersive cyberspace and open challenges. arXiv (2022)

  • Li, J., Li, S.: Gaze estimation from color image based on the eye model with known head pose. IEEE Trans. Hum.-Mach. Syst. 46(3), 414–423 (2015)

    Article  Google Scholar 

  • Mehrubeoglu, M., Pham, L.M., Le, H.T., Muddu, R., Ryu, D.: Real-time eye tracking using a smart camera. In: Proceedings of the 2011 IEEE Applied Imagery Pattern Recognition Workshop (AIPR). IEEE, Washington, DC, USA (2011)

  • Ohno, T., Mukawa, N., Yoshikawa, A.: Freegaze: a gaze tracking system for everyday gaze interaction. In: Proceedings of the 2002 Symposium on Eye Tracking Research & Applications (ETRA), pp. 125–132. ACM, New Orleans, USA (2002)

  • O’Reilly, J., Khan, A.S., Li, Z., Cai, J., Hu, X., Chen, M., Tong, Y.: A novel remote eye gaze tracking system using line illumination sources. In: Proceedings of the 2nd IEEE Conference on Multimedia Information Processing and Retrieval (MIPR), pp. 449–454. IEEE, San Jose, CA, USA (2019)

  • Patney, A., Salvi, M., Kim, J., Kaplanyan, A., Wyman, C., Benty, N., Luebke, D., Lefohn, A.: Towards foveated rendering for gaze-tracked virtual reality. ACM Trans. Graph. 35(6), 1–12 (2016)

    Article  Google Scholar 

  • Rayner, K.: Eye movements in reading and information processing: 20 years of research. Psychol. Bull. 124(3), 372–422 (1998)

    Article  Google Scholar 

  • Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: Towards real-time object detection with region proposal networks. In: Proceedings of 29th Annual Conference on Neural Information Processing Systems (NIPS), pp. 91–99. MIT Press, Montreal, Quebec, Canada (2015)

  • Ronneberger, O., Fischer, P., Brox, T.: U-Net: Convolutional networks for biomedical image segmentation. In: Proceedings of the 18th International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), pp. 234–241. Springer, Munich, Germany (2015)

  • Safaee-Rad, R., Tchoukanov, I., Smith, K.C., Benhabib, B.: Three-dimensional address estimation of circular features for machine vision. IEEE Trans. Robot. Autom. 8(5), 624–640 (1992)

    Article  Google Scholar 

  • Savela, N., Oksanen, A., Kaakinen, M., Noreikis, M., Xiao, Y.: Does augmented reality affect sociability, entertainment, and learning? A field experiment. Appl. Sci. 10(4), 1392 (2020)

    Article  Google Scholar 

  • Shatilov, K.A., Chatzopoulos, D., Lee, L.-H., Hui, P.: Emerging ExG-based NUI inputs in extended realities: a bottom-up survey. ACM Trans. Interact. Intell. Syst. 11(2), 1–49 (2021)

    Article  Google Scholar 

  • Sun, L., Song, M., Liu, Z., Sun, M.-T.: Real-time gaze estimation with online calibration. IEEE Multimed. 21(4), 28–37 (2014)

    Article  Google Scholar 

  • Sun, L., Liu, Z., Sun, M.-T.: Real time gaze estimation with a consumer depth camera. Inf. Sci. 320, 346–360 (2015)

    Article  MathSciNet  Google Scholar 

  • Villanueva, A., Cabeza, R.: A novel gaze estimation system with one calibration point. IEEE Trans. Syst. Man Cybern. 38(4), 1123–1138 (2008)

    Article  Google Scholar 

  • Wang, K., Ji, Q.: Real time eye gaze tracking with kinect. In: Proceedings of the IEEE 23rd International Conference on Pattern Recognition (ICPR), pp. 2752–2757. IEEE, Cancun, Mexico (2016)

  • Yang, C., Sun, J., Liu, J., Yang, X., Wang, D., Liu, W.: A gray difference-based pre-processing for gaze tracking. In: Proceedings of the 10th IEEE International Conference on Signal Processing (ICSP). IEEE, Beijing, China (2010)

  • Yang, S., He, Y., Zheng, X.: FoVR: attention-based VR streaming through bandwidth-limited wireless networks. In: Proceedings of the 16th Annual IEEE International Conference on Sensing, Communication, and Networking (SECON). IEEE, Boston, MA, USA (2019)

  • Yang, S., Jin, M., He, Y.: Continuous gaze tracking with implicit saliency-aware calibration on mobile devices. IEEE Trans. Mob. Comput. 22(10), 5816–5828 (2023)

    Article  Google Scholar 

  • Yiu, Y.-H., Aboulatta, M., Raiser, T., Ophey, L., Flanagin, V.L., Zu Eulenburg, P., Ahmadi, S.-A.: DeepVOG: open-source pupil segmentation and gaze estimation in neuroscience using deep learning. J. Neurosci. Methods 324, 108307 (2019)

    Article  Google Scholar 

  • Zhou, X., Cai, H., Shao, Z., Yu, H., Liu, H.: 3D eye model-based gaze estimation from a depth sensor. In: Proceedings of the 2016 IEEE International Conference on Robotics and Biomimetics (ROBIO), pp. 369–374. IEEE, Qingdao, China (2016)

  • Zhou, X., Cai, H., Li, Y., Liu, H.: Two-eye model-based gaze estimation from a kinect sensor. In: Proceedings of the 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 1646–1653. IEEE, Singapore (2017)

Download references

Acknowledgements

This work is supported by the National Science Fund of China under grant No. U21B2007. We thank all the anonymous reviewers for their valuable comments and helpful suggestions.

Author information

Authors and Affiliations

  1. School of Software, Tsinghua University, Qinghua Road, Beijing, 100084, Beijing, China

    Songzhou Yang, Yuan He & Yulong Chen

Authors
  1. Songzhou Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yuan He
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yulong Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yuan He.

Ethics declarations

Conflict of interest

On behalf of all authors, the corresponding author states that there is no conflict of interest.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yang, S., He, Y. & Chen, Y. Spatialgaze: towards spatial gaze tracking for extended reality. CCF Trans. Pervasive Comp. Interact. 5, 430–446 (2023). https://doi.org/10.1007/s42486-023-00139-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s42486-023-00139-4

Keywords

Search

Navigation