Skip to main content

Advertisement

Springer Nature Link
Log in

A CNN based real-time eye tracker for web mining applications

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Eye gaze tracking is an increasingly important technology in the field of human-computer interaction. Individuals’ preferences, tendencies, and attention can be measured by processing the data obtained from face and eye images. This technology is used in advertising, market research, web page design, education, learning methods, and various neurological-psychiatric studies of medical research. Many different methods have been used in eye gaze tracking tasks. Today, commonly model-shape and appearance-based methods are used. Model-shape based methods require less workload than appearance-based methods. But it is more sensitive to environmental conditions. Appearance-based methods require powerful hardware, but they are less susceptible to environmental conditions. Developments in technology have paved the way for applying appearance-based models in eye gaze tracking. In this paper, a CNN-based real-time eye tracking system was designed to overcome environmental problems in eye gaze tracking. The designed system is used to determine the areas of interest of the user in web pages. The performance of the designed CNN-based system is evaluated during the training and testing phases. In the training phase, the difference between the desired and determined points on the screen is 32 pixels and in testing phase, the difference between the desired and determined points on the screen is 53 pixels. The results of the test trials have shown that the proposed system could be used successfully in eye tracking studies on web pages.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

References

  1. Akhtar N, Ragavendran U (2020) Interpretation of intelligence in CNN-pooling processes: a methodological survey. Neural Comput Appl 32(3):879–898

    Article  Google Scholar 

  2. Aljaafreh A, Alaqtash M, Al-Oudat N, Abukhait J, Saleh M (2020) A low-cost webcam-based eye tracker and saccade measurement system. Int J Circuits Syst Signal Process 14. https://doi.org/10.46300/9106.2020.14.16

  3. Baluja, S. and Pomerleau, D. (1993) Non-Intrusive Gaze Tracking using Artificial Neural Networks. In Cowan J.D., Tesauro, G. and Alspector, J. (eds.) Advances in Neural Information Processing Systems, (NIPS) 6. 1994. pp. 753–760. Morgan Kaufmann Publishers, San Francisco, CA.

  4. Barea R, Boquete L, Mazo M, López E (2002) System for assisted mobility using eye movements based on electrooculography. IEEE Trans Neural Syst Rehabil Eng 10(4):209–218

    Article  Google Scholar 

  5. Chen J, Ji Q (2008) 3D gaze estimation with a single camera without IR illumination. Proceedings-International Conference on Pattern Recognition, pp 1–4. https://doi.org/10.1109/icpr.2008.4761343

  6. Chen HH, Hwang BJ, Wu JS, Liu PT (2020) The effect of different deep network architectures upon CNN-based gaze tracking. Algorithms 13(5). https://doi.org/10.3390/A13050127

  7. Collewijn H, van der Mark F, Jansen TC (1975) Precise recording of human eye movements. Vision Res 15(3):447–450. https://doi.org/10.1016/0042-6989(75)90098-X

    Article  Google Scholar 

  8. Cooles TF, Edwards GJ, Taylor CJ (2001) Active appearance models. IEEE Trans Pattern Anal Mach Intell 23(6):681–685. https://doi.org/10.1109/34.927467

    Article  Google Scholar 

  9. Cun YL, Guyon I, Jackel LD, Henderson D, Boser B, Howard RE, Denker JS, Hubbard W, Graf HP (1989) Handwritten digit recognition: applications of neural network chips and automatic learning. IEEE Commun Mag 27(11):41–46. https://doi.org/10.1109/35.41400

    Article  Google Scholar 

  10. dlib C + + Library. http://www.dlib.net. Accessed 23 Sept 2020

  11. Damji O, Lee-Nobbee P, Borkenhagen D, Cheng A (2019) Analysis of eye-tracking behaviours in a pediatric trauma simulation. Can J Emerg Med 21(1):138–140

    Article  Google Scholar 

  12. de Araújo FMA, Fonseca Ferreira NM, Mascarenhas VTOC, Adad Filho JA, Viana Filho PRF (2020) Eye tracking in framework for the development of games for people with motor disabilities. Lect Notes Electr Eng 621:239–247. https://doi.org/10.1007/978-981-15-1465-4_25

    Article  Google Scholar 

  13. Doğan F, Türkoğlu İ (2019) Derin Öğrenme Modelleri ve Uygulama Alanlarına İlişkin Bir Derleme. DÜMF Mühendislik Dergisi 10(2):409–445. https://doi.org/10.24012/dumf.411130

    Article  Google Scholar 

  14. Donuk K, Hanbay D (2020) Video Based Real-Time Eye Tracking. In: 28th IEEE Signal Processing and Communications Applications (SIU-2020), pp 21–24, Turkey

  15. Duchowski AT (2017) In eye tracking methodology: Theory and practice: 3rd edn. Springer International Publishing, Berlin. https://doi.org/10.1007/978-3-319-57883-5

  16. Dutta P, Upadhyay P, De M, Khalkar RG (2020) Medical Image Analysis using Deep Convolutional Neural Networks: CNN architectures and transfer learning. Proceedings of the 5th International Conference on Inventive Computation Technologies, ICICT 2020, pp 175–180

  17. Elmas, Ç. (2011). “Yapay Zekâ Uygulamaları: Yapay Sinir Ağı, Bulanık Mantık, Genetik Algoritma”, Seçkin Yayıncılık,Ankara.

  18. Eye Tracking System Technology For Everyone - UX Testing | Gazepoint. https://www.gazept.com. Accessed 30 Dec 2020

  19. Fukushima K (1980) Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biol Cybern 36(4):193–202

    Article  Google Scholar 

  20. Gurgu E, Gurgu I-A, Tonis RBM (2020) Neuromarketing for a better understanding of consumer needs and emotions. Indep J Manage Prod 11(1):208

    Article  Google Scholar 

  21. Hansen DW, Ji Q (2010) In the eye of the beholder: a survey of models for eyes and gaze. IEEE Trans Pattern Anal Mach Intell 32(3):478–500

    Article  Google Scholar 

  22. Hinton GE, Salakhutdinov RR (2006) Reducing the dimensionality of data with neural networks. Science 313(5786):504–507. https://doi.org/10.1126/science.1127647

    Article  MathSciNet  MATH  Google Scholar 

  23. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780

    Article  Google Scholar 

  24. Home - OpenCV. https://opencv.org/. Accessed 30 Dec 2020

  25. Hung JC, Wang CC (2020) Exploring the website object layout of responsive web design: results of eye tracking evaluations. J Supercomput 1–23. https://doi.org/10.1007/s11227-020-03283-1

  26. i·bug-Facial point annotations. https://ibug.doc.ic.ac.uk/resources/facial-point-annotations/. Accessed 23 Sept 2020

  27. Ioffe S, Szegedy C (2015) Batch normalization: Accelerating deep network training by reducing internal covariate shift. 32nd International Conference on Machine Learning, ICML 2015, vol 1

  28. Ishikawa T, Baker S, Matthews I, Kanade T (2004) Passive driver gaze tracking with active appearance models. Proc. World Congress on Intelligent Transportation Systems

  29. Ivakhnenko, A.; Lapa, V. (1965) Cybernetic predicting devices; CCM Information Corp.: New York, NY, USA.

  30. Janssen RGJ, Rothkrantz L, Wiggers IP, Geers IH (2010) Real time eye blink detection using a configurable processor. Ph.D. Dissertation, Doctoral Dissertation, TU Delft, Delft University of Technology

  31. Jie HJ, Wanda P (2020) Runpool: A dynamic pooling layer for convolution neural network. Int J Comput Intell Syst 13(1):66–76. https://doi.org/10.2991/ijcis.d.200120.002

    Article  Google Scholar 

  32. Kazemi V, Sullivan J (2014) One millisecond face alignment with an ensemble of regression trees. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. https://doi.org/10.1109/CVPR.2014.241

  33. Kim M, Wang O, Ng N (2017) Convolutional neural network architectures for gaze. Estimation on Mobile Devices, Stanford

    Google Scholar 

  34. Kimme C, Ballard D, Sklansky J (1975) Finding circles by an array of accumulators. Commun ACM 18(2):120–122. https://doi.org/10.1145/360666.360677

    Article  MATH  Google Scholar 

  35. Kingma DP, Ba JL (2015) Adam: A method for stochastic optimization. 3rd International Conference on Learning Representations, ICLR 2015- Conference Track Proceedings

  36. Krafka K, Khosla A, Kellnhofer P, Kannan H, Bhandarkar S, Matusik W, Torralba A (2016) Eye tracking for everyone. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. https://doi.org/10.1109/CVPR.2016.239

  37. Krizhevsky A, Sutskever I, Hinton GE (2012) ImageNet classification with deep convolutional neural networks. Adv Neural Inf Process Syst 2:1097–1105

    Google Scholar 

  38. LeCun Y, Boser B, Denker J, Henderson D, Howard R, Hubbard W, Jackel L (1989) Handwritten digit recognition with a back-propagation network. pp 396–404

  39. LeCun Y, Bottou L, Bengio Y, Haffner P (1998)Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324. https://doi.org/10.1109/5.726791

  40. Lu F, Sugano Y, Okabe T, Sato Y (2014) Adaptive linear regression for appearance-based gaze estimation. IEEE Trans Pattern Anal Mach Intell 36(10):2033–2046

    Article  Google Scholar 

  41. Majaranta P, Bulling A (2014) Eye Tracking and Eye-BasedHuman–Computer Interaction. 39–65. https://doi.org/10.1007/978-1-4471-6392-3_3

  42. Martens J, Sutskever I (2011) Learning recurrent neural networks with Hessian-free optimization. Proceedings of the 28th International Conference on Machine Learning, ICML 2011

  43. Massin L, Nourrit V, Lahuec C, Seguin F, Adam L, Daniel E, De J-L, De B, Tocnaye LA (2020) Development of a new scleral contact lens with encapsulated photodetectors for eye tracking. Opt Express 28(19):28635–28647. https://doi.org/10.1364/OE.399823

    Article  Google Scholar 

  44. Mora KAF, Monay F, Odobez JM (2014) EYEDIAP: A database for the development and evaluation of gaze estimation algorithms from RGB and RGB-D cameras. Eye Tracking Research and Applications Symposium (ETRA), pp 255–258. https://doi.org/10.1145/2578153.2578190

  45. Morimoto CH, Amir A, Flickner M (2002) Detecting eye position and gaze from a single camera and 2 light sources. Proceedings - International Conference on Pattern Recognition 16(4):314–317

  46. Nair V, Hinton GE (2010) Rectified linear units improve restricted Boltzmann machines. In: ICML’10: Proceedings of the 27th International Conference on International Conference on Machine Learning, pp 807–814

  47. Niemann J, Fussenecker C, Schlösser M (2019) Eye tracking for quality control in automotive manufacturing. Commun Comput Inform Sci 1060:289–298

    Article  Google Scholar 

  48. Pfiffelmann J, Dens N, Soulez S (2020) Personalized advertisements with integration of names and photographs: An eye-tracking experiment. J Bus Res 111:196–207

    Article  Google Scholar 

  49. Rayner K (1998) Eye movements in reading and information processing: 20 years of research. Psychol Bull 124(3). https://doi.org/10.1037/0033-2909.124.3.372

  50. Reyes F, Fajardo A, Hernandez A (2020) Convolutional neural network for automatic speech recognition of Filipino language. Int J Adv Trends Comput Sci Eng 9(1.1 Special Issue):34–40. https://doi.org/10.30534/ijatcse/2020/0791.12020

  51. Robinson DA (1963) A method of measuring eye movement using a scleral search coil in a magnetic field. IEEE Trans Bio-med Electron 10(4):137–145

    Article  Google Scholar 

  52. Rosenblatt F (1958) The perceptron: A probabilistic model for information storage and organization in the brain. Psychol Rev 65(6):386–408. https://doi.org/10.1037/h0042519

    Article  Google Scholar 

  53. Salakhutdinov, R. and Hinton, G. (2009). Deep Boltzmann machines. In Proceedings of the Twelfth International Conference on Artificial Intelligence and Statistics (AISTATS 2009), vol. 8

  54. Saraee E, Jalal M, Betke M (2020) Visual complexity analysis using deep intermediate-layer features. Comput Vis Image Underst 195:102949. https://doi.org/10.1016/j.cviu.2020.102949

    Article  Google Scholar 

  55. Schmidhuber J (2015) Deep Learning in neural networks: An overview. Neural Netw 61:85–117. https://doi.org/10.1016/j.neunet.2014.09.003

    Article  Google Scholar 

  56. Şeker A, Diri B, Balık H (2017) Derin Öğrenme Yöntemleri ve Uygulamaları Hakkında Bir İnceleme. Gazi J Eng Sci 3(3):47–64

    Google Scholar 

  57. Sewell W, Komogortsev O (2010)Real-time eye gaze tracking with an unmodified commodity webcam employing a neural network. Conference on Human Factors in Computing Systems – Proceedings, pp 3739–3744. https://doi.org/10.1145/1753846.1754048

  58. Shokishalov Z, Wang H (2019) Applying eye tracking in information security. Procedia Comput Sci 150:347–351. https://doi.org/10.1016/j.procs.2019.02.062

    Article  Google Scholar 

  59. Slanzi G, Balazs JA, Velásquez JD (2017) Combining eye tracking, pupil dilation and EEG analysis for predicting web users click intention. Inform Fusion 35:51–57

    Article  Google Scholar 

  60. Specializing in eye tracking - EyeLink Eye Trackers - SR Research. https://www.sr-research.com. Accessed 30 Dec 2020

  61. Sugano Y, Matsushita Y, Sato Y (2014)Learning-by-synthesis for appearance-based 3D gaze estimation. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp 1821–1828. https://doi.org/10.1109/CVPR.2014.235

  62. Sultana F, Sufian A, Dutta P (2020) A review of object detection models based on convolutional neural network. In: Advances in Intelligent Systems and Computing 1157:1–16. Springer

  63. Sun S, Hu B, Yu Z, Song XN (2020) A stochastic max pooling strategy for convolutional neural network trained by noisy samples. Intern J Comput Commun Contr 15(1):9

  64. Suzuki S, Be KA (1985) Topological structural analysis of digitized binary images by border following. Comput Vis Graphics Image Process 30(1):32–46

    Article  Google Scholar 

  65. Tobii is the world leader in eye tracking. https://www.tobii.com. Accessed 30 Dec 2020

  66. Tsantekidis A, Passalis N, Tefas A, Kanniainen J, Gabbouj M, Iosifidis A (2020) Using Deep Learning for price prediction by exploiting stationary limit order book features. Appl Soft Comput J 93. https://doi.org/10.1016/j.asoc.2020.106401

  67. Uhm KH, Kang MC, Kim JY, Ko SJ (2020) Improving the robustness of gaze tracking under unconstrained illumination conditions. Multimed Tools Appl 79(29–30):20603–20616

    Article  Google Scholar 

  68. Velásquez JD (2013) Combining eye-tracking technologies with web usage mining for identifying Website Keyobjects. Eng Appl Artif Intell 26(5–6):1469–1478

    Article  Google Scholar 

  69. Vincent P, Larochelle H, Bengio Y, Manzagol PA (2008) Extracting and composing robust features with denoising autoencoders. Proceedings of the 25th International Conference on Machine Learning, pp 1096–1103. https://doi.org/10.1145/1390156.1390294

  70. Viola P, Jones M (2001) Rapid object detection using a boosted cascade of simple features. Proc  IEEE Comput Soc Conf Comput Vision Pattern Recogn 1:511–518

  71. Welcome to Colaboratory - Colaboratory. https://colab.research.google.com. Accessed 3 Nov 2020

  72. Whitmire E, Trutoiu L, Cavin R, Perek D, Scally B, Phillips J, Patel S (2016) EyeContact: Scleral coil eye tracking for virtual reality. International Symposium on Wearable Computers, pp 184–191

  73. Wilkinson L, Friendly M (2009) History corner the history of the cluster heat map. Am Stat 63(2):179–184. https://doi.org/10.1198/tas.2009.0033

    Article  Google Scholar 

  74. Wong ET, Yean S, Hu Q, Lee BS, Liu J, Deepu R, Workshops C (2019) PerCom Workshops 2019. https://doi.org/10.1109/PERCOMW.2019.8730846

  75. Xu L-Q, Machin D, Sheppard P (1998) A novel approach to real-time non-intrusive gaze finding. In: Proc. BMVC. https://doi.org/10.5244/c.12.43

  76. Ye L, Zhu M, Xia S, Pan H (2014) Cascaded convolutional neural network for eye detection under complex scenarios. Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture. Notes in Bioinformatics) 8833:473–480. https://doi.org/10.1007/978-3-319-12484-1_54

  77. Yoo DH, Chung MJ (2005) A novel non-intrusive eye gaze estimation using cross-ratio under large head motion. Comput Vis Image Underst 98(1):25–51

    Article  Google Scholar 

  78. Zhang X, Sugano Y, Fritz M, Bulling A (2015)Appearance-based gaze estimation in the wild. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp 4511–4520. https://doi.org/10.1109/CVPR.2015.7299081

  79. Zhang K, Zhang Z, Li Z, Qiao Y (2016) Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Process Lett 23(10):1499–1503

    Article  Google Scholar 

  80. Zhang Z, Zhao Y, Liao X, Shi W, Li K, Zou Q, Peng S (2019) Deep learning in omics: A survey and guideline. Brief Funct Genomics 18(1):41–57. https://doi.org/10.1093/bfgp/ely030

    Article  Google Scholar 

  81. Zhang X, Wang L, Su Y (2020) Visual place recognition: a survey from deep learning perspective pattern recognition. https://doi.org/10.1016/j.patcog.2020.107760

  82. Zhou Y (2020) Psychological analysis of online teaching in colleges based on eye-tracking technology. Rev Argenti Clin Psicol 29(2):523–529. https://doi.org/10.24205/03276716.2020.272

    Article  Google Scholar 

Download references

Funding

This study was supported by Inonu University Scientific Research Projects Coordination Unit (BAP) with the project coded FDK-2020-2110.

Author information

Authors and Affiliations

  1. Comp. Prog. Programme, Voc. School of Cizre, Sirnak University, 73000, Sirnak, Turkey

    Kenan Donuk

  2. Computer Engineering Department, Engineering Faculty, Inonu University, 1 Ali Ari - 2, 44280, Malatya, Turkey

    Ali Ari & Davut Hanbay

Authors
  1. Kenan Donuk
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Ali Ari
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Davut Hanbay
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Ali Ari.

Ethics declarations

Conflict of interest

The authors declare that there is no conflict to interest related to this paper.

Ethical approval

Not required.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Donuk, K., Ari, A. & Hanbay, D. A CNN based real-time eye tracker for web mining applications. Multimed Tools Appl 81, 39103–39120 (2022). https://doi.org/10.1007/s11042-022-13085-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-13085-7

Keywords