Skip to main content

Advertisement

SpringerLink
Log in

Learning to Draw Sight Lines

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

In this paper, we are concerned with the task of gaze following. Given a scene (e.g. a girl playing soccer on the field) and a human subject’s head position, this task aims to infer where she is looking (e.g. at the soccer ball). An existing method adopts a saliency model conditioned on the head position. However, this methodology is intrinsically troubled with dataset bias issues, which we will reveal in detail. In order to resolve these issues, we argue that the right methodology is to simulate how human beings follow gazes. Specifically, we propose the hypothesis that a human follows gazes by searching for salient objects along the subject’s sight line direction. To algorithmically embody this hypothesis, a two-stage method is proposed, which is dubbed as learning to draw sight lines. In the first stage, a fully convolutional network is trained to directly regress the existence strength of sight lines. It may seem counterintuitive at a first glance as these so-called sight lines do not really exist in the form of learnable image gradients. However, with the large-scale dataset GazeFollow, we demonstrate that this highly abstract concept can be grounded into neural network activations. An extensive study is conducted on the design of this sight line grounding network. We show that the best model we visited can already outperform the state-of-the-arts by a large margin, using a naive greedy inference strategy. We attribute these improvements to modern architecture design philosophies. However, no matter how strong the sight line grounding network is, the greedy inference strategy cannot handle a bunch of failure cases caused by dataset bias issues. We identify these issues and demonstrate that those grounded sight lines, which is a unique ingredient of our method, is the key to overcome them. Specifically, an algorithm termed as RASP is introduced as a second stage. RASP has five intriguing features: (1) it explicitly embodies the aforementioned hypothesis; (2) it involves no hyper-parameters, thus guaranteeing its robustness; (3) if needed, it can be implemented as an integrated layer for end-to-end inference; (4) it improves the performances of all sight line grounding networks we inspected; (5) further analyses confirm that RASP works by alleviating those spotted dataset biases. Strong results are achieved on the GazeFollow benchmark. Combining RASP and the best sight line grounding network can bring mean distance, minimum distance and mean angle difference 45.85%, 42.60%, and 49.23% closer towards human performance when compared with state-of-the-arts. We also contribute a video gaze following benchmark called GazeShift, on which we further demonstrate the importance of RASP in video applications. Codes and models will be released, encouraging further research on the important task of gaze following. Along with our implementation, we contribute a well-engineered toolbox for joint subject tracking and gaze following.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24
Fig. 25

Similar content being viewed by others

Notes

  1. Lowering thresholds can make subsets larger. However, they are not realistic settings because in the wild we don’t have the ground truth to filter a large box set with many false positives.

  2. The optical center is assumed to be aligned with the image center and focal lengths are assumed to be 500. We first convert the 2D head position into a bearing vector using the intrinsic matrix. Then an arbitrary 3D point is selected on the vector as a 3D starting point. Using the estimated 3D face direction we can get an ending point. Projecting these two points back to the image plane using the same intrinsic matrix gives us a 2D direction.

References

  • Achanta, R., Hemami, S., Estrada, F., & Süsstrunk, S. (2009). Frequency-tuned salient region detection. In IEEE international conference on computer vision and pattern recognition (CVPR 2009) (pp. 1597–1604). CONF.

  • Agrawal, A., Batra, D., Parikh, D., & Kembhavi, A. (2017). Don’t just assume; look and answer: Overcoming priors for visual question answering. ArXiv preprint arXiv:1712.00377.

  • Badrinarayanan, V., Kendall, A., & Cipolla, R. (2017). Segnet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(12), 2481–2495.

    Article  Google Scholar 

  • Blanz, V., & Vetter, T. (1999). A morphable model for the synthesis of 3d faces. In Proceedings of the 26th annual conference on computer graphics and interactive techniques (pp. 187–194). ACM Press/Addison-Wesley Publishing Co.

  • Borji, A., Cheng, M. M., Jiang, H., & Li, J. (2015). Salient object detection: A benchmark. IEEE Transactions on Image Processing, 24(12), 5706–5722.

    Article  MathSciNet  Google Scholar 

  • Breitenstein, M. D., Kuettel, D., Weise, T., Van Gool, L., & Pfister, H. (2008). Real-time face pose estimation from single range images. In IEEE conference on computer vision and pattern recognition, 2008. CVPR 2008 (pp. 1–8). IEEE.

  • Brooks, R., & Meltzoff, A. N. (2005). The development of gaze following and its relation to language. Developmental Science, 8, 535–543.

    Article  Google Scholar 

  • Bruce, N., & Tsotsos, J. (2006). Saliency based on information maximization. In Advances in neural information processing systems (pp. 155–162).

  • Cao, Z., Simon, T., Wei, S. E., & Sheikh, Y. (2017). Realtime multi-person 2d pose estimation using part affinity fields. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 7291–7299)

  • Chen, L. C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2018). Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 40(4), 834–848.

    Article  Google Scholar 

  • Cheng, M. M., Mitra, N. J., Huang, X., Torr, P. H., & Hu, S. M. (2015). Global contrast based salient region detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(3), 569–582.

    Article  Google Scholar 

  • Chong, E., Ruiz, N., Wang, Y., Zhang, Y., Rozga, A., & Rehg, J. M. (2018). Connecting gaze, scene, and attention: Generalized attention estimation via joint modeling of gaze and scene saliency. In Proceedings of the European conference on computer vision (ECCV) (pp. 383–398).

  • Erdem, E., & Erdem, A. (2013). Visual saliency estimation by nonlinearly integrating features using region covariances. Journal of Vision, 13(4), 11–11.

    Article  Google Scholar 

  • Everingham, M., Van Gool, L., Williams, C. K., Winn, J., & Zisserman, A. (2010). The pascal visual object classes (VOC) challenge. International Journal of Computer Vision, 88(2), 303–338.

    Article  Google Scholar 

  • Fan, L., Chen, Y., Wei, P., Wang, W., Zhu, S. C. (2018). Inferring shared attention in social scene videos. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 6460–6468).

  • Flom, R., Deák, G. O., Phill, C. G., & Pick, A. D. (2004). Nine-month-olds’ shared visual attention as a function of gesture and object location. Infant Behavior and Development, 27, 181–194.

    Article  Google Scholar 

  • Fouhey, D. F., Delaitre, V., Gupta, A., Efros, A. A., Laptev, I., & Sivic, J. (2014). People watching: Human actions as a cue for single view geometry. International Journal of Computer Vision, 110(3), 259–274.

    Article  Google Scholar 

  • Funes Mora, K. A., Monay, F., & Odobez, J. M. (2014). Eyediap: A database for the development and evaluation of gaze estimation algorithms from rgb and rgb-d cameras. In Proceedings of the symposium on eye tracking research and applications (pp. 255–258). ACM.

  • Ghiasi, G., & Fowlkes, C. C. (2016). Laplacian pyramid reconstruction and refinement for semantic segmentation. In European conference on computer vision (pp. 519–534). Springer.

  • Harel, J., Koch, C., & Perona, P. (2007). Graph-based visual saliency. In Advances in neural information processing systems (pp. 545–552).

  • He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770–778).

  • Henriques, J. F., Caseiro, R., Martins, P., & Batista, J. (2015). High-speed tracking with kernelized correlation filters. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(3), 583–596.

    Article  Google Scholar 

  • Hoiem, D., Efros, A. A., & Hebert, M. (2005). Geometric context from a single image. In Tenth IEEE international conference on computer vision, 2005. ICCV 2005 Vol 1 (pp. 654–661). IEEE.

  • Hou, X., & Zhang, L. (2007). Saliency detection: A spectral residual approach. In 2007 IEEE conference on computer vision and pattern recognition (pp. 1–8). IEEE.

  • Hou, X., & Zhang, L. (2009). Dynamic visual attention: Searching for coding length increments. In Advances in neural information processing systems (pp. 681–688).

  • Itti, L., Koch, C., & Niebur, E. (1998). A model of saliency-based visual attention for rapid scene analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 11, 1254–1259.

    Article  Google Scholar 

  • Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. (2014). Caffe: Convolutional architecture for fast feature embedding. In Proceedings of the 22nd ACM international conference on multimedia (pp. 675–678). ACM.

  • Jiang, M., Huang, S., Duan, J., & Zhao, Q. (2015). Salicon: Saliency in context. In Proceedings of the IEEE conference on computer vision and pattern recognition.

  • Joachims, T., Finley, T., & Yu, C. N. J. (2009). Cutting-plane training of structural SVMs. Machine Learning, 77(1), 27–59.

    Article  Google Scholar 

  • Judd, T., Ehinger, K., Durand, F., & Torralba, A. (2009). Learning to predict where humans look. In 2009 IEEE 12th international conference on computer vision.

  • Kalal, Z., Mikolajczyk, K., Matas, J., et al. (2012). Tracking-learning-detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(7), 1409.

    Article  Google Scholar 

  • Krafka, K., Khosla, A., Kellnhofer, P., Kannan, H., Bhandarkar, S., Matusik, W., & Torralba, A. (2016). Eye tracking for everyone. In 2016 IEEE conference on computer vision and pattern recognition (CVPR) (pp. 2176–2184). IEEE.

  • Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems (pp. 1097–1105).

  • LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11), 2278–2324.

    Article  Google Scholar 

  • Li, Y., Fathi, A., & Rehg, J. M. (2013). Learning to predict gaze in egocentric video. In Proceedings of the IEEE international conference on computer vision (pp. 3216–3223).

  • Li, Y., Hou, X., Koch, C., Rehg, J. M., & Yuille, A. L. (2014). The secrets of salient object segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 280–287).

  • Liu, T., Yuan, Z., Sun, J., Wang, J., Zheng, N., Tang, X., et al. (2011). Learning to detect a salient object. IEEE Transactions on Pattern analysis and machine intelligence, 33(2), 353–367.

    Article  Google Scholar 

  • Long, J., Shelhamer, E., & Darrell, T. (2015). Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 3431–3440).

  • Lu, F., Okabe, T., Sugano, Y., & Sato, Y. (2011). A head pose-free approach for appearance-based gaze estimation. In BMVC (pp. 1–11).

  • Lu, F., Sugano, Y., Okabe, T., & Sato, Y. (2014). Adaptive linear regression for appearance-based gaze estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(10), 2033–2046.

    Article  Google Scholar 

  • Lu, J., Yang, J., Batra, D., & Parikh, D. (2018). Neural baby talk. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 7219–7228).

  • Lukežič, A., Vojíř, T., Zajc, L. Č., Matas, J., & Kristan, M. (2018). Discriminative correlation filter tracker with channel and spatial reliability. International Journal of Computer Vision, 126(7), 671–688.

    Article  MathSciNet  Google Scholar 

  • Mallya, A., & Lazebnik, S. (2015). Learning informative edge maps for indoor scene layout prediction. In Proceedings of the IEEE international conference on computer vision (pp. 936–944).

  • Marín-Jiménez, M. J., Zisserman, A., Eichner, M., & Ferrari, V. (2014). Detecting people looking at each other in videos. International Journal of Computer Vision, 106(3), 282–296.

    Article  Google Scholar 

  • Mathe, S., & Sminchisescu, C. (2012). Dynamic eye movement datasets and learnt saliency models for visual action recognition. In European conference on computer vision (pp. 842–856). Springer.

  • Mathe, S., & Sminchisescu, C. (2015). Actions in the eye: Dynamic gaze datasets and learnt saliency models for visual recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(7), 1408–1424.

    Article  Google Scholar 

  • Morimoto, C. H., & Mimica, M. R. (2005). Eye gaze tracking techniques for interactive applications. Computer Vision and Image Understanding, 98(1), 4–24.

    Article  Google Scholar 

  • Perazzi, F., Krähenbühl, P., Pritch, Y., & Hornung, A. (2012). Saliency filters: Contrast based filtering for salient region detection. In 2012 IEEE conference on computer vision and pattern recognition (pp. 733–740). IEEE.

  • Recasens, A., Khosla, A., Vondrick, C., & Torralba, A. (2015). Where are they looking? In NIPS.

  • Recasens, A., Vondrick, C., Khosla, A., & Torralba, A. (2017). Following gaze in video. In The IEEE international conference on computer vision (ICCV) vol. 4.

  • Rehg, J., Abowd, G., Rozga, A., Romero, M., Clements, M., Sclaroff, S., Essa, I., Ousley, O., Li, Y., & Kim, C., et al. (2013). Decoding children’s social behavior. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 3414–3421).

  • Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., et al. (2015). Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3), 211–252.

    Article  MathSciNet  Google Scholar 

  • Schneider, T., Schauerte, B., & Stiefelhagen, R. (2014). Manifold alignment for person independent appearance-based gaze estimation. In 2014 22nd international conference on pattern recognition (pp. 1167–1172). IEEE.

  • Schwing, A. G., Hazan, T., Pollefeys, M., & Urtasun, R. (2012). Efficient structured prediction for 3d indoor scene understanding. In 2012 IEEE conference on computer vision and pattern recognition (CVPR) (pp 2815–2822). IEEE.

  • Senju, A., & Csibra, G. (2008). Gaze following in human infants depends on communicative signals. Current Biology, 18, 668–671.

    Article  Google Scholar 

  • Shrivastava, A., Pfister, T., Tuzel, O., Susskind, J., Wang, W., & Webb, R. (2017). Learning from simulated and unsupervised images through adversarial training. In The IEEE conference on computer vision and pattern recognition (CVPR) Vol 3 (p. 6).

  • Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. ArXiv preprint arXiv:1409.1556.

  • Song, S., Yu, F., Zeng, A., Chang, A. X., Savva, M., & Funkhouser, T. (2017). Semantic scene completion from a single depth image. In 2017 IEEE conference on computer vision and pattern recognition (CVPR) (pp. 190–198). IEEE.

  • Sugano, Y., Matsushita, Y., Sato, Y., & Koike, H. (2008). An incremental learning method for unconstrained gaze estimation. In European conference on computer vision (pp. 656–667). Springer.

  • Sugano, Y., Matsushita, Y., & Sato, Y. (2014). Learning-by-synthesis for appearance-based 3d gaze estimation. In 2014 IEEE conference on computer vision and pattern recognition (CVPR) (pp. 1821–1828). IEEE.

  • Wei, P., Liu, Y., Shu, T., Zheng, N., & Zhu, S. C. (2018). Where and why are they looking? jointly inferring human attention and intentions in complex tasks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 6801–6809).

  • Wood, E., Baltrušaitis, T., Morency, L. P., Robinson, P., Bulling, A. (2016). Learning an appearance-based gaze estimator from one million synthesised images. In Proceedings of the ninth biennial ACM symposium on eye tracking research and applications (pp. 131–138). ACM.

  • Wu, Z., Shen, C., & Van Den Hengel, A. (2016). Wider or deeper: Revisiting the resnet model for visual recognition. ArXiv preprint arXiv:1611.10080.

  • Yan, Q., Xu, L., Shi, J., & Jia, J. (2013). Hierarchical saliency detection. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1155–1162).

  • Yao, A., & Chen, Y. (2018). Combinatorial shape regression for face alignment in images. US Patent App. 15/573,631

  • Yu, F., & Koltun, V. (2015). Multi-scale context aggregation by dilated convolutions. ArXiv preprint arXiv:1511.07122.

  • Yu, F., Koltun, V., & Funkhouser, T. A. (2017). Dilated residual networks. In CVPR Vol 2 (p. 3).

  • Zagoruyko, S., & Komodakis, N. (2016). Wide residual networks. ArXiv preprint arXiv:1605.07146.

  • Zhang, L., Tong, M. H., Marks, T. K., Shan, H., & Cottrell, G. W. (2008). Sun: A bayesian framework for saliency using natural statistics. Journal of Vision, 8(7), 32–32.

    Article  Google Scholar 

  • Zhang, S., Zhu, X., Lei, Z., Shi, H., Wang, X., & Li, S. Z. (2017). \(\text{S}^{\wedge }\) 3fd: Single shot scale-invariant face detector. In 2017 IEEE international conference on computer vision (ICCV) (pp. 192–201). IEEE.

  • Zhang, X., Sugano, Y., Fritz, M., & Bulling, A. (2015). Appearance-based gaze estimation in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4511–4520).

  • Zhang, Y., Bai, M., Kohli, P., Izadi, S., & Xiao, J. (2016). Deepcontext: Context-encoding neural pathways for 3d holistic scene understanding. ArXiv preprint arXiv:1603.04922.

  • Zhao, H., Lu, M., Yao, A., Guo, Y., Chen, Y., & Zhang, L. (2017a). Physics inspired optimization on semantic transfer features: An alternative method for room layout estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 10–18).

  • Zhao, H., Shi, J., Qi, X., Wang, X., & Jia, J. (2017b). Pyramid scene parsing network. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2881–2890).

  • Zhao, Y., & Zhu, S. C. (2013). Scene parsing by integrating function, geometry and appearance models. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 3119–3126).

  • Zhou, B., Lapedriza, A., Xiao, J., Torralba, A., & Oliva, A. (2014). Learning deep features for scene recognition using places database. In Advances in neural information processing systems (pp. 487–495).

Download references

Acknowledgements

We thank anonymous reviewers for suggestions on literature review and experimental design. This work was jointly supported by National Natural Science Foundation of China (Grant Nos. 61132007, 61172125 and U1533132).

Author information

Authors and Affiliations

  1. Tsinghua University, Beijing, China

    Hao Zhao, Ming Lu & Li Zhang

  2. Intel Labs China, Beijing, China

    Anbang Yao & Yurong Chen

Authors
  1. Hao Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ming Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Anbang Yao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yurong Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Li Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hao Zhao.

Additional information

Communicated by Cristian Sminchisescu.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (txt 0 KB)

Supplementary material 2 (txt 0 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhao, H., Lu, M., Yao, A. et al. Learning to Draw Sight Lines. Int J Comput Vis 128, 1076–1100 (2020). https://doi.org/10.1007/s11263-019-01263-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-019-01263-4

Keywords

Search

Navigation