Skip to main content
SpringerLink
Log in

Mining Videos for Features that Drive Attention

  • Chapter
  • First Online:
Multimedia Data Mining and Analytics

Abstract

Certain features of a video capture human attention and this can be measured by recording eye movements of a viewer. Using this technique combined with extraction of various types of features from video frames, one can begin to understand what features of a video may drive attention. In this chapter we define and assess different types of feature channels that can be computed from video frames, and compare the output of these channels to human eye movements. This provides us with a measure of how well a particular feature of a video can drive attention. We then examine several types of channel combinations and learn a set of weightings of features that can best explain human eye movements. A linear combination of features with high weighting on motion and color channels was most predictive of eye movements on a public dataset.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Baluch F, Itti L (2010) Training top-down attention improves performance on a triple-conjunction search task. PLoS One 5(2):e9127

    Article  Google Scholar 

  2. Baluch F, Itti L (2011) Mechanisms of top-down attention. Trends Neurosci 34(4):210–240

    Article  Google Scholar 

  3. Berg DJ, Boehnke SE, Marino RA, Munoz DP, Itti L (2009) Free viewing of dynamic stimuli by humans and monkeys. J Vis 9 5(19):1–15

    Google Scholar 

  4. Borji A, Itti L (2013) State-of-the-art in visual attention modeling. IEEE Trans Pattern Anal Mach Intell 35(1):185–207

    Article  MathSciNet  Google Scholar 

  5. Chiang A-YD, Berg D, Itti L (2011) Saliency, memory, and attention capture in marketing. J Vis 11(11):493–493

    Article  Google Scholar 

  6. Connor CE, Egeth HE, Yantis S (2004) Visual attention: bottom-up versus top-down. Curr Biol 14(19):R850–R852

    Article  Google Scholar 

  7. Elazary L, Itti L (2008) Interesting objects are visually salient. J Vis 8(3):3

    Article  Google Scholar 

  8. Fecteau J, Bell A, Munoz D (2004) Neural correlates of the automatic and goal-driven biases in orienting spatial attention. J Neurophysiol 92(3):1728–1737

    Article  Google Scholar 

  9. Gilbert C, Sigman M (2007) Brain states: top-down influences in sensory processing. Neuron 54(5):677–696

    Article  Google Scholar 

  10. Grant WS, Itti L (2012) Saliency mapping enhanced by symmetry from local phase. In: Proceedings of IEEE international conference on image processing (ICIP). Florida, pp 653–656

    Google Scholar 

  11. Itti L (2008) Crcns data sharing: eye movements during free-viewing of natural videos. In: Collaborative research in computational neuroscience annual meeting. California

    Google Scholar 

  12. Itti L, Dhavale N, Pighin F (2003) Realistic avatar eye and head animation using a neurobiological model of visual attention. In: Bosacchi B, Fogel DB, Bezdek JC (eds) Proceedings of SPIE 48th annual international symposium on optical science and technology, vol 5200. SPIE Press, Bellingham, pp 64–78

    Google Scholar 

  13. Itti L et al (1998) The ilab neuromorphic vision C++ toolkit (INVT), http://ilab.usc.edu/toolkit

  14. Itti L, Koch C (2001) Computational modelling of visual attention. Nat Rev Neurosci 2(3):194–203

    Google Scholar 

  15. Itti L, Koch C (2001) Feature combination strategies for saliency-based visual attention systems. J Electron Imaging 10(1):161–169

    Article  Google Scholar 

  16. Itti L, Koch C, Niebur E (1998) A model of saliency-based visual attention for rapid scene analysis. IEEE Trans Pattern Anal Mach Intell 20(11):1254–1259

    Article  Google Scholar 

  17. Kienzle W, Franz MO, Schölkopf B, Wichmann FA (2009) Center-surround patterns emerge as optimal predictors for human saccade targets. J Vis 9(5):7

    Article  Google Scholar 

  18. Koch C, Ullman S (1985) Shifts in selective visual attention: towards the underlying neural circuitry. Hum Neurobiol 4(4):219–227

    Google Scholar 

  19. Koene AR, Zhaoping L (2007) Feature-specific interactions in salience from combined feature contrasts: evidence for a bottom-up saliency map in v1. J Vis 7(7):6

    Article  Google Scholar 

  20. Li Z, Itti L (2011) Saliency and gist features for target detection in satellite images. IEEE Trans Image Process 20(7):2017–2029

    Article  MathSciNet  Google Scholar 

  21. Li Z, Qin S, Itti L (2011) Visual attention guided bit allocation in video compression. Image Vis Comput 29(1):1–14

    Article  MATH  Google Scholar 

  22. Moore T (2006) The neurobiology of visual attention: finding sources. Curr Opin Neurobiol 16(2):159–165

    Article  Google Scholar 

  23. Mundhenk TN, Itti L (2005) Computational modeling and exploration of contour integration for visual saliency. Biolo Cybern 93(3):188–212

    Article  MATH  MathSciNet  Google Scholar 

  24. Navalpakkam V, Itti L (2006) An integrated model of top-down and bottom-up attention for optimal object detection. In: Proceedings of IEEE conference on computer vision and pattern recognition (CVPR). New York, pp 2049–2056

    Google Scholar 

  25. Peters RJ, Iyer A, Itti L, Koch C (2005) Components of bottom-up gaze allocation in natural images. Vis Res 45(8):2397–2416

    Article  Google Scholar 

  26. Rajashekar U, Bovik AC, Cormack LK (2006) Visual search in noise: revealing the influence of structural cues by gaze-contingent classification image analysis. J Vis 6(4):7

    Article  Google Scholar 

  27. Treisman A, Gelade G (1980) A feature-integration theory of attention. Cogn Psychol 12(1):97–136

    Google Scholar 

  28. Tseng P, Cameron IGM, Pari G, Reynolds JN, Munoz DP, Itti L (2013) High-throughput classification of clinical populations from natural viewing eye movements. J Neurol 260:275–284

    Google Scholar 

  29. Tseng P, Carmi R, Cameron IGM, Munoz D, Itti L (2009) Quantifying center bias of observers in free viewing of dynamic natural scenes. J Vis 9 7(4):1–16

    Google Scholar 

  30. Ts’o D, Gilbert CD (1988) The organization of chromatic and spatial interactions in the primate striate cortex. J Neurosci 8(5):1712–1727

    Google Scholar 

  31. Walther D (2006) Interactions of visual attention and object recognition: Computational modeling, algorithms, and psychophysics. PhD thesis, California Institute of Technology

    Google Scholar 

  32. Walther D, Itti L, Riesenhuber M, Poggio T, Koch C (2002) Attentional selection for object recognition—a gentle way. In: Biologically motivated computer vision. Springer, pp 472–479

    Google Scholar 

  33. White BJ, Boehnke SE, Marino RA, Itti L, Munoz DP (2009) Color-related signals in the primate superior colliculus. J Neurosci 29(39):12159–12166

    Article  Google Scholar 

  34. Wolfe JM, Horowitz TS (2004) What attributes guide the deployment of visual attention and how do they do it? Nat Rev Neurosci 5(6):495–501

    Article  Google Scholar 

  35. Yarbus AL, Haigh B, Rigss LA (1967) Eye Mov Vis. Plenum Press, New York

    Book  Google Scholar 

  36. Yoshida M, Itti L, Berg DJ, Ikeda T, Kato R, Takaura K, White BJ, Munoz DP, Isa T (2012) Residual attention guidance in blindsight monkeys watching complex natural scenes. Curr Biol 22(15):1429–1434

    Article  Google Scholar 

  37. Yubing T, Cheikh FA, Guraya FFE, Konik H, Trémeau A (2011) A spatiotemporal saliency model for video surveillance. Cogn Comput 3(1):241–263

    Article  Google Scholar 

Download references

Acknowledgments

This work was supported by the National Science Foundation (grant numbers CCF-1317433), the Office of Naval Research (N00014-13-1-0563), and the Army Research Office (W911NF-12-1-0433). The authors affirm that the views expressed herein are solely their own, and do not represent the views of the United States government or any agency thereof.

Author information

Authors and Affiliations

  1. Research and Development Group, Opera Solutions, 12230 El Camino Real, San Diego, CA, 92130, USA

    Farhan Baluch

  2. Department of Computer Science, Psychology & Neuroscience Graduate Program, University of Southern California, 3641 Watt Way, HNB 10, Los Angeles, CA, 90089, USA

    Laurent Itti

Authors
  1. Farhan Baluch
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Laurent Itti
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Farhan Baluch .

Editor information

Editors and Affiliations

  1. IBM Corp., Durham, North Carolina, USA

    Aaron K. Baughman

  2. Nokia Inc., Sunnyvale, California, USA

    Jiang Gao

  3. Google Inc., Mountain View, California, USA

    Jia-Yu Pan

  4. 4i, Inc., Carlsbad, California, USA

    Valery A. Petrushin

Rights and permissions

Reprints and permissions

Copyright information

© 2015 Springer International Publishing Switzerland

About this chapter

Cite this chapter

Baluch, F., Itti, L. (2015). Mining Videos for Features that Drive Attention. In: Baughman, A., Gao, J., Pan, JY., Petrushin, V. (eds) Multimedia Data Mining and Analytics. Springer, Cham. https://doi.org/10.1007/978-3-319-14998-1_14

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-14998-1_14

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-14997-4

  • Online ISBN: 978-3-319-14998-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation