Skip to main content
SpringerLink
Log in

Face for Ambient Interface

  • Chapter
Ambient Intelligence in Everyday Life

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 3864))

Abstract

The human face is used to identify other people, to regulate the conversation by gazing or nodding, to interpret what has been said by lip reading, and to communicate and understand social signals, including affective states and intentions, on the basis of the shown facial expression. Machine understanding of human facial signals could revolutionize user-adaptive social interfaces, the integral part of ambient intelligence technologies. Nonetheless, development of a face-based ambient interface that detects and interprets human facial signals is rather difficult. This article summarizes our efforts in achieving this goal, enumerates the scientific and engineering issues that arise in meeting this challenge and outlines recommendations for accomplishing this objective.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Aarts, E.: Ambient Intelligence – Visualizing the Future. In: Proc. Conf. Smart Objects & Ambient Intelligence (2005), http://www.soc-eusai2005.org/

  2. Ambady, N., Rosenthal, R.: Thin Slices of Expressive Behavior as Predictors of Interpersonal Consequences: A Meta-Analysis. Psychological Bulletin 111(2), 256–274 (1992)

    Article  Google Scholar 

  3. Anderson, K., McOwan, P.W.: A Real-Time Automated System for Recognition of Human Facial Expressions. IEEE Trans. Systems, Man, and Cybernetics, Part B 36(1), 96–105 (2006)

    Article  Google Scholar 

  4. Aristotle: Physiognomonica. In: Ross, W.D. (ed.) The works of Aristotle, pp. 805–813. Clarendon, Oxford (1913)

    Google Scholar 

  5. Baker, S., Matthews, I., Xiao, J., Gross, R., Kanade, T.: Real-time non-rigid driver head tracking for driver mental state estimation. In: Proc. World Congress on Intelligent Transportation Systems (2004), http://www.ri.cmu.edu/projects/project_448.html

  6. Barron, J., Fleet, D., Beauchemin, S.: Performance of optical flow techniques. J. Computer Vision 12(1), 43–78 (1994)

    Article  Google Scholar 

  7. Bartlett, M.S., Hager, J.C., Ekman, P., Sejnowski, T.J.: Measuring facial expressions by computer image analysis. Psychophysiology 36, 253–263 (1999)

    Article  Google Scholar 

  8. Bartlett, M.S., Littlewort, G., Lainscsek, C., Fasel, I., Movellan, J.R.: Machine Learning Methods for Fully Automatic Recognition of Facial Expressions and Facial Actions. In: Proc. Conf. Systems, Man, and Cybernetics, vol. 1, pp. 592–597 (2004)

    Google Scholar 

  9. Bassili, J.N.: Facial Motion in the Perception of Faces and of Emotional Expression. J. Experimental Psychology 4, 373–379 (1978)

    Google Scholar 

  10. Black, M., Yacoob, Y.: Recognizing facial expressions in image sequences using local parameterized models of image motion. Computer Vision 25(1), 23–48 (1997)

    Article  Google Scholar 

  11. Bobick, A.F., Davis, J.W.: The Recognition of Human Movement Using Temporal Templates. IEEE Trans. Pattern Analysis and Machine Intelligence 23(3), 257–267 (2001)

    Article  Google Scholar 

  12. Bowyer, K.W.: Face Recognition Technology – Security vs. Privacy. IEEE Technology and Society Magazine 23(1), 9–19 (2004)

    Article  Google Scholar 

  13. Bruce, V.: Recognizing Faces. Lawrence Erlbaum Assoc., Hove (1986)

    Google Scholar 

  14. Cohen, I., Sebe, N., Garg, A., Chen, L.S., Huang, T.S.: Facial expression recognition from video sequences – temporal and static modeling. Computer Vision and Image Understanding 91, 160–187 (2003)

    Article  Google Scholar 

  15. Cohn, J.F., Reed, L.I., Ambadar, Z., Xiao, J., Moriyama, T.: Automatic analysis and recognition of brow actions and head motion in spontaneous facial behavior. In: Proc. Conf. Systems, Man and Cybernetics, vol. 1, pp. 610–616 (2004)

    Google Scholar 

  16. Cohn, J.F., Zlochower, A.J., Lien, J., Kanade, T.: Automated face analysis by feature point tracking has high concurrent validity with manual faces coding. Psychophysiology 36, 35–43 (1999)

    Article  Google Scholar 

  17. Cristinacce, D., Cootes, T.F.: A Comparison of Shape Constrained Facial Feature Detectors. In: Proc. Conf. Automatic Face and Gesture Recognition, pp. 375–380 (2004)

    Google Scholar 

  18. Darwin, C.: The expression of the emotions in man and animals. University of Chicago Press, Chicago (1965 / 1872)

    Book  Google Scholar 

  19. DeCarlo, D., Metaxas, D.: The integration of optical flow and deformable models with applications to human face shape and motion estimation. In: Proc. Conf. Computer Vision and Pattern Recognition, pp. 231–238 (1996)

    Google Scholar 

  20. Dishman, E.: Inventing wellness systems for aging in place. IEEE Computer Magazine, Spec. Issue on Computing and the Aging 37(5), 34–41 (2004)

    Article  Google Scholar 

  21. Donato, G., Bartlett, M.S., Hager, J.C., Ekman, P., Sejnowski, T.J.: Classifying Facial Actions. IEEE Trans. Pattern Analysis and Machine Intelligence 21(10), 974–989 (1999)

    Article  Google Scholar 

  22. Ekman, P.: Emotions Revealed. Times Books, New York (2003)

    Google Scholar 

  23. Ekman, P., Friesen, W.V.: The repertoire of nonverbal behavior. Semiotica 1, 49–98 (1969)

    Article  Google Scholar 

  24. Ekman, P., Friesen, W.V.: Unmasking the face. Prentice-Hall, New Jersey (1975)

    Google Scholar 

  25. Ekman, P., Friesen, W.V.: Facial Action Coding System. Consulting Psychologist Press, Palo Alto (1978)

    Google Scholar 

  26. Ekman, P., Friesen, W.V., Hager, J.C.: Facial Action Coding System. A Human Face, Salt Lake City (2002)

    Google Scholar 

  27. Fasel, I., Fortenberry, B., Movellan, J.R.: GBoost: A generative framework for boosting with applications to real-time eye coding. Computer Vision and Image Understanding, under review, http://mplab.ucsd.edu/publications/

  28. Friedman, J., Hastie, T., Tibshirani, R.: Additive logistic regression: A statistical view of boosting. Annals of Statistics 28(2), 337–374 (2000)

    Article  MathSciNet  MATH  Google Scholar 

  29. Gokturk, S.B., Bouguet, J.Y., Tomasi, C., Girod, B.: Model-based face tracking for view-independent facial expression recognition. In: Proc. Conf. Automatic Face and Gesture Recognition, pp. 272–278 (2002)

    Google Scholar 

  30. Gross, T.: Ambient Interfaces – Design Challenges and Recommendations. In: Proc. Conf. Human-Computer Interaction, pp. 68–72 (2003)

    Google Scholar 

  31. Gu, H., Ji, Q.: Information extraction from image sequences of real-world facial expressions. Machine Vision and Applications 16(2), 105–115 (2005)

    Article  Google Scholar 

  32. Guo, G., Dyer, C.R.: Learning From Examples in the Small Sample Case – Face Expression Recognition. IEEE Trans. Systems, Man, and Cybernetics, Part B 35(3), 477–488 (2005)

    Article  Google Scholar 

  33. Haykin, S., de Freitas, N., (eds): Special Issue on Sequential State Estimation. Proceedings of the IEEE 92(3), 399–574 (2004)

    Google Scholar 

  34. Isard, M., Blake, A.: Condensation – conditional density propagation for visual tracking. J. Computer Vision 29(1), 5–28 (1998)

    Article  Google Scholar 

  35. Jacobs, D.W., Osadchy, M., Lindenbaum, M.: What Makes Gabor Jets Illumination Insensitive, http://rita.osadchy.net/papers/gabor-3.pdf

  36. Kalman, R.E.: A new approach to linear filtering and prediction problems. Trans. ASME J. Basic Eng. 82, 35–45 (1960)

    Article  Google Scholar 

  37. Kanade, T., Cohn, J., Tian, Y.: Comprehensive database for facial expression analysis. In: Proc. Conf. Automatic Face and Gesture Recognition, pp. 46–53 (2000)

    Google Scholar 

  38. Keltner, D., Ekman, P.: Facial Expression of Emotion. In: Lewis, M., Haviland-Jones, J.M. (eds.) Handbook of Emotions, 2nd edn., pp. 236–249. The Guilford Press, New York (2004)

    Google Scholar 

  39. Li, S.Z., Jain, A.K. (eds.): Handbook of Face Recognition. Springer, New York (2005)

    MATH  Google Scholar 

  40. van Loenen, E.J.: On the role of Graspable Objects in the Ambient Intelligence Paradigm. In: Proc. Conf. Smart Objects (2003), http://www.grenoble-soc.com/

  41. Lucas, B.D., Kanade, T.: An iterative image registration technique with an application to stereo vision. In: Proc. Conf. Artificial Intelligence, pp. 674–679 (1981)

    Google Scholar 

  42. Martinez, A.M.: Matching expression variant faces. Vision Research 43, 1047–1060 (2003)

    Article  Google Scholar 

  43. Mase, K.: Recognition of facial expression from optical flow. IEICE Transactions E74(10), 3474–3483 (1991)

    Google Scholar 

  44. Moghaddam, B., Pentland, A.: Probabilistic Visual Learning for Object Recognition. IEEE Trans. Pattern Analysis and Machine Intelligence 19(7), 696–710 (1997)

    Article  Google Scholar 

  45. Norman, D.A.: The Invisible Computer. MIT Press, Cambridge (1999)

    Google Scholar 

  46. Ortony, A., Turner, T.J.: What is basic about basic emotions? Psychological Review 74, 315–341 (1990)

    Article  Google Scholar 

  47. Pantic, M.: Face for Interface. In: Pagani, M. (ed.) The Encyclopedia of Multimedia Technology and Networking 1, pp. 308–314. Idea Group Reference, Hershy (2005)

    Google Scholar 

  48. Pantic, M., Patras, I.: Detecting facial actions and their temporal segments in nearly frontal-view face image sequences. In: Proc. Conf. Systems, Man, and Cybernetics (2005)

    Google Scholar 

  49. Pantic, M., Patras, I.: Dynamics of Facial Expressions – Recognition of Facial Actions and their Temporal Segments from Face Profile Image Sequences. IEEE Trans. Systems, Man, and Cybernetics, Part B 36 (2006)

    Google Scholar 

  50. Pantic, M., Rothkrantz, L.J.M.: Expert system for automatic analysis of facial expression. Image and Vision Computing 18(11), 881–905 (2000)

    Article  Google Scholar 

  51. Pantic, M., Rothkrantz, L.J.M.: Automatic Analysis of Facial Expressions – The State of the Art. IEEE Trans. Pattern Analysis and Machine Intelligence 22(12), 1424–1445 (2000)

    Article  Google Scholar 

  52. Pantic, M., Rothkrantz, L.J.M.: Toward an Affect-Sensitive Multimodal Human-Computer Interaction. Proceedings of the IEEE, Spec. Issue on Human-Computer Multimodal Interface 91(9), 1370–1390 (2003)

    Google Scholar 

  53. Pantic, M., Rothkrantz, L.J.M.: Facial Action Recognition for Facial Expression Analysis from Static Face Images. IEEE Trans. Systems, Man, and Cybernetics, Part B 34(3), 1449–1461 (2004)

    Article  Google Scholar 

  54. Pantic, M., Rothkrantz, L.J.M.: Case-based reasoning for user-profiled recognition of emotions from face images. In: Proc. Conf. Multimedia and Expo, vol. 1, pp. 391–394 (2005)

    Google Scholar 

  55. Pantic, M., Sebe, N., Cohn, J.F., Huang, T.: Affective Multimodal Human-Computer Interaction. In: Proc. ACM Conf. Multimedia (2005)

    Google Scholar 

  56. Pantic, M., Valstar, M.F., Rademaker, R., Maat, L.: Web-based database for facial expression analysis. In: Proc. Conf. Multimedia and Expo (2005), http://www.mmifacedb.com/

  57. Patras, I., Pantic, M.: Particle Filtering with Factorized Likelihoods for Tracking Facial Features. In: Proc. Conf. Automatic Face and Gesture Recognition, pp. 97–102 (2004)

    Google Scholar 

  58. Patras, I., Pantic, M.: Tracking Deformable Motion. In: Proc. Conf. Systems, Man, and Cybernetics (2005)

    Google Scholar 

  59. Pentland, A.: Looking at people – Sensing for ubiquitous and wearable computing. IEEE Trans. Pattern Analysis and Machine Intelligence 22(1), 107–119 (2000)

    Article  Google Scholar 

  60. Pentland, A., Moghaddam, B., Starner, T.: View-Based and Modular Eigenspaces for Face Recognition. In: Proc. Conf. Computer Vision and Pattern Recognition, pp. 84–91 (1994)

    Google Scholar 

  61. Picard, R.W.: Affective Computing. MIT Press, Cambridge (1997)

    Book  Google Scholar 

  62. Pitt, M.K., Shephard, N.: Filtering via simulation: auxiliary particle filtering. J. Amer. Stat. Assoc. 94, 590–599 (1999)

    Article  MATH  Google Scholar 

  63. Preece, J., Rogers, Y., Sharp, H.: Interaction Design – Beyond Human-Computer Interaction. John Wiley & Sons, New York (2002)

    Google Scholar 

  64. Raisinghani, M.S., Benoit, A., Ding, J., Gomez, M., Gupta, K., Gusila, V., Power, D., Schmedding, O.: Ambient Intelligence – Changing Forms of Human-Computer Interaction and their Social Implications. J. Digital Information 5(4), 1–8 (2004)

    Google Scholar 

  65. Remagnino, P., Foresti, G.L.: Ambient Intelligence – A New Multidisciplinary Paradigm. IEEE Trans. Systems, Man, and Cybernetics, Part A, Spec. Issue on Ambient Intelligence 35(1), 1–6 (2005)

    Article  Google Scholar 

  66. Rowley, H., Baluja, S., Kanade, T.: Neural Network-Based Face Detection. IEEE Trans. Pattern Analysis and Machine Intelligence 20(1), 23–38 (1998)

    Article  Google Scholar 

  67. Russell, J.A., Fernandez-Dols, J.M. (eds.): The Psychology of Facial Expression. Cambridge University Press, Cambridge (1997)

    Google Scholar 

  68. Samal, A., Iyengar, P.A.: Automatic recognition and analysis of human faces and facial expressions: A survey. Pattern Recognition 25(1), 65–77 (1992)

    Article  Google Scholar 

  69. Scherer, K.R., Ekman, P. (eds.): Handbook of methods in non-verbal behavior research. Cambridge University Press, Cambridge (1982)

    Google Scholar 

  70. Schmidt, K.L., Cohn, J.F.: Dynamics of facial expression: Normative characteristics and individual differences. In: Proc. Conf. Multimedia and Expo, pp. 547–550 (2001)

    Google Scholar 

  71. Shadbolt, N.: Ambient Intelligence. IEEE Intelligent Systems 18(4), 2–3 (2003)

    Article  Google Scholar 

  72. Shi, J., Tomasi, C.: Good features to track. In: Proc. Conf. Computer Vision and Pattern Recognition, pp. 593–600 (1994)

    Google Scholar 

  73. Stephanidis, C., Akoumianakis, D., Sfyrakis, M., Paramythis, A.: Universal accessibility in HCI. In: Proc. ERCIM Workshop. User Interfaces For All (1998), http://ui4all.ics.forth.gr/UI4ALL-98/proceedings.html

  74. Streitz, N., Nixon, P.: The Disappearing Computer. ACM Communications, Spec. Issue on The Disappearing Computer 48(3), 33–35 (2005)

    Google Scholar 

  75. Sung, K.K., Poggio, T.: Example-Based Learning for View-Based Human Face Detection. IEEE Trans. Pattern Analysis and Machine Intelligence 20(1), 39–51 (1998)

    Article  Google Scholar 

  76. Tao, H., Huang, T.S.: Connected vibrations – a model analysis approach to non-rigid motion tracking. In: Proc. Conf. Computer Vision and Pattern Recognition, pp. 735–740 (1998)

    Google Scholar 

  77. Tian, Y., Kanade, T., Cohn, J.F.: Recognizing action units for facial expression analysis. IEEE Trans. Pattern Analysis & Machine Intelligence 23(2), 97–115 (2001)

    Article  Google Scholar 

  78. Tian, Y.L., Kanade, T., Cohn, J.F.: Facial Expression Analysis. In: Li, S.Z., Jain, A.K. (eds.) Handbook of Face Recognition. Springer, New York (2005)

    Google Scholar 

  79. Tscheligi, M.: Ambient Intelligence – The Next Generation of User Centeredness. ACM Interactions, Spec. Issue on Ambient Intelligence 12(4), 20–21 (2005)

    Article  Google Scholar 

  80. Valstar, M., Pantic, M., Patras, I.: Motion History for Facial Action Detection from Face Video. In: Proc. Conf. Systems, Man and Cybernetics, vol. 1, pp. 635–640 (2004)

    Google Scholar 

  81. Valstar, M., Patras, I., Pantic, M.: Facial Action Unit Detection using Probabilistic Actively Learned Support Vector Machines on Tracked Facial Point Data. In: Proc. Conf. Computer Vision and Pattern Recognition (2005)

    Google Scholar 

  82. Viola, P., Jones, M.: Robust real-time object detection. In: Proc. Int’l Conf. Computer Vision, Workshop on Statistical and Computation Theories of Vision (2001)

    Google Scholar 

  83. Vukadinovic, D., Pantic, M.: Fully automatic facial feature point detection using Gabor feature based boosted classifiers. In: Proc. Conf. Systems, Man and Cybernetics (2005)

    Google Scholar 

  84. Weiser, M.: The world is not a desktop. ACM Interactions 1(1), 7–8 (1994)

    Article  Google Scholar 

  85. Xiao, J., Baker, S., Matthews, I., Kanade, T.: Real-time Combined 2D+3D Active Appearance Models. In: Proc. Conf. Computer Vision and Pattern Recognition, vol. 2, pp. 535–542 (2004)

    Google Scholar 

  86. Yacoob, Y., Davis, L., Black, M., Gavrila, D., Horprasert, T., Morimoto, C.: Looking at People in Action. In: Cipolla, R., Pentland, A. (eds.) Computer Vision for Human-Machine Interaction, pp. 171–187. Cambridge University Press, Cambridge (1998)

    Chapter  Google Scholar 

  87. Yang, M.H., Kriegman, D.J., Ahuja, N.: Detecting faces in images: A survey. IEEE Trans. Pattern Analysis and Machine Intelligence 24(1), 34–58 (2002)

    Article  Google Scholar 

  88. Zhai, S., Bellotti, V.: Introduction to Sensing-Based Interaction. ACM Trans. Computer-Human Interaction, Spec. Issue on Sensing-Based Interaction 12(1), 1–2 (2005)

    Article  Google Scholar 

  89. Zhang, Y., Ji, Q.: Active and Dynamic Information Fusion for Facial Expression Understanding from Image Sequence. IEEE Trans. Pattern Analysis & Machine Intelligence 27(5), 699–714 (2005)

    Article  Google Scholar 

  90. Zhao, W., Chellappa, R., Rosenfeld, A., Phillips, P.J.: Face Recognition – A literature survey. ACM Computing Surveys 35(4), 399–458 (2003)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Computing Department, Imperial College, 180 Queens Gate, London, SW7 2AZ, U.K.

    Maja Pantic

Authors
  1. Maja Pantic
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Ambient Intelligence Lab, Carnegie Mellon University, CIC-2218, 4720 Forbes Avenue, 15213, Pittsburgh, PA, USA

    Yang Cai

  2. Informatika Fakultatea, University of the Basque Country, Manuel Lardizabal 1, 20018, Donostia, Spain

    Julio Abascal

Rights and permissions

Reprints and permissions

Copyright information

© 2006 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Pantic, M. (2006). Face for Ambient Interface. In: Cai, Y., Abascal, J. (eds) Ambient Intelligence in Everyday Life. Lecture Notes in Computer Science(), vol 3864. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11825890_2

Download citation

  • DOI: https://doi.org/10.1007/11825890_2

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-37785-6

  • Online ISBN: 978-3-540-37788-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation