Skip to main content

Advertisement

Springer Nature Link
Log in

Affective-aware tutoring platform for interactive digital television

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Interactive Digital TeleVision (IDTV) is emerging as a potentially important medium for learning at home. This paper presents a novel affective-aware tutoring platform for IDTV which makes use of automatic facial emotion recognition to improve the tutor-student relationship. The system goes further than simply broadcasting an interactive educational application by allowing the personalization of the course content. The tutor can easily access academic information relating to the students and also emotional information captured from learners’ facial expressions. In this way, depending on their academic and affective progress, the tutor can send personal messages or extra educational contents for improving students’ learning. In order to include these features it was necessary to address some important technical challenges derived from IDTV hardware and software restrictions. The system has been successfully tested with real students and tutors in a non-laboratory environment. Our system tries to advance in the challenge of providing to distance learning systems with the perceptual abilities of human teachers with the final aim of improving students learning experience and outcome. Nevertheless, there is still relatively little understanding of the impact of affect on students’ behaviour and learning and of the dynamics of affect during learning with software. Systems like ours would make it possible to attack these relevant open questions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  1. Abadía D, Navamuel JJ, Vea-Murguia J, Alvarez F, Menendez JM, Sanchez FA, Fernandez G, Rovira M, Domech A (2009) i-LAB: Distributed laboratory network for interactive TV set-top box and services testing, IEEE 13th International Symposium on Consumer Electronics 773–776

  2. Affdex (2013) Available: http://www.affectiva.com/affdex/

  3. Alexander S, Sarrafzadeh A (2004) Interfaces that adapt like humans, computer human interaction 641–645

  4. An KH, Chung MJ (2009) Cognitive face analysis system for future interactive TV. IEEE Transactions on Consumer Eectronics 55(4):2271–2279

    Article  Google Scholar 

  5. Baker RR, D’Mello SK, Rodrigo MMT, Graesser AC (2010) Better to be frustrated than bored: The incidence, persistence,a nd impact of learners’cognitive–affective states during interactions with three different computer-based learning environments. Int J Human-Computer Studies 68:223–241

    Article  Google Scholar 

  6. Bates PJ (2005) Learning through iDTV–results of t-learning study. Europ Confer Interact Telev 137–138

  7. Bellotti F, Pellegrino M, Tsampoulatidis I, Vrochidis S, Lhoas P, Bo G, De Gloria A, Kompatsiaris I (2010) An integrated framework for personalized T-Learning in Cases on transnational learning and technologically enabled environments 118

  8. Brusilovsky P, Schwarz E, Weber G (1996) ELM–ART: An intelligent tutoring system on World Wide Web. In Proceedings 3rd international conference on intelligent tutoring systems, ITS 96, pp. 261–269.

  9. Brusilovsky P, Vassileva J (2003) Course sequencing techniques for large-scale web-based education. International Journal of Continuing Engineering Education and Lifelong Learning 13(1–2):75–94

    Article  Google Scholar 

  10. Burleson W (2006) Affective learning companions: Strategies for empathetic agents with real-time multimodal affective sensing to foster meta-cognitive and meta-affective approaches to learning, motivation, and perseverance. Doctoral Thesis, Massachusetts In-stitute of Technology.

  11. Caridakis G, Malatesta L, Kessous L, Amir N, Paouzaiou A, Karpouzis K (2006) Modeling naturalistic affective states via facial and vocal expression recognition. Intern Confer Multim Interf 146–154

  12. Chen C (2008) Intelligent web-based learning system with personalized learning path guidance. Computers & Education vol 51:787–814

    Article  Google Scholar 

  13. Clark D (2013) MOOCs: Taxonomy of 8 types of MOOC. Donald Clark Paln B. http://donaldclarkplanb.blogspot.com.es/2013/04/moocs-taxonomy-of-8-types-of-mooc.html

  14. Cmolik L, Mikovec Z, Slavik P, Mannova B (2007) Personalized e-learning in interactive digital television environment, in IADIS International Conference WWW/Internet, 35–39

  15. Conati C (2002) Probabilistic assessment of user’s emotions in educational games. Journal of Applied Artificial Intelligence 16:555–575

    Article  Google Scholar 

  16. Conati C, Zhao X (2004) Building and evaluating an intelligent pedagogical agent to improve the effectiveness of an educational game, Proceedings of the Ninth International Conference on Intelligent User Interface, pp. 6–13

  17. D’Mello S, Jackson T, Craig S, Morgan B, Chipman P, White H, El Kaliouby R, Picard RW, Graesser A (2008) Auto tutor detects and responds to learners affective and cognitive states. In: Proceedings of the Workshop on Emotional and Cognitive Issues at the International Conference of Intelligent Tutoring Systems, pp. 23–27

  18. D’Mello SK, Lehman B, Graesser A (2011) A motivationally supportive affect-sensitive autotutor. In New Perspectives on Affect and Learning Technologies, 113–126. New York: Springer

  19. Damásio M, Quico C (2004) T-learning and interactive television edutainment: The Portuguese case study. ED-Media 4511–4518

  20. IPEA: Instituto de Pesquisa Economica Aplicada (2013) Panorama da Comunicacao e das telecomunicacoes no Brasil, (in portuguesse): http://www4.planalto.gov.br/brasilconectado/noticias/panorama-da-comunicacoes-e-telecomunicacoes-no-brasil/

  21. de Vicente A, Pain H (2002) Informing the detection of the students’ motivational state: An empirical study, Proceedings of the Sixth International Conference on Intelligent Tutoring Systems, pp. 933–943

  22. Desmarais MC, d Baker RS (2012) A review of recent advances in learner and skill modeling in intelligent learning environments. User Model User-Adap Inter 22(1–2):9–38

    Article  Google Scholar 

  23. dos Santos DT, do Vale DT, Meloni LG (2007) Digital TV and distance learning: Potentials and limitations. In: Proceedings of the 36 th Annual Conference on Frontiers in Education pp. 1–6

  24. Dosi A, Prario B (2004) New frontiers of T-learning: The emergence of interactive digital broadcasting learning services in Europe. ED-Media, pp. 4831–4836

  25. Ekman P, Friesen WV, Hager JC (2002) Facial action coding system, Research Nexus eBook

    Google Scholar 

  26. eMotion©: emotion recognition software, 2013. Available: http://www.visual-recognition.nl/Demo.html

  27. Face API technical specifications brochure. Available:http://www.seeingmachines.com/pdfs/brochures/faceAPI-Brochure.pdf

  28. Fragopanagos N, Taylor JG (2005) Emotion recognition in human–computer interaction. Neural Netw 18:389–405

    Article  Google Scholar 

  29. Gee JP (2004) Situated language and learning: A critique of traditional schooling. Routledge, Taylor & Francis, London

    Google Scholar 

  30. Graesser AC, D’Mello S, Person NK (2009) Metaknowledge in tutoring. In: Hacker D, Donlosky J, Graesser AC (eds) Handbook of metacognition in education. Taylor & Francis, Mahwah

    Google Scholar 

  31. Hammal Z, Couvreur L, Caplier A, Rombaut M (2007) Facial expression classification: An approach based on the fusion of facial deformations using the transferable belief model. Int J Approx Reason 46:542–567

    Article  Google Scholar 

  32. Henze N, Nejdl W (2001) Adaptation in open corpus hypermedia. Int J Artif Intell Educ 12(4):325–350

    Google Scholar 

  33. Hone K (2006) Empathic agents to reduce user frustration: the effects of varying agent characteristics. Interact Comput 18:227–245

    Article  Google Scholar 

  34. Hupont I, Baldassarri S, Cerezo E (2013) Facial emotional classification: From a discrete perspective to a continuous emotional space. Pattern Anal Applic 16(1):41–54

    Article  MathSciNet  Google Scholar 

  35. Hupont I, Ballano S, Baldassarri S, Cerezo E (2011) Scalable multimodal fusion for continuous affect sensing, Proc. IEEE Workshop on Affective Computational Intelligence (WACI 2011), Symposium Series on Computational Intelligence, ISBN 978-1-61284-082-6, pp. 68-75

  36. Hupont I, Ballano S, Cerezo E, Baldassarri S (2013) From a discrete perspective of emotions to continuous, dynamic and multimodal affect sensing, Advances in Emotion Recognition, ISBN 978-1118130667, Wiley-Blackwell

  37. Isen AM (2000) Positive affect and decision making. In: Lewis M, Haviland J (eds) Handbook of emotions. Guilford, New York

    Google Scholar 

  38. Kalman RE (1960) A new approach to linear filtering and prediction problems. Transactions of the ASME - Journal of Basic Engineering, Series D 82:34–45

    Google Scholar 

  39. Kapoor A, Burleson W, Picard RW (2007) Automatic prediction of frustration. International Journal of Human-Computer Studies 65:724–736

    Article  Google Scholar 

  40. Kim J, Kang S (2011) An ontology-based personalized target advertisement system on interactive TV. IEEE International Conference on Consumer Electronics, pp. 895–896

  41. Kort B, Reilly R, Picard R (2001) An affective model of interplay between emotions and learning: reengineering educational pedagogy—building a learning companion, Proceedings IEEE International Conference on Advanced Learning Technology: Issues, Achievements and Challenges. IEEE Computer Society, pp.43–48.

  42. Kumar P, Yildirim EA (2005) Minimum-volume enclosing ellipsoids and core sets. J Optim Theory Appl 126:1–21

    Article  MATH  MathSciNet  Google Scholar 

  43. López-Nores M, Blanco-Fernández Y, Pazos-Arias JJ, García-Duque J, Ramos-Cabrer M, Gil-Solla A, Díaz-Redondo RP, Fernández-Vilas A (2009) Receiver-side semantic reasoning for digital TV personalization in the absence of return channels. Multimed Tools Appl 41:407–436

    Article  Google Scholar 

  44. Mandryk RL, Atkins MS (2007) A fuzzy physiological approach for continuously modeling emotion during interaction with play technologies. International Journal of Human-Computer Studies 65(4):329–347

    Article  Google Scholar 

  45. McDuff D, Kaliouby RE, Picard RW (2012) Crowdsourcing facial responses to online videos. IEEE Trans on Affective Computing 3(4):456–468

    Article  Google Scholar 

  46. Merrill DC, Reiser BJ, Ranney M, Trafton JG (1992) Effective tutoring techniques: A comparison of human tutors and intelligent tutoring systems. J Learn Sci 2(3):277–305

    Article  Google Scholar 

  47. Montpetit M, Klym N, Mirlacher T (2011) The future of IPTV. Multimedia Tools and Applications 53:519–532

    Article  Google Scholar 

  48. Morrell DR, Stirling WC (2003) An extended set-valued Kalman filter, In Proceedings of ISIPTA, 396–407

  49. Nicolaou MA, Gunes H, Pantic M (2011) Continuous prediction of spontaneous affect from multiple cues and modalities in valence-arousal space. IEEE Trans Affect Comput 2(2):92–105

    Article  Google Scholar 

  50. Papanikolaou KA, Grigoriadou M (2002) Towards new forms of knowledge communication: The adaptive dimension of a webbased learning environment. Computers and Education 39:333–360

    Article  Google Scholar 

  51. Pazos-Arias JJ, López-Nores M, García-Duque J, Díaz-Redondo RP, Blanco-Fernández Y, Ramos-Cabrer M, Gil-Solla A, Fernández-Vilas A (2008) Provision of distance learning services over Interactive Digital TV with MHP. Comput Educ 50:927–949

    Article  Google Scholar 

  52. Petridis S, Gunes H, Kaltwang S, Pantic M, Static vs. dynamic modeling of human nonverbal behavior from multiple cues and modalities, Proceedings of the International Conference on Multimodal Interfaces, 2009, pp. 23–30

  53. Picard RW (2000) Affective computing, The MIT Press

  54. Picard RW, Papert S, Bender W, Blumberg B, Breazeal C, Cavallo D, Machover T, Resnick M, Roy D, Strohecker C (2004) Affective learning-a manifesto. BT Technology Journal 22(4)

  55. Plutchik R (1980) Emotion: A psychoevolutionary synthesis. Harper & Row, New York

    Google Scholar 

  56. Porayska-Pomsta K, Mavrikis M, Pain H (2008) Diagnosing and acting on student affect: The tutor’s perspective. User Model User-Adap Inter 18(1–2):125–173

    Article  Google Scholar 

  57. Rey-López M, Díaz-Redondo RP, Fernández-Vilas A, Pazos-Arias JJ, López-Nores M, García-Duque J, Gil-Solla A, Ramos-Cabrer M (2008) T-MAESTRO and its authoring tool: Using adaptation to integrate entertainment into personalized t-learning. Multimedia Tools and Applications 40:409–451

    Article  Google Scholar 

  58. Rey-López M, Fernández-Vilas A, Díaz-Redondo RP (2006) A model for personalized learning through IDTV. In: Wade V, Ashman H, Smyth B (eds.) AH 2006, LNCS 4018, pp. 457–461

  59. Roland H (2000) Logically optimal curriculum sequences for adaptive hypermedia systems. In International conference on adaptive hypermedia and adaptive web-based system, pp. 121–132.

  60. Russell JA (1980) A circumplex model of affect. Journal of personality and social psychology 39(6):1161–1178

    Article  Google Scholar 

  61. Russell T (1997) Technology wars: Winners and losers. Educom Review 32(2):44–46

    Google Scholar 

  62. Sarrafzadeh A, Alexander S, Dadgostar F, Fan C, Bigdeli A (2008) How do you know that I don’t understand? A look at the future of intelligent tutoring systems, Computers in Human Behavior vol 24:1342–1363

    Google Scholar 

  63. Schiaffino S, Garcia P, Amandi A (2008) eTeacher: Providing personalized assistance to e-learning students. Computers & Education vol 51:1744–1754

    Article  Google Scholar 

  64. Smith ASG, Blandford A (2003) MLTutor: An application of machine learning algorithms for an adaptive web-based information system. Int J Artif Intell Educ 13(2–4):233–260

    Google Scholar 

  65. Snow R, Farr M (1987) Cognitive-conative-affective processes in aptitude, learning, and instruction: An introduction. Conative and affective process analysis 3:1–10

    Google Scholar 

  66. Soleymani M, Pantic M (2013) Emotional Aware TV, Proceedings of TVUX-2013: Workshop on Exploring and Enhancing the User Experience for TV at ACM CHI 2013

  67. Soyel H, Demirel H (2007) Facial expression recognition using 3D facial feature distances. Lect Notes Comput Sci 4633:831–883

    Article  Google Scholar 

  68. Suraweera P, Mitrovic A (2002) KERMIT: A constraint-based tutor for database modeling. In Intelligent Tutoring Systems, 377–387. Springer Berlin Heidelberg

  69. VanLehn K (2006) The behavior of tutoring systems. International journal of artificial intelligence in education 16(3):227–265

    Google Scholar 

  70. Videodrips. Available: http://www.interactivetvweb.org/tutorials/mhp/content_referencing

  71. Wallhoff F (2006) Facial expressions and emotion database, Technische Universität München Available: http://www.mmk.ei.tum.de/~waf/fgnet/feedtum.html

  72. Whissell CM (1989) The dictionary of affect in language, Emotion: Theory, research and experience, vol 4. The Measurement of Emotions, New York

    Google Scholar 

  73. Whissell C, Whissell’s dictionary of affect in language technical manual and user’s guide. Available: http://www.hdcus.com/manuals/wdalman.

  74. Witten I, Frank E (2005) Data mining: Practical machine learning tools and techniques, 2nd edn. Morgan Kaufmann, San Francisco

    Google Scholar 

  75. Yeasin M, Bullot B, Sharma R (2006) Recognition of facial expressions and measurement of levels of interest from video. IEEE Transactions on Multimedia 8:500–508

    Article  Google Scholar 

  76. Zapata-Ros M, El diseño instruccional de los MOOCs y el de los nuevos cursos online abiertos personalizados (POOCs), 2013 (In Spanish) [Preprint] http://hdl.handle.net/10760/19744

Download references

Acknowledgments

This work has been partly financed by the Spanish “Dirección General de Investigación”, contract number TIN2011-24660, by the CYTED, contract number 512RT0461, by the Mechatronics and Systems Group (SISTRONIC) of the Aragon Institute of Technology and by the Spanish “Ministerio de Ciencia e Innovación” in the context of the QuEEN project, contract number IPT-2011-1235-430000.

Author information

Authors and Affiliations

  1. GIGA-AffectiveLab, Computer Science Department, Engineering Research Institute of Aragon (I3A), Universidad of Zaragoza, Zaragoza, Spain

    Sandra Baldassarri & Eva Cerezo

  2. Multimedia Technologies Division, Aragon Institute of Technology, Zaragoza, Spain

    Isabelle Hupont & David Abadía

Authors
  1. Sandra Baldassarri
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Isabelle Hupont
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. David Abadía
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Eva Cerezo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sandra Baldassarri.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Baldassarri, S., Hupont, I., Abadía, D. et al. Affective-aware tutoring platform for interactive digital television. Multimed Tools Appl 74, 3183–3206 (2015). https://doi.org/10.1007/s11042-013-1779-z

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-013-1779-z

Keywords