Abstract
In marketing research, one of the most exciting, innovative, and promising trends is quantification of customer interest. This paper presents a deep learning-based system for monitoring customer behavior specifically for detection of interest. The proposed system first measures customer attention through head pose estimation. For those customers whose heads are oriented toward the advertisement or the product of interest, the system further analyzes the facial expressions and reports customers’ interest. The proposed system starts by detecting frontal face poses; facial components important for facial expression recognition are then segmented and an iconized face image is generated; finally, facial expressions are analyzed using the confidence values of obtained iconized face image combined with the raw facial images. This approach fuses local part-based features with holistic facial information for robust facial expression recognition. With the proposed processing pipeline, using a basic imaging device, such as a webcam, head pose estimation, and facial expression recognition is possible. The proposed pipeline can be used to monitor emotional response of focus groups to various ideas, pictures, sounds, words, and other stimuli.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Ali G, Iqbal MA, Choi T-S (2016) Boosted NNE collections for multicultural facial expression recognition. Pattern Recognit 55:14–27. https://doi.org/10.1016/j.patcog.2016.01.032
Alioua N, Amine A, Rogozan A et al (2016) Driver head pose estimation using efficient descriptor fusion. EURASIP J Image Video Process 2016:2. https://doi.org/10.1186/s13640-016-0103-z
Al-Milaji Z, Ersoy I, Hafiane A et al (2017) Integrating segmentation with deep learning for enhanced classification of epithelial and stromal tissues in H&E images. Pattern Recognit Lett. https://doi.org/10.1016/j.patrec.2017.09.015
Arigbabu OA, Mahmood S, Ahmad SMS, Arigbabu AA (2016) Smile detection using hybrid face representation. J Ambient Intell Humaniz Comput 7:415–426. https://doi.org/10.1007/s12652-015-0333-4
Ba SO, Odobez J-M (2009) Recognizing visual focus of attention from head pose in natural meetings. IEEE Trans Syst Man, Cybern Part B 39:16–33. https://doi.org/10.1109/TSMCB.2008.927274
Ba SO, Odobez J (2011) Multiperson visual focus of attention from head pose and meeting contextual cues. IEEE Trans Pattern Anal Mach Intell 33:101–116. https://doi.org/10.1109/TPAMI.2010.69
Baldassarri S, Hupont I, Abadía D, Cerezo E (2015) Affective-aware tutoring platform for interactive digital television. Multimed Tools Appl 74:3183–3206. https://doi.org/10.1007/s11042-013-1779-z
Barretto AM (2017) Application of facial expression studies on the field of marketing. FEELab Science Books, Porto
Bayrakdar S, Akgün D, Yücedağ İ (2017) An accelerated approach for facial expression analysis on video files. Pamukkale Univ J Eng Sci 23:602–613. https://doi.org/10.5505/pajes.2016.00908
Benmohamed A, Neji M, Ramdani M et al (2015) Feast: face and emotion analysis system for smart tablets. Multimed Tools Appl 74:9297–9322. https://doi.org/10.1007/s11042-014-2082-3
Bijlstra G, Dotsch R (2011) FaceReader 4 emotion classification performance on images from the Radboud Face Database
Cheng Chen, Odobez J (2012) We are not contortionists: Coupled adaptive learning for head and body orientation estimation in surveillance video. In: IEEE conference on computer vision and pattern recognition, pp 1544–1551
Choi J-S, Lee W-H, Lee J-H et al (2017) Deep Learning Based NLOS Identification with Commodity WLAN Devices. IEEE Trans Veh Technol. https://doi.org/10.1109/tvt.2017.2780121
Cook IA, Warren C, Pajot SK et al (2011) Regional brain activation with advertising images. J Neurosci Psychol Econ 4:147–160. https://doi.org/10.1037/a0024809
De Carolis B, Ferilli S, Palestra G (2017) Simulating empathic behavior in a social assistive robot. Multimed Tools Appl 76:5073–5094. https://doi.org/10.1007/s11042-016-3797-0
Ekman P, Friesen WV (1971) Constants across cultures in the face and emotion. J Pers Soc Psychol 17:124–129. https://doi.org/10.1037/h0030377
Ekman P, Friesen WV (1975) Unmasking the face: a guide to recognising emotions from facial clues. Consulting Psychologists Press, New Jersey (Prentice Hall [Palo Alto, CA])
Face ++ SC (2017) Face ++ Cognitive Services. https://www.faceplusplus.com/. Accessed 12 Nov 2017
Gaschler A, Jentzsch S, Giuliani M, et al (2012) Social behavior recognition using body posture and head pose for human-robot interaction. In: IEEE/RSJ international conference on intelligent robots and systems, pp 2128–2133
Ghimire D, Lee J (2012) Histogram of orientation gradient feature-based facial expression classification using bagging with extreme learning machine. Adv Sci Lett 17:156–161. https://doi.org/10.1166/asl.2012.4257
Ghimire D, Lee J (2013) Geometric feature-based facial expression recognition in image sequences using multi-class Adaboost and support vector machines. Sensors 13:7714–7734. https://doi.org/10.3390/s130607714
Goodfellow I, Bengio Y, Courville A (2016) Deep learning. MIT Press, Cambridge
Gu J, Wang Z, Kuen J et al (2017) Recent advances in convolutional neural networks. Pattern Recognit. https://doi.org/10.1016/j.patcog.2017.10.013
Hinton GE, Srivastava N, Krizhevsky A, et al (2012) Improving neural networks by preventing co-adaptation of feature detectors. arXiv
Ilbeygi M, Shah-Hosseini H (2012) A novel fuzzy facial expression recognition system based on facial feature extraction from color face images. Eng Appl Artif Intell 25:130–146. https://doi.org/10.1016/j.engappai.2011.07.004
Jung-Bae Kim, Youngkyoo Hwang, Won-Chul Bang, et al (2013) Real-time realistic 3D facial expression cloning for smart TV. In: IEEE international conference on consumer electronics (ICCE), pp 240–241
Kahneman D (2011) Thinking fast and slow. Allen Lane, Bristol
Kang S-K, Chung K-Y, Lee J-H (2014) Development of head detection and tracking systems for visual surveillance. Pers Ubiquitous Comput 18:515–522. https://doi.org/10.1007/s00779-013-0668-9
Karu-salo I (2013) The effect of universal emotions on customer behaviour. Estonian Business School, Tallinn
Kasiran Z, Yahya S (2007) Facial expression as an implicit customers’ feedback and the challenges. In: Computer graphics, imaging and visualisation, IEEE, pp 377–381
Langleben DD, Loughead JW, Ruparel K et al (2009) Reduced prefrontal and temporal processing and recall of high “sensation value” ads. Neuroimage 46:219–225. https://doi.org/10.1016/j.neuroimage.2008.12.062
Langner O, Dotsch R, Bijlstra G et al (2010) Presentation and validation of the Radboud Faces Database. Cogn Emot 24:1377–1388. https://doi.org/10.1080/02699930903485076
Laros FJM, Steenkamp J-BEM (2005) Emotions in consumer behavior: a hierarchical approach. J Bus Res 58:1437–1445. https://doi.org/10.1016/j.jbusres.2003.09.013
Lawrence S, Giles CL, Tsoi Ah Chung, Back AD (1997) Face recognition: a convolutional neural-network approach. IEEE Trans Neural Netw 8:98–113. https://doi.org/10.1109/72.554195
Lee C-C, Shih C-Y, Lai W-P, Lin P-C (2012) An improved boosting algorithm and its application to facial emotion recognition. J Ambient Intell Humaniz Comput 3:11–17. https://doi.org/10.1007/s12652-011-0085-8
Li R, Liu P, Jia K, Wu Q (2015) Facial Expression Recognition under Partial Occlusion Based on Gabor Filter and Gray-Level Cooccurrence Matrix. In: International conference on computational intelligence and communication networks (CICN). IEEE, pp 347–351
Liew CF, Yairi T (2013) A comparison study of feature spaces and classification methods for facial expression recognition. In: IEEE international conference on robotics and biomimetics (ROBIO), pp 1294–1299
Liew CF, Yairi T (2015) Facial expression recognition and analysis: a comparison study of feature descriptors. IPSJ Trans Comput Vis Appl 7:104–120. https://doi.org/10.2197/ipsjtcva.7.104
Lin Zhong, Qingshan Liu, Peng Yang, et al (2012) Learning active facial patches for expression analysis. In: IEEE conference on computer vision and pattern recognition, pp 2562–2569
Liu J, Gu Y, Kamijo S (2017) Customer behavior classification using surveillance camera for marketing. Multimed Tools Appl 76:6595–6622. https://doi.org/10.1007/s11042-016-3342-1
Liu J, Gu Y, Kamijo S (2018) Integral customer pose estimation using body orientation and visibility mask. Multimed Tools Appl. https://doi.org/10.1007/s11042-018-5839-2
Lopes AT, de Aguiar E, De Souza AF, Oliveira-Santos T (2017) Facial expression recognition with convolutional neural networks: coping with few data and the training sample order. Pattern Recognit 61:610–628. https://doi.org/10.1016/j.patcog.2016.07.026
Lozano-Monasor E, López MT, Vigo-Bustos F, Fernández-Caballero A (2017) Facial expression recognition in ageing adults: from lab to ambient assisted living. J Ambient Intell Humaniz Comput 8:567–578. https://doi.org/10.1007/s12652-017-0464-x
Lundqvist D, Flykt A, Öhman A (1998) The Karolinska Directed Emotional Faces (KDEF). CD ROM from Department of Clinical Neuroscience, Psychology section, Karolinska Institutet, Solna
Matsugu M, Mori K, Mitari Y, Kaneda Y (2003) Subject independent facial expression recognition with robust face detection using a convolutional neural network. Neural Netw 16:555–559. https://doi.org/10.1016/S0893-6080(03)00115-1
McKenna S, Gong S (1998) Real-time face pose estimation. Real-Time Imaging 4:333–347. https://doi.org/10.1016/S1077-2014(98)90003-1
Menon K, Dubé L (2000) Ensuring greater satisfaction by engineering salesperson response to customer emotions. J Retail 76:285–307. https://doi.org/10.1016/S0022-4359(00)00034-8
Micu AC, Plummer JT (2010) Measurable emotions: how television ads really work. J Advert Res 50:137–153. https://doi.org/10.2501/S0021849910091300
Missaglia AL, Oppo A, Mauri M et al (2017) The impact of emotions on recall: an empirical study on social ads. J Consum Behav 16:424–433. https://doi.org/10.1002/cb.1642
Mukherjee SS, Robertson NM (2015) Deep head pose: gaze-direction estimation in multimodal video. IEEE Trans Multimed 17:2094–2107. https://doi.org/10.1109/TMM.2015.2482819
Murphy-Chutorian E, Trivedi MM (2009) Head pose estimation in computer vision: a survey. IEEE Trans Pattern Anal Mach Intell 31:607–626. https://doi.org/10.1109/TPAMI.2008.106
Murphy-Chutorian E, Doshi A, Trivedi MM (2007) Head pose estimation for driver assistance systems: a robust algorithm and experimental evaluation. In: IEEE Intelligent transportation systems conference, pp 709–714
Nair V, Hinton GE (2010) Rectified Linear Units Improve Restricted Boltzmann Machines. In: Proceedings of the 27th international conference on international conference on machine learning. Omnipress, USA, pp 807–814
Niforatos E, Karapanos E (2015) EmoSnaps: a mobile application for emotion recall from facial expressions. Pers Ubiquitous Comput 19:425–444. https://doi.org/10.1007/s00779-014-0777-0
Ohme R, Reykowska D, Wiener D, Choromanska A (2009) Analysis of neurophysiological reactions to advertising stimuli by means of EEG and galvanic skin response measures. J Neurosci Psychol Econ 2:21–31. https://doi.org/10.1037/a0015462
Oztel I, Yolcu G, Ersoy I, et al (2017) Mitochondria segmentation in electron microscopy volumes using deep convolutional neural network. In: ieee international conference on bioinformatics and biomedicine (BIBM). pp 1195–1200
Oztel I, Yolcu G, Ersoy I et al (2018a) Deep learning approaches in electron microscopy imaging for mitochondria segmentation. Int J Data Min Bioinform 21:91. https://doi.org/10.1504/IJDMB.2018.096398
Oztel I, Yolcu G, Oz C et al (2018b) iFER: facial expression recognition using automatically selected geometric eye and eyebrow features. J Electron Imaging 27:1. https://doi.org/10.1117/1.JEI.27.2.023003
Patacchiola M, Cangelosi A (2017) Head pose estimation in the wild using Convolutional Neural Networks and adaptive gradient methods. Pattern Recognit 71:132–143. https://doi.org/10.1016/j.patcog.2017.06.009
Perumal Ramalingam S, Chandra Mouli PVSSR (2018) Modified dimensionality reduced local directional pattern for facial analysis. J Ambient Intell Humaniz Comput 9:725–737. https://doi.org/10.1007/s12652-017-0473-9
Pitaloka DA, Wulandari A, Basaruddin T, Liliana DY (2017) Enhancing CNN with preprocessing stage in automatic emotion recognition. Proc Comput Sci 116:523–529. https://doi.org/10.1016/j.procs.2017.10.038
Pons G, Masip D (2017) Supervised committee of convolutional neural networks in automated facial expression analysis. IEEE Trans Affect Comput. https://doi.org/10.1109/taffc.2017.2753235
Popa M, Rothkrantz L, Yang Z, et al (2010) Analysis of shopping behavior based on surveillance system. In: IEEE international conference on systems, man and cybernetics, pp 2512–2519
Rao Q, Qu X, Mao Q, Zhan Y (2015) Multi-pose facial expression recognition based on SURF boosting. In: 2015 international conference on affective computing and intelligent interaction (ACII). IEEE, pp 630–635
Riener A, Sippl A (2014) Head-pose-based attention recognition on large public displays. IEEE Comput Graph Appl 34:32–41. https://doi.org/10.1109/MCG.2014.9
Sahney S (2011) Module-6 consumer behavior. Vinod Gupta School of Management, Kharagpur, pp 1–24
Samara A, Galway L, Bond R, Wang H (2017) Affective state detection via facial expression analysis within a human–computer interaction context. J Ambient Intell Humaniz Comput. https://doi.org/10.1007/s12652-017-0636-8
Santra B, Mukherjee DP (2016) Local saliency-inspired binary patterns for automatic recognition of multi-view facial expression. In: IEEE international conference on image processing (ICIP). pp 624–628
Shaykha I, Menkara A, Nahas M, Ghantous M (2015) FEER: Non-intrusive facial expression and emotional recognition for driver’s vigilance monitoring. In: 57th international symposium ELMAR (ELMAR). IEEE, pp 233–237
Sheikhi S, Odobez J-M (2015) Combining dynamic head pose–gaze mapping with the robot conversational state for attention recognition in human–robot interactions. Pattern Recognit Lett 66:81–90. https://doi.org/10.1016/j.patrec.2014.10.002
Shelhamer E, Long J, Darrell T (2016) Fully convolutional networks for semantic segmentation. Cogn Emot 24:1377–1388
Sobia MC, Brindha V, Abudhahir A (2014) Facial expression recognition using PCA based interface for wheelchair. In: International conference on electronics and communication systems (ICECS). IEEE, pp 1–6
Srinivas S, Sarvadevabhatla RK, Mopuri KR et al (2016) A taxonomy of deep convolutional neural nets for computer vision. Front Robot AI. https://doi.org/10.3389/frobt.2015.00036
Sun Z, Hu Z-P, Wang M, Zhao S-H (2017) Discriminative feature learning-based pixel difference representation for facial expression recognition. IET Comput Vis 11:675–682. https://doi.org/10.1049/iet-cvi.2016.0505
Takahashi M, Clippingdale S, Naemura M, Shibata M (2015) Estimation of viewers’ ratings of TV programs based on behaviors in home environments. Multimed Tools Appl 74:8669–8684. https://doi.org/10.1007/s11042-014-2352-0
Teixeira T, Wedel M, Pieters R (2012) Emotion-induced engagement in internet video advertisements. J Mark Res 49:144–159. https://doi.org/10.1509/jmr.10.0207
Terzis V, Moridis CN, Economides AA (2013) Measuring instant emotions based on facial expressions during computer-based assessment. Pers Ubiquitous Comput 17:43–52. https://doi.org/10.1007/s00779-011-0477-y
Teyeb I, Jemai O, Zaied M, Ben Amar C (2015) Vigilance measurement system through analysis of visual and emotional driver’s signs using wavelet networks. In: 15th international conference on intelligent systems design and applications (ISDA). IEEE, pp 140–147
Tran BH, Kim Y-G (2017) Deep head pose estimation for faces in the wile and its transfer learning. In: Seventh international conference on information science and technology (ICIST). IEEE, pp 187–193
Ungureanu F, Lupu RG, Cadar A, Prodan A (2017) Neuromarketing and visual attention study using eye tracking techniques. In: 2017 21st international conference on system theory, control and computing (ICSTCC). pp 553–557
Vedaldi A, Lenc K (2015) MatConvNet. In: Proceedings of the 23rd ACM international conference on multimedia—MM’15. ACM Press, New York, New York, USA, pp 689–692
Viola P, Jones MJ (2004) Robust real-time face detection. Int J Comput Vis 57:137–154. https://doi.org/10.1023/B:VISI.0000013087.49260.fb
Wedel M, Pieters R (2000) Eye fixations on advertisements and memory for brands: a model and findings. Mark Sci 19:297–312. https://doi.org/10.1287/mksc.19.4.297.11794
Wei B, Sun X, Ren X, Xu J (2017) Minimal Effort Back Propagation for Convolutional Neural Networks. Comput Res Repos
Xu X, Kakadiaris IA (2017) Joint head pose estimation and face alignment framework using global and local CNN features. In: 12th ieee international conference on automatic face & gesture recognition (FG 2017). pp 642–649
Yan C, Zhang Y, Xu J et al (2014a) A highly parallel framework for HEVC coding unit partitioning tree decision on many-core processors. IEEE Signal Process Lett 21:573–576. https://doi.org/10.1109/LSP.2014.2310494
Yan C, Zhang Y, Xu J et al (2014b) Efficient parallel framework for HEVC motion estimation on many-core processors. IEEE Trans Circuits Syst Video Technol 24:2077–2089. https://doi.org/10.1109/TCSVT.2014.2335852
Yan C, Xie H, Chen J et al (2018a) A fast uyghur text detector for complex background images. IEEE Trans Multimed 20:3389–3398. https://doi.org/10.1109/TMM.2018.2838320
Yan C, Xie H, Liu S et al (2018b) Effective uyghur language text detection in complex background images for traffic prompt identification. IEEE Trans Intell Transp Syst 19:220–229. https://doi.org/10.1109/TITS.2017.2749977
Yan C, Xie H, Yang D et al (2018c) Supervised hash coding with deep neural network for environment perception of intelligent vehicles. IEEE Trans Intell Transp Syst 19:284–295. https://doi.org/10.1109/TITS.2017.2749965
Yan C, Li L, Zhang C et al (2019) Cross-modality bridging and knowledge transferring for image understanding. IEEE Trans Multimed. https://doi.org/10.1109/tmm.2019.2903448
Yolcu G, Oztel I, Kazan S, et al (2017) Deep learning-based facial expression recognition for monitoring neurological disorders. In: IEEE international conference on bioinformatics and biomedicine (BIBM). pp 1652–1657
Zalewski L, Shaogang Gong (2005) 2D Statistical Models of Facial Expressions for Realistic 3D Avatar Animation. In: IEEE computer society conference on computer vision and pattern recognition (CVPR’05). pp 217–222
Zhang Y, Hua C (2015) Driver fatigue recognition based on facial expression analysis using local binary patterns. Opt Int J Light Electron Opt 126:4501–4505. https://doi.org/10.1016/j.ijleo.2015.08.185
Zhao G, Pietikainen M (2007) Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans Pattern Anal Mach Intell 29:915–928. https://doi.org/10.1109/TPAMI.2007.1110
Zhao Y, Wang X, Goubran M et al (2013) Human emotion and cognition recognition from body language of the head using soft computing techniques. J Ambient Intell Humaniz Comput 4:121–140. https://doi.org/10.1007/s12652-012-0107-1
Acknowledgements
This research is supported by The Scientific and Technological Research Council of Turkey (TUBITAK-BIDEB 2214/A) and Sakarya University Scientific Research Projects Unit (Project Number: 2015-50-02-039).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Yolcu, G., Oztel, I., Kazan, S. et al. Deep learning-based face analysis system for monitoring customer interest. J Ambient Intell Human Comput 11, 237–248 (2020). https://doi.org/10.1007/s12652-019-01310-5
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12652-019-01310-5