Abstract
In this work, a multi-modal pain intensity recognition system based on both audio and video channels is presented. The system is assessed on a newly recorded dataset consisting of several individuals, each subjected to 3 gradually increasing levels of painful heat stimuli under controlled conditions. The assessment of the dataset consists of the extraction of a multitude of features from each modality, followed by an evaluation of the discriminative power of each extracted feature set. Finally, several fusion architectures, involving early and late fusion, are assessed. The temporal availability of the audio channel is taken in consideration during the assessment of the fusion architectures.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Amirian, M., Kächele, M., Schwenker, F.: Using radial basis function neural networks for continuous and discrete pain estimation from bio-physiological signals. In: Schwenker, F., Abbas, H.M., El Gayar, N., Trentin, E. (eds.) ANNPR 2016. LNCS, vol. 9896, pp. 269–284. Springer, Cham (2016). doi:10.1007/978-3-319-46182-3_23
Aung, M.S.H., Kaltwang, S., Romera-Paredes, B., Martinez, B., Singh, A., Cella, M., Valstar, M., Meng, H., Kemp, A., Shafizadeh, M., Elkins, A.C., Kanakam, N., de Rothschild, A., Tyler, N., Watson, P.J., Williams, A.C., Pantic, M., Bianchi-Berthouze, N.: The automatic detection of chronic pain-related expression: requirements, challenges and multimodal dataset. IEEE Trans. Affect. Comput. 7, 435–451 (2016)
Baltrusaitis, T., Robinson, P., Morency, L.P.: OpenFace: an open source facial behavior analysis toolkit. In: 2016 IEEE Winter Conference on Applications of Computer Vision, pp. 1–10 (2016)
Bosch, A., Zisserman, A., Munoz, X.: Representing shape with a spatial pyramid kernel. In: Proceedings of the 6th ACM International Conference on Image and Video Retrieval, pp. 401–408 (2007)
Breiman, L.: Random forests. Mach. Learn. 45, 5–32 (2001)
Chu, Y., Zhao, X., Yao, J., Zhao, Y., Wu, Z.: Physiological signals based quantitative evaluation method of the pain. In: Proceedings of the 19th IFAC World Congress, pp. 2981–2986 (2014)
Eyben, F., Weninger, F., Gross, F., Schuller, B.: Recent developments in openSMILE, the Munich open-source multimedia feature extractor. In: ACM Multimedia (MM), pp. 835–838 (2013)
Florea, C., Florea, L., Vertan, C.: Learning pain from emotion: transferred HoT data representation for pain intensity estimation. In: Agapito, L., Bronstein, M.M., Rother, C. (eds.) ECCV 2014. LNCS, vol. 8927, pp. 778–790. Springer, Cham (2015). doi:10.1007/978-3-319-16199-0_54
Gruss, S., Treister, R., Werner, P., Traue, H.C., Crawcour, S., Andrade, A., Walter, S.: Pain intensity recognition rates via biopotential feature patterns with support vector machines. PLoS ONE 10, e0140330 (2015)
Hermansky, H., Morgan, N., Bayya, A., Kohn, P.: RASTA-PLP speech analysis technique. In: Proceedings of the 1992 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 121–124 (1992)
Jagan Mohan, B., Badu N., R.: Speech recognition using MFCC and DTW. In: International Conference on Advances in Electrical Engineering (ICAEE), pp. 1–4 (2014)
Kächele, M., Amirian, M., Thiam, P., Werner, P., Walter, S., Palm, G., Schwenker, F.: Adaptive confidence learning for the personalization of pain intensity estimation systems. Evol. Syst. 8, 1–13 (2016)
Kächele, M., Schwenker, F.: Cascaded fusion of dynamic, spatial, and textural feature sets for person-independent facial emotion recognition. In: 2014 22nd Internation Conference on Pattern Recognition, pp. 4660–4665 (2014)
Kächele, M., Thiam, P., Amirian, M., Schwenker, F., Palm, G.: Methods for person-centered continuous pain intensity assessment from bio-physiological channels. IEEE J. Sel. Top. Signal Process. 10, 854–864 (2016)
Kächele, M., Thiam, P., Amirian, M., Werner, P., Walter, S., Schwenker, F., Palm, G.: Multimodal data fusion for person-independent, continuous estimation of pain intensity. In: Iliadis, L., Jayne, C. (eds.) EANN 2015. CCIS, vol. 517, pp. 275–285. Springer, Cham (2015). doi:10.1007/978-3-319-23983-5_26
Kächele, M., Werner, P., Al-Hamadi, A., Palm, G., Walter, S., Schwenker, F.: Bio-visual fusion for person-independent recognition of pain intensity. In: Schwenker, F., Roli, F., Kittler, J. (eds.) MCS 2015. LNCS, vol. 9132, pp. 220–230. Springer, Cham (2015). doi:10.1007/978-3-319-20248-8_19
Kaltwang, S., Rudovic, O., Pantic, M.: Continuous pain intensity estimation from facial expressions. In: Bebis, G., et al. (eds.) ISVC 2012. LNCS, vol. 7432, pp. 368–377. Springer, Heidelberg (2012). doi:10.1007/978-3-642-33191-6_36
Krothapalli, S.R., Koolagudi, S.G.: Emotion recognition using vocal tract information. In: Krothapalli, S.R., Koolagudi, S.G. (eds.) Emotion Recognition using Speech Features, pp. 67–78. Springer, New York (2013)
Krothapalli, S.R., Koolagudi, S.G.: Speech emotion recognition: a review. In: Krothapalli, S.R., Koolagudi, S.G. (eds.) Emotion Recognition using Speech Features, pp. 15–34. Springer, New York (2013)
Kuncheva, L.I.: Combining Pattern Classifiers: Methods and Algorithms. Wiley, Hoboken (2004)
Meudt, S., Schwenker, F.: On instance selection in audio based emotion recognition. In: Mana, N., Schwenker, F., Trentin, E. (eds.) ANNPR 2012. LNCS, vol. 7477, pp. 186–192. Springer, Heidelberg (2012). doi:10.1007/978-3-642-33212-8_17
Olugbade, T.A., Bianchi-Berthouze, N., Marquardt, N., Williams, A.C.: Pain level recognition using kinematics and muscle activity for physical rehabilitation in chronic pain. In: IEEE Proceedings of International Conference on Affective Computing and Intelligent Interaction, pp. 243–249 (2015)
Sun, B., Li, L., Zhou, G., Wu, X., He, J., Yu, L., Li, D., Wei, Q.: Combining multimodal features within a fusion network for emotion recognition in the wild. In: Proceedings of the 2015 ACM International Conference on Multimodal Interaction, pp. 497–502 (2015)
Werner, P., Al-Hamadi, A., Niese, R., Walter, S., Gruss, S., Traue, H.C.: Towards pain monitoring: facial expression, head pose, a new database, an automatic system and remaining challenges. In: Proceedings of the British Machine Vision Conference, pp. 1–13 (2013)
Werner, P., Al-Hamadi, A., Niese, R., Walter, S., Gruss, S., Traue, H.C.: Automatic pain recognition from video and biomedical signals. In: Proceedings of the International Conference on Pattern Recognition (ICPR), pp. 4582–4587 (2014)
Zhao, G., Pietikaeinen, M.: Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans. Pattern Anal. Mach. Intell. 29, 915–928 (2007)
Acknowledgments
Viktor Kessler and Friedhelm Schwenker are active within the Transregional Collaborative Research Centre SFB/TRR 62 Companion-Technology for Cognitive Technical Systems, funded by the German Research Foundation (DFG).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Thiam, P., Kessler, V., Walter, S., Palm, G., Schwenker, F. (2017). Audio-Visual Recognition of Pain Intensity. In: Schwenker, F., Scherer, S. (eds) Multimodal Pattern Recognition of Social Signals in Human-Computer-Interaction. MPRSS 2016. Lecture Notes in Computer Science(), vol 10183. Springer, Cham. https://doi.org/10.1007/978-3-319-59259-6_10
Download citation
DOI: https://doi.org/10.1007/978-3-319-59259-6_10
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-59258-9
Online ISBN: 978-3-319-59259-6
eBook Packages: Computer ScienceComputer Science (R0)