Skip to main content
Springer Nature Link
Log in

PainFusion: Multimodal Pain Assessment from RGB and Sensor Data

  • Conference paper
  • First Online:
The 19th International Conference on Soft Computing Models in Industrial and Environmental Applications SOCO 2024 (SOCO 2024)

Part of the book series: Lecture Notes in Networks and Systems ((LNNS,volume 888))

  • 249 Accesses

Abstract

Traditional pain assessment tools often rely on subjective self-reporting methods, hindering the work of healthcare professionals. However, the patient’s facial expressions and biomedical data provide a reliable source of information for caregivers. In this work, we present a multimodal architecture that utilizes both RGB video and biomedical sensor data from the BioVid Heat Pain dataset. We use video transformer architectures in conjunction with a thorough analysis of biomedical signals, including galvanic skin response, electromyography, and electrocardiogram, for comprehensive feature extraction. These features are then fused to create a robust model for pain assessment. Experimental results show that our multimodal architecture outperforms unimodal video-based methods in pain detection. Furthermore, our study highlights the potential of combining non-invasive video analysis with physiological data to facilitate pain prediction and management in clinical settings, paving the way for more accurate and efficient pain assessment methods that can be used in various healthcare applications.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    https://github.com/3dperceptionlab/tfg_mdlopez.

References

  1. Amirian, M., Kächele, M., Schwenker, F.: Using radial basis function neural networks for continuous and discrete pain estimation from bio-physiological signals. In: Schwenker, F., Abbas, H.M., El Gayar, N., Trentin, E. (eds.) Artificial Neural Networks in Pattern Recognition: 7th IAPR TC3 Workshop, ANNPR 2016, Ulm, Germany, September 28–30, 2016, Proceedings, pp. 269–284. Springer International Publishing, Cham (2016). https://doi.org/10.1007/978-3-319-46182-3_23

    Chapter  Google Scholar 

  2. Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lučić, M., Schmid, C.: ViViT: a video vision transformer. In: Proceedings of IEEE/CVF ICCV, pp. 6836–6846 (2021)

    Google Scholar 

  3. Aung, M.S.H., et al.: The automatic detection of chronic pain-related expression: requirements, challenges and the multimodal EmoPain dataset. IEEE Trans. Affect. Comput. 7(4), 435–451 (2016)

    Article  Google Scholar 

  4. Babarro, A.A.: La importancia de evaluar adecuadamente el dolor. Atención primaria 43(11), 575 (2011)

    Article  Google Scholar 

  5. Bao, H., Dong, L., Piao, S., Wei, F.: BEiT: BERT pre-training of image transformers. arXiv:2106.08254 (2021)

  6. Benavent-Lledo, M., et al.: A comprehensive study on pain assessment from multimodal sensor data. Sensors 23(24) (2023)

    Google Scholar 

  7. Bertasius, G., Wang, H., Torresani, L.: Is space-time attention all you need for video understanding? In: ICML, vol. 2, p. 4 (2021)

    Google Scholar 

  8. Dosovitskiy, A., et al.: An image is worth 16\(\,\times \,\)16 words: transformers for image recognition at scale. arXiv:2010.11929 (2020)

  9. Fan, H., Ling, H.: SANet: structure-aware network for visual tracking (2017)

    Google Scholar 

  10. Gomez-Donoso, F., et al.: A robotic platform for customized and interactive rehabilitation of persons with disabilities. Pattern Recognit. Lett. 99, 105–113 (2017)

    Article  Google Scholar 

  11. Haque, M.A., et al.: Deep multimodal pain recognition: a database and comparison of spatio-temporal visual modalities. In: FG, pp. 250–257 (2018)

    Google Scholar 

  12. Ibáñez, R.M., et al.: Escalas de valoración del dolor. Jano 25(1), 41–44 (2005)

    Google Scholar 

  13. Kächele, M., et al.: Multimodal data fusion for person-independent, continuous estimation of pain intensity. In: Iliadis, L., Jayne, C. (eds.) Engineering Applications of Neural Networks: 16th International Conference, EANN 2015, Rhodes, Greece, September 25-28 2015.Proceedings, pp. 275–285. Springer International Publishing, Cham (2015). https://doi.org/10.1007/978-3-319-23983-5_26

    Chapter  Google Scholar 

  14. Kessler, V., Thiam, P., Amirian, M., Schwenker, F.: Pain recognition with camera photoplethysmography. In: IPTA, IEEE (2017)

    Google Scholar 

  15. Li, X., Zhang, X., Yang, H., Duan, W., Dai, W., Yin, L.: An EEG-based multi-modal emotion database with both posed and authentic facial actions for emotion analysis. In: Face and Gestures, pp. 336–343 (2020)

    Google Scholar 

  16. López, J.A., et al.: A novel prediction method for early recognition of global human behaviour in image sequences. Neural Process. Lett. 43(2), 363–387 (2016)

    Article  Google Scholar 

  17. Mende-Siedlecki, P., et al.: The delaware pain database: a set of painful expressions and corresponding norming data. PAIN Rep. 5(6), e853 (2020)

    Article  Google Scholar 

  18. Moreno-Serrano, N.L.R., et al.: Medicina del dolor y cuidado paliativo. Editorial Universidad del Rosario (2022)

    Google Scholar 

  19. Ochs, M., Kretz, A., Mester, R.: SDNet: semantically guided depth estimation network (2019)

    Google Scholar 

  20. Olugbade, T.A., et al.: Bi-modal detection of painful reaching for chronic pain rehabilitation systems. In: International Conference on Multimodal Interaction (2014)

    Google Scholar 

  21. Ortiz-Perez, D., Ruiz-Ponce, P., Tomás, D., Garcia-Rodriguez, J., Vizcaya-Moreno, M.F., Leo, M.: A deep learning-based multimodal architecture to predict signs of dementia. Neurocomputing 548, 126, 413 (2023)

    Google Scholar 

  22. Othman, E., et al.: Automatic vs. human recognition of pain intensity from facial expression on the X-ITE pain database. Sensors 21(9), 3273 (2021)

    Google Scholar 

  23. Prkachin, K.M., Solomon, P.E.: The structure, reliability and validity of pain expression: evidence from patients with shoulder pain. Pain 139(2), 267–274 (2008)

    Article  Google Scholar 

  24. Revuelta, F.F., et al.: Representation of 2D objects with a topology preserving network. In: 2nd International Workshop on Pattern Recognition in Information Systems, April 2002, pp. 267–276 (2002)

    Google Scholar 

  25. Ruiz-Ponce, P., et al.: POSEIDON: a data augmentation tool for small object detection datasets in maritime environments. Sensors 23(7), 3691 (2023)

    Article  Google Scholar 

  26. Santiago, A.J., Sánchez, S.B.: Experiencia diferencial del dolor según género, edad, adscripción religiosa y pertenencia étnica. Archivos en Medicina Familiar 16(3), 49–55 (2017)

    Google Scholar 

  27. Selva, J., et al.: Video transformers: a survey. TPAMI (2023)

    Google Scholar 

  28. Semwal, A., et al.: Computer aided pain detection and intensity estimation using compact CNN based fusion network. Appl. Soft Comput. 112, 107, 780 (2021)

    Google Scholar 

  29. Tong, Z., et al.: VideoMAE: masked autoencoders are data-efficient learners for self-supervised video pre-training. NeurIPS 35, 10078–10093 (2022)

    Google Scholar 

  30. Tsai, F.S., Hsu, Y.L., Chen, W.C., Weng, Y.M., Ng, C.J., Lee, C.C.: Toward development and evaluation of pain level-rating scale for emergency triage based on vocal characteristics and facial expressions. In: Interspeech 2016. ISCA (2016)

    Google Scholar 

  31. Vaswani, A., et al.: Attention is all you need. NeurIPS 30 (2017)

    Google Scholar 

  32. Velana, M., et al.: The senseemotion database: a multimodal database for the development and systematic validation of an automatic pain- and emotion-recognition system. In: MPRSS Workshop, pp. 127–139 (2017)

    Google Scholar 

  33. Viejo, D., et al.: Using GNG to improve 3D feature extraction - application to 6dof egomotion. Neural Netw. 32, 138–146 (2012)

    Article  Google Scholar 

  34. Walter, S., et al.: The biovid heat pain database data for the advancement and systematic validation of an automated pain recognition system. In: 2013 IEEE International Conference on Cybernetics (CYBCO), pp. 128–131 (2013)

    Google Scholar 

  35. Werner, P., Al-Hamadi, A., Limbrecht-Ecklundt, K., Walter, S., Gruss, S., Traue, H.C.: Automatic pain assessment with facial activity descriptors. IEEE Trans. Affect. Comput. 8(3), 286–299 (2017)

    Article  Google Scholar 

  36. Werner, P., Al-Hamadi, A., Niese, R., Walter, S., Gruss, S., Traue, H.C.: Automatic pain recognition from video and biomedical signals. In: ICPR 2014 (2014)

    Google Scholar 

Download references

Acknowledgment

We would like to thank CIAICO/2022/132 Consolidated group project “AI4-Health” funded by the Valencian government. This work has also been supported by a Spanish national and a regional grants for PhD studies, FPU21/00414, CIACIF/2021/430 and CIACIF/2022/175.

Author information

Authors and Affiliations

  1. Department of Computer Technology, University of Alicante, Alicante, Spain

    Manuel Benavent-Lledo, Maria Dolores Lopez-Valle, David Ortiz-Perez, David Mulero-Perez & Jose Garcia-Rodriguez

  2. School of Computer Science and Engineering, University of Westminster, London, UK

    Alexandra Psarrou

Authors
  1. Manuel Benavent-Lledo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Maria Dolores Lopez-Valle
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. David Ortiz-Perez
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. David Mulero-Perez
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jose Garcia-Rodriguez
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Alexandra Psarrou
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jose Garcia-Rodriguez .

Editor information

Editors and Affiliations

  1. University of A Coruña, Ferrol, La Coruña, Spain

    Héctor Quintián

  2. University of Salamanca, Salamanca, Spain

    Emilio Corchado

  3. Pablo de Olavide University, Seville, Spain

    Alicia Troncoso Lora

  4. University of León, León, Spain

    Hilde Pérez García

  5. University of A Coruña, Ferrol, La Coruña, Spain

    Esteban Jove

  6. University of A Coruña, A Coruña, Spain

    José Luis Calvo Rolle

  7. University of La Rioja, Logroño, Spain

    Francisco Javier Martínez de Pisón

  8. University of Deusto, Bilbao, Spain

    Pablo García Bringas

  9. Pablo de Olavide University, Seville, Spain

    Francisco Martínez Álvarez

  10. University of Burgos, Burgos, Spain

    Álvaro Herrero Cosío

  11. University of Bergamo, Dalmine, Bergamo, Italy

    Paolo Fosci

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Benavent-Lledo, M., Lopez-Valle, M.D., Ortiz-Perez, D., Mulero-Perez, D., Garcia-Rodriguez, J., Psarrou, A. (2024). PainFusion: Multimodal Pain Assessment from RGB and Sensor Data. In: Quintián, H., et al. The 19th International Conference on Soft Computing Models in Industrial and Environmental Applications SOCO 2024. SOCO 2024. Lecture Notes in Networks and Systems, vol 888. Springer, Cham. https://doi.org/10.1007/978-3-031-75013-7_30

Download citation

Publish with us

Policies and ethics