Skip to main content
SpringerLink
Log in

Video-based neonatal pain expression recognition with cross-stream attention

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Facial expression is considered as the most specific pain indicator, which has been effectively employed for neonatal pain assessment. Since neonates cannot verbalize their subjective pain experiences, recognizing neonatal pain expression automatically has great value and meaning. The Two-Stream Convolutional Network (TS-ConvNet) can effectively aggregate the spatial and temporal information in the neonatal pain expression videos by adopting the two-stream structure. However, traditional TS-ConvNet is unable to exploit the correlation across the spatial stream and temporal stream, due to the spatial and temporal streams being independent of each other. To overcome this drawback, this paper presents a Cross-Stream Attention (CSA) mechanism with non-local operations to model the correlation of the two streams and proposes a new model called TS-ConvNet with CSA units (TSCN-CSA) by introducing CSA mechanism into TS-ConvNet. TSCN-CSA enables spatial information and temporal information to interact with each other at different semantic levels, and employs ResNet-50 pre-trained on ImageNet as the backbone to extract neonatal pain expression features. In addition, to evaluate the performance of the proposed model, we collected a video dataset named Dynamic Facial Expression of Pain in Neonates (DFEPN), which is composed of 1897 video clips with four categories of expression labels: calmness, crying, moderate pain, and severe pain. The experimental results on the DFEPN dataset demonstrate that CSA units have a positive effect and improve the accuracy of TS-ConvNets for neonatal pain expression recognition. As a result, the proposed method achieves the promising recognition performance (66.20%) for the four categories based neonatal pain expression recognition.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Data Availability

The data that support the findings of this study are available from Children’s Hospital of Nanjing Medical University but restrictions apply to the availability of these data, which were used under license for the current study, and so are not publicly available. Data are however available from the authors upon reasonable request and with permission of Children’s Hospital of Nanjing Medical University.

References

  1. Brahnam S, Chuang CF, Sexton RS et al (2007) Machine assessment of neonatal facial expressions of acute pain. Decis Support Syst 43(4):1242–1254. https://doi.org/10.1016/j.dss.2006.02.004

    Article  Google Scholar 

  2. Brahnam S, Chuang CF, Shih FY et al (2006) Machine recognition and representation of neonatal facial displays of acute pain. Artif Intell Med 36(3):211–222. https://doi.org/10.1016/j.artmed.2004.12.003

    Article  Google Scholar 

  3. Brahnam S, Nanni L, McMurtrey S et al (2019) Neonatal pain detection in videos using the iCOPEvid dataset and an ensemble of descriptors extracted from Gaussian of Local Descriptors. Applied Computing and Informatics. https://doi.org/10.1016/j.aci.2019.05.003

  4. Feichtenhofer C, Fan H, Malik J et al (2019) Slowfast networks for video recognition. Paper presented at the IEEE International Conference on Computer Vision (ICCV), Seoul, Korea, 2019, pp 6202–6211

  5. Gholami B, Haddad WM, Tannenbaum AR (2010) Relevance vector machine learning for neonate pain intensity assessment using digital imaging. IEEE Trans Biomed Eng 57(6):1457–1466. https://doi.org/10.1109/TBME.2009.2039214

    Article  Google Scholar 

  6. Grunau RE, Craig KD (1987) Pain expression in neonates: facial action and cry. Pain 28(3):395–410. https://doi.org/10.1016/0304-3959(87)90073-X

    Article  Google Scholar 

  7. Grunau RE, Oberlander T, Holsti L et al (1998) Bedside application of the Neonatal Facial Coding System in pain assessment of premature infants. Pain 76(3):277–286. https://doi.org/10.1016/S0304-3959(98)00046-3

    Article  Google Scholar 

  8. Hartley KA, Miller CS, Gephart SM (2015) Facilitated tucking to reduce pain in neonates: evidence for best practice. Adv Neonat Care 15(3):201–208. https://doi.org/10.1097/ANC.0000000000000193

    Article  Google Scholar 

  9. Hatfield LA, Meyers MA, Messing TM (2013) A systematic review of the effects of repeated painful procedures in infants: is there a potential to mitigate future pain responsivity? J Nurs Educ Pract 3(8):99–112. https://doi.org/10.5430/jnep.v3n8p99

    Google Scholar 

  10. He K, Zhang X, Ren S et al (2016) Deep residual learning for image recognition. Paper presented at the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 2016, pp 770–778

  11. Hu J, Shen L, Albanie S et al (2020) Squeeze-and-excitation networks. IEEE T Pattern Anal 42(8):2011–2023. https://doi.org/10.1109/TPAMI.2019.2913372

    Article  Google Scholar 

  12. Hummel P, Dijk M (2006) Pain assessment: current status and challenges. Semin Fetal Neonat M 11(4):237–245. https://doi.org/10.1016/j.siny.2006.02.004

    Article  Google Scholar 

  13. Kazemi V, Sullivan J (2014) One millisecond face alignment with an ensemble of regression trees. Paper presented at the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Columbus, OH, USA, 2014, pp 1867–1874

  14. Li C, Zhong Q, Xie D et al (2019) Collaborative spatiotemporal feature learning for video action recognition. Paper presented at the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 2019, pp 7872–7881

  15. Lu G, Hao Q, Kong K et al (2018) Deep convolutional neural networks with transfer learning for neonatal pain expression recognition. Paper presented at the 14th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery (ICNC-FSKD), Huangshan, China, 28–30 July 2018, pp 251–256

  16. Lu G, Yang C, Chen M et al (2016) Sparse representation based facial expression classification for pain assessment in neonates. Paper presented at the 12th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery (ICNC-FSKD), Changsha, China, 13–15 August 2016, pp 1615–1619

  17. Lu G, Yuan L, Li X et al (2008) Facial expression recognition of pain in neonates. Paper presented at the International Conference on Computer Science and Software Engineering, CSSE, Wuhan, China, 12–14 December 2008, pp 756–759

  18. Nanni L, Brahnam S, Lumini A (2010) A local approach based on a Local Binary Patterns variant texture descriptor for classifying pain states. Expert Syst Appl 37(12):7888–7894. https://doi.org/10.1016/j.eswa.2010.04.048

    Article  Google Scholar 

  19. Prkachin KM, Solomon P, Hwang T et al (2001) Does experience influence judgements of pain behaviour? Evidence from relatives of pain patients and therapists. Pain Res Manage 6(2):105–112. https://doi.org/10.1155/2001/108098

    Article  Google Scholar 

  20. Qiu Z, Yao T, Mei T (2017) Learning spatio-temporal representation with pseudo-3D residual networks. Paper presented at the IEEE International Conference on Computer Vision (ICCV), Venice, Italy, 2017, pp 5533–5541

  21. Rostami M, Berahmand K, Nasiri E et al (2021) Review of swarm intelligence-based feature selection methods. Eng Appl Artif Intel 100:104–210. https://doi.org/10.1016/j.engappai.2021.104210

    Article  Google Scholar 

  22. Rostami M, Forouzandeh S, Berahmand K et al (2020) Integration of multi-objective pSO based feature selection and node centrality for medical datasets. Genomics 112(6):4370–4384. https://doi.org/10.1016/j.ygeno.2020.07.027

    Article  Google Scholar 

  23. Russakovsky O, Deng J, Su H et al (2015) ImageNet large scale visual recognition challenge. Int J Comput Vis 115:211–252. https://doi.org/10.1007/s11263-015-0816-y

    Article  MathSciNet  Google Scholar 

  24. Salekin MS, Zamzmi G, Goldgof D et al (2021) Multimodal spatio-temporal deep learning approach for neonatal postoperative pain assessment. Comput Biol Med 129:104–150. https://doi.org/10.1016/j.compbiomed.2020.104150

    Article  Google Scholar 

  25. Schwaller F, Fitzgerald M (2014) The consequences of pain in early life: injury-induced plasticity in developing pain pathways. Eur J Neurosci 39 (3):344–352. https://doi.org/10.1111/ejn.12414

    Article  Google Scholar 

  26. Simonyan K, Zisserman A (2014) Two-stream convolutional networks for action recognition in videos. Paper presented at the 28th International Conference on Neural Information Processing Systems (NIPS), Montreal, Canada, December 2014, pp 568–576

  27. Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. Paper presented at the International Conference on Learning Representations (ICLR) San Diego, CA, USA, 7–9 May 2015

  28. Thiam P, Kestler HA, Schwenker F (2020) Two-stream attention network for pain recognition from video sequences. Sensors 20(3):839. https://doi.org/10.3390/s20030839

    Article  Google Scholar 

  29. Tran D, Bourdev L, Fergus R et al (2015) Learning spatiotemporal features with 3D convolutional networks. Paper presented at the IEEE International Conference on Computer Vision (ICCV), Santiago, Chile, 2015, pp 4489–4497

  30. Tran A, Cheong LF (2017) Two-stream flow-guided convolutional attention networks for action recognition. Paper presented at the IEEE International Conference on Computer Vision (ICCV), Venice, Italy, 2017, pp 3110–3119

  31. Vaswani A, Shazeer N, Parmar N et al (2017) Attention is all you need. Paper presented at the 31st International Conference on Neural Information Processing Systems (NIPS), Long Beach, CA, USA, 4–9 December 2017, pp 6000–6010

  32. Virrey RA, Liyanage CDS, Petra MIPH et al (2019) Visual data of facial expressions for automatic pain detection. J Vis Commun Image R 61:209–217. https://doi.org/10.1016/j.jvcir.2019.03.023

    Article  Google Scholar 

  33. Wang X, Girshick R, Gupta A et al (2018) Non-local neural networks. Paper presented at the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA, 18–21 June 2018, pp 7794–7803

  34. Wei J, Lu G, Yan J (2021) A comparative study on movement feature in different directions for micro-expression recognition. Neurocomputing 449:159–171. https://doi.org/10.1016/j.neucom.2021.03.063. https://www.sciencedirect.com/science/article/pii/S0925231221004495

    Article  Google Scholar 

  35. Wei J, Lu G, Yan J et al (2022) Learning two groups of discriminative features for micro-expression recognition. Neurocomputing 479:22–36. https://doi.org/10.1016/j.neucom.2021.12.088

    Article  Google Scholar 

  36. Wei J, Peng W, Lu G et al (2022) Geometric graph representation with learnable graph structure and adaptive au constraint for micro-expression recognition. arXiv:220500380

  37. Yan J, Lu G, Li X et al (2020) FEN P: a database of neonatal facial expression for pain analysis. Preprint at https://doi.org/10.1109/TAFFC.2020.3030296

  38. Zach C, Pock T, Bischof H (2007) A duality based approach for realtime tV-L1 optical flow. Paper presented at the 29th DAGM Conference on Pattern Recognition, Heidelberg, Germany, 12–14 September 2007, pp 214–223

  39. Zamzmi G, Kasturi R, Goldgof D et al (2017) A review of automated pain assessment in infants: features, classification tasks, and databases. IEEE Rev Biomed Eng 11:77–96. https://doi.org/10.1109/RBME.2017.2777907

    Article  Google Scholar 

  40. Zamzmi G, Ruiz G, Goldgof D et al (2015) Pain assessment in infants: towards spotting pain expression based on infants’ facial strain. Paper presented at the 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), Ljubljana, Slovenia, 4–8 May 2015

  41. Zhi R, Zamzmi G, Goldgof D et al (2018) Automatic infants’ pain assessment by dynamic facial representation: effects of profile view, gestational age, gender, and race. J Clin Med 7(7):173. https://doi.org/10.3390/jcm7070173

    Article  Google Scholar 

  42. Zhi R, Zhou C, Yu J et al (2021) Multi-stream integrated neural networks for facial expression-based pain recognition. Paper presented at the International Conference on Pattern Recognition (ICPR), Milan, Italy, 10–15 January 2021, pp 28–35

Download references

Acknowledgements

This work was partly supported by National Natural Science Foundation of China (Grant Nos. 72074038 and 61971236). The authors would like to acknowledge the cooperation and generous assistance of all the nurses in the neonatal unit at Children’s Hospital of Nanjing Medical University, the Second Affiliated Hospital of Nanjing Medical University, and Children’s Hospital of Chongqing Medical University. We are especially grateful to the parents who had agreed to allow their children to take part in this study. Thanks to Jinsheng Wei (third author) who makes contributions to the technical guidance and theoretical analysis, the design and implementation of experiments, and the writing and technical editing, especially during the major revision, responsible for revision work, including supplementary experiments, rewriting most sections, rearranging structure, and clarification technical details.

Author information

Authors and Affiliations

  1. College of Telecommunications and Information Engineering, Nanjing University of Posts and Telecommunications, Nanjing, 210003, China

    Guanming Lu, Haoxia Chen, Jinsheng Wei, Yimo Lou & Jingjie Yan

  2. Children’s Hospital of Nanjing Medical University, Nanjing, 210008, China

    Xiaonan Li

  3. Key Laboratory of Child Development Disease Research, Ministry of Education / National Clinical Research Center for Child Health and Disorders, Children’s Hospital of Chongqing Medical University, Chongqing, 400014, China

    Xianlan Zheng & Hongyao Leng

Authors
  1. Guanming Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Haoxia Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jinsheng Wei
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xiaonan Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xianlan Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Hongyao Leng
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Yimo Lou
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Jingjie Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Guanming Lu.

Ethics declarations

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lu, G., Chen, H., Wei, J. et al. Video-based neonatal pain expression recognition with cross-stream attention. Multimed Tools Appl 83, 4667–4690 (2024). https://doi.org/10.1007/s11042-023-15403-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-15403-z

Keywords

Search

Navigation