Skip to main content
SpringerLink
Log in

Detecting dressing failures using temporal–relational visual grammars

  • Original Research
  • Published:
Journal of Ambient Intelligence and Humanized Computing Aims and scope Submit manuscript

Abstract

Evaluation of dressing activities is essential in the assessment of the performance of patients with psycho-motor impairments. However, the current practice of monitoring dressing activity (performed by the patients in front of the therapist) has a number of disadvantages when considering the personal nature of dressing activity as well as inconsistencies between the recorded performance of the activity and performance of the same activity carried out in the patients’ natural environment, such as their home. As such, a system that can evaluate dressing activities automatically and objectively would alleviate some of these issues. However, a number of challenges arise, including difficulties in correctly identifying garments, their position in the body (partially of fully worn) and their position in relation to other garments. To address these challenges, we have developed a novel method based on visual grammars to automatically detect dressing failures and explain the type of failure. Our method is based on the analysis of image sequences of dressing activities and only requires availability of a video recording device. The analysis relies on a novel technique which we call temporal–relational visual grammar; it can reliably recognize temporal dressing failures, while also detecting spatial and relational failures. Our method achieves 91% precision in detecting dressing failures performed by 11 subjects. We explain these results and discuss the challenges encountered during this work.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. The grammar does not contain rules describing failures, meaning that if a configuration of a dressing activity cannot be explained by the grammar it is marked as a failure.

References

  • Afsar P, Cortez P, Santos H (2015) Automatic visual detection of human behavior: a review from 2000 to 2014. Expert Syst Appl 42(20):6936–6956. https://doi.org/10.1016/j.eswa.2015.05.023

    Article  Google Scholar 

  • Allen JF (1983) Maintaining knowledge about temporal intervals. Commun ACM 26(11):832–843. https://doi.org/10.1145/182.358434

    Article  MATH  Google Scholar 

  • Bahle G, Gruenerbl A, Lukowicz P, Bignotti E, Zeni M, Giunchiglia F (2014) Recognizing hospital care activities with a coat pocket worn smartphone. In: Mobile computing, applications and services (MobiCASE), 2014 6th international conference on, IEEE, pp 175–181

  • Bay H, Ess A, Tuytelaars T, Gool LJV (2008) Speeded-up robust features (SURF). Comput Vis Image Underst 110(3):346–359. https://doi.org/10.1016/j.cviu.2007.09.014

    Article  Google Scholar 

  • Breiman L (2001) Random forests. Mach Learn 45(1):5–32. https://doi.org/10.1023/A:1010933404324

    Article  MATH  Google Scholar 

  • Brown C, Moore WP, Hemman D, Yunek A (1996) Influence of instrumental activities of daily living assessment method on judgments of independence. Am J Occup Ther 50(3):202–206

    Article  Google Scholar 

  • Chen C, Zhang D, Sun L, Hariz M, Yuan Y (2012) Does location help daily activity recognition? In: Donnelly M, Paggetti C, Nugent C, Mokhtari M (eds) Impact analysis of solutions for chronic disease prevention and management. ICOST 2012. Lecture notes in computer science, vol 7251. Springer, Berlin, pp 83–90

    Chapter  Google Scholar 

  • Chernbumroong S, Cang S, Atkins A, Yu H (2013) Elderly activities recognition and classification for applications in assisted living. Expert Syst Appl 40(5):1662–1674

    Article  Google Scholar 

  • Chomsky N (2002) Syntactic structures. A Mouton Classic, Mouton de Gruyter, Berlin. http://books.google.com/books?id=SNeHkMXHcd8C

  • Cortes C, Vapnik V (1995) Support-vector networks. Mach Learn 20(3):273–297. https://doi.org/10.1007/BF00994018

    Article  MATH  Google Scholar 

  • Costagliola G, Deufemia V, Ferrucci F, Gravino C (2002) Using extended positional grammars to develop visual modeling languages. In: Proceedings of the 14th international conference on software engineering and knowledge engineering, SEKE 2002, Ischia, Italy, July 15–19, 2002, ACM, pp 201–208. https://doi.org/10.1145/568760.568795

  • Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In: 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR 2005), 20–26 June 2005, San Diego, CA, USA, IEEE Computer Society, pp 886–893. https://doi.org/10.1109/CVPR.2005.177

  • Edmans J, Lincoln N (1990) The relation between perceptual deficits after stroke and independence in activities of daily living. Br J Occup Ther 53(4):139–142

    Article  Google Scholar 

  • Edmans J, Towle D, Lincoln NB (1991) The recovery of perceptual problems after stroke and the impact on daily life. Clin Rehabil 5(4):301–309

    Article  Google Scholar 

  • Ferrucci F, Pacini G, Satta G, Sessa MI, Tortora G, Tucci M, Vitiello G (1996) Symbol-relation grammars: a formalism for graphical languages. Inf Comput 131(1):1–46. https://doi.org/10.1006/inco.1996.0090

    Article  MathSciNet  MATH  Google Scholar 

  • Feyereisen P (1999) Disorders of everyday actions in subjects suffering from senile dementia of Alzheimer’s type: an analysis of dressing performance. Neuropsychol Rehabil 9(2):169–188

    Article  Google Scholar 

  • Foncubierta-Rodríguez A, Müller H, Depeursinge A (2017) From visual words to a visual grammar: using language modelling for image classification. CoRR arXiv:abs/1703.05571

  • Friedman A, Ron S (2017) Unlocking the power of visual grammar theory: analyzing social media political advertising messages in the 2016 US election. J Vis Lit 36(2):90–103. https://doi.org/10.1080/1051144X.2017.1379758

    Article  Google Scholar 

  • Girshick RB, Felzenszwalb PF, McAllester DA (2011) Object detection with grammar models. In: Zemel RS, Bartlett PL, Pereira FCN, Weinberger KQ Shawe-Taylor J (eds) NIPS, pp 442–450

  • Gottfried B (2015) The systematic design of visual languages applied to logical reasoning. J Vis Lang Comput 28:212–225. https://doi.org/10.1016/j.jvlc.2015.02.001

    Article  Google Scholar 

  • Haralick RM, Shanmugam KS, Dinstein I (1973) Textural features for image classification. IEEE Trans Syst Man Cybern 3(6):610–621. https://doi.org/10.1109/TSMC.1973.4309314

    Article  Google Scholar 

  • Kong J, Zhang K, Zeng X (2006) Spatial graph grammars for graphical user interfaces. ACM Trans Comput-Hum Interact 13(2):268–307. https://doi.org/10.1145/1165734.1165739

    Article  Google Scholar 

  • Lakin F (1987) Visual grammars for visual languages. In: Forbus KD, Shrobe HE (eds) Proceedings of the 6th national conference on artificial intelligence. Seattle, WA, July 1987, Morgan Kaufmann, pp 683–688. http://www.aaai.org/Library/AAAI/1987/aaai87-122.php

  • Leborg C (2006) Visual grammar. Princeton Architectural Press, New York

    Google Scholar 

  • Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vis 60(2):91–110

    Article  Google Scholar 

  • Maio CD, Fenza G, Gallo M, Loia V, Parente M (2017a) Time-aware adaptive tweets ranking through deep learning. Future Gener Comput Syst. https://doi.org/10.1016/j.future.2017.07.039

    Article  Google Scholar 

  • Maio CD, Fenza G, Loia V, Orciuoli F (2017b) Unfolding social content evolution along time and semantics. Future Gener Comput Syst 66:146–159. https://doi.org/10.1016/j.future.2016.05.039

    Article  Google Scholar 

  • Marriott K, Meyer B (1996) Towards a hierarchy of visual languages. In: Proceedings of the 1996 IEEE symposium on visual languages, Boulder, Colorado, USA, September 3–6, 1996, IEEE Computer Society, pp 196–203. https://doi.org/10.1109/VL.1996.545288

  • Matic A, Mehta P, Rehg JM, Osmani V, Mayora O (2012) Monitoring dressing activity failures through RFID and video. J Methods Inf Med 51:45–54

    Article  Google Scholar 

  • Matic A, Mehta P, Rehg JM, Osmani V, Mayora O (2010) Aid-me: Automatic identification of dressing failures through monitoring of patients and activity evaluation. In: Pervasive computing technologies for healthcare (PervasiveHealth), 2010 4th international conference on, IEEE, pp 1–8

  • Mikolajczyk K, Schmid C (2005) A performance evaluation of local descriptors. IEEE Trans Pattern Anal Mach Intell 27(10):1615–1630. https://doi.org/10.1109/TPAMI.2005.188

    Article  Google Scholar 

  • Mjolsness E (1991) Visual grammars and their neural nets. In: Moody JE, Hanson SJ, Lippmann R (eds) Advances in neural information processing systems, vol 4 (NIPS conference, Denver, Colorado, USA, December 2–5, 1991), Morgan Kaufmann, pp 428–435. http://papers.nips.cc/paper/499-visual-grammars-and-their-neural-nets

  • Namazi KH, Johnson BD (1992) Dressing independently: a closet modification model for Alzheimer’s disease patients. Am J Alzheimers Dis Other Dement 7(1):22–28

    Google Scholar 

  • Ojala T, Pietikäinen M, Harwood D (1996) A comparative study of texture measures with classification based on featured distributions. Pattern Recognit 29(1):51–59. https://doi.org/10.1016/0031-3203(95)00067-4

    Article  Google Scholar 

  • Osmani V (2015) Smartphones in mental health: detecting depressive and manic episodes. IEEE Pervasive Comput 14(3):10–13

    Article  Google Scholar 

  • Qi S, Huang S, Wei P, Zhu S (2017) Predicting human activities using stochastic grammar. In: IEEE international conference on computer vision, ICCV 2017, Venice, Italy, October 22–29, 2017, IEEE Computer Society, pp 1173–1181. https://doi.org/10.1109/ICCV.2017.132

  • Rublee E, Rabaud V, Konolige K, Bradski GR (2011) ORB: an efficient alternative to SIFT or SURF. In: Metaxas DN, Quan L, Sanfeliu A, Gool LJV (eds) IEEE International conference on computer vision, ICCV 2011, Barcelona, Spain, November 6–13, 2011, IEEE, pp 2564–2571. https://doi.org/10.1109/ICCV.2011.6126544

  • Sunderland A, Walker CM, Walker MF (2006) Action errors and dressing disability after stroke: an ecological approach to neuropsychological assessment and intervention. Neuropsycholl Rehabil 16(6):666–683

    Article  Google Scholar 

  • Vedaldi A, Fulkerson B (2010) Vlfeat: an open and portable library of computer vision algorithms. In: Proceedings of the 18th ACM international conference on multimedia, ACM, New York, NY, USA, MM’10, pp 1469–1472. https://doi.org/10.1145/1873951.1874249

  • Walker M, Lincoln N (1991) Factors influencing dressing performance after stroke. J Neurol Neurosurg Psychiatry 54(8):699–701

    Article  Google Scholar 

  • Walker C, Walker M (2001) Dressing ability after stroke: a review of the literature. Br J Occup Ther 64(9):449–454

    Article  Google Scholar 

  • Walker M, Drummond A, Lincoln N (1996) Evaluation of dressing practice for stroke patients after discharge from hospital: a crossover design study. Clin Rehabil 10(1):23–31

    Article  Google Scholar 

  • Walker CM, Walker MF, Sunderland A (2003) Dressing after a stroke: a survey of current occupational therapy practice. Br J Occup Ther 66(6):263–268

    Article  Google Scholar 

  • Wittenburg K, Weitzman L (1996) Relational grammars: theory and practice in a visual language interface for process modeling. In: Workshop on theory of visual languages, Springer, Berlin pp 27–29

  • Wu YN, Si Z, Gong H, Zhu SC (2010) Learning active basis model for object detection and recognition. Int J Comput Vis 90(2):198–235

    Article  MathSciNet  Google Scholar 

  • Zhu SC, Mumford D (2006) A stochastic grammar of images. Found Trends Comput Graph Vis 2(4):259–362. https://doi.org/10.1561/0600000018

    Article  MATH  Google Scholar 

  • Zhu L, Chen Y, Yuille AL (2009) Unsupervised learning of probabilistic grammar-markov models for object categories. Trans Pattern Anal Mach Intell 31(1):114–128. https://doi.org/10.1109/TPAMI.2008.67

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Computer Science Department, Instituto Nacional de Astrofísica, Óptica y Electrónica, Luis Enrique Erro No. 1, Sta. María Tonanzintla, C.P. 72840, Puebla, Mexico

    Elias Ruiz & L. Enrique Sucar

  2. Fondazione Bruno Kessler (FBK), 38123, Trento, TN, Italy

    Venet Osmani & Oscar Mayora

Authors
  1. Elias Ruiz
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Venet Osmani
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. L. Enrique Sucar
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Oscar Mayora
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Venet Osmani.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ruiz, E., Osmani, V., Sucar, L.E. et al. Detecting dressing failures using temporal–relational visual grammars. J Ambient Intell Human Comput 10, 2757–2770 (2019). https://doi.org/10.1007/s12652-018-0975-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12652-018-0975-0

Keywords

Search

Navigation