Skip to main content
SpringerLink
Log in

Uniform Evaluation of Properties in Activity Recognition

  • Conference paper
  • First Online:
Advances in Knowledge Discovery and Data Mining (PAKDD 2022)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 13281))

Included in the following conference series:

Abstract

The main additional problem in activity recognition (AR) systems in contrast to traditional ones is the importance of duration: a predicted concept in AR is durative and can be correct in a period and incorrect in another one. Therefore, it is fundamental to extend the correctness vocabulary and to formalize a new evaluation system including these extensions. Even in similar areas, few empirical attempts are proposed which are confronted with the problems of correctness and completeness. In this paper, we propose the first formal multi-modal evaluation approach for durative concepts. This novel mathematical method evaluates the performance of an AR system from multiple perspectives, including detection, total duration, relative duration, boundary alignment, and uniformity. It extracts the properties considered in the state-of-the-art and redefines the well-known true-positive, false-positive and false-negative terms for durative events. Our proposed method is extensible, interpretable, customizable, open source and improves the expressiveness of the evaluation while its computation complexity remains linear. Comprehensive experimental evaluations are conducted to show the usefulness of our proposed method.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    \(\mathrm {overfill}_{\mathrm {start}}\!\mathrm {(r,ps)}\!=\!\mathrm {max}\mathrm {(0,start(r)}-\mathrm {start(ps))}\quad \mathrm {underfill}_{\mathrm {start}}\!(\mathrm {r}\!,\mathrm {ps})\!=\!\mathrm {max}\mathrm {(0},\!\mathrm {start(ps)}\!-\!\mathrm {start(r))}\)

    \(\mathrm {overfill}_{\mathrm {end}}\mathrm {(r,ps)}=\mathrm {max}\mathrm {(0,end(ps)}-\mathrm {end(r))}\qquad \mathrm {underfill}_{\mathrm {end}}\mathrm {(r,ps)}\!=\!\mathrm {max}\mathrm {(0,end(r)}-\mathrm {end(ps))}\).

  2. 2.

    We use feature extraction in [13] and three layers perceptron for classifier step.

  3. 3.

    The internal steps are not important since the concentration is on the metrics.

  4. 4.

    For saving the space, the analysis of other classes are existed in our repository.

  5. 5.

    If the used segmentation algorithm generates more segments for longer events which is the case with the well-used sliding window method.

References

  1. Alemdar, H., Tunca, C., Ersoy, C.: Daily life behaviour monitoring for health assessment using machine learning: bridging the gap between domains. Pers. Ubiquit. Comput. 19(2), 303–315 (2014). https://doi.org/10.1007/s00779-014-0823-y

    Article  Google Scholar 

  2. Asghari, P., Soleimani, E., Nazerfard, E.: Online human activity recognition employing hierarchical hidden Markov models. J. Ambient. Intell. Humaniz. Comput. 11(3), 1141–1152 (2020). https://doi.org/10.1007/s12652-019-01380-5

    Article  Google Scholar 

  3. Awad, G., et al.: TRECVID 2020: a comprehensive campaign for evaluating video retrieval tasks across multiple application domains. In: Proceedings of TRECVID, pp. 1–55. NIST, USA (2021)

    Google Scholar 

  4. Bilen, C., Ferroni, G., Tuveri, F., Azcarreta, J., Krstulovic, S.: A framework for the robust evaluation of sound event detection. In: ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 61–65 (2020). https://doi.org/10.1109/ICASSP40776.2020.9052995

  5. Cook, D.J., Narayanan, C.K.: Activity Learning: Discovering, Recognizing, and Predicting Human Behavior from Sensor Data. Wiley Series on Parallel and Distributed Computing, 1st edn. Wiley (2015)

    Google Scholar 

  6. Dutta, J., Banerjee, B.: Online detection of abnormal events using incremental coding length. In: AAAI Conference on Artificial Intelligence (2015). https://ojs.aaai.org/index.php/AAAI/article/view/9799

  7. Fu, T.C.: A review on time series data mining. Eng. Appl. Artif. Intell. 24(1), 164–181 (2011). https://doi.org/10.1016/j.engappai.2010.09.007

    Article  Google Scholar 

  8. Gjoreski, H., et al.: Competitive live evaluations of activity-recognition systems. IEEE Pervasive Comput. 14(1), 70–77 (2015). https://doi.org/10.1109/MPRV.2015.3

    Article  Google Scholar 

  9. Hein, A., Kirste, T.: Generic performance metrics for continuous activity recognition. In: Bach, J., Edelkamp, S. (eds.) KI 2011. LNCS (LNAI), vol. 7006, pp. 139–143. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-24455-1_13

    Chapter  Google Scholar 

  10. Hwang, W.S., Yun, J.H., Kim, J., Kim, H.C.: Time-series aware precision and recall for anomaly detection. In: Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pp. 2241–2244. ACM, New York (2019). https://doi.org/10.1145/3357384.3358118

  11. Ionescu, R.T., Khan, F.S., Georgescu, M.I., Shao, L.: Object-centric auto-encoders and dummy anomalies for abnormal event detection in video. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), vol. 2019-June, pp. 7834–7843. IEEE (2019). https://doi.org/10.1109/CVPR.2019.00803, https://ieeexplore.ieee.org/document/8954309/

  12. Kasteren, T.V., Alemdar, H., Ersoy, C.: Effective performance metrics for evaluating activity recognition methods. In: ARCS (2011)

    Google Scholar 

  13. Krishnan, N.C., Cook, D.J.: Activity recognition on streaming sensor data. Pervasive Mobile Comput. 10(PART B), 138–154 (2014). https://doi.org/10.1016/j.pmcj.2012.07.003

    Article  Google Scholar 

  14. Lavin, A., Ahmad, S.: Evaluating real-time anomaly detection algorithms - the Numenta Anomaly Benchmark. In: IEEE 14th International Conference on Machine Learning and Applications (ICMLA), pp. 38–44. IEEE (2015). https://doi.org/10.1109/ICMLA.2015.141, http://ieeexplore.ieee.org/document/7424283/

  15. Lu, Y., Kumar, K.M., Nabavi, S.S., Wang, Y.: Future frame prediction using convolutional VRNN for anomaly detection. In: IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS). IEEE (2019). https://doi.org/10.1109/AVSS.2019.8909850

  16. Mesaros, A., Heittola, T., Virtanen, T.: Metrics for polyphonic sound event detection. Appl. Sci. (Switzerland) 6(6) (2016). https://doi.org/10.3390/app6060162

  17. Minnen, D., Westeyn, T.L., Starner, T., Ward, J.A., Lukowicz, P.: Performance metrics and evaluation issues for continuous activity recognition. In: Performance Metrics for Intelligent Systems, pp. 141–148. NIST, Gaithersburg (2006)

    Google Scholar 

  18. Modaresi, S., Osmani, A., Razzazi, M., Chibani, A.: Multimodal evaluation method for sound event detection. In: IEEE International Conference on Acoustics, Speech and Signal Processing, (ICASSP). IEEE (2022)

    Google Scholar 

  19. Ni, Q., García Hernando, A., de la Cruz, I.: The elderly’s independent living in smart homes: a characterization of activities and sensing infrastructure survey to facilitate services development. Sensors 15(5), 11312–11362 (2015). https://doi.org/10.3390/s150511312

  20. NIST: TRECVID 2004 Evaluation (2004). https://www-nlpir.nist.gov/projects/tv2004/index.html

  21. Osmani, A.: STCSP: a representation model for sequential patterns. Foundations and Applications of Spatio-Temporal Reasoning (FASTR) (2003). https://www.aaai.org/Library/Symposia/Spring/2003/ss03-03-010.php

  22. Perera, C., Zaslavsky, A., Christen, P., Georgakopoulos, D.: Context aware computing for the Internet of Things: a survey. IEEE Commun. Surv. Tutor. 16(1), 414–454 (2014). https://doi.org/10.1109/SURV.2013.042313.00197, http://ieeexplore.ieee.org/document/6512846/

  23. Qian, H., Pan, S.J., Miao, C.: Latent independent excitation for generalizable sensor-based cross-person activity recognition. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 13, pp. 11921–11929 (2021). https://ojs.aaai.org/index.php/AAAI/article/view/17416

  24. Ross, R.J., Kelleher, J.: Accuracy and timeliness in ML based activity recognition. In: Proceedings of the 13th AAAI Conference on Plan, Activity, and Intent Recognition, AAAIWS’13-13, vol. WS-13-13, pp. 39–46. AAAI Press (2013). https://doi.org/10.5555/2908241.2908247

  25. Stowell, D., Giannoulis, D., Benetos, E., Lagrange, M., Plumbley, M.D.: Detection and classification of acoustic scenes and events. IEEE Trans. Multimedia 17(10), 1733–1746 (2015). https://doi.org/10.1109/TMM.2015.2428998

    Article  Google Scholar 

  26. Tatbul, N., Lee, T.J., Zdonik, S., Alam, M., Gottschlich, J.: Precision and recall for time series. In: Neural Information Processing Systems (NIPS) (2018). https://papers.nips.cc/paper/7462-precision-and-recall-for-time-series

  27. Ward, J.A., Lukowicz, P., Gellersen, H.W.: Ward: performance metrics for activity recognition. ACM Trans. Intell. Syst. Technol. (2011). https://doi.org/10.1145/1889681.1889687

Download references

Author information

Authors and Affiliations

  1. SSRD Lab., Computer Engineering Department, Amirkabir University of Technology, Tehran, Iran

    Seyed M. R. Modaresi & Mohammadreza Razzazi

  2. LIPN-UMR-CNRS 7030 Lab., Sorbonne University Paris Nord, Paris, France

    Seyed M. R. Modaresi & Aomar Osmani

  3. LISSI Lab., Université Paris-Est Créteil, Paris, France

    Abdelghani Chibani

  4. Computer Science, Institute for Research in Fundamental Sciences, Tehran, Iran

    Mohammadreza Razzazi

Authors
  1. Seyed M. R. Modaresi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Aomar Osmani
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mohammadreza Razzazi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Abdelghani Chibani
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Seyed M. R. Modaresi .

Editor information

Editors and Affiliations

  1. Laboratory of Artificial Intelligence and Decision Support, University of Porto, Porto, Portugal

    João Gama

  2. School of Computing and Artificial Intelligence, Southwest Jiaotong University, Chengdu, China

    Tianrui Li

  3. National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China

    Yang Yu

  4. School of Computer Science and Technology, University of Science and Technology of China, Hefei, China

    Enhong Chen

  5. JD iCity, JD Technology & JD Intelligent Cities Research, Beijing, China

    Yu Zheng

  6. School of Computing and Artificial Intelligence, Southwest Jiaotong University, Chengdu, China

    Fei Teng

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Modaresi, S.M.R., Osmani, A., Razzazi, M., Chibani, A. (2022). Uniform Evaluation of Properties in Activity Recognition. In: Gama, J., Li, T., Yu, Y., Chen, E., Zheng, Y., Teng, F. (eds) Advances in Knowledge Discovery and Data Mining. PAKDD 2022. Lecture Notes in Computer Science(), vol 13281. Springer, Cham. https://doi.org/10.1007/978-3-031-05936-0_7

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-05936-0_7

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-05935-3

  • Online ISBN: 978-3-031-05936-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation