Skip to main content

Advertisement

SpringerLink
Log in

CSE database: extended annotations and new recommendations for ECG software testing

  • Original Article
  • Published:
Medical & Biological Engineering & Computing Aims and scope Submit manuscript

Abstract

Nowadays, cardiovascular diseases represent the most common cause of death in western countries. Among various examination techniques, electrocardiography (ECG) is still a highly valuable tool used for the diagnosis of many cardiovascular disorders. In order to diagnose a person based on ECG, cardiologists can use automatic diagnostic algorithms. Research in this area is still necessary. In order to compare various algorithms correctly, it is necessary to test them on standard annotated databases, such as the Common Standards for Quantitative Electrocardiography (CSE) database. According to Scopus, the CSE database is the second most cited standard database. There were two main objectives in this work. First, new diagnoses were added to the CSE database, which extended its original annotations. Second, new recommendations for diagnostic software quality estimation were established. The ECG recordings were diagnosed by five new cardiologists independently, and in total, 59 different diagnoses were found. Such a large number of diagnoses is unique, even in terms of standard databases. Based on the cardiologists’ diagnoses, a four-round consensus (4R consensus) was established. Such a 4R consensus means a correct final diagnosis, which should ideally be the output of any tested classification software. The accuracy of the cardiologists’ diagnoses compared with the 4R consensus was the basis for the establishment of accuracy recommendations. The accuracy was determined in terms of sensitivity = 79.20–86.81%, positive predictive value = 79.10–87.11%, and the Jaccard coefficient = 72.21–81.14%, respectively. Within these ranges, the accuracy of the software is comparable with the accuracy of cardiologists. The accuracy quantification of the correct classification is unique. Diagnostic software developers can objectively evaluate the success of their algorithm and promote its further development. The annotations and recommendations proposed in this work will allow for faster development and testing of classification software. As a result, this might facilitate cardiologists’ work and lead to faster diagnoses and earlier treatment.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Al-Aweel IC, Krishnamurthy KB, Hausdorff JM, Mietus JE, Ives JR, Blum AS, Schomer DL, Goldberger AL (1999). Post-Ictal Heart Rate Oscillations in Partial Epilepsy. Neurology 53(7):1590-1592. Available from: https://www.physionet.org/physiobank/database/szdb/szdb.shtml. Accessed 21 Oct 2016

  2. Albrecht P (1983). S-T segment characterization for long-term automated ECG analysis. Master’s thesis, MIT Dept. of Electrical Engineering and Computer Science

  3. American Heart Association ECG Database DVD (2015) ECRI Institute. Available from: https://www.ecri.org/components/Pages/AHA_ECG_DVD.aspx?tab=1. Accessed 21 Oct 2016

  4. Arif M, Malagore IA, Afsar FA, Ray AK, Chakraborty C (2012) Detection and Localization of Myocardial Infarction using K-nearest Neighbor Classifier. J Med Syst 36:279–289. doi:10.1007/s10916-010-9474-3

    Article  PubMed  Google Scholar 

  5. Arif M, Malagore IA, Afsar FA, Ray AK, Chakraborty C (2010). Automatic Detection and Localization of Myocardial Infarction Using Back Propagation Neural Networks. 2010 4th International Conference on Bioinformatics and Biomedical Engineering 8:1-4. doi:10.1109/ICBBE.2010.5514664

  6. Baim DS, Colucci WS, Monrad ES, Smith HS, Wright RF, Lanoue A, Gauthier DF, Ransil BJ, Grossman W, Braunwald E (1986) Survival of patients with severe congestive heart failure treated with oral milrinone. J Am Coll Cardiol 7:661–670. doi:10.1016/S0735-1097(86)80478-8

    Article  CAS  PubMed  Google Scholar 

  7. Bousseljot R, Kreiseler D, Schnabel A, Moody GB, Mazzei MG, Pisani P, Roggero N, Varanini M, Mark RG, Braaksma L, Zeelenberg C, Marchesi C (1997) Nutzung der EKG-Signaldatenbank CARDIODAT der PTB über das Internet: development, distribution and use. Biomed Tech/Biomed Eng 20:317–318. doi:10.1515/bmte.1995.40.s1.317

    Google Scholar 

  8. Cardiac monitors, heart rate meters, and alarms [American National Standard (ANSI/AAMI EC13:2002)] (2002) Association for the Advancement of Medical Instrumentation, Arlington. Available from: www.pauljbennett.com/pbennett/work/ec13/ec13.pdf. Accessed 21 Oct 2016

  9. Castro D, Felix P, Presedo J (2015) A Method for Context-Based Adaptive QRS Clustering in Real Time. IEEE J Biomed Health Inform 19:1660–1671. doi:10.1109/JBHI.2014.2361659

    Article  PubMed  Google Scholar 

  10. Dawoud F, Wagner GS, Moody G, Horacek BM (2007) Using inverse electrocardiography to image myocardial infarction-reflecting on the 2007 PhysioNet/Computers in Cardiology Challenge. J Electocardiol 41:630–635. doi:10.1016/j.jelectrocard.2008.07.022

    Article  Google Scholar 

  11. de Chazal P, O’Dwyer M, Reilly RB (2004) Automatic Classification of Heartbeats Using ECG Morphology and Heartbeat Interval Features. IEEE Trans Biomed Eng 51:1196–1206. doi:10.1109/TBME.2004.827359

    Article  PubMed  Google Scholar 

  12. Gath I, Inbar GF (2013). Advances in processing and pattern analysis of biological signals: revue littéraire mensuelle. Springer, S.l. Available from: http://www.springer.com/cn/book/9780306452154. Accessed 21 Oct 2016

  13. Goldberger AL, Amaral LAN, Glass L, Hausdorff JM, Ivanov PC, Mark RG, Mietus JE, Moody GB, Peng C-K, Stanley HE (2000) PhysioBank, physiotoolkit, and physionet: components of a new research resource for complex physiologic signals. Circulation 101:e215–e220. doi:10.1161/01.CIR.101.23.e215

    Article  CAS  PubMed  Google Scholar 

  14. Greenwald SD (1990). Improved detection and classification of arrhythmias in noise-corrupted electrocardiograms using contextual information. PhD thesis, Harvard-MIT Division of Health Sciences and Technology

  15. Greenwald SD (1986). Development and analysis of a ventricular fibrillation detector. Master’s thesis, MIT Dept. of Electrical Engineering and Computer Science

  16. Jager F, Taddei A, Moody GB, Emdin M, Antolic G, Dorn R, Smrdel A, Marchesi C, and Mark RG (2003) Long-term ST database: a reference for the development and evaluation of automated ischaemia detectors and for the study of the dynamics of myocardial ischaemia. Medical and Biological Engineering and Computing 41:172–183. Available from: https://physionet.org/physiobank/database/ltstdb/lt03/lt03.pdf. Accessed 21 Oct 2016

  17. Jenkins J et al. (2016) Ann Arbor Electrogram Libraries. Ann Arbor, Michigan USA. Available from: electrogram.com. Accessed 21 Oct 2016

  18. Jenkins D, Gerred S (1995–2014) ECG library. Available from: http://www.ecglibrary.com

  19. Korhonen I, Ojaniemi J, Niieminen K, Van Gils M, Heikela A, Kari A (1997) Building the IMPROVE data library. IEEE Eng Med Biol Mag 16:25–32. doi:10.1109/51.637114

    Article  CAS  PubMed  Google Scholar 

  20. Kutlu Y, Kuntalp D, Min LC (2011) A multi-stage automatic arrhythmia recognition and classification system. Comput Biol Med 41:37–45. doi:10.1016/j.compbiomed.2010.11.003

    Article  PubMed  Google Scholar 

  21. Laguna P, Mark R, Goldbergerg A, Moody G (1997) A database for evaluation of algorithms for measurement of QT and Other Waveform Intervals in the ECG. Comput Cardiol 24:673–676. Available from: https://physionet.org/physiobank/database/qtdb/doc/index.shtml. Accessed 27 Oct 2016

  22. Lu HL, Ong K, Chia P, Ray AK, Chakraborty C (2000) An automated ECG classification system based on a neuro-fuzzy system. Comput Cardiol 36:387–390. doi:10.1109/CIC.2000.898538

    Google Scholar 

  23. Lugovaya TS (2005). Biometric human identification based on electrocardiogram. Master’s thesis, Faculty of Computing Technologies and Informatics, Electrotechnical University, Saint-Petersburg, Russian Federation

  24. Martis RJ, Acharya UR, Mandana KM, Ray AK, Chakraborty C (2012) Application of principal component analysis to ECG signals for automated diagnosis of cardiac health. Expert Syst Appl 39:11792–11800. doi:10.1016/j.eswa.2012.04.072

    Article  Google Scholar 

  25. Martis RJ, Acharya UR, Min LC (2013) ECG beat classification using PCA, LDA, ICA and Discrete Wavelet Transform. Biomed Signal Process Control 8:437–448. doi:10.1016/j.bspc.2013.01.005

    Article  Google Scholar 

  26. Mason JW, Hancock EW, Gettes LS, Moody GB, Mazzei MG, Pisani P, Roggero N, Varanini M, Mark RG, Braaksma L, Zeelenberg C, Marchesi C (1997) Recommendations for the standardization and interpretation of the electrocardiogram. Part II: electrocardiography diagnostic statement list. A Scientific Statement from the American Heart Association Electrocardiography and Arrhythmias Committee, Council on Clinical Cardiology; The American College of Cardiology Foundation; and The Heart Rhythm Society. Endorsed by the International Society for Computerized Electrocardiology. Circulation. doi:10.1161/CIRCULATIONAHA.106.180201

    Google Scholar 

  27. Matonia A, Jezewski J, Kupka T, Horoba K, Wrobel J, Gacek A (2006) The influence of coincidence of fetal and maternal QRS complexes on fetal heart rate reliability. Med Biol Eng Comput 44(5):393–403. doi:10.1007/s11517-006-0054-0

    Article  PubMed  Google Scholar 

  28. Melillo P, Izzo R, Orrico A, Scala P, Attanasio M, Mirra M, De Luca N, Pecchia L, Frasch MG (2015-3-20). Automatic Prediction of Cardiovascular and Cerebrovascular Events Using Heart Rate Variability Analysis. Plos One 10:e0118504-. doi:10.1371/journal.pone.0118504

  29. Moody G, Goldberger A, McClennen S, Swiryn S (2001) Predicting the onset of paroxysmal atrial fibrillation: the Computers in Cardiology Challenge 2001. Comput Cardiol 28:113–116. doi:10.1109/CIC.2001.977604

    Google Scholar 

  30. Moody GB, Goldberger A, McClennen S, Swiryn S (2008) The Physionet/Computers in Cardiology challenge 2008: t-wave alternans. Comput Cardiol. doi:10.1109/CIC.2008.4749089

    Google Scholar 

  31. Moody GB, Mark RG, Glass L, Hausdorff JM, Ivanov PC, Mark RG, Mietus JE, Moody GB, Peng C-K, Stanley HE (2000) The impact of the MIT-BIH arrhythmia database: components of a new research resource for complex physiologic signals. IEEE Eng Med Biol Mag 20:45–50. doi:10.1109/51.932724

    Article  Google Scholar 

  32. Moody GB, Muldrow WE, Mark RG (1984). A noise stress test for arrhythmia detectors. Comput Cardiol 11:381–384. Available from: http://ecg.mit.edu/george/publications/nst-cinc-1984.pdf. Accessed 21 Oct 2016

  33. Moody GB, Mark RG (1983). A new method for detecting atrial fibrillation using R-R intervals. Comput in Cardiol 36:227-230. Available from: http://ecg.mit.edu/george/publications/afib-cinc-1983.pdf. Accessed 21 Oct 2016

  34. Moody GB, Mark RG, Goldberger AL (1988). Evaluation of the “TRIM” ECG data compressor. Comput Cardiol 15:167–170. Available from: http://ecg.mit.edu/george/publications/trim-cinc-1988.pdf. Accessed 21 Oct 2016

  35. Moody GE (2004). Spontaneous termination of atrial fibrillation: A challenge from Physionet and Computers in Cardiology 2004. Comput Cardiol 101–104. doi:10.1109/CIC.2004.1442881

  36. Petrutiu S, Sahakian AV, Swiryn S, Scala P, Attanasio M, Mirra M, De Luca N, Pecchia L, Frasch MG (2007) Abrupt changes in fibrillatory wave characteristics at the termination of paroxysmal atrial fibrillation in humans. Europace 9:466–470. doi:10.1093/europace/eum096

    Article  PubMed  Google Scholar 

  37. PhysioNet (2016) The research resource for complex physiologic signals. Available from: www.physionet.org. Accessed 21 Oct 2016

  38. Priori SG, Wilde AA, Horie M, Cho Y, Behr ER, Berul C, Blom N, Brugada J, Chiang C-E, Huikuri H, Kannankeril P, Krahn A, Leenhardt A, Moss A, Schwartz PJ, Shimizu W, Tomaselli G, Tracy C (2013) HRS/EHRA/APHRS Expert Consensus Statement on the Diagnosis and Management of Patients with Inherited Primary Arrhythmia Syndromes. Heart Rhythm 10:1932–1963. doi:10.1016/j.hrthm.2013.05.014

    Article  PubMed  Google Scholar 

  39. Scopus (2016) Available from: www.scopus.com. Accessed 21 Oct 2016

  40. Sheldon RS, Grubb BP, Olshansky B, Shen W-K, Calkins H, Brignole M, Raj SR, Krahn AD, Morillo CA, Stewart JM, Sutton R, Sandroni P, Friday KJ, Hachul DT, Cohen MI, Lau DH, Mayuga KA, Moak JP, Sandhu RK, Kanjwal K (2015) 2015 Heart Rhythm Society Expert Consensus Statement on the Diagnosis and Treatment of Postural Tachycardia Syndrome, Inappropriate Sinus Tachycardia, and Vasovagal Syncope. Heart Rhythm 12:e41–e63. doi:10.1016/j.hrthm.2015.03.029

    Article  PubMed  PubMed Central  Google Scholar 

  41. Taddei A, Biagini A, Distante G, Emdin M, Mazzei MG, Pisani P, Roggero N, Varanini M, Mark RG, Moody GB, Braaksma L, Zeelenberg C, Marchesi C (1991) The European ST-T database: development, distribution and use. Comput Cardiol 20:177–180. doi:10.1109/CIC.1990.144191

    Google Scholar 

  42. Taddei A, Distante G, Emdin M, Pisani P, Moody GB, Zeelenberg C, Marchesi C (1992) The European ST-T database: standard for evaluating systems for the analysis of ST-T changes in ambulatory electrocardiography. Eur Heart J 13:1164–1172. Available from: http://eurheartj.oxfordjournals.org/content/13/9/1164.long. Accessed 21 Oct 2016

  43. The CSE Working Party (1985) Recommendations for measurement standards in quantitative electrocardiography. Eur Heart J 10:815–825. Available from: http://eurheartj.oxfordjournals.org/content/6/10/815.long. Accessed 21 Oct 2016

  44. The CSE Working Party (1990) Common standards for quantitative electrocardiography. CD-ROM version of the CSE database

  45. University Rochester Medical Center (2012) Telemetric and holter ECG warehouse. Available from: http://thew-project.org/. Accessed 21 Oct 2016

  46. Willems JL (1988) CSE Multilead Atlas: measurement results–data set 3. Leuven

  47. Willems JL, Arnaud P, van Bemmel JH, et al. (1985) Assessment of the performance of electrocardiographic computer programs with the use of a reference database. Circulation 1985; 71:523-534. Available from: http://circ.ahajournals.org/content/71/3/523. Accessed 21 Oct 2016

  48. Willems JL, Abreu-Lima C, Arnaud P, van Bemmel JH, Brohet C, Degani R, Denis B, Gehring J, Graham I, van Herpen G. The diagnostic performance of computer programs for the interpretation of electrocardiograms. N Engl J Med. 1991;325:1767–1773. Available from: http://www.nejm.org/doi/pdf/10.1056/NEJM199112193252503. Accessed 27 Oct 2016

Download references

Acknowledgements

This work was supported by grant project GAČR P102/12/2034.

Author information

Authors and Affiliations

  1. Department of Biomedical Engineering, The Faculty of Electrical Engineering and Communication, Brno University of Technology, Technická 3058/10, 61600, Brno, Czech Republic

    Radovan Smíšek, Lucie Maršánová, Andrea Němcová, Martin Vítek & Jiří Kozumplík

  2. Department of Physiology, Faculty of Medicine, Masaryk University, Kamenice 753/5, 62500, Brno, Czech Republic

    Marie Nováková

Authors
  1. Radovan Smíšek
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lucie Maršánová
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Andrea Němcová
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Martin Vítek
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jiří Kozumplík
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Marie Nováková
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Radovan Smíšek.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (DOCX 23 kb)

Supplementary material 2 (XLSX 15 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Smíšek, R., Maršánová, L., Němcová, A. et al. CSE database: extended annotations and new recommendations for ECG software testing. Med Biol Eng Comput 55, 1473–1482 (2017). https://doi.org/10.1007/s11517-016-1607-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11517-016-1607-5

Keywords

Search

Navigation