Skip to main content

Advertisement

SpringerLink
Log in

OMR metrics and evaluation: a systematic review

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Music is rhythm, timbre, tones, intensity and performance. Conventional Western Music Notation (CWMN) is used to generate Music Scores in order to register music on paper. Optical Music Recognition (OMR) studies techniques and algorithms for converting music scores into a readable format for computers. This work presents a systematic literature review (SLR) searching for metrics and methods of evaluation and comparing for OMR systems and algorithms. The most commonly used metrics on OMR works are described. A research protocol is elaborated and executed. From 802 publications found, 94 are evaluated. All results are organized and classified focusing on metrics, stages, comparisons, OMR datasets and related works. Although there is still no standard methodology for evaluating OMR systems, a good number of datasets and metrics are already available and apply to all the stages of OMR. Some of the analyzed works can give good directions for future works.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Notes

  1. https://www.mendeley.com/

  2. http://www.cvc.uab.es/cvcmuscima/competition/index.htm

  3. http://www.cvc.uab.es/cvcmuscima/competition2013/

  4. http://gamera.informatik.hsnr.de/addons/musicstaves/testset-musicstaves.tar.gz

  5. http://www.cvc.uab.es/cvcmuscima/index_database.html

  6. http://imslp.org/wiki/Main_Page

  7. http://grfia.dlsi.ua.es/homus/

  8. http://mipal.snu.ac.kr/index.php/Repository

  9. http://www.digital-scriptorium.org/

  10. http://www.disit.org/5932

  11. http://www.scribeserver.com/NEUMES/index.html

  12. https://sites.google.com/site/elyorkodirovresearch/omr-chsr6306

References

  1. Adamska J, Piecuch M, Podgórski M, Walkiewicz P, Lukasik E (2015) Mobile system for optical music recognition and music sound generation. In: IFIP International conference on computer information systems and industrial management. Springer , pp 571–582

  2. AG SIP (2017) Springer link. https://link.springer.com/

  3. Bainbridge D, Bell T (1997) Dealing with superimposed objects in optical music recognition. In: Proceedings of the 6th International Conference on image processing and its applications. IET

  4. Baró A, Riba P, Fornés A (2016) Towards the recognition of compound music notes in handwritten music scores. In: Proceedings of the International Conference on frontiers in handwriting recognition (ICFHR). IEEE, pp 465–470

  5. Baumann S (1995) A simplified attributed graph grammar for high-level music recognition. In: Proceedings of the 3rd International Conference on document analysis and recognition. IEEE, vol 2, pp 1080–1083

  6. Bellini P, Bruno I, Nesi P (2007) Assessing optical music recognition tools. Comput Music J 31(1):68–93

    Article  Google Scholar 

  7. Bruder I, Ignatova T, Milewski L (2004) Knowledge-based scribe recognition in historical music archives. Lecture notes in computer science, pp 304–316

  8. Bugge EP, Juncher KL, Mathiasen BS, Simonsen JG (2011) Using sequence alignment and voting to improve optical music recognition from multiple recognizers. In: Proceedings of the 12th international society for music information retrieval (ISMIR) conference

  9. Burgoyne JA, Pugin L, Eustace G, Fujinaga I (2007) A comparative survey of image binarisation algorithms for optical recognition on degraded musical sources. In: Proceedings of the international Society for music information retrieval (ISMIR) conference, pp 509–512

  10. Scopus EBV (2017) https://www.scopus.com/

  11. Byrd D, Schindele M (2006) Prospects for improving omr with multiple recognizers. In: Proceedings of the int. Society for music information retrieval (ISMIR) conference, pp 41–46

  12. Byrd D, Simonsen JG (2015) Towards a standard testbed for optical music recognition Definitions, metrics, and page images. J Music Res 44(3):169–195

    Article  Google Scholar 

  13. Calvo-Zaragoza J, Barbancho I, Tardón LJ, Barbancho AM (2015) Avoiding staff removal stage in optical music recognition: application to scores written in white mensural notation. Pattern Anal Appl 18(4):933–943

    Article  MathSciNet  Google Scholar 

  14. Calvo-Zaragoza J, Micó L, Oncina J (2016) Music staff removal with supervised pixel classification. Int J Doc Anal Recogn (IJDAR) 19(3):211–219

    Article  Google Scholar 

  15. Calvo-Zaragoza J, Oncina J (2017) Recognition of pen-based music notation with finite-state machines. Expert Syst Appl 72:395–406

    Article  Google Scholar 

  16. Calvo-Zaragoza J, Pertusa A, Oncina J (2017) Staff-line detection and removal using a convolutional neural network. Mach Vis Appl 28:1–10

  17. Calvo-Zaragoza J, Vigliensoni G, Fujinaga I (2017) Staff-line detection on grayscale images with pixel classification. In: Iberian conference on pattern recognition and image analysis. Springer, pp 279–286

  18. Campos VB, Calvo-Zaragoza J, Toselli AH, Ruiz EV (2016) Sheet music statistical layout analysis. In: Proceedings of the International Conference on frontiers in handwriting recognition (ICFHR). IEEE, pp 313–318

  19. Cardoso J, Rebelo A (2010) Robust staffline thickness and distance estimation in binary and gray-level music scores. In: Proceedings of the International Conference on pattern recognition (ICPR). IEEE, pp 1856–1859

  20. Chen Y-S, Chen F-S, Teng C-H (2013) AN optical music recognition system for skew or inverted musical scores. International Journal of Pattern Recognition and Artificial Intelligence 27(7):1353005

  21. Chen L, Duan K (2016) Midi-assisted egocentric optical music recognition. In: Proceedings of the IEEE winter conference on applications of computer vision (WACV). IEEE, pp 1–9

  22. Dalitz C, Droettboom M, Pranzas B, Fujinaga I (2008) A comparative study of staff removal algorithms. IEEE Trans Pattern Anal Mach Intell 30(5):753–766

    Article  Google Scholar 

  23. Dalitz C, Michalakis GK, Pranzas C (2008) Optical recognition of psaltic byzantine chant notation. Int J Doc Anal Recogn (IJDAR) 11(3):143–158

    Article  Google Scholar 

  24. Davis J, Goadrich M (2006) The relationship between precision-recall and roc curves. In: Proceedings of the 23rd international conference on Machine learning. ACM, pp 233–240

  25. Dinh CM, Yang HJ, Lee G, Kim SH (2016) Fast lyric area extraction from images of printed korean music scores. IEICE Trans Inf Syst 99(6):1576–1584

    Article  Google Scholar 

  26. dos Santos Montagner I, Hirata R, Hirata NS (2014) A machine learning based method for staff removal. In: Proceedings of the International Conference on pattern recognition (ICPR). IEEE, pp 3162–3167

  27. Dutta A, Pal U, Fornes A, Llados J (2010) An efficient staff removal approach from printed musical documents. In: Proceedings of the International Conference on pattern recognition (ICPR). IEEE, pp 1965–1968

  28. Fahn C-S, Lu K-J (2014) Humanoid recognizing piano scores techniques. In: Proceedings of the International Conference on information science, electronics and electrical engineering (ISEEE). IEEE, vol 3, pp 1397–1402

  29. Fang Y, Gui-fa T (2015) Visual music score detection with unsupervised feature learning method based on k-means. Int J Mach Learn Cybern 6(2):277–287

    Article  Google Scholar 

  30. A. for Computing Machinery. Acm digital library (2017) https://dl.acm.org/

  31. Fornes A, Lladós J, Sánchez G, Bunke H (2009) On the use of textural features for writer identification in old handwritten music scores. In: Proceedings of the 10th International Conference on document analysis and recognition (ICDAR). IEEE, pp 996–1000

  32. Fornes A, Dutta A, Gordo A, Llados J (2011) The icdar 2011 music scores competition: Staff removal and writer identification. In: Proceedings of the International Conference on document analysis and recognition (ICDAR). IEEE, pp 1511–1515

  33. Fornés A, Dutta A, Gordo A, Lladós J (2012) Cvc-muscima: a ground truth of handwritten music score images for writer identification and staff removal. Int J Doc Anal Recognit 15:1–9

  34. Fornés A, Dutta A, Gordo A, Lladós J (2013) The 2012 music scores competitions: staff removal and writer identification. In: Graphics recognition. New trends and challenges. Springer, pp 173–186

  35. Fornes A, Dutta A, Gordo A, Llados J (2013) The 2012 music scores competitions Staff removal and writer identification. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 7423 LNCS: 173–186

  36. Fornés A, Lladós J, Sánchez G (2005) Primitive segmentation in old handwritten music scores. In: International workshop on graphics recognition. Springer, pp 279–290

  37. Fornés A, Lladós J, Sánchez G (2007) Old handwritten musical symbol classification by a dynamic time warping based method. In: International workshop on graphics recognition. Springer, pp 51–60

  38. Fornés A, Kieu VC, Visani M, Journet N, Dutta A (2013) The icdar/grec 2013 music scores competition: Staff removal. In: International workshop on graphics recognition. Springer, pp 207–220

  39. Fornés A, Sánchez G (2014) Analysis and recognition of music scores. In: Handbook of document image processing and recognition. Springer, pp 749–774

  40. Google (2017) Google scholar. https://scholar.google.com/

  41. IEEE (2017) Ieee xplore digital library. http://ieeexplore.ieee.org/search/

  42. Izmirli Ö, Sharma G (2012) Bridging printed music and audio through alignment using a mid-level score representation. In: Proceedings of the Int. Society for Music Information Retrieval (ISMIR) Conference, pp 61–66

  43. Jastrzebska A, Lesinski W (2016) Optical music recognition as the case of imbalanced pattern recognition: a study of single classifiers. In: Knowledge, information and creativity support systems: Recent trends, advances and solutions. Springer, pp 493–505

  44. Kato H, Inokuchi S (1992) A recognition system for printed piano music using musical knowledge and constraints. Structured Document Image Analysis:435–455

  45. Kitchenham B (2004) Procedures for performing systematic reviews. Keele UK Keele Univ 33(2004):1–26

    Google Scholar 

  46. Kodirov E, Han S, Lee G-S, Kim Y (2014) Music with harmony: chord separation and recognition in printed music score images. In: Proceedings of the 8th International Conference on Ubiquitous Information Management and Communication. ACM, pp 50

  47. Lesinski W, Jastrzebska A (2015) Optical music recognition: music Standard and cost-sensitive learning with imbalanced data. In: IFIP International conference on computer information systems and industrial management. Springer, pp 601–612

  48. Limited IU (2017) Taylor & francis online. http://www.tandfonline.com/

  49. Liu X (2012) Note symbol recognition for music scores. Intelligent Information and Database Systems 7197:263–273

  50. Liu X, Zhou M, Xu P (2015) A robust method for musical note recognition. In: Proceedings of the 14th International Conference on computer-aided design and computer graphics (CAD/graphics). IEEE, pp 212–213

  51. López B, et al. (2005) Staff and graphical primitive segmentation in old handwritten music scores. Artif Intell Res Dev 131:83

    Google Scholar 

  52. Luangnapa N, Silpavarangkura T, Nukoolkit C, Mongkolnam P (2012) Optical music recognition on android platform. Adv Inf Technol 344:106–115

    Article  Google Scholar 

  53. Luckner M (2006) Recognition of noised patterns using non-disruption learning set. In: Proceedings of the 6th International Conference on intelligent systems design and applications (ISDA). IEEE, vol 1, pp 557–562

  54. Malik R, Roy PP, Pal U, Kimura F (2013) Handwritten musical document retrieval using music-score spotting. In: Proceedings of the 12th international conference on document analysis and recognition (ICDAR). IEEE, pp 832–836

  55. Märgner V, Abed HE (2014) Tools and metrics for document analysis systems evaluation. Handbook of Document Image Processing and Recognition:1011–1036

  56. Mehta AA, Bhatt MS (2015) Optical music notes recognition for printed piano music score sheet. In: Proceedings of the International Conference on computer communication and informatics (ICCCI). IEEE, pp 1–6

  57. Miyao H, Nakano Y (1995) Head and stem extraction from printed music scores using a neural network approach. In: Proceedings of the 3rd International Conference on document analysis and recognition. IEEE, vol 2, pp 1074–1079

  58. Modayur BR, Ramesh V, Haralick RM, Shapiro LG (1993) MUSER A prototype musical score recognition system using mathematical morphology. Mach Vis Appl 6(2-3):140–150

    Article  Google Scholar 

  59. Modayur BR, Ramesh V, Haralick RM, Shapiro LG (1993) Muser: A prototype musical score recognition system using mathematical morphology. Mach Vis Appl 6(2):140–150

    Article  Google Scholar 

  60. Montagner IS, Hirata R, Hirata NS (2014) Learning to remove staff lines from music score images. In: Proceedings of the IEEE international conference on image processing (ICIP). IEEE, pp 2614– 2618

  61. Montagner IS, Hirata NS, Hirata R (2017) Staff removal using image operator learning. Pattern Recogn 63:310–320

    Article  Google Scholar 

  62. MuseScore BVBA. Musescore. https://musescore.org/

  63. Na IS, Kim SH (2016) Music symbol recognition by a lag-based combination model. Multimedia Tools and Applications 76:1–17

  64. Nhat VQ, Lee G (2014) Adaptive line fitting for staff detection in handwritten music score images. In: Proceedings of the 8th aInternational Conference on Ubiquitous Information Management and Communication. ACM, pp 99

  65. Novotn? J., Pokorn? J (2015) Introduction to optical music recognition: Overview and practical challenges. In: Proceedings of the int. Workshop on databases, texts, specifications, and objects (DATESO), pp 65–76

  66. Oh J, Son SJ, Lee S, Kwon J-W, Kwak N (2017) Online recognition of handwritten music symbols. Int J Doc Anal Recogn (IJDAR) 20(2):79–89

    Article  Google Scholar 

  67. Otsu N (1979) A threshold selection method from gray-level histograms. IEEE Trans Syst Man Cybern 9(1):62–66

    Article  Google Scholar 

  68. Padilla V, Marsden A, McLean A, Ng K (2014) Improving omr for digital music libraries with multiple recognisers and multiple sources. In: Proceedings of the 1st International Workshop on Digital Libraries for Musicology. ACM, pp 1–8

  69. Pedersoli F, Tzanetakis G (2016) Document segmentation and classification into musical scores and text. Int J Doc Anal Recogn (IJDAR) 19(4):289–304

    Article  Google Scholar 

  70. Piatkowska W, Nowak L, Pawlowski M, Ogorzalek M (2012) Stafflines pattern detection using the swarm intelligence algorithm. Computer Vision and Graphics 7594:557–564

  71. Pinheiro Pereira RM, Matos CE, Braz Junior G, de Almeida JD, de Paiva AC (2016) A deep approach for handwritten musical symbols recognition. In: Proceedings of the 22nd Brazilian Symposium on Multimedia and the Web. ACM, pp 191–194

  72. Pinto H (1978) Iniciacao ao violao, vol 1. Ed. Ricordi, Sao Paulo

    Google Scholar 

  73. Pinto T, Rebelo A, Giraldi GA, Cardoso J (2011) Music score binarization based on domain knowledge. In: IbPRIA. Springer, pp 700–708

  74. Pugin L, Burgoyne JA, Fujinaga I (2007) Goal-directed evaluation for the improvement of optical music recognition on early music prints. In: Proceedings of the 7th ACM/IEEE-CS joint conference on Digital libraries. ACM, pp 303–304

  75. Pugin L, Burgoyne JA, Fujinaga I (2007) Reducing costs for digitising early music with dynamic adaptation. In: International conference on theory and practice of digital libraries. Springer, pp 471– 474

  76. Pugin L, Hockman J, Burgoyne JA, Fujinaga I (2008) Gamera versus aruspix–two optical music recognition approaches. In: Proceedings of the int. Society for music information retrieval (ISMIR) conference. Citeseer

  77. Ramirez C, Ohya J (2010) Symbol classification approach for omr of square notation manuscripts. In: Proceedings of the int. Society for music information retrieval (ISMIR) conference, pp 549–554

  78. Raphael C, Wang J (2011) New approaches to optical music recognition. In: Proceedings of the int. Society for music information retrieval (ISMIR) conference, pp 305–310

  79. Raphael C, Jin R (2014) Optical music recognition on the international music score library project. In: Proceedings of the documento recognition and treieval (DDR) conference, pp 90210f–90210f

  80. Rebelo A, Capela A, Da Costa JFP, Guedes C, Carrapatoso E, Cardoso J (2007) A shortest path approach for staff line detection. In: Proceedings of the 3rd int. Cont. on automated production of cross media content for multi-channel distribution (AXMEDIS). IEEE, pp 79–85

  81. Rebelo A, Capela G, Cardoso J (2010) Optical recognition of music symbols. Int J Doc Anal Recognit 13(1):19–31

    Article  Google Scholar 

  82. Rebelo A, Fujinaga I, Paszkiewicz F, Marcal ARS, Guedes C, Cardoso J (2012) Optical music recognition: state-of-the-art and open issues. Int J Multimed Inf Retrieval 1(3):173–190

    Article  Google Scholar 

  83. Rebelo A, Marcal AR, Cardoso J (2013) Global constraints for syntactic consistency in omr: an ongoing approach. In: International conference image analysis and recognition. Springer, pp 734– 741

  84. Reed KT, Parker JR (1996) Automatic computer recognition of printed music. In: Proceedings of the 13th International Conference on Pattern Recognition. IEEE, vol 3, pp 803–807

  85. Riba P, Fornés A, Lladós J (2015) Towards the alignment of handwritten music scores. In: International workshop on graphics recognition. Springer, pp 103–116

  86. Rossant F, Bloch I (2004) A fuzzy model for optical recognition of musical scores. Fuzzy Sets Syst 141(2):165–201

    Article  MathSciNet  Google Scholar 

  87. Rossant F, Bloch I (2005) Optical music recognition based on a fuzzy modeling of symbol classes and music writing rules. In: Proceedings of the IEEE international conference on image processing (ICIP). IEEE, vol 2, pp II–538

  88. Rossant F, Bloch I (2007) Robust and adaptive omr system including fuzzy modeling, fusion of musical rules, and possible error detection. EURASIP J Appl Signal Process 2007(1):160–160

    MATH  Google Scholar 

  89. Schmucker M (2002) Staff line features as information carrier. In: Proceedings of the First International Symposium on Cyber Worlds. IEEE, pp 168–175

  90. Sharif M, Arshad Q-A, Raza M, Khan WZ (2009) [comscan]: an optical music recognition system. In: Proceedings of the 7th International Conference on Frontiers of Information Technology. ACM, pp 34

  91. Su M-C, Tew C-Y, Chen H-H (2001) Musical symbol recognition using som-based fuzzy systems. In: Proceedings of the joint 9th IFSA world congress and 20th NAFIPS international conference. IEEE, vol 4, pp 2150–2153

  92. Su B, Lu S, Pal U, Tan CL (2012) An effective staff detection and removal technique for musical documents. In: Proceedings of the 10th IAPR international workshop on document analysis systems (DAS). IEEE, pp 160–164

  93. Szwoch M (2005) A robust detector for distorted music staves. Lect Notes Comput Sci 3691:701–708

    Article  Google Scholar 

  94. Szwoch M (2008) Using musicxml to evaluate accuracy of omr systems. Diagrammatic Representation and Inference 5223:419–422

  95. Tambouratzis T (2011) Identification of key music symbols for optical music recognition and on-screen presentation. In: Proceedings of the international joint conference on neural networks (IJCNN). IEEE, pp 1935–1942

  96. Tambouratzis T (2013) The digital music stand as a minimal processing custom-made optical music recognition system, part 1: Key music symbol recognition. Int J Intell Syst 28(5):474–504

    Article  Google Scholar 

  97. Thomas V, Wagner C, Clausen M (2011) Ocr based post processing of omr for the recovery of transposing instruments in complex orchestral scores. In: Proceedings of the int. Society for music information retrieval (ISMIR) conference, pp 411–416

  98. Timofte R, Van Gool L (2012) Automatic stave discovery for musical facsimiles. In: Asian conference on computer vision. Springer, pp 510–523

  99. Viera AJ, Garrett JM, et al. (2005) Understanding interobserver agreement: the kappa statistic. Fam Med 37(5):360–363

    Google Scholar 

  100. Vigliensoni G, Burgoyne JA, Hankinson A, Fujinaga I (2011) Automatic pitch recognition in printed square-note notation. In: Proceedings of the int. Society for music information retrieval (ISMIR) conference

  101. Visaniy M, Kieu VC, Fornés A, Journet N (2013) Icdar 2013 music scores competition: Staff removal. In: Proceedings of the 12th international conference on document analysis and recognition (ICDAR). IEEE, pp 1407–1411

  102. Vo QN, Kim SH, Yang HJ, Lee G (2016) An mrf model for binarization of music scores with complex background. Pattern Recogn Lett 69:88–95

    Article  Google Scholar 

  103. Wen C, Rebelo A, Zhang J, Cardoso J (2015) A new optical music recognition system based on combined neural network. Pattern Recogn Lett 58:1–7

    Article  Google Scholar 

  104. Wijaya K, Bainbridge D (1999) Staff line restoration. Proceedings of the 7th International Conference on Image Processing and Its Applications

  105. Wu F-HF, Jang J-SR (2014) An architecture for optical music recognition of numbered music notation. In: Proceedings of International Conference on Internet Multimedia Computing and Service. ACM, pp 241

  106. Wu F-HF, Jang J-SR (2014) On the way to ambient media for sheet music by techniques of information retrieval. In: Proceedings of the IEEE international conference on multimedia and expo workshops (ICMEW). IEEE, pp 1–6

  107. Wu F-HF (2016) An evaluation framework of optical music recognition in numbered music notation. In: Proceedings of the IEEE international symposium on multimedia (ISM). IEEE, pp 626–631

  108. Yadid-Pecht O, Gerner M, Dvir L, Brutman E, Shimony U (1996) Recognition of handwritten musical notes by a modified neocognitron. Mach Vis Appl 9(2):65–72

    Article  Google Scholar 

  109. Yoo J, Kim G, Lee G (2008) Mask matching for low resolution musical note recognition. In: Proceedings of the IEEE international symposium on signal processing and information technology. IEEE, pp 223–226

Download references

Acknowledgments

We would like to thank CAPES (Coordination for the Improvement of Higher Level Personnel) and CNPq (National Council for Scientific and Technological Development) from the Brazilian government and Fundação Araucária for their financial support on the project.

Author information

Authors and Affiliations

  1. Postgraduate Program in Computer Science — PPGIa — Polytechnic School, Pontifícia Universidade Católica do Paraná - PUCPR, Av. Imaculada Conceição, 1155 - Prado Velho, Curitiba, Parana, 80215-901, Brazil

    Luciano Mengarelli, Bruno Kostiuk, João G. Vitório, Maicon A. Tibola, William Wolff & Carlos N. Silla Jr.

Authors
  1. Luciano Mengarelli
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bruno Kostiuk
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. João G. Vitório
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Maicon A. Tibola
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. William Wolff
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Carlos N. Silla Jr.
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Luciano Mengarelli.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mengarelli, L., Kostiuk, B., Vitório, J.G. et al. OMR metrics and evaluation: a systematic review. Multimed Tools Appl 79, 6383–6408 (2020). https://doi.org/10.1007/s11042-019-08200-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-019-08200-0

Keywords

Search

Navigation