Abstract
Music is rhythm, timbre, tones, intensity and performance. Conventional Western Music Notation (CWMN) is used to generate Music Scores in order to register music on paper. Optical Music Recognition (OMR) studies techniques and algorithms for converting music scores into a readable format for computers. This work presents a systematic literature review (SLR) searching for metrics and methods of evaluation and comparing for OMR systems and algorithms. The most commonly used metrics on OMR works are described. A research protocol is elaborated and executed. From 802 publications found, 94 are evaluated. All results are organized and classified focusing on metrics, stages, comparisons, OMR datasets and related works. Although there is still no standard methodology for evaluating OMR systems, a good number of datasets and metrics are already available and apply to all the stages of OMR. Some of the analyzed works can give good directions for future works.










Similar content being viewed by others
Notes
References
Adamska J, Piecuch M, Podgórski M, Walkiewicz P, Lukasik E (2015) Mobile system for optical music recognition and music sound generation. In: IFIP International conference on computer information systems and industrial management. Springer , pp 571–582
AG SIP (2017) Springer link. https://link.springer.com/
Bainbridge D, Bell T (1997) Dealing with superimposed objects in optical music recognition. In: Proceedings of the 6th International Conference on image processing and its applications. IET
Baró A, Riba P, Fornés A (2016) Towards the recognition of compound music notes in handwritten music scores. In: Proceedings of the International Conference on frontiers in handwriting recognition (ICFHR). IEEE, pp 465–470
Baumann S (1995) A simplified attributed graph grammar for high-level music recognition. In: Proceedings of the 3rd International Conference on document analysis and recognition. IEEE, vol 2, pp 1080–1083
Bellini P, Bruno I, Nesi P (2007) Assessing optical music recognition tools. Comput Music J 31(1):68–93
Bruder I, Ignatova T, Milewski L (2004) Knowledge-based scribe recognition in historical music archives. Lecture notes in computer science, pp 304–316
Bugge EP, Juncher KL, Mathiasen BS, Simonsen JG (2011) Using sequence alignment and voting to improve optical music recognition from multiple recognizers. In: Proceedings of the 12th international society for music information retrieval (ISMIR) conference
Burgoyne JA, Pugin L, Eustace G, Fujinaga I (2007) A comparative survey of image binarisation algorithms for optical recognition on degraded musical sources. In: Proceedings of the international Society for music information retrieval (ISMIR) conference, pp 509–512
Scopus EBV (2017) https://www.scopus.com/
Byrd D, Schindele M (2006) Prospects for improving omr with multiple recognizers. In: Proceedings of the int. Society for music information retrieval (ISMIR) conference, pp 41–46
Byrd D, Simonsen JG (2015) Towards a standard testbed for optical music recognition Definitions, metrics, and page images. J Music Res 44(3):169–195
Calvo-Zaragoza J, Barbancho I, Tardón LJ, Barbancho AM (2015) Avoiding staff removal stage in optical music recognition: application to scores written in white mensural notation. Pattern Anal Appl 18(4):933–943
Calvo-Zaragoza J, Micó L, Oncina J (2016) Music staff removal with supervised pixel classification. Int J Doc Anal Recogn (IJDAR) 19(3):211–219
Calvo-Zaragoza J, Oncina J (2017) Recognition of pen-based music notation with finite-state machines. Expert Syst Appl 72:395–406
Calvo-Zaragoza J, Pertusa A, Oncina J (2017) Staff-line detection and removal using a convolutional neural network. Mach Vis Appl 28:1–10
Calvo-Zaragoza J, Vigliensoni G, Fujinaga I (2017) Staff-line detection on grayscale images with pixel classification. In: Iberian conference on pattern recognition and image analysis. Springer, pp 279–286
Campos VB, Calvo-Zaragoza J, Toselli AH, Ruiz EV (2016) Sheet music statistical layout analysis. In: Proceedings of the International Conference on frontiers in handwriting recognition (ICFHR). IEEE, pp 313–318
Cardoso J, Rebelo A (2010) Robust staffline thickness and distance estimation in binary and gray-level music scores. In: Proceedings of the International Conference on pattern recognition (ICPR). IEEE, pp 1856–1859
Chen Y-S, Chen F-S, Teng C-H (2013) AN optical music recognition system for skew or inverted musical scores. International Journal of Pattern Recognition and Artificial Intelligence 27(7):1353005
Chen L, Duan K (2016) Midi-assisted egocentric optical music recognition. In: Proceedings of the IEEE winter conference on applications of computer vision (WACV). IEEE, pp 1–9
Dalitz C, Droettboom M, Pranzas B, Fujinaga I (2008) A comparative study of staff removal algorithms. IEEE Trans Pattern Anal Mach Intell 30(5):753–766
Dalitz C, Michalakis GK, Pranzas C (2008) Optical recognition of psaltic byzantine chant notation. Int J Doc Anal Recogn (IJDAR) 11(3):143–158
Davis J, Goadrich M (2006) The relationship between precision-recall and roc curves. In: Proceedings of the 23rd international conference on Machine learning. ACM, pp 233–240
Dinh CM, Yang HJ, Lee G, Kim SH (2016) Fast lyric area extraction from images of printed korean music scores. IEICE Trans Inf Syst 99(6):1576–1584
dos Santos Montagner I, Hirata R, Hirata NS (2014) A machine learning based method for staff removal. In: Proceedings of the International Conference on pattern recognition (ICPR). IEEE, pp 3162–3167
Dutta A, Pal U, Fornes A, Llados J (2010) An efficient staff removal approach from printed musical documents. In: Proceedings of the International Conference on pattern recognition (ICPR). IEEE, pp 1965–1968
Fahn C-S, Lu K-J (2014) Humanoid recognizing piano scores techniques. In: Proceedings of the International Conference on information science, electronics and electrical engineering (ISEEE). IEEE, vol 3, pp 1397–1402
Fang Y, Gui-fa T (2015) Visual music score detection with unsupervised feature learning method based on k-means. Int J Mach Learn Cybern 6(2):277–287
A. for Computing Machinery. Acm digital library (2017) https://dl.acm.org/
Fornes A, Lladós J, Sánchez G, Bunke H (2009) On the use of textural features for writer identification in old handwritten music scores. In: Proceedings of the 10th International Conference on document analysis and recognition (ICDAR). IEEE, pp 996–1000
Fornes A, Dutta A, Gordo A, Llados J (2011) The icdar 2011 music scores competition: Staff removal and writer identification. In: Proceedings of the International Conference on document analysis and recognition (ICDAR). IEEE, pp 1511–1515
Fornés A, Dutta A, Gordo A, Lladós J (2012) Cvc-muscima: a ground truth of handwritten music score images for writer identification and staff removal. Int J Doc Anal Recognit 15:1–9
Fornés A, Dutta A, Gordo A, Lladós J (2013) The 2012 music scores competitions: staff removal and writer identification. In: Graphics recognition. New trends and challenges. Springer, pp 173–186
Fornes A, Dutta A, Gordo A, Llados J (2013) The 2012 music scores competitions Staff removal and writer identification. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 7423 LNCS: 173–186
Fornés A, Lladós J, Sánchez G (2005) Primitive segmentation in old handwritten music scores. In: International workshop on graphics recognition. Springer, pp 279–290
Fornés A, Lladós J, Sánchez G (2007) Old handwritten musical symbol classification by a dynamic time warping based method. In: International workshop on graphics recognition. Springer, pp 51–60
Fornés A, Kieu VC, Visani M, Journet N, Dutta A (2013) The icdar/grec 2013 music scores competition: Staff removal. In: International workshop on graphics recognition. Springer, pp 207–220
Fornés A, Sánchez G (2014) Analysis and recognition of music scores. In: Handbook of document image processing and recognition. Springer, pp 749–774
Google (2017) Google scholar. https://scholar.google.com/
IEEE (2017) Ieee xplore digital library. http://ieeexplore.ieee.org/search/
Izmirli Ö, Sharma G (2012) Bridging printed music and audio through alignment using a mid-level score representation. In: Proceedings of the Int. Society for Music Information Retrieval (ISMIR) Conference, pp 61–66
Jastrzebska A, Lesinski W (2016) Optical music recognition as the case of imbalanced pattern recognition: a study of single classifiers. In: Knowledge, information and creativity support systems: Recent trends, advances and solutions. Springer, pp 493–505
Kato H, Inokuchi S (1992) A recognition system for printed piano music using musical knowledge and constraints. Structured Document Image Analysis:435–455
Kitchenham B (2004) Procedures for performing systematic reviews. Keele UK Keele Univ 33(2004):1–26
Kodirov E, Han S, Lee G-S, Kim Y (2014) Music with harmony: chord separation and recognition in printed music score images. In: Proceedings of the 8th International Conference on Ubiquitous Information Management and Communication. ACM, pp 50
Lesinski W, Jastrzebska A (2015) Optical music recognition: music Standard and cost-sensitive learning with imbalanced data. In: IFIP International conference on computer information systems and industrial management. Springer, pp 601–612
Limited IU (2017) Taylor & francis online. http://www.tandfonline.com/
Liu X (2012) Note symbol recognition for music scores. Intelligent Information and Database Systems 7197:263–273
Liu X, Zhou M, Xu P (2015) A robust method for musical note recognition. In: Proceedings of the 14th International Conference on computer-aided design and computer graphics (CAD/graphics). IEEE, pp 212–213
López B, et al. (2005) Staff and graphical primitive segmentation in old handwritten music scores. Artif Intell Res Dev 131:83
Luangnapa N, Silpavarangkura T, Nukoolkit C, Mongkolnam P (2012) Optical music recognition on android platform. Adv Inf Technol 344:106–115
Luckner M (2006) Recognition of noised patterns using non-disruption learning set. In: Proceedings of the 6th International Conference on intelligent systems design and applications (ISDA). IEEE, vol 1, pp 557–562
Malik R, Roy PP, Pal U, Kimura F (2013) Handwritten musical document retrieval using music-score spotting. In: Proceedings of the 12th international conference on document analysis and recognition (ICDAR). IEEE, pp 832–836
Märgner V, Abed HE (2014) Tools and metrics for document analysis systems evaluation. Handbook of Document Image Processing and Recognition:1011–1036
Mehta AA, Bhatt MS (2015) Optical music notes recognition for printed piano music score sheet. In: Proceedings of the International Conference on computer communication and informatics (ICCCI). IEEE, pp 1–6
Miyao H, Nakano Y (1995) Head and stem extraction from printed music scores using a neural network approach. In: Proceedings of the 3rd International Conference on document analysis and recognition. IEEE, vol 2, pp 1074–1079
Modayur BR, Ramesh V, Haralick RM, Shapiro LG (1993) MUSER A prototype musical score recognition system using mathematical morphology. Mach Vis Appl 6(2-3):140–150
Modayur BR, Ramesh V, Haralick RM, Shapiro LG (1993) Muser: A prototype musical score recognition system using mathematical morphology. Mach Vis Appl 6(2):140–150
Montagner IS, Hirata R, Hirata NS (2014) Learning to remove staff lines from music score images. In: Proceedings of the IEEE international conference on image processing (ICIP). IEEE, pp 2614– 2618
Montagner IS, Hirata NS, Hirata R (2017) Staff removal using image operator learning. Pattern Recogn 63:310–320
MuseScore BVBA. Musescore. https://musescore.org/
Na IS, Kim SH (2016) Music symbol recognition by a lag-based combination model. Multimedia Tools and Applications 76:1–17
Nhat VQ, Lee G (2014) Adaptive line fitting for staff detection in handwritten music score images. In: Proceedings of the 8th aInternational Conference on Ubiquitous Information Management and Communication. ACM, pp 99
Novotn? J., Pokorn? J (2015) Introduction to optical music recognition: Overview and practical challenges. In: Proceedings of the int. Workshop on databases, texts, specifications, and objects (DATESO), pp 65–76
Oh J, Son SJ, Lee S, Kwon J-W, Kwak N (2017) Online recognition of handwritten music symbols. Int J Doc Anal Recogn (IJDAR) 20(2):79–89
Otsu N (1979) A threshold selection method from gray-level histograms. IEEE Trans Syst Man Cybern 9(1):62–66
Padilla V, Marsden A, McLean A, Ng K (2014) Improving omr for digital music libraries with multiple recognisers and multiple sources. In: Proceedings of the 1st International Workshop on Digital Libraries for Musicology. ACM, pp 1–8
Pedersoli F, Tzanetakis G (2016) Document segmentation and classification into musical scores and text. Int J Doc Anal Recogn (IJDAR) 19(4):289–304
Piatkowska W, Nowak L, Pawlowski M, Ogorzalek M (2012) Stafflines pattern detection using the swarm intelligence algorithm. Computer Vision and Graphics 7594:557–564
Pinheiro Pereira RM, Matos CE, Braz Junior G, de Almeida JD, de Paiva AC (2016) A deep approach for handwritten musical symbols recognition. In: Proceedings of the 22nd Brazilian Symposium on Multimedia and the Web. ACM, pp 191–194
Pinto H (1978) Iniciacao ao violao, vol 1. Ed. Ricordi, Sao Paulo
Pinto T, Rebelo A, Giraldi GA, Cardoso J (2011) Music score binarization based on domain knowledge. In: IbPRIA. Springer, pp 700–708
Pugin L, Burgoyne JA, Fujinaga I (2007) Goal-directed evaluation for the improvement of optical music recognition on early music prints. In: Proceedings of the 7th ACM/IEEE-CS joint conference on Digital libraries. ACM, pp 303–304
Pugin L, Burgoyne JA, Fujinaga I (2007) Reducing costs for digitising early music with dynamic adaptation. In: International conference on theory and practice of digital libraries. Springer, pp 471– 474
Pugin L, Hockman J, Burgoyne JA, Fujinaga I (2008) Gamera versus aruspix–two optical music recognition approaches. In: Proceedings of the int. Society for music information retrieval (ISMIR) conference. Citeseer
Ramirez C, Ohya J (2010) Symbol classification approach for omr of square notation manuscripts. In: Proceedings of the int. Society for music information retrieval (ISMIR) conference, pp 549–554
Raphael C, Wang J (2011) New approaches to optical music recognition. In: Proceedings of the int. Society for music information retrieval (ISMIR) conference, pp 305–310
Raphael C, Jin R (2014) Optical music recognition on the international music score library project. In: Proceedings of the documento recognition and treieval (DDR) conference, pp 90210f–90210f
Rebelo A, Capela A, Da Costa JFP, Guedes C, Carrapatoso E, Cardoso J (2007) A shortest path approach for staff line detection. In: Proceedings of the 3rd int. Cont. on automated production of cross media content for multi-channel distribution (AXMEDIS). IEEE, pp 79–85
Rebelo A, Capela G, Cardoso J (2010) Optical recognition of music symbols. Int J Doc Anal Recognit 13(1):19–31
Rebelo A, Fujinaga I, Paszkiewicz F, Marcal ARS, Guedes C, Cardoso J (2012) Optical music recognition: state-of-the-art and open issues. Int J Multimed Inf Retrieval 1(3):173–190
Rebelo A, Marcal AR, Cardoso J (2013) Global constraints for syntactic consistency in omr: an ongoing approach. In: International conference image analysis and recognition. Springer, pp 734– 741
Reed KT, Parker JR (1996) Automatic computer recognition of printed music. In: Proceedings of the 13th International Conference on Pattern Recognition. IEEE, vol 3, pp 803–807
Riba P, Fornés A, Lladós J (2015) Towards the alignment of handwritten music scores. In: International workshop on graphics recognition. Springer, pp 103–116
Rossant F, Bloch I (2004) A fuzzy model for optical recognition of musical scores. Fuzzy Sets Syst 141(2):165–201
Rossant F, Bloch I (2005) Optical music recognition based on a fuzzy modeling of symbol classes and music writing rules. In: Proceedings of the IEEE international conference on image processing (ICIP). IEEE, vol 2, pp II–538
Rossant F, Bloch I (2007) Robust and adaptive omr system including fuzzy modeling, fusion of musical rules, and possible error detection. EURASIP J Appl Signal Process 2007(1):160–160
Schmucker M (2002) Staff line features as information carrier. In: Proceedings of the First International Symposium on Cyber Worlds. IEEE, pp 168–175
Sharif M, Arshad Q-A, Raza M, Khan WZ (2009) [comscan]: an optical music recognition system. In: Proceedings of the 7th International Conference on Frontiers of Information Technology. ACM, pp 34
Su M-C, Tew C-Y, Chen H-H (2001) Musical symbol recognition using som-based fuzzy systems. In: Proceedings of the joint 9th IFSA world congress and 20th NAFIPS international conference. IEEE, vol 4, pp 2150–2153
Su B, Lu S, Pal U, Tan CL (2012) An effective staff detection and removal technique for musical documents. In: Proceedings of the 10th IAPR international workshop on document analysis systems (DAS). IEEE, pp 160–164
Szwoch M (2005) A robust detector for distorted music staves. Lect Notes Comput Sci 3691:701–708
Szwoch M (2008) Using musicxml to evaluate accuracy of omr systems. Diagrammatic Representation and Inference 5223:419–422
Tambouratzis T (2011) Identification of key music symbols for optical music recognition and on-screen presentation. In: Proceedings of the international joint conference on neural networks (IJCNN). IEEE, pp 1935–1942
Tambouratzis T (2013) The digital music stand as a minimal processing custom-made optical music recognition system, part 1: Key music symbol recognition. Int J Intell Syst 28(5):474–504
Thomas V, Wagner C, Clausen M (2011) Ocr based post processing of omr for the recovery of transposing instruments in complex orchestral scores. In: Proceedings of the int. Society for music information retrieval (ISMIR) conference, pp 411–416
Timofte R, Van Gool L (2012) Automatic stave discovery for musical facsimiles. In: Asian conference on computer vision. Springer, pp 510–523
Viera AJ, Garrett JM, et al. (2005) Understanding interobserver agreement: the kappa statistic. Fam Med 37(5):360–363
Vigliensoni G, Burgoyne JA, Hankinson A, Fujinaga I (2011) Automatic pitch recognition in printed square-note notation. In: Proceedings of the int. Society for music information retrieval (ISMIR) conference
Visaniy M, Kieu VC, Fornés A, Journet N (2013) Icdar 2013 music scores competition: Staff removal. In: Proceedings of the 12th international conference on document analysis and recognition (ICDAR). IEEE, pp 1407–1411
Vo QN, Kim SH, Yang HJ, Lee G (2016) An mrf model for binarization of music scores with complex background. Pattern Recogn Lett 69:88–95
Wen C, Rebelo A, Zhang J, Cardoso J (2015) A new optical music recognition system based on combined neural network. Pattern Recogn Lett 58:1–7
Wijaya K, Bainbridge D (1999) Staff line restoration. Proceedings of the 7th International Conference on Image Processing and Its Applications
Wu F-HF, Jang J-SR (2014) An architecture for optical music recognition of numbered music notation. In: Proceedings of International Conference on Internet Multimedia Computing and Service. ACM, pp 241
Wu F-HF, Jang J-SR (2014) On the way to ambient media for sheet music by techniques of information retrieval. In: Proceedings of the IEEE international conference on multimedia and expo workshops (ICMEW). IEEE, pp 1–6
Wu F-HF (2016) An evaluation framework of optical music recognition in numbered music notation. In: Proceedings of the IEEE international symposium on multimedia (ISM). IEEE, pp 626–631
Yadid-Pecht O, Gerner M, Dvir L, Brutman E, Shimony U (1996) Recognition of handwritten musical notes by a modified neocognitron. Mach Vis Appl 9(2):65–72
Yoo J, Kim G, Lee G (2008) Mask matching for low resolution musical note recognition. In: Proceedings of the IEEE international symposium on signal processing and information technology. IEEE, pp 223–226
Acknowledgments
We would like to thank CAPES (Coordination for the Improvement of Higher Level Personnel) and CNPq (National Council for Scientific and Technological Development) from the Brazilian government and Fundação Araucária for their financial support on the project.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Mengarelli, L., Kostiuk, B., Vitório, J.G. et al. OMR metrics and evaluation: a systematic review. Multimed Tools Appl 79, 6383–6408 (2020). https://doi.org/10.1007/s11042-019-08200-0
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-019-08200-0