Skip to main content
SpringerLink
Log in

Searching for musical features using natural language queries: the C@merata evaluations at MediaEval

  • Original Paper
  • Published:
Language Resources and Evaluation Aims and scope Submit manuscript

Abstract

Musicological texts about classical music frequently include detailed technical discussions concerning the works being analysed. These references can be specific (e.g. C sharp in the treble clef) or general (fugal passage, Thor’s Hammer). Experts can usually identify the features in question in music scores but a means of performing this task automatically could be very useful for experts and beginners alike. Following work on textual question answering over many years as co-organisers of the QA tasks at the Cross Language Evaluation Forum, we decided in 2013 to propose a new type of task where the input would be a natural language phrase, together with a music score in MusicXML, and the required output would be one or more matching passages in the score. We report here on 3 years of the C@merata task at MediaEval. We describe the design of the task, the evaluation methods we devised for it, the approaches adopted by participant systems and the results obtained. Finally, we assess the progress which has been made in aligning natural language text with music and map out the main steps for the future. The novel aspects of this work are: (1) the task itself, linking musical references to actual music scores, (2) the evaluation methods we devised, based on modified versions of precision and recall, applied to demarcated musical passages, and (3) the progress which has been made in analysing and interpreting detailed technical references to music within texts.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Notes

  1. http://www.clef-initiative.eu/.

  2. http://trec.nist.gov/.

  3. http://www.ted.com.

  4. http://www.gutenberg.org/.

  5. http://en.wikisource.org/wiki/Wikisource:WikiProject_1911_Encyclop%C3%A6dia_Britannica.

  6. http://www.wikipedia.org/.

  7. http://www.oxfordmusiconline.com/grovemusic/.

  8. http://guidolib.sourceforge.net/GUIDO/.

  9. http://wordnet.princeton.edu/.

  10. http://sentiwordnet.isti.cnr.it/.

  11. http://nlp.stanford.edu/software/lex-parser.shtml.

  12. http://www.dbpedia-spotlight.org/.

  13. http://www.ims.uni-stuttgart.de/forschung/ressourcen/werkzeuge/matetools.en.html.

  14. http://www.last.fm/.

  15. http://babelfy.org/.

  16. http://www.songfacts.com/.

  17. http://tagme.d4science.org/tagme/.

  18. http://www.upf.edu/web/mtg/flabase.

  19. http://musicbrainz.org/.

  20. http://acousticbrainz.org/.

  21. http://jmcauley.ucsd.edu/data/amazon/.

  22. http://labrosa.ee.columbia.edu/millionsong/.

  23. http://lyricfind.com.

  24. http://nlp.stanford.edu/projects/glove/.

  25. http://spacy.io/.

  26. http://www.midi.org/techspecs/.

  27. http://www.musicxml.com/.

  28. http://music-encoding.org/home.

  29. http://www.tei-c.org/index.xml.

  30. www.whitehouse.gov/administration/president-obama.

  31. http://musescore.org/.

References

  • Artstein, R., & Poesio, M. (2008). Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4), 555–596.

    Article  Google Scholar 

  • Asooja, K., Ernala, S. K., & Buitelaar, P. (2014a). UNLP at the MediaEval 2014 C@merata task. In Proceedings of the MediaEval 2014 workshop Barcelona, Catalunya, Spain, October 16–17, 2014. http://ceur-ws.org/Vol-1263/mediaeval2014_submission_52.pdf. Accessed 15 Mar 2017.

  • Asooja, K., Ernala, S. K., & Buitelaar, P. (2014b). UNLP at the C@merata task: Question answering on musical scores. In Proceedings of the C@merata task at MediaEval 2015. http://csee.essex.ac.uk/camerata/unlp_asooja_long_14.pdf. Accessed 15 Mar 2017.

  • Asooja, K., Ernala, S. K., & Buitelaar, P. (2015a). UNLP at the MediaEval 2015 C@merata task. In Proceedings of the MediaEval 2015 workshop, Wurzen, Germany, September 14–15 2015. http://ceur-ws.org/Vol-1436/Paper86.pdf. Accessed 15 Mar 2017.

  • Asooja, K., Ernala, S. K., & Buitelaar, P. (2015b). UNLP at the 2015 C@merata task: question answering on musical scores by matching the passage sequences to musical entity sequences. http://csee.essex.ac.uk/camerata/unlp_asooja_long_15.pdf. Accessed 15 Mar 2017.

  • Baumann, S. (2003). Cultural metadata for artist recommendation. In Proceedings of the WEDELMUSIC.

  • Brochu, E., & de Freitas, N. (2003). “Name that song!”: A probabilistic approach to querying music and text. In: S. Becker, S. Thrun, & K. Obermayer (Eds.), Advances in neural information processing systems (Vol. 15, pp. 1505–1512). Cambridge, MA: MIT Press.

  • Buffa, M., & Cabrio, E. (2016). Natural language processing of song lyrics—WASABI project (web audio semantic aggregated in the browser for indexation). http://wimmics.inria.fr/node/60, http://wasabihome.i3s.unice.fr/. Accessed 22 Mar 2017.

  • Cleverdon, C. W. (1962). Report on the testing and analysis of an investigation into the comparative efficiency of indexing systems. Cranfield, England: College of Aeronautics.

    Google Scholar 

  • Collins, T. (2014a). Stravinsqi/De Montfort University at the MediaEval 2014 C@merata task. In Proceedings of the MediaEval 2014 Workshop Barcelona, Catalunya, Spain, October 16–17, 2014. http://ceur-ws.org/Vol-1263/mediaeval2014_submission_50.pdf. Accessed 29 Mar 2017.

  • Collins, T. (2014b). Stravinsqi/De Montfort University at the MediaEval 2014 C@merata task. In Proceedings of the C@merata Task at MediaEval 2015. http://csee.essex.ac.uk/camerata/dmun_collins_long_14.pdf.

  • Cooke, D. (1995). Bruckner, (Joseph) Anton. In S. Sadie (Ed.), New Grove dictionary of music and musicians (Vol. 3, Section 7, pp. 362–366). London, UK: Macmillan.

    Google Scholar 

  • Cuthbert, M. S., & Ariza C. (2010). music21: A toolkit for computer-aided musicology and symbolic music data. In Proceedings of the international symposium on music information retrieval, Utrecht, The Netherlands, August 09–13, 2010 (pp. 637–642).

  • Downie, J. S. (2003). Music information retrieval. Annual Review of Information Science and Technology, 37, 295–340.

    Article  Google Scholar 

  • Downie, J. S. (2008). The music information retrieval evaluation exchange (2005–2007): A window into music information retrieval research. Acoustical Science and Technology, 29(4), 247–255. https://doi.org/10.1250/ast.29.247.

    Article  Google Scholar 

  • Downie, S., & Nelson, M. (2000). Evaluation of a simple and effective music information retrieval method. In Proceedings of the ACM international conference on research and development in information retrieval (SIGIR) (pp. 73–80).

  • Dunning, T. (1993). Statistical identification of language. ACM Transactions on Programming Languages and Systems, 15(5), 745–770.

    Article  Google Scholar 

  • Ekman, P. (1993). Facial expression of emotion. American Psychologist, 48, 384–392.

    Article  Google Scholar 

  • Forte, A. (1973). The structure of atonal music. New Haven: Yale University Press.

    Google Scholar 

  • Fux, J. J. (1725). Gradus ad Parnassum (practical rules for learning composition translated from a work intitled Gradus ad Parnassum written originally in Latin by John Joseph Feux). Translated around 1750 by unknown translator. London: Welcker. http://imslp.nl/imglnks/usimg/3/31/IMSLP370587-PMLP187246-practicalrules%20fo00fuxj.pdf.

  • Gravier, G., Demarty, C. -H., Bredin, H., Ionescu, B., Boididou, C., Dellandrea, E., Choi, J., Riegler, M., Sutcliffe, R., Szoke, I., Jones, G., & Larson, M. (2016). Proceedings of the MediaEval 2016 Workshop, Hilversum, The Netherlands, October 20–21 2016. http://ceur-ws.org/Vol-1739/. Accessed 1 Apr 2017.

  • Hopkins, A. (1982). The nine symphonies of Beethoven. London: Pan Books.

    Google Scholar 

  • Hu, X., Choi, K., Hao, Y., Cunningham, J., Lee, J. H., Laplante, A., Bainbridge, D., & Downie, J. S. (2017). Exploring the music library association mailing list: A text mining approach. In Proceedings of ISMIR, Suzhou, China, 2017.

  • Hu, X., Downie, J. S., & Ehman, A. F. (2009). Lyric text mining in music mood classification. In Proceedings of ISMIR (pp. 411–416).

  • Huron, D. (1997). Humdrum and Kern: Selective feature encoding. In E. Selfridge-Field (Ed.), Beyond MIDI (pp. 375–401). Cambridge, MA: MIT Press.

    Google Scholar 

  • Huron, D. (2002). Music information processing using the Humdrum toolkit: Concepts, examples, and lessons. Computer Music Journal, 26(2), 11–26.

    Article  Google Scholar 

  • Katsiavalos, A., (2016). DMUN: A textual interface for content-based music information retrieval in the C@merata task for MediaEval 2016. In Proceedings of the MediaEval 2016 workshop, Hilversum, The Netherlands, October 20–21, 2016. http://ceur-ws.org/Vol-1739/MediaEval_2016_paper_56.pdf. Accessed 26 Mar 2017.

  • Katsiavalos, A., & Collins, T. (2015a). DMUN at the MediaEval 2015 C@merata task: The Stravinsqi algorithm. In Proceedings of the MediaEval 2015 workshop, Wurzen, Germany, September 14–15 2015. http://ceur-ws.org/Vol-1436/Paper88.pdf. Accessed 29 Mar 2017.

  • Katsiavalos, A., & Collins, T. (2015b). DMUN at the C@merata 2015 task: A full description of the Stravinsqi-June2015 algorithm. In Proceedings of the C@merata task at MediaEval 2014. http://csee.essex.ac.uk/camerata/dmun_collins_long_15.pdf. Accessed 29 Mar 2017.

  • Kini, N. (2014a). TCSL at the MediaEval 2014 C@merata task. In Proceedings of the MediaEval 2014 workshop Barcelona, Catalunya, Spain, October 16–17, 2014. http://ceur-ws.org/Vol-1263/mediaeval2014_submission_51.pdf. Accessed 29 Mar 2017.

  • Kini, N. (2014b). TCSL at the C@merata 2014 task: A tokenizing and parsing framework to understand queries on sheet music. In Proceedings of the C@merata task at MediaEval 2015. http://csee.essex.ac.uk/camerata/tcsl_kini_long_14.pdf. Accessed 29 Mar 2017.

  • Kini, N. (2015). TNKG at the MediaEval 2015 C@merata task. In Proceedings of the MediaEval 2015 workshop, Wurzen, Germany, September 14–15 2015. http://ceur-ws.org/Vol-1436/Paper89.pdf. Accessed 29 Mar 2017.

  • Kirkpatrick, R. (1953). Domenico Scarlatti. Princeton, NJ: Princeton University Press.

    Google Scholar 

  • Kitson, C. H. (1907). The art of counterpoint and its application as a decorative principle. Oxford, UK: Clarendon Press. https://archive.org/details/artofcounterpoin00kitsuoft. Accessed 1 Jan 2017.

  • Kuribayashi, T., Asano, Y., & Yoshikawa, M. (2013). Ranking method specialized for content descriptions of classical music. In Proceedings of the 22nd international world wide web conference, Rio de Janeiro, Brazil, 13–17 May, 2013 (pp. 141–142). http://dl.acm.org/citation.cfm?id=2487856. Accessed 1 Apr 2017.

  • Kuribayashi, T., Asano, Y., & Yoshikawa, M. (2015). Towards support for understanding classical music: Alignment of content descriptions on the web. In Proceedings of ISMIR 2015. http://ismir2015.uma.es/articles/287_Paper.pdf. Accessed 1 Apr 2017.

  • Larson, M., Ionescu, B., Anguera, X., Eskevich, M., Korshunov, P., Schedl, Soleymani, M., Petkos, G., Sutcliffe, R., Choi, J., & Jones, G. F. (2014). Proceedings of the MediaEval 2014 workshop, Barcelona, Spain, October 16–17 2014. http://ceur-ws.org/Vol-1263/. Accessed 1 Mar 2017.

  • Larson, M., Ionescu, B., Sjöberg, M., Anguera, X., Poignant, J., Riegler, M., Eskevich, M., Hauff, C., Sutcliffe, R., Jones, G. F., Yang, Y. -H., Soleymani, M., & Papadopoulos, S. (2015). Proceedings of the MediaEval 2015 workshop, Wurzen, Germany, September 14–15 2015. http://ceur-ws.org/Vol-1436/. Accessed 1 Mar 2017.

  • Lavecchia, C., Smaïli, K., & Langlois, D. (2007). Building parallel corpora from movies. In The 4th international workshop on natural language processing and cognitive science - NLPCS 2007, Jun 2007. Funchal, Madeira.

  • Logan, B., Kositsky, A., & Moreno, P. (2004). Semantic analysis of song lyrics. In IEEE ICME.

  • Mahedero, J. P. G., MartÍnez, Á., Cano, P., Koppenberger, M., & Gouyon, F. (2005). Natural language processing of lyrics. In Proceedings of the 13th annual ACM international conference on multimedia (MULTIMEDIA 2005) (pp. 475–478). New York, NY, USA: ACM.

  • McKay, C., et al. (2010). Evaluating the genre classification performance of lyrical features relative to audio, symbolic and cultural features. In: Proceedings of ISMIR.

  • Mihalcea, R., & Strapparava, C. (2012). Lyrics, music, and emotions. In Proceedings of the 2012 joint conference on empirical methods in natural language processing and computational natural language learning (EMNLP-CoNLL 2012) (pp. 590–599). Stroudsburg, PA, USA: Association for Computational Linguistics.

  • Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient estimation of word representations in vector space. In Proceedings of the international conference on learning representations, ICLR, Scottsdale, AZ, 2013. arXiv:1301.3781.

  • Mollá, D., & Vicedo, J. L. (2007). Question answering in restricted domains: An overview. Computational Linguistics, 33(1), 41–61.

    Article  Google Scholar 

  • Mytrova, M. (2016). The KIAM system in the C@merata task at MediaEval 2016. In Proceedings of the MediaEval 2016 workshop, Hilversum, The Netherlands, October 20–21, 2016. http://ceur-ws.org/Vol-1739/MediaEval_2016_paper_57.pdf. Accessed 14 Mar 2017.

  • Nadeau, D., & Sekine, S. (2007). A survey of named entity recognition and classification. Journal of Linguisticae Investigationes, 30(1), 3–26.

    Article  Google Scholar 

  • Ó Maidín, D. S. (2014a). OMDN at the MediaEval 2014 C@merata task. In Proceedings of the MediaEval 2014 workshop Barcelona, Catalunya, Spain, October 16–17, 2014. http://ceur-ws.org/Vol-1263/mediaeval2014_submission_48.pdf. Accessed 12 Mar 2017.

  • Ó Maidín, D. S. (2014b). OMDN at the MediaEval 2014 C@merata task: Using CPNView to answer questions about scores. In Proceedings of the C@merata task at MediaEval 2015. http://csee.essex.ac.uk/camerata/omdn_omaidin_long_14.pdf. Accessed 12 Mar 2017.

  • Ó Maidín, D. S. (2015a). OMDN at the MediaEval 2015 C@merata task. In Proceedings of the MediaEval 2015 workshop, Wurzen, Germany, September 14–15 2015. http://ceur-ws.org/Vol-1436/Paper88.pdf. Accessed 12 Mar 2017.

  • Ó Maidín, D. S. (2015b). OMDN at the C@merata 2015 task: A description of the CPNView approach to answering natural language questions about music scores. In Proceedings of the C@merata task at MediaEval 2014. http://csee.essex.ac.uk/camerata/omdn_omaidin_long_15.pdf. Accessed 12 Mar 2017.

  • Ó Maidín, D. (2016). OMDN at the C@merata 2016 task: Approaches to and issues arising from answering natural language questions about music scores. In Proceedings of the MediaEval 2016 workshop, Hilversum, The Netherlands, October 20–21, 2016. http://ceur-ws.org/Vol-1739/MediaEval_2016_paper_58.pdf. Accessed 16 Mar 2017.

  • O’Hara, T. (2011). Inferring the meaning of chord sequences via lyrics. In Proceedings of 2nd workshop on music recommendation and discovery (WOMRAD 2011), Chicago, IL, October.

  • Oramas, S. (2017). Knowledge extraction and representation learning for music recommendation and classification. Ph.D. Thesis, Universitat Pompeu Fabra Barcelona, 14th November, 2017. https://zenodo.org/record/1100973. Accessed 1 Mar 2018.

  • Oramas, S., Espinosa-Anke, L., Sordo, M., Saggion, H., & Serra, X. (2016a). Information extraction for knowledge base construction in the music domain. Journal of Knowledge & Data Engineering, 106, 70–83.

  • Oramas, S., Espinosa-Anke, L., Saggion, H., & Serra, X. (2016b). ELMD: An automatically generated entity linking gold standard dataset in the music domain. In Proceedings of LREC 2016.

  • Oramas, S., Espinosa-Anke, L., Lawlor, A., Serra, X., & Saggion, H. (2016c). Exploring music reviews for music genre classification and evolutionary studies. In 17th international society for music information retrieval conference. ISMIR 2016.

  • Oramas, S., Espinosa-Anke, L., Zhang, S., Saggion, H., & Serra, X. (2016d). Natural language processing for music information retrieval. Tutorial held at 17th international society for music information retrieval conference (ISMIR 2016). http://mtg.upf.edu/system/files/Tutorial_NLP4MIR.pdf. Accessed 1 September 2016.

  • Oramas, S., Espinosa-Anke, L., Saggion, H., & Serra, X. (2017a). Natural language processing for music information retrieval. Tutorial, Universitat Pompeu Fabra, Barcelona, 30th January 2017. http://mtg.upf.edu/system/files/Tutorial_NLP4MIR_2017.pdf. Accessed 1 Feb 2017.

  • Oramas, S., Nieto, O., Barbieri, F., & Serra, X. (2017b). Multi-label music genre classification from audio, text, and images using deep features. In Proceedings of the 18th international society of music information retrieval conference ISMIR 2017.

  • Oramas, S., & Sordo, M. (2016). Knowledge is out there: A new step in the evolution of music digital libraries. Fontes Artis Musicae, 63(4), 285–298.

    Google Scholar 

  • Oramas, S., Sordo, M., Espinosa-Anke, L., & Serra, X. (2015). A semantic-based approach for artist similarity. In 16th international society for music information retrieval conference (ISMIR 2015).

  • Oramas, S., Sordo, M., & Serra, X. (2014). Automatic creation of knowledge graphs from digital musical document libraries. In Conference in Interdisciplinary Musicology (CIM 2014).

  • Orio, N. (2006). Music retrieval: A tutorial and review. Foundations and Trends in Information Retrieval, 1(1), 1–90.

    Article  Google Scholar 

  • Peñas, A., Forner, P., Rodrigo, A., Sutcliffe, R., Forăscu, C., Mota, C. (2010). Overview of ResPubliQA 2010: Question answering evaluation over European Legislation. In Proceedings of ResPubliQA 2010. Held as part of CLEF 2010. http://csee.essex.ac.uk/staff/rsutcl/ResPubliQA2010_Overview-final.pdf. Accessed 10 Mar 2017.

  • Peñas, A., Forner, P., Sutcliffe, R., Rodrigo, A., Forascu, C., Alegria, I., Giampiccolo, D., Moreau, N., & Osenova, P. (2009). Overview of ResPubliQA 2009: Question answering evaluation over European legislation. In Notebook of the cross language evaluation forum, CLEF 2009, Corfu, Greece, 30 September–2 October. http://csee.essex.ac.uk/staff/rsutcl/CLEF2009wn-QACLEF-PenasEt2009.pdf. Accessed 10 Mar 2017.

  • Peñas, A., Hovy, E., Forner, P., Rodrigo, A., Sutcliffe, R., Forascu, C., & Sporleder, C. (2011). Overview of QA4MRE at CLEF 2011: question answering for machine reading evaluation. In Proceedings of QA4MRE-2011. Held as part of CLEF 2011. http://csee.essex.ac.uk/staff/rsutcl/QA4MRE-2011_Overview-final.doc. Accessed 10 Mar 2017.

  • Peñas, A., Hovy, E., Forner, P., Rodrigo, Á., Sutcliffe, R., & Morante, R. (2013). QA4MRE 2011–2013: Overview of question answering for machine reading evaluation. In P. Forner, H. Müller, R. Paredes, P. Rosso, & B. Stein (Eds.), Information access evaluation. Multilinguality, multimodality, and visualization. CLEF 2013. (Vol. 8138)., Lecture notes in computer science Berlin: Springer. https://doi.org/10.1007/978-3-642-40802-1_29.

    Google Scholar 

  • Peñas, A., Hovy, E., Forner, P., Rodrigo, A., Sutcliffe, R., Sporleder, C., Forascu, C., Benajiba, Y., & Osenova, P. (2012a). Overview of QA4MRE at CLEF 2012: Question answering for machine reading evaluation. In Proceedings of QA4MRE-2012. Held as part of CLEF 2012. http://csee.essex.ac.uk/staff/rsutcl/QA4MRE-2012-overview-v11.pdf.

  • Peñas, A., Magnini, B., Forner, P., Sutcliffe, R., Rodrigo, A., & Giampiccolo, D. (2012b). Question answering at the cross-language evaluation forum 2003–2010. Language Resources and Evaluation Journal, 46(2), 177–217.

  • Peñas, A., & Rodrigo, A. (2011). A simple measure to assess non-response. In Proceedings of 49th annual meeting of the association for computational linguistics: Human language technologies (ACL-HLT 2011), Portland, Oregon, USA. June 19–24. 2011.

  • Read, G. (1978). Music notation—A manual of modern practice. London, UK: Victor Gollancz.

    Google Scholar 

  • Sadie, S. (Ed.). (1980). The new Grove dictionary of music and musicians. London, UK: Macmillan.

    Google Scholar 

  • Salton, G., Wong, A., & Yang, C. S. (1975). A vector space model for automatic indexing. Communications of the ACM, 18(11), 613–620.

    Article  Google Scholar 

  • Schedl, M., Gómez, E., & Urbano, J. (2014). Music information retrieval: Recent developments and applications. Foundations and Trends in Information Retrieval, 8(2–3), 127–261.

    Article  Google Scholar 

  • Socher, R., Bauer, J., Manning, C. D., & Ng, A. Y. (2013). Parsing with compositional vector grammars. In Proceedings of ACL 2013.

  • Sterckx, L., Demeester, T., Deleu, J., Mertens, L., & Develder, C. (2014). Assessing quality of unsupervised topics in song lyrics. In 36th European Conference on Information Retrieval, Lecture Notes in Computer Science (ECIR 2014).

  • Sutcliffe, R. F. E. (2014). A description of the C@merata baseline system in Python 2.7 for answering natural language queries on MusicXML scores. University of Essex Technical Report, 21st May, 2014.

  • Sutcliffe, R. F. E., Collins, T., Hovy, E., Lewis, R., Fox, C., & Root, D. L. (2016). The C@merata task at MediaEval 2016: Natural language queries derived from exam papers, articles and other sources against classical music scores in MusicXML. In Proceedings of the MediaEval 2016 workshop, Hilversum, The Netherlands, October 20–21, 2016. http://ceur-ws.org/Vol-1739/MediaEval_2016_paper_55.pdf. Accessed 4 Feb 2017.

  • Sutcliffe, R. F. E., Crawford, T., Fox, C., Root, D. L., & Hovy, E. (2014a). The C@merata task at MediaEval 2014: Natural language queries on classical music scores. In Proceedings of MediaEval 2014 workshop, Barcelona, Spain, October 16–17 2014. http://ceur-ws.org/Vol-1263/mediaeval2014_submission_46.pdf. Accessed 12 Mar 2017.

  • Sutcliffe, R. F. E., Crawford, T., Fox, C., Root, D. L., & Hovy, E. (2014b). Shared evaluation of natural language queries against classical music scores: A full description of the C@merata 2014 task. In Proceedings of the C@merata task at MediaEval 2014. http://csee.essex.ac.uk/camerata/newest_camerata_long_14.pdf. Accessed 12 Mar 2017.

  • Sutcliffe, R. F. E., Crawford, T., Fox, C., Root, D. L., Hovy, E., & Lewis, R. (2015a). Relating natural language text to musical passages. In Proceedings of the 16th international society for music information retrieval conference, Malaga, Spain, 26–30 October, 2015. http://ismir2015.uma.es/articles/263_Paper.pdf. Accessed 20 Mar 2017.

  • Sutcliffe, R. F. E., Fox, C., Root, D. L., Hovy, E., & Lewis, R. (2015b). Shared evaluation of natural language queries against classical music scores: A full description of the C@merata 2015 task. In Proceedings of the C@merata task at MediaEval 2015. http://csee.essex.ac.uk/camerata/camerata_long_15.pdf. Accessed 4 Mar 2017.

  • Sutcliffe, R. F. E., Fox, C., Root, D. L., Hovy, E., & Lewis, R. (2015c). The C@merata task at MediaEval 2015: Natural language queries on classical music scores. In Proceedings of the MediaEval 2015 workshop, Wurzen, Germany, September 14–15 2015. http://ceur-ws.org/Vol-1436/Paper12.pdf. Accessed 4 Mar 2017.

  • Sutcliffe, R. F. E., & Liem, C. (2017). Capturing the meaning of complex texts about music. In Proceedings of ISMIR 2017, Suzhou, China. https://ismir2017.smcnus.org/lbds/Sutcliffe2017.pdf. Accessed 1 Dec 2017.

  • Sutcliffe, R. F. E., Ó Maidín, D. S., & Hovy E. (2017). The C@merata task at MediaEval 2017: Natural language queries about music, their JSON representations, and matching passages in MusicXML scores. In Proceedings of the MediaEval 2017 workshop, Trinity College Dublin, Ireland, September 13–15 2017. http://ceur-ws.org/Vol-1984/Mediaeval_2017_paper_35.pdf. Accessed 2 Nov 2017.

  • Sutcliffe, R., Peñas, A., Hovy, E., Forner, P., Rodrigo, A., Forascu, C., Benajiba, Y., & Osenova, P. (2013). Overview of QA4MRE main task at CLEF 2013. online working notes. In CLEF 2013 evaluation labs and workshop, 23–26 September 2013, Valencia, Spain. http://csee.essex.ac.uk/staff/rsutcl/CLEF2013wn-QA4MRE-SutcliffeEt2013.pdf. Accessed 4 Mar 2017.

  • Tata, S., & Di Eugenio, B. (2010). Generating fine-grained reviews of songs from album reviews. In Proceedings of the 48th ACL Annual Meeting, July (pp. 1376–1385).

  • Tsaptsinos, A. (2017). Lyrics-based music genre classification using a hierarchical attention network. In Proceedings of ISMIR, Suzhou, China, 2017.

  • van Rijsbergen, K. J. (1979). Information retrieval. London, UK: Butterworth. http://www.dcs.gla.ac.uk/Keith/Preface.html. Accessed 24 Jan 2017.

  • Viglianti, R. (2015). Enhancing music notation addressability. http://mith.umd.edu/research/project/enhancing-music-notation-addressability/. Accessed 10 Mar 2017.

  • Voorhees, E. M. (2002). Overview of the TREC 2002 question answering track. In Text retrieval conference (TREC). http://trec.nist.gov/pubs/trec11/papers/QA11.pdf. Accessed 15 Mar 2017.

  • Wan, S. (2014a). The CLAS system at the MediaEval 2014 C@merata task. In Proceedings of the MediaEval 2014 workshop Barcelona, Catalunya, Spain, October 16–17, 2014. http://ceur-ws.org/Vol-1263/mediaeval2014_submission_49.pdf. Accessed 12 Apr 2017.

  • Wan, S. (2014b). A description of the CLAS system at C@merata 2014. In Proceedings of the C@merata task at MediaEval 2015. http://csee.essex.ac.uk/camerata/clas_wan_long_14.pdf. Accessed 12 Apr 2017.

  • Wan, S. (2015a). CLAS at the MediaEval 2015 C@merata task. In Proceedings of the MediaEval 2015 workshop, Wurzen, Germany, September 14–15 2015. http://ceur-ws.org/Vol-1436/Paper85.pdf. Accessed 12 Apr 2017.

  • Wan, S. (2015b). CLAS at the C@merata 2015 task: Using unification between lexico-semantic query features and musical event metadata. In Proceedings of the C@merata task at MediaEval 2014. http://csee.essex.ac.uk/camerata/clas_wan_long_15.pdf. Accessed 12 Apr 2017.

  • Wang, Y., Kan, M., Nwe, T., Shenoy, A., & Yin, J. (2004). LyricAlly: Automatic synchronization of acoustic musical signals and textual lyrics. In Proceedings of MM’04, New York, October.

  • Whitman, B., & Ellis, D. (2004). Automatic record reviews. In Proceedings of ISMIR.

  • Yang, Z., Yang, D., Dyer, C., He, X., Smola, A., & Hovy, E. (2016). Hierarchical attention networks for document classification. In NAACL-HLT (pp. 1480–1489).

Download references

Author information

Authors and Affiliations

  1. School of CSEE, University of Essex, Colchester, UK

    Richard Sutcliffe

  2. Language Technologies Institute, Carnegie-Mellon University, Pittsburgh, PA, USA

    Eduard Hovy

  3. Department of Computer Science, Lafayette College, Easton, PA, USA

    Tom Collins

  4. CSIRO, Epping, NSW, Australia

    Stephen Wan

  5. Department of Computing, Goldsmiths, University of London, London, UK

    Tim Crawford

  6. Department of Music, University of Pittsburgh, Pittsburgh, PA, USA

    Deane L. Root

Authors
  1. Richard Sutcliffe
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Eduard Hovy
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tom Collins
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Stephen Wan
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Tim Crawford
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Deane L. Root
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Richard Sutcliffe.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sutcliffe, R., Hovy, E., Collins, T. et al. Searching for musical features using natural language queries: the C@merata evaluations at MediaEval. Lang Resources & Evaluation 53, 87–140 (2019). https://doi.org/10.1007/s10579-018-9422-2

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10579-018-9422-2

Keywords

Search

Navigation