Skip to main content
SpringerLink
Log in

Toward an understanding of the history and impact of user studies in music information retrieval

  • Published:
Journal of Intelligent Information Systems Aims and scope Submit manuscript

Abstract

Most Music Information Retrieval (MIR) researchers will agree that understanding users’ needs and behaviors is critical for developing a good MIR system. The number of user studies in the MIR domain has been gradually increasing since the early 2000s, reflecting this growing appreciation of the need for empirical studies of users. However, despite the growing number of user studies and the wide recognition of their importance, it is unclear how great their impact has been in the field: on how systems are developed, how evaluation tasks are created, and how MIR system developers in particular understand critical concepts such as music similarity or music mood. In this paper, we present our analysis on the growth, publication and citation patterns, topics, and design of 198 user studies. This is followed by a discussion of a number of issues/challenges in conducting MIR user studies and distributing the research results. We conclude by making recommendations to increase the visibility and impact of user studies in the field.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Notes

  1. Music Information Retrieval Evaluation eXhange (MIREX) is the annual evaluation campaign for various music information retrieval algorithms hosted by the International Music Information Retrieval Systems Evaluation Lab (IMIRSEL) at the University of Illinois at Urbana-Champaign.

  2. http://www.nzdl.org/greenstone3-mir-user-study/library/collection/mir-user-study/

References

  • Agresti, A., & Finlay, B. (2009). Statistical methods for the social sciences. Upper Saddle River: Pearson Prentice Hall.

    Google Scholar 

  • Berenzweig, A., Logan, B., Ellis, D.P.W., Whitman, B. (2004). A large-scale evaluation of acoustic and subjective music-similarity measures. Computer Music Journal, 28(2), 63–76.

    Article  Google Scholar 

  • Boltz, M., Schulkind, M., Kantra, S. (1991). Effects of background music on the remembering of filmed events. Memory & Cognition, 19(6), 593–606.

    Article  Google Scholar 

  • Bradford, S. (1985). Sources of information on specific subjects. Journal of Information Science, 10(4), 173–180.

    Article  Google Scholar 

  • Bull, M. (2005). No dead air! The iPod and the culture of mobile listening. Leisure Studies, 24(4), 343–355.

    Article  Google Scholar 

  • Cunningham, S.J. (2003). User studies: A first step in designing an MIR testbed. In The MIR/MDL evaluation project white paper collection, GSLIS (pp. 19–21). Champaign, IL.

  • Cunningham, S.J., Reeves, N., Britland, M. (2003). An ethnographic study of music information seeking: Implications for the design of a music digital library. In Proceedings of the 3rd ACM/IEEE-CS joint conference on digital libraries (pp. 5–16).

  • Dervin, B. (2003). Human studies and user studies: A call for methodological inter-disciplinarity. Information Research, 9(1). http://informationr.net/ir/9-1/paper166.html..

  • Dickson, W.J., & Roethlisberger, F.J. (1966). Counseling in an organization: A sequel to the Hawthorn Researches. Boston: Harvard University.

    Google Scholar 

  • Downie, J.S. (2003). Music information retrieval. In: B. Cronin (Ed.), Annual review of information science and technology 37 (pp. 295–340). Medford: Information Today.

    Google Scholar 

  • Drott, M.C. (1981). Bradford’s law: Theory, empiricism and the gaps between. In Library trends summer (Special issue on bibliometrics) (pp. 41–52).

  • Ellis, D.P.W., Whitman, B., Berenzweig, A., Lawrence, S. (2002). The quest for ground truth in musical artist similarity. In Proceedings of the 3rd international society for music information retrieval (ISMIR) conference (pp. 170–177).

  • Fanelli, D. (2012). Negative results are disappearing from most disciplines and countries. Scientometrics, 90, 891–904.

    Article  Google Scholar 

  • Futrelle, F., & Downie, J.S. (2002). Interdisciplinary communities and research issues in music information retrieval. In Proceedings of the 3rd ISMIR conference (pp. 215–221).

  • Greenberg, S., & Buxton, B. (2008). Usability evaluation considered harmful (some of the time). In Proceedings of the CHI’08 (pp. 111–120).

  • Gomes, C., Schneider, D., Moraes, K., De Souza, J. (2012). Crowdsourcing for music: Survey and taxonomy. In Proceedings of the IEEE SMCS conference.

  • Hansen, D.L., Schneiderman, B., Smith, M.A. (2011). Analyzing social media networks with NodeXL: Insights from a connected world. Burlington: Morgan Kaufmann.

    Google Scholar 

  • Hiemstra, D., Hauff, C., De Jong, F., Kraaij, W. (2007). SIGIR’s 30th anniversary: An analysis of trends in IR research and the topology of its community. ACM SIGIR Forum, 41(2), 18–24.

    Article  Google Scholar 

  • Lee, J.H. (2011). How similar is too similar?: Exploring users’ perceptions of similarity in playlist evaluation. In Proceedings of the 12th ISMIR conference (pp. 109–114).

  • Lee, J.H., & Downie, J.S. (2004). Survey of music information needs, uses, and seeking behaviours: Preliminary findings. In Proceedings of the 5th ISMIR conference (pp. 441–446).

  • Lee, J.H., Hill, T., Work, L. (2012). What does music mood mean for real users? In Proceedings of the 2012 iConference (pp. 112–119).

  • Lee, J.H., Jones, M.C., Downie, J.S. (2009). An analysis of ISMIR proceedings: Patterns of authorship, topic, and citation. In Proceedings of the 10th ISMIR conference (pp. 57–62).

  • Lee, J.H., & Waterman, N.M. (2012). Understanding user requirements for music information services. In Proceedings of the 13th ISMIR conference (pp. 253–258).

  • Levitin, D.J. (1994). Absolute memory for musical pitch: evidence from the production of learned melodies. Perception & Psychophysics, 54(4), 414–423.

    Article  Google Scholar 

  • Levitt, J.M., & Thelwall, M. (2009). The most highly cited Library and Information Science articles: interdisciplinary, first authors and citation patterns. Scientometrics, 78(1), 45–67.

    Article  Google Scholar 

  • Liu, X., Bollen, J., Nelson, M.L., Van de Sompel, H. (2005). Co-authorship networks in the digital library research community. Information Processing & Management, 41(6), 1462–1480.

    Article  Google Scholar 

  • McNab, R.J., Smith, L.A., Witten, I.H., Henderson, C.L., Cunningham, S.J. (1996). Towards the digital music library: Tune retrieval from acoustic input. In Proceedings of the 1st ACM international conference on digital libraries (pp. 11–18).

  • North, A.C., Hargreaves, D.J., Hargreaves, J.J. (2004). Uses of music in everyday life. Music Perception: An Interdisciplinary Journal, 22(1), 41–77.

    Article  Google Scholar 

  • North, A.C., Hargreaves, D.J., O’Neill, S.A. (2000). The importance of music to adolescents. British Journal of Educational Psychology, 70, 255–272.

    Article  Google Scholar 

  • Pauws, S., & Eggen, B. (2002). PATS: Realization and user evaluation of an automatic playlist generator. In Proceedings of the 3rd ISMIR conference (pp. 13–17).

  • Porter, M. (1997). An algorithm for suffix stripping. In: K. Sparck-Jones, & P. Willet (Eds.), Readings in information retrieval (pp. 313–316). San Francisco: Morgan Kaufmann.

    Google Scholar 

  • Rentfrow, P.J., & Gosling, S.D. (2003). The Do Re Mi’s of everyday life: the structure and personality correlates of music preference. Journal of Psychology and Social Psychology, 84(6), 1236–1256.

    Google Scholar 

  • Schedl, M., & Flexer, A. (2012). Putting the user in the center of music information retrieval. In Proceedings of the 13th ISMIR conference (pp. 385–390).

  • Shardanand, U., & Maes, P. (1995). Social information filtering: Algorithms for automating “word of mouth”. In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 210–217).

  • Sloboda, J.A., O’Neill, S.A., Ivaldi, A. (2001). Functions of music in everyday life: an exploratory study using the Experience Sampling Method. Musicae Scientiae, 5(1), 9–32.

    Google Scholar 

  • Smeaton, A.F., Keogh, G., Gurrin, C., McDonald, K., Sødring, T. (2002). Analysis of papers from twenty-five years of SIGIR conferences: what have we been doing for the last quarter of a century? ACM SIGIR Forum, 37(1), 49–53.

    Article  Google Scholar 

  • Thorne, F.C. (1977). The citation index: another case of spurious validity. Journal of Clinical Psychology, 33(4), 1157–1161.

    Article  MathSciNet  Google Scholar 

  • Voida, A., Grinter, R.E., Ducheneaut, N., Edwards, W.K., Newman, M.W. (2005). Listening in: Practices surrounding iTunes music sharing. In CHI’ 05 Proceedings of the SIGCHI conference on human factors in computing systems (pp. 191–200).

  • Weigl, D.M., & Guastavino, C. (2011). User studies in the music information retrieval literature. In Proceedings of the 12th ISMIR conference (pp. 335–340).

  • Weinstock, M. (1971). Citation indexes. In Encyclopedia of library and information science 5 (pp. 16–41). New York: Marcel Dekker.

    Google Scholar 

Download references

Acknowledgements

We thank Gary Gao, Tiffany Huang, and Ben Farabelli at University of Washington for their valuable contributions to this project, participants of the ISMIR 2012 Conference for their helpful feedback, David Bainbridge at the University of Waikato for creating the initial Greenstone digital library of MIR user studies, and the anonymous reviewers for their comments and suggestions.

Author information

Authors and Affiliations

  1. Information School, University of Washington, Mary Gates Hall, Ste 370, Seattle, WA, 98195, USA

    Jin Ha Lee

  2. Department of Computer Science, University of Waikato, Private Bag 3105, Hamilton, 3240, New Zealand

    Sally Jo Cunningham

Authors
  1. Jin Ha Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sally Jo Cunningham
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jin Ha Lee.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Lee, J.H., Cunningham, S.J. Toward an understanding of the history and impact of user studies in music information retrieval. J Intell Inf Syst 41, 499–521 (2013). https://doi.org/10.1007/s10844-013-0259-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10844-013-0259-2

Keywords

Search

Navigation