Skip to main content
SpringerLink
Log in

Learning the harmonic analysis: is visualization an effective approach?

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Understanding the structure of music compositions requires an ability built over time, through the study of the music theory and the application of countless hours of practice. In particular for beginner learners, it can be a time-consuming and a tedious task due to the steep learning curve, especially for classical music. In this work we focus on a specific type of classical music composition, that is music in chorale style. Composing such type of music requires the study of rules that are related to many structural aspects of music, such as melodic and mainly harmonic aspects. To overcome these difficulties, interdisciplinary techniques could be exploited to understand whether extra (visual) information, provided through a specific software tool, could be useful to improve learning in a quick and effective way. We introduce therefore VisualHarmony, a tool that allows users to perform the harmonic analysis of music compositions by exploiting visual clues superimposed on the music scores. Since the harmonic analysis requires to identify similar tonalities and relevant degrees, the visualization approach proposed uses closest colors to represent similar tonalities and degrees. To assess the effectiveness of our idea, we performed an evaluation study involving 60 participants among experts (with a conservatory degree) and music students from conservatory classes. We derived interesting results about the overall learning capabilities (when using visualization in supporting learning) and music information retention when using VisualHarmony in a first phase to study rules, and then move on to the classic way of performing harmonization. This result allowed us to further demonstrate the effectiveness of visualization to learn classic music rules. Finally, we also obtained positive feedback about the system usability and the satisfaction of the users with regard to the easiness and the usefulness of the tested tool.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Notes

  1. VisualHarmony is available at http://www.isislab.it/delmal/MTAP/Tool.

  2. Images obtained by using Vischeck, http://www.vischeck.com/

  3. http://www.isislab.it/delmal/MTAP/

References

  1. Al-Musawi M, Ledesma A, Nieminen H, Korhonen I (2016) Implementation and user testing of a system for visualizing continuous health data and events. In: International conference on biomedical and health informatics (BHI), pp 156–159

  2. Bergstrom T, Karahalios K, Hart JC (2007) Isochords: visualizing structure in music. In: Proceedings of graphics interface, pp 297–304

  3. Blake A, Stapleton G, Rodgers P, Howse J (2014) How should we use colour in euler diagrams? In: Proceedings of the 7th international symposium on visual information communication and interaction, pp 149:149–149:158

  4. Chan W, Qu H, Mak W (2010) Visualizing the semantic structure in classical music works. IEEE Trans Vis Comput Graph 16(1):161–173

    Article  Google Scholar 

  5. Ciuha P, Klemenc B, Solina F (2010) Visualization of concurrent tones in music with colours. In: Proceedings of the international conference on multimedia, pp 1677–1680

  6. Conati C, Carenini G, Hoque E, Steichen B, Toker D (2014) Evaluating the impact of user characteristics and different layouts on an interactive visualization for decision making. Comput. Graph Forum, 33(3)

    Article  Google Scholar 

  7. De Felice C, De Prisco R, Malandrino D, Zaccagnino G, Zaccagnino R, Zizza R (2017) Splicing music composition. Inf Sci 385:196–212

    Article  Google Scholar 

  8. De Haas W, Magalhães J, Wiering F, Veltkamp CR (2013) Automatic functional harmonic analysis. Comput Music J 37(4):37–53

    Article  Google Scholar 

  9. De Prisco R, Zaccagnino G, Zaccagnino R (2010) EvoBassComposer: a multi-objective genetic algorithm for 4-voice compositions. In: Proceedings of the 12th annual conference on genetic and evolutionary computation, GECCO ’10, pp 817–818

  10. De Prisco R, Zaccagnino G, Zaccagnino R (2011) A multi-objective differential evolution algorithm for 4-voice compositions. In: 2011 IEEE Symposium on differential evolution (SDE), pp 1–8

  11. De Prisco R, Lettieri N, Malandrino D, Pirozzi D, Zaccagnino G, Zaccagnino R (2016) Visualization of music plagiarism: analysis and evaluation. In: 20th International conference information visualisation. Lisbon, Portugal, July 19-22, pp 177–182

  12. De Prisco R, Malandrino D, Pirozzi D, Zaccagnino G, Zaccagnino R (2017) Understanding the structure of musical compositions: is visualization an effective approach? Inf Vis 16(2):139–152

    Article  Google Scholar 

  13. De Prisco R, Malandrino D, Pirozzi D, Zaccagnino G, Zaccagnino R (2018) Evaluation study of visualisations for harmonic analysis of 4-part music. In: 22th International conference information visualisation, IV’2018 (10-13 July 2018)

  14. Doolittle PE, Hicks D (2003) Constructivism as a theoretical foundation for the use of technology in social studies. Theory Res Soc Educ 31(1):72–104

    Article  Google Scholar 

  15. Erra U, Iaccarino G, Malandrino D, Scarano V (2007) Personalizable edge services for Web accessibility. Univ Access Inf Soc 6(3):285–306

    Article  Google Scholar 

  16. Erra U, Malandrino D, Pepe L (2018) A methodological evaluation of natural user interfaces for immersive 3d graph explorations. J Vis Lang Comput 44:13–27

    Article  Google Scholar 

  17. Fonteles JH, Rodrigues MAF, Basso VED (2013) Creating and evaluating a particle system for music visualization. J Vis Lang Comput 24(6):472–482

    Article  Google Scholar 

  18. Iaccarino G, Malandrino D, Scarano V (2006) Personalizable edge services for web accessibility. In: Proceedings of the 2006 international cross-disciplinary workshop on web accessibility (W4A): building the mobile web: rediscovering accessibility?, W4A ’06, pp 23–32

  19. Koelle D Music programming for Java and JVM Languages. http://www.jfugue.org/

  20. Kroger P, Passos R, Sampaio M, Cidra GD (2008) Rameau: a system for automatic harmonic analysis. In: Proceedings of the 2008 international computer music conference, pp 273–281

  21. Lazar J, Feng JH, Hochheiser H (2010) Research methods in human-computer interaction

  22. Leon P, Ur B, Shay R, Wang Y, Balebako R, Cranor L (2012) Why Johnny can’t opt out: a usability evaluation of tools to limit online behavioral advertising. In: Proceedings of the SIGCHI conference on human factors in computing systems, pp 589–598

  23. Lewis JR (1995) IBM computer usability satisfaction questionnaires: psychometric evaluation and instructions for use. Int J Hum-Comput Interact 7(1):57–78

    Article  Google Scholar 

  24. Malandrino D, Manno I, Palmieri G, Scarano V, Tateo L, Casola D, Ferrante I, Foresta F (2015) Tailorable infrastructure to enhance mobile seamless learning. IEEE Trans Learn Technol 8(1):18–30

    Article  Google Scholar 

  25. Malandrino D, Pirozzi D, Zaccagnino G, Zaccagnino R (2015) A color-based visualization approach to understand harmonic structures of musical compositions. In: 19th International conference on information visualisation, IV 2015, Barcelona, Spain, July 22-24, 2015, pp 56–61

  26. Malandrino D, Pirozzi D, Zaccagnino R (2018) Visualization and music harmony: design, implementation, and evaluation. In: 22th International conference information visualisation, IV’2018 (10-13 July 2018)

  27. Malinowki S The music animation machine — “music worth watching”. http://www.musanim.com/

  28. Mardirossian A, Chew E (2007) Visualizing music: tonal progressions and distributions. In: Proceedings of the 8th international conference on music information retrieval, pp 189–194

  29. Maxwell SE, Delaney HD, Kelley K (2018) Designing experiments and analyzing data: a model comparison perspective, 3rd ed. Routledge

  30. Miyazaki R, Fujishiro I, Hiraga R (2003) Exploring MIDI datasets. In: ACM SIGGRAPH 2003 sketches and applications, pp 1–1

  31. Muelder C, Provan T, Ma KL (2010) Content based graph visualization of audio data for music library navigation. In: Int. sym. on multimedia (ISM), pp 129–136

  32. Ono JP, Corrêa D, Ferreira M, Mello R, Nonato LG (2015) Similarity graph: visual exploration of song collections. In: SIBGRAPI

  33. Piston W (1987) Harmony: 5th edn. W. W. Norton & Company Inc

  34. Piston W, DeVoto M (1987) Harmony. Norton

  35. Prisco RD, Malandrino D, Zaccagnino G, Zaccagnino R, Zizza R (2017) A kind of bio-inspired learning of music style. In: Computational intelligence in music, sound, art and design - 6th international conference, EvoMUSART 2017, Amsterdam, The Netherlands, April 19-21, 2017, Proceedings, pp 97–113

  36. Purchase H (2012) Experimental human computer interaction: a practical guide with visual examples. Cambridge University Press

  37. Ramirez R, Canepa C, Ghisio S, Kolykhalova K, Mancini M, Volta E, Volpe G, Giraldo S, Mayor O, Perez A et al (2018) Enhancing music learning with smart technologies. In: Proceedings of the 5th international conference on movement and computing. ACM, p 49

  38. Rentfrow PJ (2012) The role of music in everyday life: current directions in the social psychology of music. Soc Person Psychol Compass 6(5):402–416

    Article  Google Scholar 

  39. Riche NH, Dwyer T (2010) Untangling euler diagrams. IEEE Trans Vis Comput Graph 16(6):1090–1099

    Article  Google Scholar 

  40. Roediger HL, Butler AC (2011) The critical role of retrieval practice in long-term retention. Trends Cogn Sci 15(1):20–27

    Article  Google Scholar 

  41. Roediger HL, Karpicke JD (2006) The power of testing memory: basic research and implications for educational practice. Perspect Psychol Sci 1(3):181–210

    Article  Google Scholar 

  42. Sapp CS (2005) Visual hierarchical key analysis. Comput Entertain 3(4):1–19

    Article  Google Scholar 

  43. Shapiro SS, Wilk MB (1965) An analysis of variance test for normality (complete samples). Biometrika 52(3/4):591–611

    Article  MathSciNet  Google Scholar 

  44. Sidorov KA, Jones A, Marshall AD (2014) Music analysis as a smallest grammar problem. In: Proceedings of the 15th international society for music information retrieval conference, ISMIR 2014, Taipei, Taiwan, October 27-31, pp 301–306

  45. Smith S, Williams G (1997) A visualization of music. In: Proceedings of the 8th conference on visualization, pp 499–503

  46. Snydal J, Hearst M (2005) ImproViz: visual explorations of jazz improvisations. In: Human factors in computing systems, CHI EA ’05, pp 1805–1808

  47. Soriano A, Paulovich F, Nonato LG, Oliveira MCF (2014) Visualization of music collections based on structural content similarity. In: 2014 27th SIBGRAPI conference on graphics, patterns and images, pp 25–32

  48. Van Der Linden J, Schoonderwaldt E, Bird J, Johnson R (2011) Musicjacket—combining motion capture and vibrotactile feedback to teach violin bowing. IEEE Trans Instrum Meas 60(1):104–113

    Article  Google Scholar 

  49. Wattenberg M (2002) Arc diagrams: visualizing structure in strings. In: IEEE Symposium on information visualization (INFOVIS), pp 110–116

  50. Wattenberg M The shape of song. http://www.turbulence.org/Works/song/

Download references

Author information

Authors and Affiliations

  1. Dipartimento di Informatica, Università di Salerno, Via Giovanni Paolo II, 132 - 84084 -, Fisciano (SA), Italy

    Delfina Malandrino, Donato Pirozzi & Rocco Zaccagnino

Authors
  1. Delfina Malandrino
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Donato Pirozzi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Rocco Zaccagnino
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Delfina Malandrino.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendices

Appendix A: The harmonic function

In an earlier work we defined an evaluation function that reflects the classical music rules and, in addition, extracts statistical information from a corpus of existing music [7]. Specifically, we extracted statistical information from a set of Bach’s chorales.

The multi-objective evaluation function is composed of two objective functions: an harmonic function fh and a melodic function fm. For the purpose of this work, we only used the harmonic function, that can be defined as follows:

$$ f_{h}(C)=\sum\limits_{i=1}^{n-1}a_{i} w_{i} $$
(2)

This function evaluates the harmonic quality of a chorale C = (c1,…, cn) by considering all pairs of consecutive chords ci, ci+ 1. The objective is to maximize fh(C). In our approach the coefficients represent the style of Bach while the weights represents well known rules from the theory of harmony. More specifically, we consider the following three possible cases:

  1. 1.

    ci and ci+ 1 are chords in the same major tonality

  2. 2.

    ci and ci+ 1 are chords in the same minor tonality

  3. 3.

    ci+ 1 is identified as a modulation change, that is ci belongs to the previous tonality while ci+ 1 belongs to a new tonality (regardless of the mode, major or minor).

For each of these cases, we have defined a set of coefficients and weights. The weights wi are used to express the objective part of the evaluation while the coefficients are used to express a subjective component. The coefficients ai are normally obtained with a statistical analysis of Bach’s chorales.

  • Weights. We relied upon well-known rules from the theory of harmony. We used as reference the description of the major harmonic progressions given by [34, page 17]. Table 3 summarizes the “rules” for chord passages in a major tonality.

    To compute the weights wi we need to define a distribution for the classes “often”, “sometimes” and “seldom” (as defined in Table 3). So, given a specific chord ci, the weight for the next chord will be a function of the preceding one according to such a distribution. More precisely, we set a probability distribution (Xoften, Xsometimes, Xseldom) and we assign the weight as follows: ci+ 1 will be one of the chords in the “often” class the Xoften percentage of the times, one of the chords in the “sometimes” class the Xsometimes percentage of the times and one of the chord in the “seldom” class the Xseldom percentage of the times.

    To choose the best distribution we have used the following approach.

    We denote by d the distribution used, k the number of iterations and by pmax the maximum cardinality of a generated set of solutions. We fix the distribution d∈ Dist, where Dist is a set of distributions (Xoften, Xsometimes, Xseldom), built according to personal considerations about the meaning of “often”, “sometimes” and “seldom”. We remark that in our experiments, we have |Dist| = 50.

    Furthermore, we fix the number of iterations k∈K= {10, 50, 100, 500, 750, 1000, 2500, 5000, 7500, 10000}, and the max size pmax ∈ P = {10, 50, 100, 500, 750, 1000}.

    For each triple (d, k, pmax) we run 5 executions of the music splicing system based on the Tonality-degree representation and we choose the best solution Cbest, according to the evaluation function described in this section. In order to choose the best distribution we observed the structure of the generated solutions. In our approach we say that a chorale is well-formed whether it starts with a I degree and ends with the cadence V − I. This is a typical condition of well composed chorales. Thus, we compute the average number of music compositions well-formed. As better results, we have obtained that with d = (80, 15, 5), 87% of the solutions are well-formed, with d = (85, 10, 5), 81% are well-formed, with d = (70, 20, 10), 76% are well-formed, with d = (60, 25, 15), 73% are well-formed, and with d = (50, 40, 10), 69% are well-formed.

    So, we set as (Xoften, Xsometimes, Xseldom) = (80, 15, 5) the distribution used for the experiments.

    For example, if ci is chord I, then the coefficient for ci+ 1 will be 0.8/3 ≃ 0.26 for each one of I, IV and V, it will be about 0.26 for vi and 0.05/2 = 0.025 for ii and iii. Table 4 shows all the weights for chord passages within a major tonality.

    Similarly, we obtain the weights for chord passages within a minor tonality. Table 5 summarizes the typical chord passages suggested by [34]. The passages are very similar to the ones in a major tonality with some difference due to the possibility of using the VII chord in a minor tonality. Table 6 shows the corresponding weights. Notice that for the row the chord vii∘, there are no chords in the “sometimes” class; hence for this row we use a split of 95%-5% among the “often” and the “seldom” class.

    To assign the weights for consecutive chords when a change of tonality occurs, we used the distance in the circle of fifths (see Fig. 13), between the starting and the ending tonality. More specifically the weight wModulation(S, E) is the length of the shortest path from the starting tonality S to the ending tonality E. For example, the distance between D major and C minor is 5 because we can go from D major to C minor either counterclockwise using 5 steps or clockwise using 7 steps. As before, the weights for the chord passages are computed by defining a distribution for the classes “often”, “sometimes” and “seldom”, and so the values are always between 0 and 1. The weights for the modulations, instead, are computed as the distance on the circle of fifths of the tonalities. Thus, in order to maintain such weights in a comparable range with the weights for chord passages (between 0 and 1), we normalize the above distance over the maximum possible value, that is 6. For example, the (normalized) harmonic distances from C to G, D and A are, respectively 1/6, 2/6 and 3/6; the (normalized) harmonic distances from C to B♭,E♭ and A♭ are, 2/6, 3/6 and 4/6 respectively; the (normalized) harmonic distances from A♭ to B♭ minor, D♯ minor and G♯ minor are, 1/6, 2/6 and 3/6 respectively.

  • Coefficients. The coefficients have been obtained by performing a statistical analysis over a large corpus of Bach’s chorales. In details, we have written a program that analyses chorales and that extracts information from the used harmonization. In particular, we looked for adjacent chords and we counted the percentage of passages from one chord to the subsequent one. We analyzed a corpus of Bach’s chorales, ranging from chorale BWV 253 through chorale BWV 306 and from chorale BWV 314 to chorale BWV 438.

    Tables 7, 8, and 9 summarize the result of the analysis. Notice that the sum of all the percentages in Tables 7, 8 is smaller than 100 because there have been cases where our program was not able to identify the chords; those cases have not been classified, but simply ignored. In Table 9 many entries are not specified because those specific changes of tonality were not encountered.

Table 3 Typical harmonic chord passages in a major tonality
Full size table
Table 4 Weights for chord passages in a major tonality
Full size table
Table 5 Typical harmonic chord passages in a minor tonality (see [34])
Full size table
Table 6 Weights for chord passages in a minor tonality
Full size table
Table 7 Coefficients for consecutive chords in the same major tonality
Full size table
Table 8 Coefficients for consecutive chords in the same minor tonality
Full size table
Table 9 Coefficients aModulation for change of tonalities
Full size table
Fig. 13
figure 13

Circle of fifths

Full size image

Appendix B: Evaluation study

1.1 B.1 Demographic information

figure d

1.2 B.2 Preliminary survey

  • Q1: How many hours per day do you spend by playing music?

    • Less than 1 hour

    • Between 1 and 2 hours

    • More than 2 hours

  • Q2: How do you consider your experience with music?

    Inexpert ○ ○ ○ ○ ○ More than Expert

  • Q3: Do you play any musical instrument? If “Yes”, which one?

    • Piano

    • Guitar

    • Trumpet

    • Saxophone

    • Flute

    • Bass

    • Drums

    • Other (Add here which one)

    • None (I do not play any instrument)

  • Q4: How do you consider your experience with classical music?

    Inexpert ○ ○ ○ ○ ○ More than Expert

  • Q5: How do you consider your experience with harmonic analysis?

    Inexpert ○ ○○ ○ ○ More than Expert

1.3 B.3 Summary survey

  • Q1: In general, VisualHarmony was very easy to use

    • Strongly disagree

    • Disagree

    • Neither agree nor disagree

    • Agree

    • Strongly agree

  • Q2: Do you will continue to use VisualHarmony in a future?

    • Strongly disagree

    • Disagree

    • Neither agree nor disagree

    • Agree

    • Strongly agree

1.4 B.4 CSUQ: computer system usability questionnaire

Based on: Lewis, J. R. (1995) IBM Computer Usability Satisfaction Questionnaires: Psychometric Evaluation and Instructions for Use. International Journal of Human-Computer Interaction, 7:1, 57-78.

Ratings were on a 7-point Likert scale from Strongly disagree (1) to Strongly Agree (7).

  1. 1.

    Overall, I am satisfied with how easy it is to use VisualHarmony

    • Strongly Disagree / Strongly Agree

  1. 2.

    It was simple to use VisualHarmony

    • Strongly Disagree / Strongly Agree

  1. 3.

    I can effectively complete my work using VisualHarmony

    • Strongly Disagree / Strongly Agree

  1. 4.

    I am able to complete my work quickly using VisualHarmony

    • Strongly Disagree / Strongly Agree

  1. 5.

    I am able to efficiently complete my work using VisualHarmony

    • Strongly Disagree / Strongly Agree

  1. 6.

    I feel comfortable using VisualHarmony

    • Strongly Disagree / Strongly Agree

  1. 7.

    It was easy to learn to use VisualHarmony

    • Strongly Disagree / Strongly Agree

  1. 8.

    believe I became productive quickly using VisualHarmony

    • Strongly Disagree / Strongly Agree

  1. 9.

    The information (such as online help, On-screen messages, and other Documentation) provided with VisualHarmony is clear

    • Strongly Disagree / Strongly Agree

  1. 10.

    It is easy to find information I needed

    • Strongly Disagree / Strongly Agree

  1. 11.

    The information provided for VisualHarmony is easy to understand

    • Strongly Disagree / Strongly Agree

  1. 12.

    The organization of information on VisualHarmony screen is clear

    • Strongly Disagree / Strongly Agree

  1. 13.

    The interface of VisualHarmony is pleasant

    • Strongly Disagree / Strongly Agree

  1. 14.

    I like using the interface of VisualHarmony

    • Strongly Disagree / Strongly Agree

  1. 15.

    VisualHarmony has all the functions and capabilities I expect it to have

    • Strongly Disagree / Strongly Agree

  1. 16.

    Overall, I am satisfied with VisualHarmony

    • Strongly Disagree / Strongly Agree

Appendix C: Tasks

1.1 C.1 Task 1

Given the following bass line, provide your harmonization:

Fig. 14
figure 14

Bass line for Task 1

Full size image

1.2 C.2 Task 2

Given the following bass line, provide your harmonization:

Fig. 15
figure 15

Bass line for Task 2

Full size image

1.3 C.3 Task 3

Given the following bass line, provide your harmonization:

Fig. 16
figure 16

Bass line for Task 3

Full size image

1.4 C.4 Task 4

Given the following bass line, provide your harmonization:

Fig. 17
figure 17

Bass line for Task 4

Full size image

1.5 C.5 Task 5

Given the following bass line, provide your harmonization:

Fig. 18
figure 18

Bass line for Task 5

Full size image

1.6 C.6 Task 6

Given the following bass line, provide your harmonization:

Fig. 19
figure 19

Bass line for Task 6

Full size image

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Malandrino, D., Pirozzi, D. & Zaccagnino, R. Learning the harmonic analysis: is visualization an effective approach?. Multimed Tools Appl 78, 32967–32998 (2019). https://doi.org/10.1007/s11042-019-07879-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-019-07879-5

Keywords

Search

Navigation