Abstract
Understanding the structure of music compositions requires an ability built over time, through the study of the music theory and the application of countless hours of practice. In particular for beginner learners, it can be a time-consuming and a tedious task due to the steep learning curve, especially for classical music. In this work we focus on a specific type of classical music composition, that is music in chorale style. Composing such type of music requires the study of rules that are related to many structural aspects of music, such as melodic and mainly harmonic aspects. To overcome these difficulties, interdisciplinary techniques could be exploited to understand whether extra (visual) information, provided through a specific software tool, could be useful to improve learning in a quick and effective way. We introduce therefore VisualHarmony, a tool that allows users to perform the harmonic analysis of music compositions by exploiting visual clues superimposed on the music scores. Since the harmonic analysis requires to identify similar tonalities and relevant degrees, the visualization approach proposed uses closest colors to represent similar tonalities and degrees. To assess the effectiveness of our idea, we performed an evaluation study involving 60 participants among experts (with a conservatory degree) and music students from conservatory classes. We derived interesting results about the overall learning capabilities (when using visualization in supporting learning) and music information retention when using VisualHarmony in a first phase to study rules, and then move on to the classic way of performing harmonization. This result allowed us to further demonstrate the effectiveness of visualization to learn classic music rules. Finally, we also obtained positive feedback about the system usability and the satisfaction of the users with regard to the easiness and the usefulness of the tested tool.












Similar content being viewed by others
Notes
VisualHarmony is available at http://www.isislab.it/delmal/MTAP/Tool.
Images obtained by using Vischeck, http://www.vischeck.com/
References
Al-Musawi M, Ledesma A, Nieminen H, Korhonen I (2016) Implementation and user testing of a system for visualizing continuous health data and events. In: International conference on biomedical and health informatics (BHI), pp 156–159
Bergstrom T, Karahalios K, Hart JC (2007) Isochords: visualizing structure in music. In: Proceedings of graphics interface, pp 297–304
Blake A, Stapleton G, Rodgers P, Howse J (2014) How should we use colour in euler diagrams? In: Proceedings of the 7th international symposium on visual information communication and interaction, pp 149:149–149:158
Chan W, Qu H, Mak W (2010) Visualizing the semantic structure in classical music works. IEEE Trans Vis Comput Graph 16(1):161–173
Ciuha P, Klemenc B, Solina F (2010) Visualization of concurrent tones in music with colours. In: Proceedings of the international conference on multimedia, pp 1677–1680
Conati C, Carenini G, Hoque E, Steichen B, Toker D (2014) Evaluating the impact of user characteristics and different layouts on an interactive visualization for decision making. Comput. Graph Forum, 33(3)
De Felice C, De Prisco R, Malandrino D, Zaccagnino G, Zaccagnino R, Zizza R (2017) Splicing music composition. Inf Sci 385:196–212
De Haas W, Magalhães J, Wiering F, Veltkamp CR (2013) Automatic functional harmonic analysis. Comput Music J 37(4):37–53
De Prisco R, Zaccagnino G, Zaccagnino R (2010) EvoBassComposer: a multi-objective genetic algorithm for 4-voice compositions. In: Proceedings of the 12th annual conference on genetic and evolutionary computation, GECCO ’10, pp 817–818
De Prisco R, Zaccagnino G, Zaccagnino R (2011) A multi-objective differential evolution algorithm for 4-voice compositions. In: 2011 IEEE Symposium on differential evolution (SDE), pp 1–8
De Prisco R, Lettieri N, Malandrino D, Pirozzi D, Zaccagnino G, Zaccagnino R (2016) Visualization of music plagiarism: analysis and evaluation. In: 20th International conference information visualisation. Lisbon, Portugal, July 19-22, pp 177–182
De Prisco R, Malandrino D, Pirozzi D, Zaccagnino G, Zaccagnino R (2017) Understanding the structure of musical compositions: is visualization an effective approach? Inf Vis 16(2):139–152
De Prisco R, Malandrino D, Pirozzi D, Zaccagnino G, Zaccagnino R (2018) Evaluation study of visualisations for harmonic analysis of 4-part music. In: 22th International conference information visualisation, IV’2018 (10-13 July 2018)
Doolittle PE, Hicks D (2003) Constructivism as a theoretical foundation for the use of technology in social studies. Theory Res Soc Educ 31(1):72–104
Erra U, Iaccarino G, Malandrino D, Scarano V (2007) Personalizable edge services for Web accessibility. Univ Access Inf Soc 6(3):285–306
Erra U, Malandrino D, Pepe L (2018) A methodological evaluation of natural user interfaces for immersive 3d graph explorations. J Vis Lang Comput 44:13–27
Fonteles JH, Rodrigues MAF, Basso VED (2013) Creating and evaluating a particle system for music visualization. J Vis Lang Comput 24(6):472–482
Iaccarino G, Malandrino D, Scarano V (2006) Personalizable edge services for web accessibility. In: Proceedings of the 2006 international cross-disciplinary workshop on web accessibility (W4A): building the mobile web: rediscovering accessibility?, W4A ’06, pp 23–32
Koelle D Music programming for Java and JVM Languages. http://www.jfugue.org/
Kroger P, Passos R, Sampaio M, Cidra GD (2008) Rameau: a system for automatic harmonic analysis. In: Proceedings of the 2008 international computer music conference, pp 273–281
Lazar J, Feng JH, Hochheiser H (2010) Research methods in human-computer interaction
Leon P, Ur B, Shay R, Wang Y, Balebako R, Cranor L (2012) Why Johnny can’t opt out: a usability evaluation of tools to limit online behavioral advertising. In: Proceedings of the SIGCHI conference on human factors in computing systems, pp 589–598
Lewis JR (1995) IBM computer usability satisfaction questionnaires: psychometric evaluation and instructions for use. Int J Hum-Comput Interact 7(1):57–78
Malandrino D, Manno I, Palmieri G, Scarano V, Tateo L, Casola D, Ferrante I, Foresta F (2015) Tailorable infrastructure to enhance mobile seamless learning. IEEE Trans Learn Technol 8(1):18–30
Malandrino D, Pirozzi D, Zaccagnino G, Zaccagnino R (2015) A color-based visualization approach to understand harmonic structures of musical compositions. In: 19th International conference on information visualisation, IV 2015, Barcelona, Spain, July 22-24, 2015, pp 56–61
Malandrino D, Pirozzi D, Zaccagnino R (2018) Visualization and music harmony: design, implementation, and evaluation. In: 22th International conference information visualisation, IV’2018 (10-13 July 2018)
Malinowki S The music animation machine — “music worth watching”. http://www.musanim.com/
Mardirossian A, Chew E (2007) Visualizing music: tonal progressions and distributions. In: Proceedings of the 8th international conference on music information retrieval, pp 189–194
Maxwell SE, Delaney HD, Kelley K (2018) Designing experiments and analyzing data: a model comparison perspective, 3rd ed. Routledge
Miyazaki R, Fujishiro I, Hiraga R (2003) Exploring MIDI datasets. In: ACM SIGGRAPH 2003 sketches and applications, pp 1–1
Muelder C, Provan T, Ma KL (2010) Content based graph visualization of audio data for music library navigation. In: Int. sym. on multimedia (ISM), pp 129–136
Ono JP, Corrêa D, Ferreira M, Mello R, Nonato LG (2015) Similarity graph: visual exploration of song collections. In: SIBGRAPI
Piston W (1987) Harmony: 5th edn. W. W. Norton & Company Inc
Piston W, DeVoto M (1987) Harmony. Norton
Prisco RD, Malandrino D, Zaccagnino G, Zaccagnino R, Zizza R (2017) A kind of bio-inspired learning of music style. In: Computational intelligence in music, sound, art and design - 6th international conference, EvoMUSART 2017, Amsterdam, The Netherlands, April 19-21, 2017, Proceedings, pp 97–113
Purchase H (2012) Experimental human computer interaction: a practical guide with visual examples. Cambridge University Press
Ramirez R, Canepa C, Ghisio S, Kolykhalova K, Mancini M, Volta E, Volpe G, Giraldo S, Mayor O, Perez A et al (2018) Enhancing music learning with smart technologies. In: Proceedings of the 5th international conference on movement and computing. ACM, p 49
Rentfrow PJ (2012) The role of music in everyday life: current directions in the social psychology of music. Soc Person Psychol Compass 6(5):402–416
Riche NH, Dwyer T (2010) Untangling euler diagrams. IEEE Trans Vis Comput Graph 16(6):1090–1099
Roediger HL, Butler AC (2011) The critical role of retrieval practice in long-term retention. Trends Cogn Sci 15(1):20–27
Roediger HL, Karpicke JD (2006) The power of testing memory: basic research and implications for educational practice. Perspect Psychol Sci 1(3):181–210
Sapp CS (2005) Visual hierarchical key analysis. Comput Entertain 3(4):1–19
Shapiro SS, Wilk MB (1965) An analysis of variance test for normality (complete samples). Biometrika 52(3/4):591–611
Sidorov KA, Jones A, Marshall AD (2014) Music analysis as a smallest grammar problem. In: Proceedings of the 15th international society for music information retrieval conference, ISMIR 2014, Taipei, Taiwan, October 27-31, pp 301–306
Smith S, Williams G (1997) A visualization of music. In: Proceedings of the 8th conference on visualization, pp 499–503
Snydal J, Hearst M (2005) ImproViz: visual explorations of jazz improvisations. In: Human factors in computing systems, CHI EA ’05, pp 1805–1808
Soriano A, Paulovich F, Nonato LG, Oliveira MCF (2014) Visualization of music collections based on structural content similarity. In: 2014 27th SIBGRAPI conference on graphics, patterns and images, pp 25–32
Van Der Linden J, Schoonderwaldt E, Bird J, Johnson R (2011) Musicjacket—combining motion capture and vibrotactile feedback to teach violin bowing. IEEE Trans Instrum Meas 60(1):104–113
Wattenberg M (2002) Arc diagrams: visualizing structure in strings. In: IEEE Symposium on information visualization (INFOVIS), pp 110–116
Wattenberg M The shape of song. http://www.turbulence.org/Works/song/
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendices
Appendix A: The harmonic function
In an earlier work we defined an evaluation function that reflects the classical music rules and, in addition, extracts statistical information from a corpus of existing music [7]. Specifically, we extracted statistical information from a set of Bach’s chorales.
The multi-objective evaluation function is composed of two objective functions: an harmonic function fh and a melodic function fm. For the purpose of this work, we only used the harmonic function, that can be defined as follows:
This function evaluates the harmonic quality of a chorale C = (c1,…, cn) by considering all pairs of consecutive chords ci, ci+ 1. The objective is to maximize fh(C). In our approach the coefficients represent the style of Bach while the weights represents well known rules from the theory of harmony. More specifically, we consider the following three possible cases:
- 1.
ci and ci+ 1 are chords in the same major tonality
- 2.
ci and ci+ 1 are chords in the same minor tonality
- 3.
ci+ 1 is identified as a modulation change, that is ci belongs to the previous tonality while ci+ 1 belongs to a new tonality (regardless of the mode, major or minor).
For each of these cases, we have defined a set of coefficients and weights. The weights wi are used to express the objective part of the evaluation while the coefficients are used to express a subjective component. The coefficients ai are normally obtained with a statistical analysis of Bach’s chorales.
Weights. We relied upon well-known rules from the theory of harmony. We used as reference the description of the major harmonic progressions given by [34, page 17]. Table 3 summarizes the “rules” for chord passages in a major tonality.
To compute the weights wi we need to define a distribution for the classes “often”, “sometimes” and “seldom” (as defined in Table 3). So, given a specific chord ci, the weight for the next chord will be a function of the preceding one according to such a distribution. More precisely, we set a probability distribution (Xoften, Xsometimes, Xseldom) and we assign the weight as follows: ci+ 1 will be one of the chords in the “often” class the Xoften percentage of the times, one of the chords in the “sometimes” class the Xsometimes percentage of the times and one of the chord in the “seldom” class the Xseldom percentage of the times.
To choose the best distribution we have used the following approach.
We denote by d the distribution used, k the number of iterations and by pmax the maximum cardinality of a generated set of solutions. We fix the distribution d∈ Dist, where Dist is a set of distributions (Xoften, Xsometimes, Xseldom), built according to personal considerations about the meaning of “often”, “sometimes” and “seldom”. We remark that in our experiments, we have |Dist| = 50.
Furthermore, we fix the number of iterations k∈K= {10, 50, 100, 500, 750, 1000, 2500, 5000, 7500, 10000}, and the max size pmax ∈ P = {10, 50, 100, 500, 750, 1000}.
For each triple (d, k, pmax) we run 5 executions of the music splicing system based on the Tonality-degree representation and we choose the best solution Cbest, according to the evaluation function described in this section. In order to choose the best distribution we observed the structure of the generated solutions. In our approach we say that a chorale is well-formed whether it starts with a I degree and ends with the cadence V − I. This is a typical condition of well composed chorales. Thus, we compute the average number of music compositions well-formed. As better results, we have obtained that with d = (80, 15, 5), 87% of the solutions are well-formed, with d = (85, 10, 5), 81% are well-formed, with d = (70, 20, 10), 76% are well-formed, with d = (60, 25, 15), 73% are well-formed, and with d = (50, 40, 10), 69% are well-formed.
So, we set as (Xoften, Xsometimes, Xseldom) = (80, 15, 5) the distribution used for the experiments.
For example, if ci is chord I, then the coefficient for ci+ 1 will be 0.8/3 ≃ 0.26 for each one of I, IV and V, it will be about 0.26 for vi and 0.05/2 = 0.025 for ii and iii. Table 4 shows all the weights for chord passages within a major tonality.
Similarly, we obtain the weights for chord passages within a minor tonality. Table 5 summarizes the typical chord passages suggested by [34]. The passages are very similar to the ones in a major tonality with some difference due to the possibility of using the VII chord in a minor tonality. Table 6 shows the corresponding weights. Notice that for the row the chord vii∘, there are no chords in the “sometimes” class; hence for this row we use a split of 95%-5% among the “often” and the “seldom” class.
To assign the weights for consecutive chords when a change of tonality occurs, we used the distance in the circle of fifths (see Fig. 13), between the starting and the ending tonality. More specifically the weight wModulation(S, E) is the length of the shortest path from the starting tonality S to the ending tonality E. For example, the distance between D major and C minor is 5 because we can go from D major to C minor either counterclockwise using 5 steps or clockwise using 7 steps. As before, the weights for the chord passages are computed by defining a distribution for the classes “often”, “sometimes” and “seldom”, and so the values are always between 0 and 1. The weights for the modulations, instead, are computed as the distance on the circle of fifths of the tonalities. Thus, in order to maintain such weights in a comparable range with the weights for chord passages (between 0 and 1), we normalize the above distance over the maximum possible value, that is 6. For example, the (normalized) harmonic distances from C to G, D and A are, respectively 1/6, 2/6 and 3/6; the (normalized) harmonic distances from C to B♭,E♭ and A♭ are, 2/6, 3/6 and 4/6 respectively; the (normalized) harmonic distances from A♭ to B♭ minor, D♯ minor and G♯ minor are, 1/6, 2/6 and 3/6 respectively.
Coefficients. The coefficients have been obtained by performing a statistical analysis over a large corpus of Bach’s chorales. In details, we have written a program that analyses chorales and that extracts information from the used harmonization. In particular, we looked for adjacent chords and we counted the percentage of passages from one chord to the subsequent one. We analyzed a corpus of Bach’s chorales, ranging from chorale BWV 253 through chorale BWV 306 and from chorale BWV 314 to chorale BWV 438.
Tables 7, 8, and 9 summarize the result of the analysis. Notice that the sum of all the percentages in Tables 7, 8 is smaller than 100 because there have been cases where our program was not able to identify the chords; those cases have not been classified, but simply ignored. In Table 9 many entries are not specified because those specific changes of tonality were not encountered.
Appendix B: Evaluation study
1.1 B.1 Demographic information

1.2 B.2 Preliminary survey
-
Q1: How many hours per day do you spend by playing music?
Less than 1 hour
Between 1 and 2 hours
More than 2 hours
-
Q2: How do you consider your experience with music?
Inexpert ○ ○ ○ ○ ○ More than Expert
-
Q3: Do you play any musical instrument? If “Yes”, which one?
Piano
Guitar
Trumpet
Saxophone
Flute
Bass
Drums
Other (Add here which one)
None (I do not play any instrument)
-
Q4: How do you consider your experience with classical music?
Inexpert ○ ○ ○ ○ ○ More than Expert
-
Q5: How do you consider your experience with harmonic analysis?
Inexpert ○ ○○ ○ ○ More than Expert
1.3 B.3 Summary survey
-
Q1: In general, VisualHarmony was very easy to use
Strongly disagree
Disagree
Neither agree nor disagree
Agree
Strongly agree
-
Q2: Do you will continue to use VisualHarmony in a future?
Strongly disagree
Disagree
Neither agree nor disagree
Agree
Strongly agree
1.4 B.4 CSUQ: computer system usability questionnaire
Based on: Lewis, J. R. (1995) IBM Computer Usability Satisfaction Questionnaires: Psychometric Evaluation and Instructions for Use. International Journal of Human-Computer Interaction, 7:1, 57-78.
Ratings were on a 7-point Likert scale from Strongly disagree (1) to Strongly Agree (7).
- 1.
Overall, I am satisfied with how easy it is to use VisualHarmony
Strongly Disagree / Strongly Agree
- 2.
It was simple to use VisualHarmony
Strongly Disagree / Strongly Agree
- 3.
I can effectively complete my work using VisualHarmony
Strongly Disagree / Strongly Agree
- 4.
I am able to complete my work quickly using VisualHarmony
Strongly Disagree / Strongly Agree
- 5.
I am able to efficiently complete my work using VisualHarmony
Strongly Disagree / Strongly Agree
- 6.
I feel comfortable using VisualHarmony
Strongly Disagree / Strongly Agree
- 7.
It was easy to learn to use VisualHarmony
Strongly Disagree / Strongly Agree
- 8.
believe I became productive quickly using VisualHarmony
Strongly Disagree / Strongly Agree
- 9.
The information (such as online help, On-screen messages, and other Documentation) provided with VisualHarmony is clear
Strongly Disagree / Strongly Agree
- 10.
It is easy to find information I needed
Strongly Disagree / Strongly Agree
- 11.
The information provided for VisualHarmony is easy to understand
Strongly Disagree / Strongly Agree
- 12.
The organization of information on VisualHarmony screen is clear
Strongly Disagree / Strongly Agree
- 13.
The interface of VisualHarmony is pleasant
Strongly Disagree / Strongly Agree
- 14.
I like using the interface of VisualHarmony
Strongly Disagree / Strongly Agree
- 15.
VisualHarmony has all the functions and capabilities I expect it to have
Strongly Disagree / Strongly Agree
- 16.
Overall, I am satisfied with VisualHarmony
Strongly Disagree / Strongly Agree
Appendix C: Tasks
1.1 C.1 Task 1
Given the following bass line, provide your harmonization:
1.2 C.2 Task 2
Given the following bass line, provide your harmonization:
1.3 C.3 Task 3
Given the following bass line, provide your harmonization:
1.4 C.4 Task 4
Given the following bass line, provide your harmonization:
1.5 C.5 Task 5
Given the following bass line, provide your harmonization:
1.6 C.6 Task 6
Given the following bass line, provide your harmonization:
Rights and permissions
About this article
Cite this article
Malandrino, D., Pirozzi, D. & Zaccagnino, R. Learning the harmonic analysis: is visualization an effective approach?. Multimed Tools Appl 78, 32967–32998 (2019). https://doi.org/10.1007/s11042-019-07879-5
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-019-07879-5