Skip to main content
SpringerLink
Log in

Identifying and evaluating conceptual representations for auditory-enhanced interactive physics simulations

  • Original Paper
  • Published:
Journal on Multimodal User Interfaces Aims and scope Submit manuscript

Abstract

Interactive simulations are tools that can help students understand and learn about complex relationships. While most simulations are primarily visual due to mostly historical reasons, sounds can be used to add to the experience. In this work, we evaluated sets of audio designs for two different, but contextually- and visually-similar simulations. We identified key aspects of the audio representations and the simulation content which needed to be evaluated, and compared designs across two simulations to understand which auditory designs could generalize to other simulations. To compare the designs and explore how audio affected a user’s experience, we measured preference (through usability, user experience, and open-ended questions) and interpretation accuracy for different aspects of the simulation (including the main relationships and control feedback). We suggest important characteristics to represent through audio for future simulations, provide sound design suggestions, and address how overlap between visual and audio representations can support learning opportunities.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  1. ’74, C.: Max/msp 7. https://cycling74.com/ (2014)

  2. Ableton: Ableton live. https://www.ableton.com/en/live/ (2013)

  3. Adams WK, Reid S, Lemaster R, McKagan SB, Perkins KK, Dubson M, Wieman CE (2008) A study of educational simulations Part 1: engagement and learning. J Interact Learn Res 19(3):397–419

    Google Scholar 

  4. Apple: Garageband. https://apple.com/mac/garageband/ (2004)

  5. Basballe DA, Breinbjerg M, Fritsch J (2012) Ekkomaten: an auditory interface to the 18 th century city of aarhus. Nordic conference on human-computer interaction, pp 742–745. https://doi.org/10.1145/2399016.2399130

  6. Bonebright TL (1996) An investigation of data collection methods for auditory stimuli: paired comparisons versus a computer sorting task. Behav Res Methods Instrum Comput 28(2):275–278. https://doi.org/10.3758/BF03204780

    Article  Google Scholar 

  7. Bonebright TL, Flowers JH (2011) Evaluation of auditory display. In: T. Hermann, A. Hunt, J.G. Neuhoff (eds.) The sonification handbook, pp 111–144. http://sonification.de/handbook

  8. Brewster SA, Wright PC, Edwards ADN (1993) An evaluation of earcons for use in auditory human-computer interfaces. In: Proceedings of the SIGCHI conference on Human factors in computing systems–CHI ’93 (April), pp 222–227. https://doi.org/10.1145/169059.169179

  9. Brooke J (1996) SUS: a quick and dirty usability scale. Usability Evaluat Ind 189(194):4–7

    Google Scholar 

  10. Brown LM, Brewster SA (2003) Drawing by ear: interpreting sonified line graphs. In: Proceedings of the 2003 international conference on auditory display, pp 152–156

  11. Colella V (2014) Participatory simulations: building collaborative understanding through immersive dynamic modeling. J Learn Sci 9(4):437–469

    Google Scholar 

  12. D’Angelo C, Rutstein D, Harris C, Haertel G, Bernard R, Borokhovski E (2014) Simulations for STEM learning: systematic review and meta-analysis report overview. Tech. rep

  13. Dorić B, Lambić D, Jovanović Ž (2019) The use of different simulations and different types of feedback and students ‘Academic performance in physics’. Research in Science Education pp 1–21. https://doi.org/10.1007/s11165-019-9858-4

  14. Edelson DC, Gordin DN, Pea RD (1999) Addressing the challenges of inquiry-based learning through technology and curriculum design. J Learn Sci 8(3–4):391–450. https://doi.org/10.1080/10508406.1999.9672075

    Article  Google Scholar 

  15. Ferati M, Pfaff MS, Mannheimer S, Bolchini D (2012) Audemes at work: investigating features of non-speech sounds to maximize content recognition. Int J Human Comput Stud 70(12):936–966. https://doi.org/10.1016/j.ijhcs.2012.09.003

    Article  Google Scholar 

  16. Finstad K (2010) The usability metric for user experience. Interact Comput 22(5):323–327

    Article  Google Scholar 

  17. Frauenberger C, Stockman T, Bourguet ML (2007) A survey on common practice in designing audio in the user interface. In: Proceedings of HCI 2007 The 21st British HCI group annual conference University of Lancaster, UK 21, pp 1–9

  18. Garzonis S, Jones S, Tim J, O’Neill E (2009) Auditory icon and earcon mobile service notifications: intuitiveness, learnability, memorability and preference. In: Proceedings of the SIGCHI conference on human factors in computing systems, pp 1513–1522. ACM

  19. Gaver WW (1986) Auditory icons: using sound in computer interfaces. Hum Comput Interact 2:167–177

    Article  Google Scholar 

  20. Giannakis K (2006) A comparative evaluation of auditory-visual mappings for sound visualisation. Organised Sound 11(3):297–307. https://doi.org/10.1017/S1355771806001531

    Article  Google Scholar 

  21. van Berkum JA, de Jong T (1991) Instructional environments for simulations. Educat Comput 6(3–4):305–358. https://doi.org/10.1016/0167-9287(91)80006-J

    Article  Google Scholar 

  22. Kramer G (1994) An introduction to auditory displays. In: G. Kramer (ed.) Auditory display: sonification, audification and auditory interfaces, chap. 1. Addison-Wesley Publishing Company

  23. Lancaster K, Moore EB, Parson R, Perkins KK (2013) Insights from using PhET’s design principles for interactive chemistry simulations. ACS Symp Ser 1142:97–126. https://doi.org/10.1021/bk-2013-1142.ch005

    Article  Google Scholar 

  24. Levy ST, Lahav O (2012) Enabling people who are blind to experience science inquiry learning through sound-based mediation. J Comput Assist Learn 28(6):499–513. https://doi.org/10.1111/j.1365-2729.2011.00457.x

    Article  Google Scholar 

  25. Levy ST, Peleg R, Lahav O, Chagav N, Talis V (2016) Listening versus looking: learning about dynamic complex systems in science among blind and sighted students. In: Annual international national association for research in science teaching (NARST). Baltimore, MD

  26. Mayer RC, Clark Richard E (2008) Learning by viewing versus learning by doing: evidence-based guidelines for principled learning environments. Perform Improv 47(9):9–16. https://doi.org/10.1002/pfi

    Article  Google Scholar 

  27. Mayer RE (2002) Multimedia learning. In: The annual report of educational psychology in Japan vol 41, pp 27–29. https://doi.org/10.1016/S0079-7421(02)80005-6

  28. McCartney J et al. (1996) Supercollider. https://supercollider.github.io/

  29. PhET Interactive Sims: Phetsims/Tambo (2018). https://github.com/phetsims/tambo

  30. PhET Interactive Simulations: PhET Interactive Simulations. http://phet.colorado.edu/

  31. Quintana C, Reiser BJ, Davis EA, Krajcik JS, Fretz E, Duncan RG, Kyza E, Edelson DC, Soloway E (2014) A scaffolding design framework for software to support science inquiry. J Learn Sci 13:37–41. https://doi.org/10.1207/s15327809jls1303

    Article  Google Scholar 

  32. Rutten N, Van Joolingen WR, Van Der Veen JT (2012) The learning effects of computer simulations in science education. Comput Educ 58(1):136–153. https://doi.org/10.1016/j.compedu.2011.07.017

    Article  Google Scholar 

  33. Schuett JH, Walker BN (2013) Measuring comprehension in sonification tasks that have multiple data streams. In: Proceedings of the 8th audio mostly conference. ACM

  34. Shortridge W, Gable TM, Noah BE, Walker BN (2017) Auditory and head-up displays for eco-driving interfaces. In: The 23rd international conference on auditory display, pp 107–114

  35. Smith TL, Lewis C, Moore EB (2017) Description strategies to make an interactive science simulation accessible. JTPD 5(22), 225–238. http://scholarworks.csun.edu/handle/10211.3/190214

  36. Sodnik J, Jakus G, Tomažič S (2011) Multiple spatial sounds in hierarchical menu navigation for visually impaired computer users. Int J Human Comput Stud 69(1–2):100–112. https://doi.org/10.1016/j.ijhcs.2010.10.004

    Article  Google Scholar 

  37. Stevens RD (1996) Principles for the design of auditory interfaces to present complex information to blind people. Ph.D. thesis, The University of York. http://www.cs.manchester.ac.uk/~stevensr/papers/thesis-rds.pdf

  38. Tomlinson BJ, Kaini P, Harden EL, Walker BN, Moore EB (2019) A multimodal physics simulation: design and evaluation with diverse learners. J Technol Persons Disabil

  39. Tomlinson BJ, Noah BE, Walker BN (2018) BUZZ: an auditory interface user experience scale. In: CHI’18 Extended Abstracts. Montreal, QC, Canada. https://doi.org/10.1145/3170427.3188659

  40. Tomlinson BJ, WalkerBN, MooreEB(2020) Auditory display in interactive science simulations: description and sonification support interaction and enhance opportunities for learning. In: Proceedings of the 2020 CHI conference on human factors in computing system. ACM, https://doi.org/10.1145/3313831.3376886

  41. W3C: Web audio api. https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_API (2011)

  42. Walker BN, Kramer G (2005) Mappings and metaphors in auditory displays: an experimental assessment. ACM Trans Appl Percept 2(4):407–412

    Article  Google Scholar 

  43. Zhao H, Plaisant C, Shneiderman B (2005) “I hear the pattern”: interactive sonification of geographical data patterns. In: Proceedings of ACM CHI 2005 conference on human factors in computing systems, vol 2, pp 1905–1908. https://doi.org/10.1145/1056808.1057052

Download references

Acknowledgements

We thank John Blanco and Taliesin L. Smith (PhET Interactive Simulations), and Mike Winters (Georgia Tech) for their significant contributions to the design and implementation of auditory display in PhET sims. We thank Jared Batterman, Prakriti Kaini, and Siyan Zhou (Georgia Tech) for their help with data collection and survey design. This material is based on work supported by the National Science Foundation under DRL-1621363.

Author information

Authors and Affiliations

  1. Georgia Institute of Technology, Atlanta, GA, USA

    Brianna J. Tomlinson & Bruce N. Walker

  2. University of Colorado Boulder, Boulder, CO, USA

    Emily B. Moore

Authors
  1. Brianna J. Tomlinson
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bruce N. Walker
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Emily B. Moore
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Brianna J. Tomlinson.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This material is based upon work supported by the National Science Foundation under Grant No. DRL-1621363.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Tomlinson, B.J., Walker, B.N. & Moore, E.B. Identifying and evaluating conceptual representations for auditory-enhanced interactive physics simulations. J Multimodal User Interfaces 15, 323–334 (2021). https://doi.org/10.1007/s12193-021-00365-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12193-021-00365-z

Keywords

Search

Navigation