Skip to main content
SpringerLink
Log in

Interactive hierarchy-based auditory displays for accessing and manipulating relational diagrams

  • Original Paper
  • Published:
Journal on Multimodal User Interfaces Aims and scope Submit manuscript

Abstract

An approach to designing hierarchy-based auditory displays that supports non-visual interaction with relational diagrams is presented. The approach is motivated by an analysis of the functional and structural properties of relational diagrams in terms of their role as external representations. This analysis informs the design of a multiple perspective hierarchy-based model that captures modality independent features of a diagram when translating it into an audio accessible form. The paper outlines design lessons learnt from two user studies that were conducted to evaluate the proposed approach.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Avanzini F, Crosato P (2006) Haptic-auditory rendering and perception of contact stiffness. In: Haptic and audio interaction design vol 4129, pp 24–35

    Chapter  Google Scholar 

  2. Bauer MI, Johnson-Laird PN (1993) How diagrams can improve reasoning. Psychol Sci 4(6):372–378

    Article  Google Scholar 

  3. Bennett D (2002) Effects of navigation and position on task when presenting diagrams to blind people using sound. In: Diagrammatic representation and inference, pp 307–319

    Google Scholar 

  4. Best V, Van Schaik A, Carlile S (2003) Two-point discrimination in auditory displays. In: Brazil E, Shinn-Cunningham B (eds) Proceedings of the 9th international conference on auditory display (ICAD2003). Boston University Publications Production Department, Boston, pp 17–20

    Google Scholar 

  5. Blattner MM, Sumikawa DA, Greenberg RM (1989) Earcons and icons: their structure and common design principles. Hum-Comput Interact 4(1):11–44

    Article  Google Scholar 

  6. Bregman AS (1994) Auditory scene analysis: the perceptual organization of sound. MIT Press, Cambridge

    Google Scholar 

  7. Brown A, Pettifer S, Stevens R (2004) Evaluation of a non-visual molecule browser. In: Proceedings of the 6th international ACM SIGACCESS conference on computers and accessibility (Assets ’04). Atlanta, GA, USA, pp 40–47

    Google Scholar 

  8. Gaver WW (1992) Using and creating auditory icons. SFI studies in the sciences of complexity. Addison/Wesley/Longman, Reading

    Google Scholar 

  9. Gurr CA, Lee J, Stenning K (1998) Theories of diagrammatic reasoning: distinguishing component problems. Minds and Machines 8(4):533–557

    Article  MATH  Google Scholar 

  10. Harary F, Norman RZ, Cartwright D (1965) Structural models: an introduction to the theory of directed graphs. Wiley, New York

    MATH  Google Scholar 

  11. Horstmann M, Lorenz M, Watkowski A, Ioannidis G, Herzog O, King A, Evans DG, Hagen C, Schlieder C, Burn AM, King N, Petrie H, Dijkstra S, Crombie D (2004) Automated interpretation and accessible presentation of technical diagrams for blind people. In: New review of hypermedia and multimedia, vol 10. Taylor & Francis, London, pp 141–163

    Google Scholar 

  12. Kramer G (1994) Auditory display: sonification, audification and auditory Interfaces. Addison-Wesley, Reading

    Google Scholar 

  13. Larkin JH, Simon HA (1987) Why a diagram is (sometimes) worth ten thousand words. Cognitive Science 11(1):65–100

    Article  Google Scholar 

  14. Leplatre G, Brewster S (2000) Designing non-speech sounds to support navigation in mobile phone menus. In: Cook PR (ed) Proceedings of the 6th international conference on auditory display (ICAD2000), Atlanta, GA, USA

    Google Scholar 

  15. Mansur DL, Blattner MM, Joy KI (1985) Sound graphs: a numerical data analysis method for the blind. J Med Syst 9(3):163–174

    Article  Google Scholar 

  16. McGookin D, Brewster SA (2006) Soundbar: exploiting multiple views in multimodal graph browsing. In: Proceedings of the 4th Nordic conference on human-computer interaction (NordiCHI ’06). New York, NY, USA, pp 145–154

    Chapter  Google Scholar 

  17. Metatla O, Bryan-Kinns N, Stockman T (2007) Using hierarchies to support non-visual access to relational diagrams. In: Proceedings of the 21st British CHI group annual conference on HCI (BCS-HCI ’07). British Computer Society, Swinton, pp 215–225

    Google Scholar 

  18. Metatla O, Bryan-Kinns N, Stockman T (2008) Constructing relational diagrams in audio: the multiple perspective hierarchical approach. In: Proceedings of the 10th international ACM SIGACCESS conference on computers and accessibility (Assets ’08). Halifax, Nova Scotia, Canada, pp 97–104

    Chapter  Google Scholar 

  19. Mynatt ED, Edwards KW (1992) The Mercator environment: a nonvisual interface to x windows and UNIX workstations. Tech rep GIT-GVU-92-05

  20. Mynatt ED, Weber G (1994) Nonvisual presentation of graphical user interfaces: contrasting two approaches. In: Proceedings of the SIGCHI conference on human factors in computing systems (CHI ’94). Boston, Massachusetts, USA, pp 166–172

    Google Scholar 

  21. Norman DA (1981) Categorization of action slips. Psychol Rev 88(1):1–15

    Article  MathSciNet  Google Scholar 

  22. Norman DA (1993) Things that make us smart: defending human attributes in the age of the machine. Addison-Wesley, Reading

    Google Scholar 

  23. Palmer SE (1977) Hierarchical structure in perceptual representation. Cogn Psychol 9(4):441–474

    Article  Google Scholar 

  24. Petrie H, Harrison C, Dev S (2005) Describing images on the web: a survey of current practice and prospects for the future. In: Stephanidis C (ed) Proceedings of 3rd international conference on universal access in human-computer interaction

    Google Scholar 

  25. Scaife M, Rogers Y (1996) External cognition: How do graphical representations work. Int J Hum-Comput Stud 45(2):185–213

    Article  Google Scholar 

  26. Shimojima A (1996) Operational constraints in diagrammatic reasoning. In: Logical reasoning with diagrams. Oxford University Press, New York, pp 27–48

    Google Scholar 

  27. Shneiderman B (1996) The eyes have it: A task by data type taxonomy for information visualizations. In: Proceedings of the 1996 IEEE symposium on visual languages (VL ’96). IEEE Computer Society, Washington, p 336

    Chapter  Google Scholar 

  28. Simon HA (1980) Cognitive science: The newest science of the artificial. Cogn Sci 4(1):33–46

    Article  Google Scholar 

  29. Stevens RD, Edwards ADN, Harling PA (1997) Access to mathematics for visually disabled students through multimodal interaction. Hum-Comput Interact 12(1):47–92

    Article  Google Scholar 

  30. Stevens SS (1946) On the theory of scales of measurement. Science 103(2684):677–680

    Article  Google Scholar 

  31. Stockman T, Metatla O (2008) The influence of screen-readers on web cognition. In: Proceedings of Accessible design in the digital world conference (ADDW 2008), York, UK

    Google Scholar 

  32. Wall SA, Brewster SA (2006) Tac-tiles: multimodal pie charts for visually impaired users. In: Proceedings of the 4th Nordic conference on human-computer interaction (NordiCHI ’06), New York, NY, USA, pp 9–18

    Chapter  Google Scholar 

  33. Weber G (1993) Adapting direct manipulation for blind users. In: Conference companion on human factors in computing systems (CHI ’93: INTERACT ’93 and CHI ’93), Amsterdam, The Netherlands, pp 21–22

    Chapter  Google Scholar 

  34. Yu W, Kangas K, Brewster SA (2003) Web-based haptic applications for blind people to create virtual graphs. In: Proceedings of 11th symposium on haptic interfaces for virtual environment and teleoperator systems (HAPTICS 2003), pp 318–325

    Chapter  Google Scholar 

  35. Zhang J (1996) A representational analysis of relational information displays. Int J Hum-Comput Stud 4(16):59–74

    Article  MATH  Google Scholar 

  36. Zhang J, Norman DA (1994) Representations in distributed cognitive tasks. Cogn Sci 18(1):87–122

    Article  Google Scholar 

  37. Zhao H, Plaisant C, Schneiderman B, Duraiswami R (2004) Sonification of geo-referenced data for auditory information seeking: Design principle and pilot study. In: Barrass S, Vickers P (eds) Proceedings of the 10th international conference on auditory display (ICAD2004), Sydney, Australia

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Electronic Engineering and Computer Science, Queen Mary University of London, Mile End Road, E1 4NS, London, UK

    Oussama Metatla, Nick Bryan-Kinns & Tony Stockman

Authors
  1. Oussama Metatla
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nick Bryan-Kinns
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tony Stockman
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Oussama Metatla.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Metatla, O., Bryan-Kinns, N. & Stockman, T. Interactive hierarchy-based auditory displays for accessing and manipulating relational diagrams. J Multimodal User Interfaces 5, 111–122 (2012). https://doi.org/10.1007/s12193-011-0067-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12193-011-0067-3

Keywords

Search

Navigation