Skip to main content
SpringerLink
Log in

Equivalent representations of multimodal user interfaces

Runtime Reification of Abstract User Interface Descriptions

  • Long paper
  • Published:
Universal Access in the Information Society Aims and scope Submit manuscript

Abstract

While providing non-visual access to graphical user interfaces has been a topic of research for over 20 years, blind users still face many obstacles when using computer systems. Furthermore, daily life has become more and more infused with devices that feature some kind of visual interface. Existing solutions for providing multimodal user interfaces that ensure accessibility are largely based on either graphical toolkit hooks, queries to the application and environment, scripting, model-driven user interface development or runtime adaptation. Parallel User Interface Rendering (PUIR) is a novel approach based on past and current research into accessibility, promoting the use of abstract user interface descriptions. Based on a single consistent conceptual model, PUIR provides a mechanism to render a user interface simultaneously in multiple modalities. Each representation of the UI provides equivalent semantics to ensure that collaboration between users of different modalities is facilitated. The possible application of this novel technique goes well beyond the realm of accessibility, as multimodal interfaces become more popular and even needed. The design presented here has been implemented as a prototype for testing and further research in this fascinating area of HCI.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21

Similar content being viewed by others

Notes

  1. Either through direct manipulation (“direct access”) or indirectly (“assisted access”) by means of some form of assistive technology solution.

  2. Edwards et al. [20] list only three layers in, but they limited themselves to a description of the layers of modelling, where the conceptual layer forms the basis for those three layers of modelling.

  3. Also known as the CAMELEON Reference Framework.

  4. The Unifying Reference Framework comprises more elements than are presented here. The discussion of the state of the art does not require all elements of the framework, and the scope has therefore been limited to what is sufficient to describe, understand, and compare the various approaches.

  5. The initial version of the Unifying Reference Framework [11]. defined the adaptation operation as a transformation between representations at the same level of abstraction. Revisions made in support of plasticity of user interfaces (being able to adapt to context changes without affecting usability) introduced adaptation as a cross-level operation.

  6. This is an follow-up study using an updated list of UIDLs. It confirms the findings of Souchon and Vanderdonckt [63].

  7. Each host where application code may be executed must have a session manager running.

  8. It is possible for all components to run on a single system.

  9. Even though some of the data capturing may take place between the application and the graphical toolkit (e.g. by means of an interposing library), and therefore prior to the graphical rendering, the data can still be considered visual because the application usually either tailors the data in function of the chosen representation or it passes it to specific functions based on a chosen visualisation.

  10. This not only involves defining objects and activities (functionality of objects), but also developing strategies to introduce the new model to the anticipated user population.

  11. “Computer reality” is defined by Gaver as [22, p. 85] “the domain in which computer events are described, either by reference to the physical hardware of the system or its operations expressed in some programming language.”

  12. Sighted users may often not even realise that a mental model is involved due to the fact that a GUI is generally presented visually using iconic elements that are closely related to the underlying conceptual (mental) model.

  13. Objects and the manipulations that are possible on and with those objects.

  14. It is safe to assume for the purpose of this discussion that the element provides user interaction functionality only. Conveying information is an aspect of abstract UI semantics that can be represented in various ways—it is not purely related to visualisation.

  15. It is important to note that although information may be lost, accessibility may not be impacted because often only perceptual information is affected.

  16. Either between blind users or between a sighted and a blind user. Sighted users often tend to depend on a visual focal point when collaborating about the interaction with a system or an application.

  17. The main requirement for the communication channel is that it provides for sufficient bandwidth to enable efficient and specific exchange of information. A phone connection is often much more constructive to collaboration than, for example, an online chat session.

  18. Or at least, not possible in an equivalent and/or efficient manner.

  19. Kochanek [34] provides a detailed description of the construction-process for an off-screen model for a GUI.

  20. Specifically, the needs of the target group as those relate to UI interaction.

  21. A pluggable component is one that can easily be replaced by an equivalent component. The term is commonly used in UI contexts to indicate exchangeable presentation components. It is a derivative of the “plug-n-play” hardware concept.

  22. This AUI description effectively defines the user interface at the conceptual and semantic level. It is a formal description of the conceptual mode.

  23. Note that this does not necessarily imply that the result of user interaction is immediate, although it has become common practice to provide near-immediate results in support of the WYSIWYG design principle.

  24. It is important to note that it is not a requirement that all representations are generated at runtime, although development time construction of any representations could imply that dynamic updates to the UI structure are not possible.

  25. The physical devices that the user employs to perform operations of user interaction with the application.

  26. Device drivers at the OS level are meant to serve all possible consumers. The events they generate are most commonly very low-level events.

  27. This is known as a contraction in English Braille, American Edition.

  28. It is obvious that even amongst the blind or the sighted, not necessarily everyone will prefer everything the same way. This has been a driving force behind the efforts to provide user customisations for UIs.

  29. Generically, this type of control is known as a “valuator.”

  30. This is a common problem in any circumstance where essentially the same information is presented in tow different locations.

  31. Although “appearance” is commonly interpreted as an aspect of visual perception, it actually carries a much broader meaning, across multiple modalities of perception.

  32. This is fundamentally different from other approaches (e.g. the HOMER UIMS [59, 60]) where UI objects are described multiple times: once in abstract form, and once or more in modality-specific forms.

  33. This is commonly known as a “call back” feature.

  34. In this context, being “visible” means that the user can note the existence of the widget. Being part of the AUI, the widget obviously has no perceptual characteristics.

  35. More specific research into the impact of mental context switching and related topics is outside the scope of this work.

  36. A common problem would be that the modality in which the user interaction was initiated might render the feedback prior to the application logic responding to the operation, whereas all other renderings would render feedback afterwards. This is also commonly observed in assistive technology solutions such as screen readers that are implemented as a derivative to the graphical representation.

  37. And in order to work towards a possible future adoption as an AT support solution.

References

  1. Ali, M.F.: A transformation-based approach to building multi-platform user interfaces using a task model and the user interface markup language. Ph.D. thesis, Virginia Polytechnic Institute and State University (2004)

  2. Barnicle, K.: Usability testing with screen reading technology in a Windows environment. In: Proceedings of the 2000 Conference on Universal Usability, CUU ’00, pp. 102–109. ACM (2000)

  3. Bergman, E., Johnson, E.: Toward accessible human-computer interaction. In: J. Nielsen (ed.) Advances in Human-Computer Interaction, vol. 5, pp. 87–113. Ablex Publishing Corp., New York (1995)

    Google Scholar 

  4. Bishop, J., Horspool, N.: Developing principles of GUI programming using views. In: Proceedings of the 35th SIGCSE technical symposium on Computer science education, SIGCSE ’04, pp. 373–377. ACM (2004)

  5. Blattner, M., Glinert, E., Jorge, J., Ormsby, G.: Metawidgets: towards a theory of multimodal interface design. In: Computer Software and Applications Conference, 1992. COMPSAC ’92. Proceedings., Sixteenth Annual International, pp. 115–120. IEEE Computer Society Press (1992)

  6. Bouillon, L., Vanderdonckt, J., Chow, K.C.: Flexible re-engineering of web sites. In: Proceedings of the 9th International Conference on Intelligent user interfaces, IUI ’04, pp. 132–139. ACM (2004)

  7. Bouraoui, A., Soufi, M.: Improving computer access for blind users. In: K. Elleithy (ed.) Advances and Innovations in Systems, Computing Sciences and Software Engineering, pp. 29–34. Springer, The Netherland (2007)

    Chapter  Google Scholar 

  8. Braille Authority of North America: Braille Formats: Principles of Print to Braille Transcription 1997. American Printing House for the Blind (1998)

  9. Brunet, P., Feigenbaum, B.A., Harris, K., Laws, C., Schwerdtfeger, R., Weiss, L.: Accessibility requirements for systems design to accommodate users with vision impairments. IBM Syst. J. 44(3), 445–466 (2005)

    Article  Google Scholar 

  10. Buxton, W., Gaver, W., Bly, S.: Auditory interfaces: The use of non-speech audio at the interface (1994). Draft manuscript

  11. Calvary, G., Coutaz, J., Thevenin, D.: A unifying reference framework for the development of plastic user interfaces. In: Proceedings of the 8th IFIP International Conference on Engineering for Human-Computer Interaction, EHCI ’01, pp. 173–192. Springer, Berlin (2001)

  12. Calvary, G., Coutaz, J., Thevenin, D., Limbourg, Q., Bouillon, L., Vanderdonckt, J.: A unifying reference framework for multi-target user interfaces. Interact. Comput. 15(3), 289–308 (2003)

    Article  Google Scholar 

  13. Calvary, G., Coutaz, J., Thevenin, D., Limbourg, Q., Souchon, N., Bouillon, L., Florins, M., Vanderdonckt, J.: Plasticity of user interfaces: a revised reference framework. In: Proceedings of the First International Workshop on Task Models and Diagrams for User Interface Design, pp. 127–134. INFOREC Publishing House Bucharest (2002)

  14. Chapman, M.: Create user interfaces with glade. Linux J. 87, 90–92,94 (2001)

    Google Scholar 

  15. Congress of the United States of America: 42 U.S.C.—The Public Health and Welfare, Section 1382(a)2). GPO (1997)

  16. Coutaz, J., Nigay, L., Salber, D.: Multimodality from the user and system perspectives. In: Proceedings of the ERCIM’95 Workshop on Multimedia Multimodal User Interfaces (1995)

  17. Draheim, D., Lutteroth, C., Weber, G.: Graphical user interfaces as documents. In: Proceedings of the 7th ACM SIGCHI New Zealand chapter’s International Conference on Computer-human interaction: design centered HCI, CHINZ ’06, pp. 67–74. ACM (2006)

  18. Edwards, A.D.N.: The difference between a blind computer user and a sighted one is that the blind one cannot see (1994). Interactionally Rich Systems Network, Working Paper No. ISS/WP2

  19. Edwards, A.D.N., Mitsopoulos, E.: A principled methodology for the specification and design of nonvisual widgets. ACM Trans. Appl. Percept. 2(4), 442–449 (2005)

    Article  Google Scholar 

  20. Edwards, W.K., Mynatt, E.D., Stockton, K.: Providing access to graphical user interfaces—not graphical screens. In: Proceedings of the First Annual ACM Conference on Assistive Technologies, Assets ’94, pp. 47–54. ACM (1994)

  21. Gajos, K., Weld, D.S.: SUPPLE: automatically generating user interfaces. In: IUI ’04: Proceedings of the 9th International Conference on Intelligent User Interfaces, pp. 93–100. ACM Press (2004)

  22. Gaver, W.W.: The sonicfinder: an interface that uses auditory icons. Hum.-Comput. Interact. 4(1), 67–94 (1989)

    Article  MathSciNet  Google Scholar 

  23. Guerrero-Garcia, J., González-Calleros, J.M., Vanderdonckt, J., Muñoz-Arteaga, J.: A theoretical survey of user interface description languages: preliminary results. In: Latin American Web Congress, 2009. LA-WEB ’09, pp. 36–43 (2009)

  24. Gunzenhäuser, R., Weber, G.: Graphical user interfaces for blind people. In: K. Brunnstein, E. Raubold (eds.) 13th World Computer Congress 94, vol. 2, pp. 450–457. Elsevier, Amsterdam (1994)

  25. Haneman, B., Mulcahy, M.: The GNOME accessibility architecture in detail (2002). Presented at the CSUN Conference on Technology and Disabilities

  26. Harness, S., Pugh, K., Sherkat, N., Whitrow, R.: Fast icon and character recognition for universal access to WIMP interfaces for the blind and partially sighted. In: Ballabio, E., Placencia-Porrero , I., Bellcasa, R.P.D.L. (eds.) Rehabilitation Technology: Strategies for the European Union (Proceedings of the First Tide Congress), pp. 19–23. IOS Press, Brussels (1993)

    Google Scholar 

  27. Hollins, M.: Understanding Blindness: An Integrative Approach. Lawrence Erlbaum Associates, Boca Raton (1989)

    Google Scholar 

  28. Institute of Electrical and Electronics Engineers: 610.12-1990, IEEE Standard Glossary of Software Engineering Terminology. IEEE, Los Alamos, CA (1990)

  29. International Organization for Standardization: ISO/IEC 9126, Information Technology, Software Product Evaluation, Quality Characteristics and Guidelines for their Use. ISO, Geneva (1991)

  30. International Organization for Standardization: ISO/IEC 9241-11, Ergonomic Requirements for Office Work with Visual Display Terminals (VDTs), Part 11: Guidance on Usability. ISO, Geneva (1998)

  31. Jacob, R.J.K.: User interfaces. In: Ralston, A., Reilly, E.D., Hemmendinger, D. (eds.) Encyclopedia of Computer Science, 4th Edition. Grove Dictionaries, Inc., Williston (2000)

    Google Scholar 

  32. Kawai, S., Aida, H., Saito, T.: Designing interface toolkit with dynamic selectable modality. In: Proceedings of the second annual ACM conference on Assistive technologies, Assets ’96, pp. 72–79. ACM (1996)

  33. Kay, A.C.: User interface: A personal view. In: Laurel, B. (ed.) The Art of Human-Computer Interface Design, pp. 191–207. Addison-Wesley Publishing Co, Boston (1990)

    Google Scholar 

  34. Kochanek, D.: Designing an offscreen model for a gui. In: Zagler, W., Busby, G., Wagner, R. (eds.) Computers for Handicapped Persons, Lecture Notes in Computer Science, vol. 860, pp. 89–95. Springer, Berlin/Heidelberg (1994)

  35. Kost, S.: Dynamically generated multi-modal application interfaces. Ph.D. thesis, Technische Universität Dresden, Dresden, Germany (2006)

  36. Kraus, M., Völkel, T., Weber, G.: An off-screen model for tactile graphical user interfaces. In: K. Miesenberger, J. Klaus, W. Zagler, A. Karshmer (eds.) Computers Helping People with Special Needs, Lecture Notes in Computer Science, vol. 5105, pp. 865–872. Springer, Berlin/Heidelberg (2008)

  37. Kurniawan, S.H., Sutcliffe, A.G.: Mental models of blind users in the Windows environment. In: K. Miesenberger, J. Klaus, W. Zagler (eds.) Computers Helping People with Special Needs, Lecture Notes in Computer Science, vol. 2398, pp. 373–386. Springer, Berlin/Heidelberg (2002)

  38. Kurniawan, S.H., Sutcliffe, A.G., Blenkhorn, P.L.: How blind users’ mental models affect their perceived usability of an unfamiliar screen reader. In: M. Rauterberg, M. Menozzi, J. Wesson (eds.) Human-Computer Interaction INTERACT ’03, pp. 631–638. IOS Press, Brussels (2003)

    Google Scholar 

  39. Laberge-Nadeau, C.: Wireless telephones and the risk of road crashes. Accid. Anal. Prev. 35(5), 649–660 (2003)

    Article  Google Scholar 

  40. Lauridsen, O.: Abstract specification of user interfaces. In: Conference companion on Human factors in computing systems, CHI ’95, pp. 147–148. ACM (1995)

  41. Lévesque, V.: Blindness, technology and haptics. Technical Report TR-CIM-05.08, McGill University, Centre for Intelligent Machines, Haptics Laboratory (2008)

  42. Limbourg, Q., Vanderdonckt, J.: UsiXML: A user interface description language supporting multiple levels of independence. In: Matera M., Comai S. (eds.) Engineering Advanced Web Applications, pp. 325–338. Rinton Press, Paramus (2004)

    Google Scholar 

  43. Limbourg, Q., Vanderdonckt, J., Michotte, B., Bouillon, L., Florins, M., Trevisan, D.: UsiXML: A user interface description language for context-sensitive user interfaces. In: K. Luyten, M. Abrams, J. Vanderdonckt, Q. Limbourg (eds.) Proceedings of the ACM AVI’2004 Workshop ”Developing User Interfaces with XML: Advances on User Interface Description Languages”, pp. 55–62 (2004)

  44. Limbourg, Q., Vanderdonckt, J., Michotte, B., Bouillon, L., López-Jaquero, V.: UsiXML: A language supporting multi-path development of user interfaces. In: R. Bastide, P. Palanque, J. Roth (eds.) Engineering Human Computer Interaction and Interactive Systems, Lecture Notes in Computer Science, vol. 3425, pp. 134–135. Springer, Berlin/Heidelberg (2005)

  45. de Melo, G., Honold, F., Weber, M., Poguntke, M., Berton, A.: Towards a flexible ui model for automotive human-machine interaction. In: Proceedings of the 1st International Conference on Automotive User Interfaces and Interactive Vehicular Applications, AutomotiveUI ’09, pp. 47–50. ACM (2009)

  46. Morley, S.: Window Concepts: An Introductory Guide for Visually Disabled Users. Royal National Institute for the Blind, London (1995)

    Google Scholar 

  47. Mynatt, E.D.: Transforming graphical interfaces into auditory interfaces for blind users. Hum.-Comput. Interact. 12(1), 7–45 (1997)

    Article  Google Scholar 

  48. Mynatt, E.D., Edwards, W.K.: Mapping guis to auditory interfaces. In: Proceedings of the 5th Annual ACM Symposium on User Interface Software and Technology, UIST ’92, pp. 61–70. ACM (1992)

  49. Mynatt, E.D., Weber, G.: Nonvisual presentation of graphical user interfaces: contrasting two approaches. In: Proceedings of the SIGCHI Conference on Human factors in Computing Systems: Celebrating Interdependence, CHI ’94, pp. 166–172. ACM (1994)

  50. Newell, A.F.: CHI for everyone. Interfaces 35, 4–5 (1997)

    Google Scholar 

  51. Nigay, L., Coutaz, J.: A design space for multimodal systems: concurrent processing and data fusion. In: Proceedings of the INTERACT ’93 and CHI ’93 Conference on Human Factors in Computing Systems, CHI ’93, pp. 172–178. ACM (1993)

  52. Parnas, D.L.: On the criteria to be used in decomposing systems into modules. Commun. ACM 15(12), 1053–1058 (1972)

    Article  Google Scholar 

  53. Pontelli, E., Gillan, D., Xiong, W., Saad, E., Gupta, G., Karshmer, A.I.: Navigation of HTML tables, frames, and XML fragments. In: Proceedings of the Fifth International ACM Conference on Assistive Technologies, Assets ’02, pp. 25–32. ACM (2002)

  54. Rose, D., Stegmaier, S., Reina, G., Weiskopf, D., Ertl, T.: Non-invasive adaptation of black-box user interfaces. In: Proceedings of the Fourth Australasian User Interface Conference on User Interfaces 2003—vol. 18, AUIC ’03, pp. 19–24. Australian Computer Society, Inc. (2003)

  55. Sacks, O.: The Mind’s Eye: What The Blind See. The New Yorker, New York, pp. 48–59 (2003)

    Google Scholar 

  56. Sadato, N., Pascual-Leone, A., Grafman, J., Deiber, M.P., Ibañez, V., Hallett, M.: Neural networks for braille reading by the blind. Brain 121, 1213–1229 (1998)

    Article  Google Scholar 

  57. Sadato, N., Pascual-Leone, A., Grafman, J., Ibañez, V., Deiber, M.P., Dold, G., Hallett, M.: Activation of the primary visual cortex by braille reading in blind subjects. Nature 380(6574), 526–528 (1996)

    Article  Google Scholar 

  58. Savidis, A., Stephanidis, C.: Building non-visual interaction through the development of the rooms metaphor. In: Conference Companion on Human Factors in Computing Systems, CHI ’95, pp. 244–245. ACM (1995)

  59. Savidis, A., Stephanidis, C.: Developing dual user interfaces for integrating blind and sighted users: the HOMER UIMS. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’95, pp. 106–113. ACM Press/Addison-Wesley Publishing Co. (1995)

  60. Savidis, A., Stephanidis, C.: The HOMER UIMS for dual user interface development: fusing visual and non-visual interactions. Interact. Comput. 11(2), 173–209 (1998)

    Article  Google Scholar 

  61. Seybold, J.: Xerox’s ”star”. The Seybold Report 10(16) (1981)

  62. Smith, D.C., Harslem, E.F., Irby, C.H., Kimball, E.B., Verplank, W.L.: Designing the Star User Interface. BYTE, pp. 242–282 (1982)

  63. Souchon, N., Venderdonckt, J.: A review of XML-compliant user interface description languages. In: Jorge, J.A., Jardim Nunes, N., Falcāo e Cunha, J. (eds.) Interactive Systems. Design, Specification, and Verification, Lecture Notes in Computer Science, vol. 2844, pp. 391–401. Springer Berlin / Heidelberg (2003)

  64. Stephanidis, C., Savidis, A.: Universal access in the information society: Methods, tools, and interaction technologies. Univ. Access Inf. Soc. 1(1), 40–55 (2001)

    Google Scholar 

  65. Sun Microsystems: GNOME 2.0 desktop: Developing with the accessibility framework. Technical report, Sun Microsystems (2003)

  66. Theofanos, M.F., Redish, J.G.: Bridging the gap: between accessibility and usability. Interactions 10(6), 36–51 (2003)

    Article  Google Scholar 

  67. Trewin, S., Zimmermann, G., Vanderheiden, G.: Abstract user interface representations: how well do they support universal access? In: Proceedings of the 2003 conference on Universal usability, CUU ’03, pp. 77–84. ACM (2003)

  68. Trewin, S., Zimmermann, G., Vanderheiden, G.: Abstract representations as a basis for usable user interfaces. Interact. Comput. 16(3), 477–506 (2004)

    Article  Google Scholar 

  69. Vanderdonckt, J., Limbourg, Q., Michotte, B., Bouillon, L., Trevisan, D., Florins, M.: UsiXML: a user interface description language for specifying multimodal user interfaces. In: WMI ’04: Proceedings of the W3C Workshop on Multimodal Interaction (2004)

  70. Weber, G.: Programming for usability in nonvisual user interfaces. In: Proceedings of the third international ACM conference on Assistive technologies, Assets ’98, pp. 46–48. ACM (1998)

  71. Weber, G., Mager, R.: Non-visual user interfaces for X Windows. In: Proceedings of the 5th international conference on Computers helping people with special needs. Part II, pp. 459–468. R. Oldenbourg Verlag GmbH (1996)

  72. Weber, G., Petrie, H., Kochanek, D., Morley, S.: Training blind people in the use of graphical user interfaces. In: W. Zagler, G. Busby, R. Wagner (eds.) Computers for Handicapped Persons, Lecture Notes in Computer Science, vol. 860, pp. 25–31. Springer, Berlin/Heidelberg (1994)

Download references

Acknowledgments

The research presented in this paper is part of the author’s doctoral work at the Katholieke Universiteit Leuven, Belgium, under supervision by Jan Engelen (ESAT-SCD-Research Group on Document Architectures).

Author information

Authors and Affiliations

  1. Department of Electrical Engineering, Katholieke Universiteit Leuven, ESAT—SCD—DocArch, Kasteelpark Arenberg 10, 3001, Heverlee, Belgium

    Kris Van Hees & Jan Engelen

Authors
  1. Kris Van Hees
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jan Engelen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kris Van Hees.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Van Hees, K., Engelen, J. Equivalent representations of multimodal user interfaces. Univ Access Inf Soc 12, 339–368 (2013). https://doi.org/10.1007/s10209-012-0282-z

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10209-012-0282-z

Keywords

Search

Navigation