1 Introduction

Accessibility enable physical or mental disabled person to live their own lives independently in society [1]. Inclusion requires the equal participation of each person in the society, without a distinction between disabled and not disabled. Everyone could find themself in the situation to be restricted in their quality of life due to barriers at any time. This viewpoint reinforces the relevance of user groups such as the elderly, temporarily sick people or people with low language skills [2] in today’s society.

In the area of information and communication technology various assistive technologies exist which are intended to facilitate the use of computer technology e.g. screen readers. Visually disabled and blind person use this supporting software to transform an application to audio and/or tactile output displayed on a braille output device.

Two-dimensional tactile braille displays (TBD), like the BrailleDis [3], offer easy access to graphical applications for blind users. In contrast to common braille lines, they allow to visualize and interact with two-dimensional content like diagrams in real-time beside textual content. The usability of TBDs is only as good as the screen reader [4], the software which transforms the regular content of an application to audio and/or tactile output. There exist many screen readers for braille lines, like JAWS and NVDA, which support a lot of different applications. So far, screen readers for TBDs are limited to selected applications, like office software, travel routes or games [5, 6]. The relevant content information differs depending on the application. That’s why screen readers for TBDs have to be customized by a software developer for every single (version of an) application in order to have a sufficient result [7].

Such adaptations and extensions are also necessary, because assistive devices are highly personalized and hardly shareable. Therefore screen reader have to be adapted individually for each application, every change in the application, and also every user.

Adapting screen readers to an application requires the capabilities of a software developer, is difficult and time-consuming. This is one reason why TBDs are not widely used. The hardware does already exist, but no adequate software for blind users. Furthermore, no framework is available to create an easy and customized screen reader for TBDs for various applications.

In this paper we present such a generic framework, its concept, toolkits and advantages [8]. It allows to customize existing software for a tactile representation without any software engineering knowledge. This paper is organized as follows: the following Sect. 2 will give an overview of the working methods of screen readers, Sect. 3 will introduce the proposed framework, the introduction of its architecture will follow in Sect. 4. The paper ends with a short conclusion and outlook section.

2 Related Work

A screen reader’s work, converting content to tactile and/or audio output, is based on a non-graphical structure, e.g. a tree.

Such a tree structure is the result of filtering an application. Figure 1 shows the filtering of the Microsoft Word application in an abstract manner. Within a filtering process all control elements (with properties such as position and size) of the application are filtered and stored in a tree-like structure (e.g. off-screen model [9]). The filtered content stored in the tree can then be used to output it auditory or tactile.

Fig. 1.
figure 1

Workflow of a screen reader.

Full size image

An application or its data can be filtered in various ways [10, 11]:

  • via interfaces provided by the operating system in order to receive information from applications (e.g. Microsoft Active AccessibilityFootnote 1, e.g. User Interface AutomationFootnote 2 under Windows or the Assistive Technology Service Provider InterfaceFootnote 3 for Linux)

  • through the Document Object Model (DOM) of the application, for example, to access HTML or XML documents

  • the Java Access Bridge interface for reading Java applications

  • the interfaces that an application provides itself

  • working with screenshots of parts of an application and the use of optical character recognition (OCR)

  • via the direct addressing of the Graphical Device Interface (GDI +) or the decoder

For a good filtering result all control elements and properties have to be detectable. For that purpose the application itself should be developed in consideration of accessibility. Guidelines therefor are available in ISO 9241-171. With regard to the blind, it is particularly important thatFootnote 4

  • alternative texts are available for all graphic controls and images,

  • content can be navigated and focused in an order that preserves meaning or operation and

  • key board shortcuts are available for all/the most important actions, taking into account the platform specific guidelines.

For transferring the filtered elements to a TBD, the following three possibilities exist [12]:

  • Direct control of the TBD

    Creation of bitonal pixel graphics (only black and white) of the application with dimensions of the TBD (e.g., 120 × 60) [13]. Each pixel corresponds to a pin. Texts must be translated beforehand in Braille. Registered events have to be transmitted into mouse actions.

  • HyperReader

    HyperReader is a screen reader [10] for audio and tactile output which was developed within the HyperBraille projectFootnote 5. For adapting filters for different applications an add-in system was created in order to develop the corresponding filter mythology depending on the possibilities of the application.

  • BrailleIO

    The BrailleIO framework [14] allows the control of various TBDs. The framework provides, among other things, the following functions: Converting texts into computer braille, rendering graphics, detecting basic gestures and detecting hardware buttons.

3 Proposed Framework

The goal of the GRANT framework is to improve the inclusion of the blind and visually impaired by making screen readers easily and individually available for different braille devices. Along this process we find three different actors with specific roles and scopes: the blind user, the assistant and the software developer (see Fig. 2). The roles and scopes of the actors will be described next. Subsequently, the work steps for creating a screen reader and the framework with its functionalities are presented.

Fig. 2.
figure 2

Roles for customizing tactile applications.

Full size image

3.1 Roles and Scopes of the Actors of GRANT

The scope of the developer is the source code of the GRANT framework. If necessary, the proposed architecture allows exchanging basic functions in the framework (e.g. libraries for filtering or data handling). The framework will be available as open source software, and developers may add any desired library or screen reader which can be used for filtering. However, this is not mandatory for further use of the framework by the assistant.

The assistant is the core user of the proposed framework. It is used to create a customized screen reader without expenditure of time, efforts for developer or programming knowledge.

The blind user relies on the framework in order to interact with the application on a given TBD. Visualization and event handling are based on the customized screen reader provided by the assistant. Thus, the blind person is the end user of the product created in this process.

3.2 Work Stages of the Actor ‘Assistant’

The process curried out by the assistant will now described in more detail with respect to the components of the GUI presented in Fig. 3. There are four main tasks to be fulfilled:

Fig. 3.
figure 3

The framework supports browsing the applications elements (top) and re-arranging them for tactile representation (bottom).

Full size image
  1. 1.

    easy and customized content filtering of any application

    The assistant can adjust the filter for any application by choosing the application to be filtered, the filter library to be used, and the content to be filtered. Therefor the assistant starts with choosing the application to be filtered. The application which is focused by the mouse (1) will be filtered by the chosen filter library (2). Based on a strategy design pattern [15], which allows to select an algorithm of certain behavior, the libraries or a common screen reader can be selected.

  2. 2.

    customized content filtering for partial elements

    After filtering the application as a whole, a tree with all discovered elements (i.e. graphical components of the user interface) is displayed (3). This tree contains the captured elements of the application and their properties (4). If necessary, each node of the tree can be filtered repeatedly by another filter library. This might be useful since the quality of the filtering results (captured elements and properties) differs for particular elements.

  3. 3.

    designing output for the braille display

    For designing a specific visualization of the GUI on the chosen braille output device (e.g. TBD), the assistant has to define the display dimensions (5). Afterwards, he can decide for one of the following design methods:

    • Creating a customized design from scratch

      For that purpose, the assistant develops an output braille tree (6) by re-arranging the elements, their properties (7) and related events (8) for visualization on the TBD (9).

    • using an existing design concept

      Instead of developing a new output braille tree from scratch, an existing design template can be used (10) and mapped to the identified elements [16, 17]. The existing design templates will be introduced in detail in Sect. 4.2.

    • updating a customized design

      Blind people are accustomed to the GUI and memorize the positions and functions of the elements. If the GUI of the underlying application changes, blind (like elder or mentally disturbed) people face the challenge to re-discover the GUI. The framework allows to easily relate changes of the underlying application to a pre-defined output design. Furthermore, the same output design can be used for different applications of similar structure and behavior.

  4. 4.

    customized user events

    Both the original application and the adapted application must handle events (e.g. pressing the ‘7’ button in the calculator (1)). The framework creates the possibility for the assistant to link the events of the output device (e.g. of a touch-sensitive braille display) with actions of the tactile representation, but also with those of the original application. Furthermore, the assistant can define additional events for the tactile display (e.g. clicking a hardware button on the TBD).

Based on the following example the workflow of the work stages of the assistant will be presented in detail:

The assistant wants to create a screen reader so a blind user is able to use the Windows Calculator on a TBD. For this purpose the assistant first set the desired filter library (2) with which the application should be filtered (in this example UIA was selected). To start the filter process, the assistant must focus the application (Windows Calculator) with the mouse (1) and press a shortcut key (e.g. F5). The result of the filtering is a tree with all filtered content (3).

In the tree view, all filtered elements are hierarchically arranged. The first node (‘Window – Rechner’) corresponds to the entire application. This has several child elements like ‘Pane’. The ‘Pane’ node itself also contains child elements like text fields and buttons of the Windows Calculator.

In this example the blind user has a TBD available. Therefor the appropriate output device with its display dimensions has to be selected from all available devices of the list in the framework (5). The simulated representation of a TBS is then displayed in the framework (9) according to the dimensions of the chosen output device.

In this example the assistant decided to use an existing design concept. Therefore the design concept template has to be loaded (10) (the design concept and its templates will be introduced more in detail in Sect. 4.2). Afterwards a new tree view for the braille output (6) is displayed in addition to the tree view of the filtered content (3). The tree view of the braille output contains all elements and properties of the content to be displayed on the output device.

Now the assistant can start designing the presentation of the application on the tactile output device. For this purpose the assistant arrange the e.g. buttons and text fields of the Windows Calculator for the braille output according to the personal desire of the blind user. The framework supports the assistant to select the elements to be displayed, there location to be set and there tactile design to be created. For example the button with the number ‘7’ of the Windows Calculator is desired to be directly under the title bar. Therefore, the assistant marks the appropriate node of the button of the number ‘7’ in the tree view (6) of the tactile representation. By marking the node:

  • the corresponding node of the filtered tree view (3) is highlighted (‘Button-7’),

  • the properties of the highlighted node in the filtered tree are displayed (4),

  • the element type and its predefined standard tactile widget representation is displayed (11),

  • the button ‘7’ is also highlighted on the calculator (1),

    the properties of the node of the tactile representation are displayed as tabular (7) next to the braille tree view and

  • the element is highlighted on the simulated braille representation (9).

If the assistant wants to change properties of the element (Button ‘7’) the appropriate property in the tabular has to be adapted. For example changing the position requires the customization of the x and y values of the ‘boundingRectangle’ (7). Consequently the position of the element on the simulated braille representation is updated.

If elements are not desired to be used (e.g. percent button), they should be deleted from the braille tree view (6). Deleted elements can be added again. Therefore the appropriate node has to be selected in the filtered tree view (3) and added to the braille view tree (6) by using the ‘Add Node’ button (11).

The assistant can store his customizing (included: filtered tree, braille tree, selected TBD and customized events) and load it again for updating it any time (12).

Thus, the presented framework offers simple adaption of various applications for blind users on different braille devices.

4 Architecture

4.1 Design Pattern of Architecture

The GRANT framework was developed with .Net 4.5 framework and is written in C#. One main goal of GRANT is that all components are (dynamically) interchangeable. For this reason, the strategy pattern was chosen as design model for the implementation, which can dynamically integrate algorithm over an interface during the runtime. Figure 4 shows an abstract representation of the used components. The GrantDesigner is responsible for displaying the User Interfaces of the wizard. The core of the application is the GrantManager. This is used to make settings (such as setting the desired strategy), basic functions (BasicFunctions), and forwarding the calls to the corresponding strategy. Forwarding to the corresponding strategy (lowest level of the figure) is done via the respective interfaces in StrategyInterfaces. For example, it is possible to dynamically change the filter library UI Automation against the Java Access Bridge during runtime.

Fig. 4.
figure 4

Components of the framework.

Full size image

UI Automation was implemented exemplary as a filter library. In order to generate the tactile output (BrailleOutput), a driver for the universal response of different output devicesFootnote 6 was used as extension for the BrailleIO framework. This solution makes it possible to develop screen reader for different output device (e.g. braille display, braille line, etc.).

4.2 Design Templates of Tactile Output

Blind people are accustomed to the GUI and memorize the positions and functions of the elements. If the GUI of the underlying application changes, blind (also elder or mentally disturbed) people face the challenge to re-discover the GUI. Therefor a design concept has been developed for two-dimensional tactile output devices within the HyperBraille project.

Based on this design concept the following views were developed [17, 18] to enable a standard tactile output design of the filtered application:

  • operating view

    completely text-based braille display, comparable to braille lines

  • outline view

    graphical representation of all structural elements using “boxes” without details

  • symbol view

    tactile widgets with braille text

  • layout view

    pixel-based representation of the original application

In addition to the different views following regions have also been created [12, 17]:

  • Header shows general information of an application, e. g. the name of the currently selected application window.

  • Body shows the content of the selected window.

  • Structure shows further information e.g. for a text box the formatting of the text.

  • Detail gives further information about the focused element or area. This output is similar to that on a Braille line.

  • Navigation bar allowed to navigate simple to other views comparable with tabs in a web browser.

This design concept was taken into account and integrated in the GRANT framework as template to support the assistant in designing the tactile output. The framework allows to easily relate changes of the underlying application to a pre-defined output design. Furthermore, the same output design can be used for different applications of similar structure and behavior.

As part of the framework the layout view, symbol view and a simplified operating view (called text view) were focused and integrated until now. Figure 5 shows the Windows Calculator exemplary presented in the three different tactile views. The tactile views are automatically generated based on the filtered content (saved as tree structure) if the design concept is enabled and before any adjustments made by the assistant.

Fig. 5.
figure 5

Exemplary application, its filtered content as tactile views of the design concept.

Full size image

From technical point of view, the symbol view represents the greatest challenge. On the one hand, all UI elements must be transferred automatically into a corresponding tactile representation. On the other hand, a sensible tactile arrangement must be determined, without knowledge of the concrete application. The prototypical implementation of this concept was therefore so far restricted to automatically arranging only the regions and most important UI elements of the filtered tree (e.g. buttons and menu bar) for the tactile representation.

In contrast to the text view the symbol view has to be completed for a suitable representation. The text view shows the complete content of the filtered application as a tree view. The layout view represents a screen shot of the application. Just like the text view a completion or adaption by the assistant is not required.

Depending on the output device an appropriate template is used in order to use the available dimensions of the hardware usefully and to receive a suitable result.

It is also possible to create new templates from scratch but using a template for adapted the output design to the needs of a blind person minimizes the effort of the assistant significantly.

5 Conclusion and Outlook

With the GRANT framework a solution to improving the inclusion of blind and visually impaired users is provided. Any application is easily accessible through individualized screen reader which are easy creatable and customizable using the framework. Since no developer is required for the adaptation and creation of a screen reader, software updates can also be reactivated more quickly.

GRANT offers not only the possibility create/customize screen readers for any software application but also for different output devices (braille display, braille line, etc.) depending to the individual needs.

The possibility of using design concepts for the tactile representation on the tactile output device reduces the working effort of an assistant but also enable creating a suitable standard design.

For a release of the framework as open source software some more extension are still necessary like completing the functions of the framework, expanding the design templates, integrating the events completely and adding the audio output. In order to evaluate the results obtained, utility tests are planned in near future.

The framework offers not only for blind and visually impaired people the possibility to adapt applications, but also for other target groups, such as seniors, people with dementia, temporarily ill people or persons with a lack of language skills. Therefore no change in the filtering process is necessary, only the output design on the desired device has to be adapted accordingly. That means that the GRANT framework provides the inclusion of any individual which meets a barrier in the field of computer technology.