Keywords

1 Introduction

The National Assessment of Adult Literacy (NAAL) is designed to measure the literacy level of adults in the United States through a series of real-world activities. Results from the most recent assessment in 2003 highlight little change from previous assessments, with 43% of participants performing at the lower end of the literacy spectrum on prose literacy tasks, and 33% on document literacy tasksFootnote 1. Adults in this range can perform only the most basic activities online, which can affect their ability to understand and complete web forms. Low literacy often appears disproportionately among the elderly, the poor, the undereducated, and racial minorities [1].

Applying for means-tested government assistance programs such as subsidized housing requires participants to read program requirements and fill out applications. People applying to these programs are more likely to be poorer and less educated than the overall population [2]. Online applications for these programs often feature large blocks of text and complicated forms, simply scanned from their paper counterparts and modified for use on a web page. Long, complicated applications may severely limit accessibility for the very people these programs intend to help [3].

Low-literate web users often rely on different strategies than stronger readers when searching for information on complex web pages and forms, often leading to increased confusion, errors, and task abandonment [4, 5]. Best practices in form optimization can reduce the occurrence of errors and improve successful completion of the desired task for all users [6]. Additional research with low-literacy populations has identified design recommendations that can improve performance for this population, including larger buttons and form fields, simplifying labels, using supporting imagery, minimizing the need for scrolling, and incorporating indicators that track user progress [5, 7, 8].

Research has also shown that low-literate users are best served through the use of a linear navigation structure in web pages and forms [5, 7]. Users of all reading levels demonstrate a preference for forms that are provided in a series of chunks as opposed to a long, scrollable page [9]. Similar to chunking content in a linear sequence of pages, accordions have become a more commonly used web element for condensing lengthy pages. Accordions provide the user with a selection of headings on a single page that they can choose to expand and read. However, there is some debate about their effectiveness, suggesting they may instead increase interaction costs [10, 11]. A recent 2014 study focused on optimizing form instructions also observed lower form completion rates for low-literate users when utilizing accordion elements; these were attributed largely to the users’ unfamiliarity with this type of web element [12].

While there are questions to be explored about the effectiveness of accordions in web forms, their value to potentially improve reading comprehension on standard web pages has not been examined. Research has been conducted on replacing large blocks of information with an incremental series of smaller chunks, with demonstrated benefits to a variety of users [4, 12]. It may be possible that an accordion element that is optimized for low-literate users can prove beneficial to this population.

2 Methods

2.1 Design

A mixed method observational study using eye tracking was conducted to evaluate the barrier effect and increased interaction costs of web-based accordion interfaces on text-based and form interfaces in low-literacy populations. Research goals focused on the effect of accordion web elements on:

  • task persistence and success of reading and form-based tasks

  • reading persistence

  • minimizing peripheral distraction

  • the reduction of user errors

Publicly accessible web-based instructions and public assistance forms from the Baltimore County, Maryland Housing and Urban Development Family Self Sufficiency Program (HUD) were redesigned to use an accordion interface and selected web-based form usability guidelines [9]. Initial evaluation of text instruction readability using the Flesch Reading Ease Score (FRES = 42.8) and Flesch-Kincaid Grade Level (FKGL = 12.7) indicated text was difficult to read and at a 12th US grade level [14,15,16]Footnote 2. While government agencies recommend using caution when assigning grade level scores and readability formulas, use of FRES and FKGL suggests difficulty for low literacy populations that could be impacted by the use of accordion web elements [15].

To reduce the context effect due to variations in literacy level between participants, a within-subjects design was used. Since this design would require the exposure of both the original and improved interface, carryover effect was a concern to researchers. To counterbalance these effects, participants were randomly assigned to one of two condition schedules (a) original interface-improved interface or (b) improved interface-original interface, with a planned two-week gap between the initial interface condition and the secondary interface exposure.

2.2 Participants

Participants were recruited for this study from a participant database maintained by the University of Baltimore. To ensure participants qualified as low literate, the Rapid Estimate of Adult Literacy in Medicine (REALM)Footnote 3 was used to ensure participants were at or below an eighth grade reading level (maximum score = 60) [14]. A total of 10 qualified participants were recruited based on historical REALM scores. Nine participants (n = 9) completed all study required observation sessions for study inclusion and met the minimum REALM threshold (<60). REALM ranges for participants in this study ranged between 36 and 57 (M = 48).

Of the 9 participants, 67% identified as female (n = 6) and 33% as male (n = 3). Of the nine participants, eight identified as African American and one as Caucasian. All reported living in the greater Baltimore-metro region. While one participant did not provide age data, reporting participants ranged from 29 to 59 (M = 50).

2.3 Procedure

Following recruitment via phone, participants were invited to the research location to give informed consent. Upon giving informed consent, a REALM test was administered by a research assistant to confirm literacy level was within the acceptable study limits. Participants were then invited into an observation room with one-way glass and asked to sit at a workstation with a Tobii T60 monitor for interaction with the text-based instructions and form.

Prior to the presentation of each interface, participants were presented with a 9 dot calibration screen from Tobii Pro 3.4.8 to determine the appropriate placement of eye tracking equipment and evaluate the use of corrective eye wear if present. Participants with bifocals or correctable vision without appropriate eye wear were provided access to single vision reading glasses. Following calibration, participants were given oral instructions on reading text-instructions or completing the web-based form as appropriate. Participants were instructed that they could view text-based instructions as long as they preferred. When completing the associated form, participants were instructed to complete the form to the best of their ability and indicate when they had completed the task.

2.4 Analysis

Qualitative Analysis.

Qualitative analysis was conducted in a two-step process. Initial coding of observation notes and session video recordings was performed by the primary and secondary researchers [21]. This identified individual participant patterns and themes related to study research questions, including: (1) task success, (2) participant behavior, (3) reading patterns, and (4) form response patterns. Initial coding was followed by pattern coding by the secondary researcher to identify themes between participants for further analysis and discussion.

Text Instruction Interface.

Evaluating task duration and end of task was determined primarily by the participant’s verbal indication of task completion. However, an early end of task was indicated by the participant clicking on a page element that would prematurely end their session in real-world conditions (e.g. visiting another web page). Criteria for task success for the text instruction stimuli was based on recorded gaze data for each text instruction segment covering 90% of text area.

Form Interface.

The end of the task was determined primarily by the participant’s verbal indication of task completion or clicking “submit” in the case of the “improved” form interface. Criteria for task success for both the original and improved stimuli was based on a participant viewing and completing form fields in each form section as appropriate. Total time spent on form field segments was recorded for each form section.

Quantitative Analysis.

Quantitative data was exported from Tobii Studio Pro 3.4.8 and analyzed in IBM SPSS v25. Measures included: (1) task success, (2) total time on task, (3) time on task by section, (4) reading persistence (text instructions), and (5) area visit duration (text instructions). Task success as defined in the qualitative review of session recordings was determined as “true” or “false” and transcribed for evaluation. Time on task and time on task by segment was recorded based on participant video timestamp codes in Tobii Studio Pro. Areas of Interest (AOIs) were created for text instruction interfaces with 5 AOIs covering page layout components (header, footer, left and right sidebars, and center content) and 11 AOIs covering section headers and section text content and paragraph sub-elements.

3 Results

3.1 Text Instruction Comparison

Task Success.

Task success is divided into two components: (1) content review and (2) reading persistence. Participants who visited/fixated on all instructional content sections were considered to have successfully completed content review. Participants who fixated on paragraph text instructions within a section, were considered to have completed the reading task associated with the stimulus.

Content Review.

There is no significant difference between “original” and “improved” interface content review rates (Table 1. Content review completion by participants). It is important to note that in both conditions, participants visited few sections beyond “Overview” (Table 2. Section content review by participants). This section appeared before the page fold as a default with no user intervention, making it difficult to skip under general conditions. In practical terms, this suggests that the accordion interface is no better than a plain text interface for reading tasks. However, it is important to note that the “next” button in the “improved” interface appeared below the page fold. This would require participants to scroll down the page to access this information. Since very few participants scrolled in the plain text interface, it is a reasonable assumption that they would not scroll in an accordion interface.

Table 1. Content review completion by participants (text instructions)
Full size table
Table 2. Section content review by participants
Full size table

Reading Persistence.

Evaluating reading persistence was performed using text associated AOIs in Tobi Studio Pro. Due to an error in fixation recording, only eight (n = 8) participants were evaluated for reading persistence. Focusing specifically on the “Overview” section, participants spent less than one minute reading in the “original” (M = 25.37 s) and “improved” (M = 59.14 s) AOI fixation durations associated with header, paragraph, and bullet text. While reading time is more than double in the “improved” interface, the difference was not statistically significant t(5) = 1.29, p = 0.25).This may be due to the small sample size. However, evaluation of heat map visualizations counters any argument for a practical difference between conditions. A visual inspection of heat maps for individual participants reveals no observable density change across text for individual participants (see Figs. 1 and 2 for representative examples).

Fig. 1.
figure 1

Reading persistence comparison (P01)

Full size image
Fig. 2.
figure 2

Reading persistence comparison (P10)

Full size image

3.2 Form Comparison

Task Success.

To establish the criteria for task success in this interface group, researchers first identified all required form inputs within the form. Since no form data was collected during research sessions, participant form input was analyzed from session recordings. Participants who completed all required fields was considered as a successful completion of the form. Not visiting a section or partially completing a section was recorded as partial completion for the condition form task. While there was a marked difference between individuals who completed the improved form over the original form (Table 3. Task completion by participants), this was not statistically significant. However, in practical terms, the quality of life benefits associated with completing such forms would likely provide stronger motivation than was apparent in the lab.

Table 3. Task completion by participants (form)
Full size table

Time on Task.

On average participants took approximately 17–21 min to complete the “original” form and 10–15 min to complete the “improved” form (Table 4. Total time on task). While this represented a 31.69% reduction in total form completion time, this difference was not statistically significant t(8) = 1.967, p = 0.085.

Table 4. Total time on task (seconds)
Full size table

Observed Form Issues.

The increased rate of partial completion by individuals in the “original” form condition, raises questions regarding form attributes that may contribute non-completion. Further examination of performance in more detail (Fig. 3) within each subsection confirms the presence of form issues when compared with the “improved” condition.

Fig. 3.
figure 3

Participants completing form sections

Full size image

Computer Skills.

As is often the case with participants with low literacy skills, there appeared to be a lack of familiarity with basic computing tasks. Eight out of 9 participants had form completion errors associated with computer skills. Seven of 9 participants had inconsistent or no use of capitalization throughout the form. Ability to back space and correct errors was an issue for two participants. Mouse issues also resulted in moderator intervention for four participants due to clicking the right mouse button or having difficulties clicking into form fields. Additionally, two participants used the browser BACK button in an attempt to navigate through the “improved” form. This suggested the accordion interface may have been a new feature that they did not understand. Thus, they may have previously encountered a form that allowed use of the back button to traverse form sections, or it may reflect a general mental model that the “back” button should return you to a previous screen in a web interface. Due to the stimulus design using Tobii Studio, use of the browser BACK button resulted in a blank screen, requiring moderator assistance to return to the test session.

Skipped Questions/Sections.

When presented with the “original” interface, participants were presented with the entire first page of the fillable PDF form, presenting sections for head of household, other family members, and income. This is contrasted with the “improved” interface, presenting only the head of household section. Manual review of participant behavior during this task revealed that two participants (one in each task condition) stopped progress on the form after they completed all on-screen elements. While the “original” form included the word “over” in the page footer and the Adobe Acrobat interface listed this page as 1 of 2, these elements did not provide enough indication that an additional page was present. The common practice of converting a paper form into a fillable PDF appeared to be problematic for other participants as well, since at least two participants clicked on the “over” text to advance to page two before proceeding to scroll the page manually.

Similarly, in the “improved” condition, the presence of the “next” button and clickable accordion headings provided no indication to one participant that additional data was required in the form. Despite looking at the headings for additional accordion sections, this participant did not look at or click the “next” button. It should also be noted that only one participant clicked the headers in the accordion-based form to navigate between each section. All other participants used the “next step” button to progress linearly through the form.

Unclear Question: Race & Ethnicity.

The “original” form included embedded instructions for completing ethnicity and race fields, requiring applicants to convert their input into numeric codes for HUD statistical purposes (Table 5. Statistical ethnicity and race codes). While this information is optional for applicants, these fields generated significant user errors in the “original” form. Most participants simply entered text in these fields. (Table 6. Selected ethnicity and race responses). Only one participant completed both fields correctly. A second participant changed the text entry into the appropriate code after reading identical instructions later in the form.

Table 5. Statistical ethnicity and race codes.
Full size table
Table 6. Selected ethnicity and race responses
Full size table

This issue was eliminated in the “improved” form by the inclusion of drop down fields. Thus, the improvement in accuracy between the “original” and “improved” forms for these fields reflects a difference between a converted paper form and a web form, rather than a full-length form versus an accordion layout.

Unclear Question: Other Members of Household.

Two participants misunderstood the purpose of this section. Instead of entering family members (or leaving blank), they filled in their own information into the form for a second time.

4 Discussion

4.1 Text Instructions

Overall, results indicate there is no difference between plain text and accordion web pages for text-based information in low literacy users. While there are numeric differences in time spent in sections and reading behavior, the lack of significance and short time spent reading raises questions regarding the reading behavior of the participants in general and in relation to this study. With the small population it is difficult to determine trends and behaviors based on REALM scores and reading behavior. It is likely that some participants with REALM scores below 45 typically avoid reading tasks [22]. While not a part of the research aims of this study, participant time spent searching off task areas of the interface (e.g. header, sidebars, etc.) was observed, as is consistent with previous research into online navigational and reading behaviors associated with low literacy [5, 12, 22].

4.2 Form

Similar to the text instructions there was no significant differences between stimuli conditions. However, the increase in successful application completion is notable, and the stakes for successful completion of this type of form are high. Given the small sample size, it is difficult to say if this improved success rate was attributed to the use of accordions or was simply the result of a native web-based form rather than a PDF, but other research confirms the benefit of “chunking” information for this population [5, 22].

Our analysis shows that utilizing drop down fields as a substitute for text fields for race/ethnicity improves data entry speed and accuracy of user data entry for fields that require standardized input. While not discussed in depth in this study, other form improvements that would be easy to implement in a web-based form are likely to improve task speed. For example, clear separation between the optional form fields and required fields is likely to improve task speed. Including a radio button on the first accordion section that skips the other members of household section if it is not relevant to the user would further increase form completion speed. Additionally, ensuring application or database level validation and data transformation routines would compensate for various data formatting issues without burdening end users.

5 Implications

Despite the lack of significant findings, this study does confirm the lack of reading in this population that underscores the need for continued investigation, as part of the effort to find structures that support and motivate low literacy populations to read without disenfranchising others.

The intentionally limited application of best practices for form design was intended to highlight the potential impact of minimal improvements to the (unfortunately frequent) deployment of long PDF forms converted from paper. However, additional work should compare forms divided accordion sections versus a division into a series of short pages. Similarly, a long but more web-optimized form could be compared to an accordion form with similar optimizations.

6 Limitations

6.1 Participant Motivation

Based on participant behavior during sessions, it is unclear if participant motivation was a factor in task performance. During observation sessions, 2 participants stopped to take phone calls during the session with one participant claiming “I’ll be out in 15 or 20 min” despite only just beginning the text-based instruction task. One participant appeared to fall asleep momentarily in an observation session. While this may be behavior that would occur naturally outside of the laboratory setting when completing similar tasks, it is unclear if the motivation of applying for needed benefits would increase attentiveness and persistence.

6.2 Comparable Designs

Text-Based Design.

While the “improved” text-based instruction interface included header, footer, and sidebar elements, the lack of interactive features (e.g. active links and a working feedback form) changed the participant interaction with the web interface. This required researchers to closely monitor participant behavior and make a determination when a task ended.

Form Design.

While the aim of the study was to evaluate the impact accordion interfaces have on reading and form completion in low-literacy populations, the creation of the “improved” web-based form also incorporated some improvements based on web-based form usability guidelines [13]. Specifically, the use of drop down menus for ethnicity and race fields and visual formatting imperfections in the “improved” form limit the ability to attribute improvements to the accordion interface.

6.3 Eye Tracking Configuration

Areas of Interest.

While Tobii Studio Pro 3.4.8 supports both static and dynamic areas of interest (AOIs), the ability of the software to adapt to scrolling web-based content is limited to static areas of interest using a consistent interface. However, the use of accordion web elements introduces dynamic interface changes within a single page. The changing interface requires the use of either dynamic AOIs to encapsulate changes in the interface or changing static AOIs to account for page scrolling. Since one format of AOIs could not universally capture interface components, approaches were stimulus specific.

Text Instruction Interfaces.

The short text passages within each accordion section resulted in an interface that required no scrolling to read text instructions at 100% magnification. Despite the ability for participants to scroll and view footer information, only 1 out of 8 participants did so. This required re-adjustment of AOI placement for one participant, creating a slight mismatch of comparison data for header AOIs across interfaces.

Form Interfaces.

The close spacing of the “original” PDF text form and the close proximity of AOI areas did not provide enough spacing between header and form input elements to reliably distinguish fixations between sections. Additional consideration is needed establishing protocols for comparison between divergent stimulus sources.

Supporting Magnification.

Eye tracking best practices necessitates the use of web stimulus at 100% magnification level to allow comparable interfaces across a range of participants. Undiagnosed vision issues and lack of appropriate eye wear in aging low literacy populations may contribute to lack of motivation or attention during reading tasks. However, limiting participants only to individuals with normal or correctible to normal vision with single vision lenses would almost eliminate the ability to research in this population. This raises questions about appropriate protocol for eye tracking in low literacy and older populations. While qualitative and other observational approaches provide an avenue for user research with populations, limiting the use of eye tracking only to populations that “track well” ultimately limits the transferability of data to general populations. Finding new approaches that moderate the technical limitations of eye tracking equipment with procedures that allow for more flexible evaluation of screen-based interfaces efficiently is an area for future exploration.