Estimating the number of subjects needed for a thinking aloud test

doi:10.1006/ijhc.1994.1065

International Journal of Human-Computer Studies

Volume 41, Issue 3, September 1994, Pages 385-397

https://doi.org/10.1006/ijhc.1994.1065 Get rights and content

Abstract

Two studies of using the thinking aloud method for user interface testing showed that experimenters who were not usability specialists could use the method. However, they found only 28-30% of known usability problems when running a single test subject. Running more test subjects increased the number of problems found, but with progressively diminishing returns; after five test subjects 77-85% of the problems had been found.

References (0)

Cited by (334)

The differences between adults and adolescents using a mobile health application for menstrual complaints: A usability and qualitative study
2024, International Journal of Medical Informatics
A “Menstruatie Educatie Kalender” application (Menstruation Education Calendar, (MEK-APP)) was developed for adults to evaluate menstrual complaints. The future aim of this app is to use it as a self-diagnostic instrument for menstrual abnormalities for both adults and adolescents. Early identification of the potential of an application for future use by both user groups would increase implementation success and adoption of the application.
To compare differences in experienced usability by adults versus adolescents and to identify factors influencing future use for both age groups in one mHealth application (in this study the MEK-APP).
This study consisted of three phases: (1) usability testing of the MEK-APP for iOS and Android by think-aloud method, (2) two-month daily use and (3) in-depth individual interview. During the think-aloud sessions, twelve tasks were performed in the application while they were thinking aloud. Usability problems were rated for their severity with Nielsen’ Severity Scale. Both the think-aloud sessions and in-depth interviews were verbatim transcribed and thematically analyzed to determine the factors influencing future use for both groups. In addition, the System Usability Scale (SUS) and Intrinsic Motivation Inventory (IMI) questionnaires were filled out during the interviews.
Seven adults (>18 years) and seven adolescents (14–18 years) evaluated the MEK-APP. There were 14 usability issues and 16 bugs in both groups. There were no differences between adults and adolescents. In the thematic qualitative analysis, the following future use factors were identified: user-expectation, motivation, privacy, understandability, and user-experience. The user-expectation, motivation and privacy differed between both groups but did not influence usage. No differences were observed in SUS and IMI scores between both groups.
There are five factors influencing the future use of a menstrual-related mHealth application for both adults and adolescents. It is possible to serve different age groups with a single application.
An online video annotation tool for optimizing secondary teachers’ motivating style: Acceptability, usability, and feasibility
2023, Teaching and Teacher Education
This paper introduces an online video annotation tool for improving teachers' motivating style (autonomy support and structure). The tool's acceptability, usability, and feasibility were tested by both pre- and in-service physical education teachers. Overall, both groups of teachers exhibited moderate to high acceptance of the tool and reported that its use stimulated reflection on their motivating style. Although the tool was user-friendly and required minimal guidance, some teachers expressed concerns about the time required to use it. In response, we have proposed several solutions. Finally, both groups of teachers agreed that an online tool could complement traditional self-reflection methods in initial teacher training programs.
Educational 360-video development for clinical practice improvement toward Deaf and Hard-of-Hearing Population
2023, Clinical Simulation in Nursing
Empathy of clinical providers is necessary for patient-centered care. Virtual reality, such as 360-videos, may enable empathy development with thoughtful design. We describe our process and outcomes developing an educational innovation for clinical providers to work with the challenges experienced by the deaf and hard-of-hearing population using 360-videos.
A user-centered design-informed process was used to (a) empathize with our patient population using a focus group and stakeholder input, (b) design content for videos with iterative feedback from stakeholders, and (c) feasibility and usability test the innovation.
We found multiple areas of concern of our patient population and subsequently created educational scripts for the videos to address these, with iterative refinement by stakeholders. Innovation delivery in the classroom setting with advanced practice nurse students was feasible and had good usability.
While initial testing of this 360-video educational innovation was promising, further testing and dissemination strategies are needed.
Comparison between two patient-reported outcome measures for patients with cervical radiculopathy: A think-aloud study
2023, Musculoskeletal Science and Practice
The Cervical Radiculopathy Impact Scale (CRIS) and Patient Specific Functional Scale 2.0 (PSFS 2.0) are patient-reported outcome measures (PROMs) used to assess activity limitations in patients with cervical radiculopathy (CR). This study a) compared the CRIS subscale 3 and the PSFS 2.0 in patients with CR with respect to completeness and patient preference, b) established the correlation between both PROMs in assessing the individual patient's level of functional limitations and c) assessed the frequency of reported functional limitations.
Participants with CR participated in semi-structured, individual, face-to-face interviews as part of a “think-aloud” process; verbalising their thoughts while completing both PROMs. Sessions were digitally recorded and transcribed verbatim for analysis.
Twenty-two patients were recruited. The most frequently reported functional limitation on the CRIS was: ‘working at a computer’ (n = 17) and overhead activities' (n = 10) for the PSFS 2.0. There was significant moderate positive correlation between the scores on the PSFS 2.0 and the CRIS (Spearman's r = 0.55, n = 22 p = .008). Most patients (n = 18; 82%) preferred the ability to present their own individual functional limitations of the PSFS 2.0. Eleven participants (50%) preferred the 11-point scale of the PSFS 2.0 over the 5-point Likert scale scoring option of the CRIS.
Both easy to complete PROMs capture functional limitations in patients with CR. Most patients prefer the PSFS 2.0 over the CRIS. The wording and layout of both PROMs need refinement to enhance user-friendliness and avoid misinterpretation.
Rendezfood: A Design Case Study of a Conversational Location-Based Approach in Restaurants
2024, SSRN
TIDAL: exploring the potential of data physicalization-based interactive environment on runners' motivation
2024, Journal of Ambient Intelligence and Humanized Computing

View all citing articles on Scopus

View full text

Regular ArticleEstimating the number of subjects needed for a thinking aloud test

Abstract

Regular Article
Estimating the number of subjects needed for a thinking aloud test