Gavin R. Philips
Abstract
Of the ∼8.8 million working-age adults with cognitive disabilities in the United States, only 28.6% are employed, contributing to a poverty rate (26.1%) for people with cognitive disabilities (PwCDs) that is more than twice that for people without disabilities. PwCDs who are employed are often still marginalized via reduced hours and pay, largely due to their more limited capability to perform work efficiently and independently. As evidence, warehouse and shelf stocking jobs that serve as employment for a significant percentage of PwCDs, often require frequent intervention from a job coach in the workplace, impacting the pay and self-esteem of these workers. This study's objective was to remove barriers to employment for PwCD through inclusive design of technology supports in warehouse and similar settings. Specifically, a nonlinear context-aware prompting system (NCAPS) was developed, iteratively refined, and tested. In a pilot crossover study, subjects with cognitive disabilities participated in simulated work sessions, picking orders in a small warehouse environment using the NCAPS and industry standard paper tickets. Their performance was assessed in terms of errors and productivity, and their insights and perspectives were gathered. The resulting NCAPS prevented or corrected all errors for all but one participant. It also increased productivity for participants with the poorest baseline (paper ticket) performance, but decreased productivity for those with high baseline performance. No significant difference was observed in system usability scale (SUS) scores between methods. Topics emerging from user input highlighted the need for technological supports for PwCD to be simple and flexible in operation, and reliable to maintain user trust. By prioritizing robustness and intuitiveness, flexible technology supports can be developed to empower workers with a broad range of abilities, including those with temporary and situational impairments. Such tools would reduce barriers to employment, including stigma, discrimination, and other barriers to equity.
1 INTRODUCTION
In the United States, more than 15 million people (4.8% of the population) live with cognitive disabilities [1, 2], meaning they are predominantly impacted in their ability to remember, concentrate, and make decisions, as well as handle other cognitive tasks relevant to planning and execution. This population is comprised of (but not limited to) individuals diagnosed with brain injury, intellectual and developmental disabilities, and dementias. Approximately 8.8 million of these individuals are of working age and living in the community, but only ∼2.5 million (28.6%) are employed. Furthermore, “employment” in this context typically involves reduced hours and pay, as compared to workers without disabilities [3, 4]. This contributes to a drastically increased poverty rate (26.1%) of working age adults with disabilities, as compared to those without disabilities (10.7%) [2]. Bringing more people with cognitive disabilities (PwCDs) into the workforce could dramatically improve their quality of life, not just through reduced poverty, but also through increased self-esteem, improved health outcomes, and reduced social isolation [5–10].
A variety of barriers impede such advancement and impact the successful integration of people with disabilities into the workforce. In particular, workers with disabilities face environmental and attitudinal barriers that include stigma and discrimination [7, 8, 11, 12]. Shier et al. indicated that primary barriers to securing employment are employer discrimination, “labeling,” and negation of human capital (overlooking the skills and abilities an individual may have developed) [12]. Here, “labeling” refers to fear of stigma resulting from being labeled as a person with a disability, which prevents many from disclosing their conditions, and significantly impacts workers with cognitive disabilities [13, 14]. However, self-disclosure is typically a prerequisite for appropriate workplace accommodations, thus this fear prevents a significant proportion of workers from requesting or receiving such accommodations [13]. This stigma also contributes to the negation of human capital, as employers discount workers’ training and skills due to the perception of disability [12]. Discrimination continues beyond job acquisition, as individuals face harassment and risk of discharge due to perceived quality of work [11]. In contrast, employment is facilitated by effective support services and accommodations, as well as other factors [15, 16], and such accommodations confer a variety of benefits on both employers and employees [17].
1.1 Vocational Support for Workers with Cognitive Disabilities
Traditionally, vocational support has taken the form of sheltered or supported employment. Sheltered employment involves segregating workers with disabilities in an environment that is intended to allow them to safely develop work-related skills and behaviors [18]. However, this practice has become controversial, due to its segregated nature and concerns that participants are not effectively prepared to transition into unsheltered employment [19]. Furthermore, workers with cognitive disabilities were shown to receive 250% higher earnings in supported employment, as compared to sheltered employment [20]. In contrast, supported employment integrates workers into mainstream environments by utilizing dedicated job coaches or natural supports (e.g., supervisors or co-workers) to provide training and continued assistance [21, 22]. This approach produces increased employment rates and improved quality of life outcomes for workers with cognitive disabilities [23, 24]. However, situations can arise in which other workers address the job coach instead of the worker [25], and the expense of training and employing job coaches reduces the availability of such services [26–28].
Human support can be supplemented or in some cases replaced, by the appropriate application of assistive technology (AT), which substantially improves the productivity and self-esteem of workers with disabilities [29–33]. A broad body of work has demonstrated the impact that such technology supports can have on workers with cognitive disabilities, improving outcomes such as productivity, independence, and generalization of skills [21, 34–39]. AT has been applied to support workers with planning, execution, attention, memory, literacy, and social/behavioral issues [36]. Devices that provide cues to remind a worker to perform a task or a series of prompts for separate steps of a task have been implemented in desktop/laptop computers [40–42], tablet computers [40, 43–46], smartphones and handheld devices [47–52], wearable watches [53, 54], augmented reality (AR) devices [55–57], and virtual reality (VR) [58].
Some prompting systems are considered “linear,” providing a set of prompts in a predetermined order, regardless of context [41, 48, 59]. While such linear prompting has been shown to be somewhat effective, further improvement is possible through the incorporation of sensing technology that can determine context [46, 55–57, 60]. A context-aware system can adapt to the user's actions in real-time, presenting prompts in a non-linear fashion as needed [36]. A subset of context-aware prompting systems sense the location of the user, device, and/or items in the workspace, and provide navigation assistance [47, 56, 57, 61–64].
One promising opportunity for the application of location and context-aware prompting systems is in small warehouses and retail environments, where workers “pick orders” (i.e., gather a list of items to be shipped or delivered). A variety of inventory and order tracking systems are utilized in these environments, but even the most supportive tools typically rely on the human order pickers to find and memorize item locations using shelf or item labels. Workers with impairments of memory, planning, execution, and attention struggle with such tasks, but could find success with appropriate technology support. Thus, order picking tasks were selected as the target of the prompting system and pilot study described herein.
1.2 Effective Selection and Implementation of AT in the Workplace
The effectiveness of AT as a technological support in the workplace is impacted by a variety of factors, including environmental and personal determinants [65, 66]. Suboptimal use or abandonment of AT by workers with cognitive disabilities is often caused by a mismatch between the technology and the specific needs and preferences of the user [34, 65].
This mismatch can be ameliorated to some degree by employing an appropriate individualized assessment and AT selection process [31], such as the Matching Person and Technology (MPT) assessment process [67]. The assessment process may also employ psychometric tools such as the Mini-Mental State Examination (MMSE) or Assistive Technology Device Predisposition Assessment (ATD PA) to quantify relevant personal factors [66, 68], including the severity of cognitive impairment, which significantly influences the effectiveness of AT use [34].
However, some interplay exists between these assessment processes and the previously discussed issues of stigma and discrimination [12]. Such services are contingent on self-disclosure of disabilities, which often leads to labeling. Furthermore, the provision of assessment and AT support services presents a financial burden, which can exacerbate the barriers to equity. A potential solution is found in inclusive design (ID), which intentionally applies a user-centered approach to develop mainstream technology that is usable by a broader segment of the population, including people with disabilities [34, 69–71]. Technology that incorporates ID features has been shown to be particularly effective in supporting workers with disabilities [36]. Furthermore, the blending of AT-relevant features into mainstream technology could help reduce complete reliance on “medical model” solutions [69], and carries the added benefit of reduced stigma when PwCDs access inclusive solutions via mainstream technology and devices [48, 72].
2 PRELIMINARY WORK
The authors’ prior work involved the development of a nonlinear context-aware prompting system (NCAPS) shown in Figure 5, designed to assist workers with cognitive disabilities in warehouse environments to facilitate their vocational independence. This work stands adjacent to the authors’ legacy work in developing a different N-CAPS to assist workers with cognitive disabilities in factory assembly tasks [45, 46]; findings from this legacy work were used to inform the design of task prompts for PwCDs in workplace settings, including considerations for prompt timing, audio/visual presentation, and affirmations for workers upon successful task completion.
The NCAPS was created via a user-centered and multi-stage design approach that entailed user research via rapid ethnography, initial prototype development based on user needs, and iterative prototype refinement based on feedback and task run-throughs with student researchers and a consultant with cognitive disabilities (COMIRB #11-0674).
The developed NCAPS was then used for small-scale pilot testing with PwCDs in a mock warehouse environment, forming the research study described in this article.
2.1 Task Analysis and Feature Specification
Careful elicitation, analysis, and specification of system requirements is vital to the development of effective tools and processes [73]. Such consideration is also crucial in the design of effective research studies. Rapid ethnography is one such method for requirements elicitation, and is defined by interactive and on-site observation techniques used to answer specific design questions in an efficient manner [74]. This approach was applied to analyze the selected work task of split-case order picking in a small warehouse or retail environment, with focus on answering (1) “What factors influence task completion by employees in a warehouse?” and (2) “What interactions with technology do employees experience in a warehouse?” [75, 76]. Semi-formal interviews and job shadowing were performed in five functioning small warehouse environments, with the participation of nine managers and 25 workers. Audio/video recordings and field notes were analyzed via inductive content analysis, producing a conceptual model of factors that impact the implementation of technology in a warehouse environment [75].
Several of the factors indicated in these rapid ethnography studies directly informed the development and subsequent user testing of the NCAPS. Replicating and expanding on observed methods of environmental communication, researchers determined the NCAPS prompts should provide navigation and task-relevant visual cues, supplementing directional/informative signs placed in the test environment. Leveraging ubiquitous equipment, the system should be integrated into standard manual carts, but retain hand-held capabilities to accommodate diverse warehouse layouts. The lack of automated feedback in typical paper ticket-based workflows suggested potential benefits from real-time error detection/correction features. Finally, no consensus was found on the question of structured vs. unstructured task flows, indicating a ripe target for investigation. The optimal rigidity of the pick path and granularity of prompts must be determined experimentally through user testing with the target population, or must adaptively accommodate the user and context. Because unstructured task flows with self-determined pick paths rely on the worker to memorize item locations, and cognitive disabilities often include memory impairments, the NCAPS should initially default to pre-determined pick paths [76].
Additional consideration of traditional prompting hierarchies in the context of efficiency-focused warehouse environments produced a set of potential cue types organized by rate of information delivery, prioritizing fast cues like static images and text over video-based prompts [76].
2.2 Initial Prompting System Design
The derived specifications guided the development of the NCAPS, which was a collaborative endeavor between the Center for Inclusive Design and Engineering (CIDE; University of Colorado Denver | Anschutz Medical Campus) and the Institute of Cognitive Science (ICS; University of Colorado Boulder). The initial prototype consisted of a web server, a controller device, a prompting device, and one or more optional scanning devices (Figure 1).
Fig. 1. Structure of NCAPS implementation. The controller device requires only a web browser, while the prompting and (optional) scanning devices utilize Android apps.
Each user- or researcher-facing device had low computing and storage requirements, and was easily replaceable, as processing and data storage was performed by the web server. The authors’ previous indoor navigation experiments leveraged this server infrastructure to design and test an Ultra High Frequency Radio Frequency Identification (UHF RFID) navigation system with seven subjects without disabilities [61]. While the UHF RFID approach was found to yield an accurate (within 0.3m) positioning system in static conditions, the sensing and processing rates were insufficient to provide positional accuracy in dynamic (walking) conditions. Additionally, the required overhead for creating RFID tags to mark shelf locations reduced the ease of system implementation in warehouse environments. Thus, an alternate approach utilizing quick response (QR) codes for spatial localization was developed. QR codes could be quickly and inexpensively printed, placed, and adjusted in the work environment, and could be scanned using the standard integrated cameras of mobile phones and tablets. These cameras were also used to scan the universal product codes (UPCs) found on the items.
The server hosted a database of items, a map of the warehouse environment (represented by a logical directed graph that can be overlaid with coordinates on a spatial map), a navigation algorithm, and a database of prompts associated with items and potential scenarios. The prompting device could be any Android-based mobile device (tablet or phone) and ran an application with a graphical user interface (GUI) that provided cues based on the user's context, as well as an animated virtual job coach (Figure 2). The integrated compass and camera of the prompting device were utilized for navigation and item scanning. Optionally, additional Android-based scanning devices could be used for QR code and/or barcode scanning (e.g., if it is inconvenient for the user to scan items with a cart-mounted tablet). The control device can be any computer or mobile device, as the control GUI runs in a web browser (Figure 3). It allows a researcher or supervisor to select pre-configured pick tickets or individual items, control the flow of prompts to the worker, and correct errors or unexpected states. Pick tickets are pre-populated with items and presented to the controller in order according to the database, but can be adjusted in real-time.
Fig. 2. Initial prototype NCAPS prompting device with AR navigation and virtual job coach.
Fig. 3. Initial prototype NCAPS browser-based controller interface, which allows a researcher or supervisor to control system flow. Shown here is the “Navigation” page of the interface, which includes the “Scan Barcode” button (green) that can manually change the participant prompting device to scan mode when they have navigated next to the target item (if the automated workflow doesn't respond), and the “[stop symbol]” button (red) that cancels the item. The “B-3-2” and alpha-numeric sequences in the squares below are shelf locations and reference numbers of the warehouse map (mostly obscured, scrolling down on the screen shows more of the map).
When prompting is initiated, the server applies the navigation algorithm, utilizing an A* (A-star) search on the warehouse map to select an optimal path from the prompting device's current location to that of the desired item. The A* search algorithm described here is a long-standing approach first developed in robotics as a means of finding the shortest feasible path of travel from point A to point B on a 2-dimensional map [77, 78]. An AR window appears on the screen of the prompting device for navigation. It displays a video feed from the camera that is scanning QR codes placed on the floor, overlaid with directional arrows that guide the user to the location of the desired item. As the user moves about the environment, QR codes are scanned and sent to the server, which recalculates the optimal path and updates the AR display accordingly. In this way, the NCAPS recognizes the context of the worker, and corrects for errors (e.g., the user moving in the wrong direction). When the user arrives at the location of the desired item, the AR navigation window is replaced by a photograph of the item, and an auditory prompt directs the user to scan the item's barcode. The user places an item in view of the camera, and the device confirms that it is the correct item or prompts the user to replace it and select the correct item. If the user does not select the correct item after successive prompts, a supervisor can be called for assistance. Upon confirmation of the correct item, the server calculates a path to the location of the next item on the ticket, and navigation begins again. Once all items in the ticket have been gathered, the delivery location (shipping table) is set as the target, and the user is guided there and instructed to deliver the items.
For initial testing and the pilot study, an arrangement of devices was mounted on a standard work cart to ensure ease of use, and to prevent damage to the devices (Figure 4). The prompting device (a Google Pixel Slate with 12.3 inch display) was mounted at the upper rear of the cart, directly in front of the push handle, where it could be easily seen and accessed by the user. A separate location scanner device (a Google Pixel 3) was mounted at the front of the cart, low for optimal view of QR codes, and protected by a makeshift bumper. A separate item scanner device (a Samsung Galaxy Tab 5) was added to the top of the cart with flexible positioning to ensure easy barcode scanning for a variety of users. This arrangement could be easily reconfigured to accommodate other scenarios and users.
Fig. 4. Initial prototype NCAPS with separate prompting device, item scanning device, and location scanning device mounted on a standard utility work cart.
Fig. 5. Final prototype NCAPS after iterative refinements. (Left) Cart modified so the location scanner “sees” the navigation QR codes right under cart. (Right) Prompting device arrows were simplified to only include current navigation directions, removing upcoming navigation directions. (Bottom) The virtual job coach has been moved to the scanning tablet so the prompting tablet can display items and directions more clearly. The item name and shelf number are also included above the image.
2.3 Iterative Refinement
The pilot NCAPS configuration was iteratively refined through mock run-throughs with student researchers, with additional user validation with one potential user with cognitive disabilities (COMIRB #11-0674). In this manner, a minimum level of function was ensured before performing the formal pilot study. Beyond minor technical issues, four significant changes were made at the direction of the potential user (note that these changes are not reflected in Figures 2–4, which depict the initial NCAPS prototype). (1) The position of the location scanner was moved to the center of the cart and a corresponding opening cut in the cart base, bringing the camera's point of view closer to the user, making navigation more intuitive. (2) Navigation arrows were simplified by removing indicators of an upcoming turn (i.e., “turn soon, but not yet”), as they were found to be confusing. (3) The AR window/item photo prompt area was expanded to occupy the whole screen on the prompting device, moving the virtual job coach to the screen of the scanner device, allowing the user to more easily see small differences in similar items. (4) The name and location code of the desired item were added in text to the screen of the prompting device, so that the user could ignore navigation guidance and move directly to the item location when able to do so. After these changes were applied, the potential user re-evaluated the system and indicated that its use was significantly more intuitive and enjoyable.
3 METHODS
3.1 Study Design
The objective of this pilot study was to examine the initial efficacy and usability of the NCAPS and to elucidate factors impacting the use of such prompting methods. Specifically, this study followed a research protocol (#18-1825) approved by the Colorado Multiple Institutional Review Board (COMIRB) that oversees university research ethics and safeguards human subject protections. The protocol employed a crossover design, with each participant engaging in simulated warehouse work sessions using both methods: the described NCAPS, and an industry-standard paper ticket method. Participants were randomly assigned to use either the NCAPS or paper ticket (Figure 7) method first, with a minimum washout period of two weeks before using the alternate method. Two lists (A and B) of pick tickets were randomly generated, with each ticket including 2–7 items. Each participant was randomly assigned list A or B for use with the NCAPS, and the alternate list for use with the paper ticket method. All instances of the assigned starting method and pick ticket list, while random, were balanced to ensure an equal distribution of test ordering and method-list combinations.
3.2 Materials and Environment
The pilot study and all user interaction occurred at the Center for Inclusive Design and Engineering (CIDE; University of Colorado Denver | Anschutz Medical Campus). The NCAPS and conventional paper ticket system were tested and evaluated by potential users in a mock retail/warehouse environment. The ∼700 square foot space included rows of multi-level shelving units, stocked with a variety of grocery items (Figure 6). 500+ items were used, including 110 unique products, each with a scannable UPC barcode. Groups of similar items were included, which required careful attention to differentiate, and provided increased opportunities for user error. Each side of an aisle was labeled with a letter code (e.g., “E”), each associated shelving unit with a number (e.g., “E-2”), and each item location with an additional number (e.g., “E-2-6”), producing a unique alphanumeric identifier for each item as is typical in such warehouses. A “shipping table” at one end of the space served as the start and end point for each trial, where participants would receive new pick tickets and drop off items once all listed items were acquired.
Fig. 6. Mock warehouse environment stocked with grocery items. Aisles, shelves, and item locations are marked with alphanumeric codes. QR codes placed on the floor are used as navigation landmarks by the prompting system.
3.3 Data Collected
This pilot study employed mixed-methods data collection, characterizing both participant performance and preference. Performance was characterized by the “productivity” and “error rate” outcome metrics, defined as the total number of warehouse items picked correctly and the number of items picked incorrectly (either by omission or commission). Preference was characterized by the participant's interpreted system (e.g., paper tickets, NCAPS) “usability”, which was measured via the standardized “System Usability Scale (SUS)” questionnaire [79] administered at the end of each session with a tablet for the participant to select their responses. The SUS and corresponding instructions were modified to utilize simple language, and clarification was provided to participants if requested. Participants were also offered the option for a researcher to administer the SUS to them verbally, and fill out their responses for them on the tablet.
Video data was also captured during the work sessions using four ceiling-mounted cameras (Lorex Technology, Markham, ON) and the Morae suite of usability testing software (TechSmith, Okemos, MI), with the primary goals of ensuring performance data collection redundancy and providing context and increased data resolution during post-hoc analyses. An observer in a separate room added event markers to the video in real-time to represent the start and end of each ticket, errors, breaks, and technical issues. Simultaneously, investigators recorded errors (incorrect or missing items) for each ticket, and manually noted behaviors or issues of interest, along with their associated time points. During NCAPS sessions, log files were generated by the web server, recording time-stamped system states.
All collected data were stored safely and securely in accordance to protocol COMIRB #18-1825. Physical information including consent and demographic forms were all stored in lockboxes and transferred to secured filing cabinets for long-term storage. Digital information including video recordings, spreadsheet data, observational notes, and tablet-collected SUS responses were all compiled after each session and stored to a secure and access-controlled research data server.
3.4 Participants and Recruitment
Seven potential users participated in the pilot study, all of whom were working age adults (18-65) with mild to moderate intellectual/development disabilities (IDD). The federally defined term for “developmental disability” [80] means a severe, chronic disability of an individual that:
(a) | is attributable to a mental or physical impairment or combination of mental and physical impairment; | ||||
(b) | is manifested before the individual attains age 22; | ||||
(c) | is likely to continue indefinitely; | ||||
(d) | results in substantial functional limitations in 3 or more of the following areas of major life activity: (i) self-care; (ii) receptive and expressive language; (iii) learning; (iv) mobility; (v) self-direction; (vi) capacity for independent living; and (vii) economic self-sufficiency; and | ||||
(e) | reflects the individual's need for a combination and sequence of special, interdisciplinary, or generic services, individualized supports, or other forms of assistance that are of lifelong or extended duration and are individually planned and coordinated. | ||||
Of the recruited participants, their diagnoses included cerebral palsy, autism spectrum disorder, Down syndrome, and intellectual disabilities (Table 1). Participants were referred to investigators by CIDE clinicians or contacted via the IRB-approved (COMIRB #19-1675) CIDE recruitment database. Their participation presented no more than minimal risk of harm to subjects and was covered under COMIRB #18-1825. Inclusion criteria were as follows:
(1) | Between the ages of 18 and 65 years. | ||||
(2) | Cognitive disability as determined by the State of Colorado Office of Disability. | ||||
(3) | Fine motor skills within gross normal limits. | ||||
(4) | Auditory abilities within (or corrected to) gross normal limits. | ||||
(5) | Visual abilities within (or corrected to) gross normal limits. | ||||
(6) | Able to lift 15 pounds. | ||||
(7) | Willingness and capability to give informed consent to participate in the study. | ||||
(8) | Able to demonstrate verbal understanding of the main elements of the research protocol during the consenting process. | ||||
(9) | Fluent in cognitively appropriate English both expressively, and receptively. | ||||
(10) | Sufficient literacy to read item names and locations. | ||||
(11) | No current history of alcohol and/or drug addiction. | ||||
3.5 Study Procedure
At the beginning of each participant's first session, he/she provided informed consent to participate as per COMIRB #18-1825, and filled out a demographics form that additionally indicated what types of (cognitive) disabilities they were diagnosed with. The participant was then briefly trained to use the pre-selected method (paper ticket or NCAPS), and allowed to practice picking items for a few tickets to become comfortable with the process. Via balanced randomization, three participants used pen and paper and four participants used the NCAPS for their first session. Once acclimated, the participant “worked” picking items to fulfill tickets for a total of two hours, using a standard, unmodified utility cart (Rubbermaid 3-shelf, 34 × 20 × 38”) for paper tickets or the retrofitted cart with the NCAPS, while pausing for breaks as desired. Items were replaced over the course of the session by the investigators, maintaining stock levels without impeding the participant's work. Furthermore, no additional assistance was provided to the participants to select the correct items, and intervention from the investigators only occurred to resolve any technical malfunction of the NCAPS system. At the end of the session, the participant completed the SUS questionnaire on a touchscreen tablet to provide timely feedback on usability, and received a gift card as appreciation for their participation. After the two-week washout period, each participant's second session followed the same timeline with the alternate method (NCAPS or paper tickets), but forgoing the consenting and demographic data gathering process.
Fig. 7. Picking warehouse items using the two different methods. (Left) Traditional paper ticket with research participant. (Right) NCAPS—Nonlinear context-aware prompting system early prototype with student researcher.
3.6 Data Analysis
The mixed-methods metrics of the study and the respective data collection and analysis are described in Table 2. As mentioned earlier, the metrics of interest are split between the high-level categories of performance and preference data.
| Metrics | Data Collected | Data Analysis |
|---|---|---|
| Performance | ||
| Errors | Number of mistakes per session | Total errors per participant |
| Productivity | Number of correct items picked per session | Total correct items per participant |
| Preference | ||
| Usability | System Usability Scale (SUS) | SUS score averages per method |
| Feedback | Unsolicited participant comments | N/A (insufficient data to require qualitative data coding) |
Table 2. Description of Study Metrics and Corresponding Data Collection and Analysis
3.6.1 Performance Data.
Investigators reviewed all video data post-session to confirm and correct initial event markers. These timestamped markers were then extracted and combined with manually recorded error logs to verify data consistency, as well as offer more contextual information for the logged errors. The number of correctly gathered items during each two-hour work session was summed per participant to represent “productivity”. Errors of omission (missing item) and commission (incorrect item) were summed for each participant across each work session, allowing comparison of error rate between the paper ticket and NCAPS methods. Calculations and statistical analyses were performed using Excel (Microsoft, Redmond, WA) and Rstudio 2023.03.0 (Posit Software, Nottingham, England).
3.6.2 Preference Data.
SUS results were exported from the Qualtrics survey software (Qualtrics, Provo, UT) and scores were calculated per participant and session using Equation (1) [79]. Calculations and statistical analyses were performed using Excel (Microsoft, Redmond, WA) and Rstudio 2023.03.0 (Posit Software, Nottingham, England). (1) \(\begin{equation} SUS\ Score = 2.5*\left[ {\begin{array}{@{}*{1}{c}@{}} {\left( {Q1 - 1} \right) + \left( {5 - Q2} \right) + \left( {Q3 - 1} \right)}\\ { + \ \left( {5 - Q4} \right) + \left( {Q5 - 1} \right) + \left( {5 - Q6} \right)}\\ { + \ \left( {Q7 - 1} \right) + \left( {5 - Q8} \right) + \left( {Q9 - 1} \right)}\\ { + \ \left( {5 - Q10} \right)} \end{array}} \right]. \end{equation}\)
4 RESULTS AND FINDINGS
4.1 Performance Data
4.1.1 Errors.
“Errors” were defined as missing or incorrect items in a completed ticket. They were quantified after an order was turned in (at the “shipping table”), with the count revealing issues that were not corrected in real-time by the worker. Such issues would require checking and correction by another worker or job coach in a real work environment. Participants produced an average of 14.29 (med 7, range 0–66) errors in a two-hour work session using paper pick tickets, and an average of 0.57 (med 0, range 0–4) errors using the NCAPS (Figure 8). The NCAPS almost completely prevented such errors, with only one participant (P06) producing errors.
Fig. 8. Reduction in errors (missing or incorrect items) from paper tickets to the NCAPS.
4.1.2 Productivity.
“Productivity” was defined as the number of items gathered in a time period, in this case, quantified over a two-hour work session. This metric mimics performance measures and quotas utilized in real work environments, and encompasses time spent correcting errors in real-time. To characterize the feasibility of applying a paired-test, the Shapiro–Wilk test was used to evaluate the normality of the difference in productivity between the paper tickets and NCAPS datasets, and failed to reject the null hypothesis of a normal distribution (p = 0.3484). The small sample size of the study (N=7) inherently lacks the power to reject this null hypothesis in most cases, but a visual assessment of the data distribution suggests an acceptable assumption that the difference between the two datasets may be normally distributed (Figure 9).
Fig. 9. Distribution of productivity difference between paper tickets and NCAPS methods for each participant.
This assumption allows the application of a paired t-test on productivity, for which a significant reduction in productivity was determined between the traditional paper tickets method (mean = 227.4, SSD = 119.3) and the NCAPS method (mean = 130.7, SSD = 30.5); t(6) = 2.44, p < 0.05. This result indicates participants had a significantly lower productivity when they employed the NCAPS method compared to the paper tickets method.
A closer examination of productivity offers more context for this significantly lower productivity, showing a trend that participants with a higher baseline productivity with the traditional paper tickets are more negatively impacted by the NCAPS method, and the converse is true for participants with the lowest baseline productivity. This relationship is illustrated via a simple linear regression (Figure 10) used to predict this within subject change from baseline, revealing a significant relationship (y = −0.852x + 97, F(1,5) = 83.74, p < 0.0002) with R2 = 0.944. The two subjects with the worst initial performance (P06, P07) showed improvement with the NCAPS. Notably, participant P01 showed high initial performance and the greatest decrease in productivity with the NCAPS, but also produced a disproportionately high number of errors when using paper tickets.
Fig. 10. Relationship between baseline productivity (number of items gathered with the traditional paper tickets) and impacted productivity (change in items gathered from paper tickets to NCAPS). Participants with better initial performance showed decreased productivity with the NCAPS.
4.2 Preference Data
4.2.1 System Usability Scale.
Summary SUS scores, calculated using Equation (1), are presented in Table 3, along with general SUS cutoff scores for “fair”, “good”, and “excellent” usability [81]. Overall, both the paper tickets and NCAPS methods can be categorized as having “good” usability, according to the sample of PwCDs, with the paper tickets outscoring the NCAPS, which is consistent with the average participant productivity observed with each system.
| \(\color{white}{\rm PARTICIPANT}\) | \(\color{white}{\rm PAPER TICKETS}\) | \(\color{white}{\rm NCAPS}\) |
|---|---|---|
| P01 | 77.5 | 35.0 |
| P02 | 82.5 | 85.0 |
| P03 | 87.5 | 87.5 |
| P04 | 100.0 | 87.5 |
| P05 | 52.5 | 80.0 |
| P06 | 57.5 | 45.0 |
| P07 | 77.5 | 75.0 |
| MEAN | 76.4 | 70.7 |
[Range] 0-100 [Benchmarks] > 51 (fair); > 71 (good); > 86 (excellent).
Table 3. System Usability Survey (SUS) Scores
[Range] 0-100 [Benchmarks] > 51 (fair); > 71 (good); > 86 (excellent).
To verify the feasibility of applying a paired t-test, the Shapiro–Wilk test was applied to evaluate the normality of the difference between the paper tickets and NCAPS datasets, and failing to reject the null hypothesis of a normal distribution (p = 0.6161). Additional visual assessment of the difference data distribution supports a normal distribution (Figure 11).
Fig. 11. Distribution of SUS score difference between paper tickets and NCAPS methods for each participant.
The paired t-test revealed no significant difference between these scores for paper tickets (mean = 76.4, sd = 16.6) and the NCAPS (mean = 70.7, sd = 21.6); t(6) = 0.72, p = 0.4996.
Examining the consistency of preference data with performance (productivity) data on an individual basis, we first recall that only participants P06 and P07 increased their productivity when using the NCAPS, while productivity was diminished for all other participants. Based on this concept, participants P02, P03, P05, P06, and P07 all scored the paper tickets and NCAPS usability contrary to the productivity they experienced with each system. Participants P02, P03, P05 all scored the NCAPS equivalent or greater in usability relative to the paper tickets. Most notably, the participant with the greatest drop in productivity when using the NCAPS, P05, scored the NCAPS 27.5 points higher than the paper tickets on the SUS. Participants P06 and P07, despite being the only participants to increase their productivity with NCAPS, rated the NCAPS lower in usability, with P06 scoring it 12.5 points lower than the paper tickets.
4.2.2 User Comments.
Comments offered by participants during study sessions further illuminate some context for their response to the NCAPS, as well as highlight some suggestions for its improvement. These comments were unsolicited, and were predominantly volunteered by the participants while on the task of gathering items or while there was technical troubleshooting of the NCAPS. Multiple participants did note that they enjoyed using the NCAPS more than paper tickets, due to a combination of its prompting assistance and the novelty of the technology itself. Prominent complaints regarding the NCAPS included:
— | Distrust of the system when it indicated that an incorrect item was selected, due to previous false negative prompts (e.g. not registering the right item when scanned). | ||||
— | Being slowed down waiting for prompts at times when they could proceed faster on their own. | ||||
— | Having to perform cumbersome movements to ensure that sensors functioned properly (i.e., navigating the cart so the floor QR codes pass through the field of view of the location scanner/camera). | ||||
Participant frustrations and confusions due to system failures were quickly mitigated by the researchers so as to not excessively impact the participant's perception of the system. The rationale behind these rapid interventions was to best emulate the user experience of an NCAPS system that had been fully streamlined, thus evaluating the NCAPS concept and method, rather than the unoptimized hardware performances.
5 DISCUSSION
Despite the modest sample size, this pilot study yielded important findings that inform the future re-design of NCAPS and similar prompting systems for vocational assistance. Performance data (error and productivity), SUS scores, and user perspectives on the NCAPS reveal several factors that may have impacted the results. These findings will contribute to the successful development and deployment of prompting systems and technological employment supports for PwCDs. On a broader scale, this article's main contribution is to provide a detailed example of the challenges faced when introducing technology as a support for PwCDs. This study's findings illustrate the complex interplay between technology reliability, accuracy, efficiency, user experience and trust. Furthermore, depending on the level of supported needed, technology can help or hinder. This demonstrates the need for small-scale evaluations of technological supports with diverse populations in order to characterize the support's universal efficacy in practice, and the design considerations required to enhance the support's flexibility and inclusivity.
5.1 Technology Supports Must Not Hinder Any User
While the NCAPS dramatically reduced errors, preventing or correcting all errors for all but one participant, its effect on productivity was not as consistent. Users with the poorest initial performance improved with the aid of the NCAPS, but those with higher initial performance experienced reduced productivity when using it. These users indicated that the system could not keep up with them, and they were slowed when waiting for prompts or fumbling with the barcode scanner. Specifically, those who could memorize the locations of items and plan efficient routes through the warehouse were hindered by strict adherence to the system's guidance.
Previous research indicates that the level of IDD influences the effective use of technology supports and should be accounted for in their design and selection [34]. Beneficial AT must either be selected for and tailored to each unique user, or it must be sufficiently flexible and adaptive. Regardless of the chosen approach, its features must match, or at least not impede the most capable user.
In the particular case of order picking, existing warehouses utilize a variety of organizational systems ranging from “structured” (strictly guided) to “unstructured” (completely unguided) [75]. An optimally flexible system is likely to be a hybrid of these two extremes. A minimal approach to “not impeding” higher performing workers would be to present all items in a ticket (allowing unstructured work), while concurrently guiding the user to individual items (providing structure as desired). This would allow the user to “work ahead” when specific guidance is not required. An improved approach would be to allow this flexibility, while also optimally sorting the items in each ticket according to the layout of the warehouse (refining the provided structure). Such sorting is a feature of some existing commercial warehouse management systems (WMS), which generate “pick lines” to minimize travel time.
5.2 Technology Supports Must be Robust to the Environment
Effective correction of errors is an obvious metric of success for a context-aware prompting system. However, impact on productivity is also a significant determining factor in applicability to real work environments. A system that eliminates all errors (Figure 8), but slows work (Figure 10), will not help to prevent employer discrimination or negation of human capital. A successful solution will account for both accuracy and speed.
Real work environments are also not as controlled or predictable as experimental simulations, or even sheltered employment settings. Effective technological support must be robust in all possible conditions. NCAPS users occasionally struggled to correct for unexpected technological malfunctions, including failure of the scanning device to read barcodes (due to reflective labels on curved containers), or to read navigational QR codes (due to the limited field of view of the scanner's camera). Such interruptions would be disruptive for any worker but are particularly problematic for those with cognitive impairments [82].
These issues are exacerbated by a plethora of environmental conditions, including but not limited to varied lighting, electromagnetic interference, unreliable wireless connectivity, soiling or scratching of camera lenses, wear or damage to navigational markers or item tags, incorrect placement of items, or stock discrepancies. Interaction with other workers or customers introduces additional variance. Technological supports must be developed for robustness in real conditions and tested accordingly, applying all relevant performance metrics.
5.3 Technology Supports Must be Intuitive, Trustworthy, and Ignorable
When workflow was interrupted by unexpected system behavior, often due to the lack of robustness to environmental conditions, participants occasionally became confused or distressed. Such situations impact not only productivity, but also the user's attitude toward, and confidence in the system. These issues were then compounded, as the user questioned prompts and focused on the system itself. In particular, false negatives (i.e., the system indicating that a correct item is incorrect) quickly caused distrust. This breakdown in trust, and its impact on user satisfaction and productivity, must be carefully balanced against the initial goal of preventing all errors [82]. Data from participants P06 and P07 (reference Figure 10 and Table 3) support this concept, as both PwCDs are the only participants to improve their productivity with the NCAPS method but score it as less usable than the paper tickets method. In other words, the NCAPS’ perceived usability was potentially scored lower due to user distrust in the system, in addition to increased cognitive load around operating new and unfamiliar technology.
As users attempted to account for technical issues, they focused on the function of the system, its components, and mitigating strategies. They learned to angle barcodes beneath the scanner, and shade them from bright lights. They performed cumbersome movements to ensure that navigation QR codes entered the field of view of the scanning camera. While the resilience of the user was demonstrated in these cases, it should not be necessary. These issues increase cognitive load, further reducing productivity and user satisfaction. Additionally, the ideal solution would be ignorable and offer a simple user experience, even if the technology itself is complex [83]. It should function intuitively and allow the user to work with complete confidence, focusing on the task and not on the tools.
5.4 A Nuanced Inclusive Design Approach is Necessary
These insights reinforce the understanding that an effective approach to the design, implementation, and provision of technology supports for employment will be nuanced. It is vital to match AT with the specific needs and preferences of the user [34, 65], but doing so requires self-disclosure of disability and incurs expense, which can lead to labeling and discrimination [12]. Inclusive (and universal) design holds promise for balancing these concerns [34, 38, 48, 69–72]. However, ID might be improved in this context by inverting its approach. Rather than starting with mainstream technology and extending its features to accommodate users with disabilities, an initial design target should be to enable potential users who require the most assistance, and then refine toward the mainstream.
Successful examples of such designs do exist, and AT need not be reserved for people with disabilities. People who can effectively walk do not typically use wheelchairs, and those who can effectively use a mouse or touchscreen do not typically use eye gaze trackers for computer access. However, average users do utilize the voice control features of smartphones and other devices. This and other accessibility features have found their way into mainstream usage by becoming sufficiently robust, intuitive, and ignorable.
This approach will extend the benefits of technology supports to a broader population of workers than those with diagnosed disabilities. The Microsoft Inclusive Design Toolkit (https://www.microsoft.com/design/inclusive/) utilizes a persona spectrum to “understand related limitations across a spectrum of permanent, temporary, and situational disabilities.” In one example, television closed captioning benefits a person who is permanently deaf, as well as people with temporary or situational hearing impairments (i.e., a person with an ear infection or in a loud bar). In the context of “order picking” in small warehouses and retail environments, the temporary analog of cognitive disability might be starting a new job and not knowing the spatial layout or procedures. The situational analog might be a lack of fluency in the local language, or working after stock has been rearranged (a common practice). An ideal prompting system would benefit all of these workers, rendering self-disclosure of disability unnecessary and making the associated expense worthwhile to employers. Furthermore, it would allow users to improve gradually over time in a supported fashion, in the long term or as a training tool.
5.5 Limitations and Future Research
Several factors are relevant to the consideration of the described pilot study and its results. Because the primary objective was to guide the refinement of the NCAPS preceding a larger study of its efficacy, the sample size was necessarily small, precluding conclusive claims of statistical significance. Further, the sample did not include a representative cross section of relevant diagnoses or severity of cognitive impairment. Representative sampling will be essential to assess the effectiveness of the finalized system, particularly with respect to the goal of supporting all users while hindering none.
One or more metrics of cognitive function seem obligatory for assessment of benefit for workers with a diverse spectrum of abilities. However, this study utilized task performance in an industry standard scenario (using paper tickets) as a baseline metric for comparison, rather than a generalized metric (e.g., MMSE or ATD PA). This approach focuses on situationally relevant ability and echoes the overarching intent to avoid self-disclosure and formalized assessment of disability.
In order to properly assess the full potential of such a prompting system, the aforementioned design principles must be applied to produce a sufficiently adaptive and robust prototype. Taking advantage of the rapidly emerging embedded systems technology will likely be an essential component to fulfilling this objective. The refined NCAPS, including improvements such as optimally sorted item orders, will be evaluated in the eventual large-scale crossover study.
6 CONCLUSION
The topic of employment supports for workers with cognitive disabilities is multifaceted, and effective solutions require a nuanced approach. Evolving definitions of “disability” focus on the mismatch between person and environment. However, finding an ideal match between the two, or effectively filling the gap with technology, is nontrivial. Individualized assessment and AT selection processes are exceedingly beneficial, but can complicate issues stemming from disclosure and discrimination.
The iterative development, testing, and refinement of the NCAPS described herein offers insights for further improvement of technology supports and their application. A tool that helps one worker may hinder another, but individual selection and customization of tools may be infeasible. Prioritization of robustness and intuitiveness could lead to flexible solutions that help when needed, and get out of the way otherwise. Designing for extreme cases across the spectrum of relevant human function will allow AT to scale effectively.
This approach could extend the utility of such technology to broader populations of workers, including those with temporary and situational impairments. This would dramatically increase its value to employers, subsequently increasing the perceived value of workers with disabilities, and reducing stigma and discrimination.
ACKNOWLEDGEMENTS
The authors would like to thank Joshua Carlin, Anna Goldberg-Richmeier, Omar Hamid, Brandon Lee, Tu Nguyen, David Pak, Samantha Thomas for assisting in data collection and analysis; Devin Benson, Eric Gunther, Rachel Fry, and Alyssa Sawyer for their valuable insights informing the NCAPS design; Sarel van Vuuren, Nattawut Ngampatipatpong, and Jariya Tuantranont for the software development of the NCAPS; Victoria Haggett for her contributions to protocol development.
- [1] . 2019. 2017 Disability Status Report: United States. Ithaca, NY.Google Scholar
- [2] . 2020. Annual Disability Statistics Compendium: 2019. Durham, NH.Google Scholar
- [3] . 2008. Entering work: Employment outcomes of people with developmental disabilities. International Journal of Rehabilitation Research 31, 3 (2008), 217–223.Google ScholarCross Ref
- [4] . 2012. VR outcome trends and the recent decline in employment for VR customers with intellectual disabilities. In Data Note Series (2012).Google Scholar
- [5] . 2020. Employment outcomes for individuals with intellectual and developmental disabilities: A literature review. Children and Youth Services Review 109 (2020), 104656.Google ScholarCross Ref
- [6] . 2018. How does employment influence health outcomes? A systematic review of the intellectual disability literature. Journal of Vocational Rehabilitation 49, 1 (2018), 1–13.Google ScholarCross Ref
- [7] . 1996. The meaning of work in the lives of people with significant disabilities: Consumer and family perspectives. Journal of Rehabilitation 62, 3 (1996), 49–55.Google Scholar
- [8] . 2009. From margins to mainstream: What do we know about work integration for persons with brain injury, mental illness and intellectual disability? Work 32, 4 (2009), 391–405.Google ScholarCross Ref
- [9] . 2013. National snapshot of adults with intellectual disabilities in the labor force. Journal of Vocational Rehabilitation 39, 3 (2013), 157–165.Google ScholarCross Ref
- [10] . 2000. Longitudinal job satisfaction of persons in supported employment. Education and Training in Mental Retardation and Developmental Disabilities 35, 4 (2000), 365–373.Google Scholar
- [11] . 2016. The National EEOC ADA research project: History, available data, and basic findings. Journal of Vocational Rehabilitation 44, 3 (2016), 333–342.Google ScholarCross Ref
- [12] . 2009. Barriers to employment as experienced by disabled people: A qualitative analysis in Calgary and Regina, Canada. Disability and Society 24, 1 (2009), 63–75.Google ScholarCross Ref
- [13] . 2004. Beyond transition: A comparison of the employment experiences of American and Canadian adults with LD. Journal of Learning Disabilities 37, 4 (2004), 283–291.Google ScholarCross Ref
- [14] . 1997. The relationship between job satisfaction and psychological health in people with an intellectual disability in competitive employment. Journal of Intellectual and Developmental Disability 22, 3 (1997), 199–211.Google ScholarCross Ref
- [15] . 2019. The effectiveness of rehabilitation interventions on the employment and functioning of people with intellectual disabilities: A systematic review. Journal of Occupational Rehabilitation 29 (2019), 773–802.Google ScholarCross Ref
- [16] . 2021. Vocational interventions to improve employment participation of people with psychosocial disability, autism and/or intellectual disability: A systematic review. International Journal of Environmental Research and Public Health 18, 22 (2021), 12083.Google ScholarCross Ref
- [17] . 2013. Implications of workplace accommodations for persons with disabilities. Journal of Workplace Behavioral Health 28, 3 (2013), 192–211.Google ScholarCross Ref
- [18] . 2011. Sheltered employment. Encyclopedia of Clinical Neuropsychology (2011), 2286–2286.Google ScholarCross Ref
- [19] . 1998. Sheltered employment for persons with disabilities. International Labour Review 137, 3 (1998), 364–365.Google Scholar
- [20] . 2002. Sheltered vs. Supported Employment: A Direct Comparison of Long-Term Earnings Outcomes for Individuals with Cognitive Disabilities.” Achievements and Challenges in Employment Services for People with Disabilities: The Longitudinal Impact of Workplace S. In (2002), 63–84.Google Scholar
- [21] . 2005. Cognitive impairments, information technology systems and the workplace. ACM SIGACCESS Accessibility and Computing 1, 83 (2005), 25–29.Google ScholarDigital Library
- [22] . 2019. Job Coaches for Adults with Disabilities: A Practical Guide. Jessica Kingsley Publishers.Google Scholar
- [23] . 2010. A comparison of quality of life outcomes for people with intellectual disabilities in supported employment, day services and employment enterprises. Journal of Applied Research in Intellectual Disabilities 23, 3 (2010), 290–295.Google ScholarCross Ref
- [24] . 2014. Effect of supported employment on vocational rehabilitation outcomes of transition-age youth with intellectual and developmental disabilities: A case control study. Intellectual and Developmental Disabilities 52, 4 (2014), 296–310.Google ScholarCross Ref
- [25] . 1997. The impact of social integration interventions and job coaches in work settings. Education and Training in Mental Retardation and Developmental Disabilities 32, 4 (1997), 281–292.Google Scholar
- [26] . 1994. The underachievement of supported employment: A call for reinvestment. Journal of Disability Policy Studies 5, 2 (1994), 1–24.Google ScholarCross Ref
- [27] . 1999. Paraprofessionals in Job Coach Roles. Research and Practice for Persons with Severe Disabilities 24, 4 (1999), 273–280.Google Scholar
- [28] . 1994. RETURN TO WORK FOR PATIENTS WITH TRAUMATIC BRAIN INJURY. American Journal of Physical Medicine and Rehabilitation 73, 4 (1994), 280– 282.Google ScholarCross Ref
- [29] . 2010. Evaluation of two instruction methods to increase employment options for young adults with autism spectrum disorders. Research in Developmental Disabilities 31, 6 (2010), 1223–1233.Google ScholarCross Ref
- [30] . 2007 Assistive Technology in the Workplace. Elsevier Health Sciences.Google Scholar
- [31] . 2001. Enhancing employment outcomes through job accommodation and assistive technology resources and services. Journal of Vocational Rehabilitation 16, 1 (2001), 27–37.Google Scholar
- [32] . 2006. Assistive technology and employment: experiences of Californians with disabilities. Work (Reading, Mass.) 27, 4 (2006), 333–44.Google Scholar
- [33] . 2022. Using wireless technology to support individuals with intellectual and developmental disabilities in vocational settings: A focus group study. Journal of Vocational Rehabilitation 56, 3 (2022), 1–10.Google Scholar
- [34] . 2019. Technology use to support employment-related outcomes for people with intellectual and developmental disability: An updated meta-analysis. International Journal of Developmental Disabilities 65, 4 (2019), 220–230.Google ScholarCross Ref
- [35] . 2015. The rehabilitation engineering research center for the advancement of cognitive technologies. American Journal of Alzheimer's Disease and other Dementias 30, 1 (2015), 6–12.Google ScholarCross Ref
- [36] . 2008. Assistive technology for cognition. IEEE Medicine in Biology Magazine 27, 2 (2008), 29–40.Google ScholarCross Ref
- [37] . 2010. Assistive technology effects on the employment outcomes for people with cognitive disabilities: A systematic review. Disability and Rehabilitation: Assistive Technology 5, 6 (2010), 377–391.Google ScholarCross Ref
- [38] . 2006. Technology use by people with intellectual and developmental disabilities to support employment activities: A single-subject design meta analysis. Journal of Vocational Rehabilitation 24, 2 (2006), 81–86.Google Scholar
- [39] . 2021. The effects of a mobile-based vocational skill building coaching technology intervention for people with cognitive disabilities: A pilot feasibility study. Journal of Rehabilitation and Assistive Technologies Engineering 8 (2021), 20556683211009731.Google ScholarCross Ref
- [40] . 2015. Teaching individuals with intellectual disability to email across multiple device platforms. Research in Developmental Disabilities 36 (2015), 645–656.Google ScholarCross Ref
- [41] . 2013. Comparing the effects of commercially available and custom-made video prompting for teaching cooking skills to high school students with autism. Remedial and Special Education 34, 6 (2013), 371–383.Google ScholarCross Ref
- [42] . 2010. Using an Extended Automatic Target Acquisition Program with dual cursor technology to assist people with developmental disabilities improve their pointing efficiency. Research in Developmental Disabilities 31, 5 (2010), 1091–1101.Google ScholarCross Ref
- [43] . 2013. Tablet-based video modeling and prompting in the workplace for individuals with autism. Journal of Vocational Rehabilitation 38, 1 (2013), 1–14.Google ScholarCross Ref
- [44] . 2014. A computer-based interactive game to train persons with cognitive impairments to perform recycling tasks independently. Research in Developmental Disabilities 35, 12 (2014), 3672–3677.Google ScholarCross Ref
- [45] . 2012. Empowering adults with a cognitive disability through inclusion of non-linear context aware prompting technology (N-CAPS). In Proceedings of the RESNA Annual Conference. 59–108.Google Scholar
- [46] . 2016. A nonlinear contextually aware prompting system (N-CAPS) to assist workers with intellectual and developmental disabilities to perform factory assembly tasks: System overview and pilot testing. Disability and Rehabilitation: Assistive Technology 11, 7 (2016), 604–612.Google ScholarCross Ref
- [47] . 2011. A location-based prompting system to transition autonomously through vocational tasks for individuals with cognitive impairments. Research in Developmental Disabilities 32, 6 (2011), 2669–2673.Google ScholarCross Ref
- [48] . 2002. Enhancing independent task performance for individuals with mental retardation through use of a handheld self-directed visual and audio prompting system. Education and Training in Mental Retardation and Developmental Disabilities 37, 2 (2002), 209–218.Google Scholar
- [49] . 2001. VICAID: Development and evaluation of a palmtop-based job aid for workers with severe developmental disabilities. British Journal of Educational Technology 32, 3 (2001), 277–287.Google ScholarCross Ref
- [50] . 1997. The planning and execution assistant and trainer (PEAT). Journal of Head Trauma Rehabilitation 12, 2 (1997), 85–91.Google ScholarCross Ref
- [51] . 2005. Promoting independent performance of transition-related tasks using a palmtop PC-Based Self-Directed visual and auditory prompting system. Journal of Special Education Technology 20, 2 (2005), 5–14.Google ScholarCross Ref
- [52] . 2009. The effectiveness of using a video ipod as a prompting device in employment settings. Journal of Behavioral Education 18 (2009), 119–141.Google ScholarCross Ref
- [53] . 2009. Effects of time management instruction on adolescents' ability to self-manage time in a vocational setting. Journal of Special Education 43, 3 (2009), 145–159.Google ScholarCross Ref
- [54] . 2011. The use of assistive technology to improve time management skills of a young adult with an intellectual disability. Journal of Special Education Technology 26, 3 (2011), 13–20.Google ScholarCross Ref
- [55] . 2013. An augmented reality (AR)-based vocational task prompting system for people with cognitive impairments. Research in Developmental Disabilities 34, 10 (2013), 3049–3056.Google ScholarCross Ref
- [56] . 2014. An assistive navigation system based on augmented reality and context awareness for people with mild cognitive impairments. IEEE Journal of Biomedical and Health Informatics 18, 1 (2014), 368–374.Google ScholarCross Ref
- [57] . 2015. Augmented reality as a navigation tool to employment opportunities for postsecondary education students with intellectual disabilities and Autism. Journal of Research on Technology in Education 47, 3 (2015), 157–172.Google ScholarCross Ref
- [58] . 2015. VR4VR: Vocational rehabilitation of individuals with disabilities in immersive virtual reality environments. In Proceedings of the 8th ACM International Conference on PErvasive Technologies Related to Assistive Environments - PETRA '15. ACM, 1–4.Google ScholarDigital Library
- [59] . 2008. Use of a handheld prompting system to transition independently through vocational tasks for students with moderate and severe intellectual disabilities. Education and Training in Developmental Disabilities 43, 1 (2008), 102–110.Google Scholar
- [60] . 2011. Rapid specification and automated generation of prompting systems to assist people with dementia. Pervasive and Mobile Computing 7, 3 (2011), 299–318.Google ScholarDigital Library
- [61] . 2017. A UHF RFID positioning system for use in warehouse navigation by employees with cognitive disability. Disability and Rehabilitation: Assistive Technology 12, 8 (2017), 832–842.Google ScholarCross Ref
- [62] . 2008. CATPro: Context-aware task prompting system with multiple modalities for individuals with cognitive impairments. In Proceedings of the 10th International ACM SIGACCESS Conference on Computers and Accessibility. 301–302.Google Scholar
- [63] . 2009. Context-aware prompting to transition autonomously through vocational tasks for individuals with cognitive impairments. In Proceedings of the 11th International ACM SIGACCESS Conference on Computers and Accessibility. 19–26.Google ScholarDigital Library
- [64] . 2008. A context aware handheld wayfinding system for individuals with cognitive impairments. In Proceedings of the 10th International ACM SIGACCESS Conference on Computers and Accessibility. 27–34.Google ScholarDigital Library
- [65] . 2015. Why people use and don't use technologies: Introduction to the special issue on assistive technologies for cognition/cognitive support technologies. NeuroRehabilitation 37, 3 (2015), 315–319.Google ScholarCross Ref
- [66] . 2011. The relationship of personal factors and subjective well-being to the use of assistive technology devices. Disability and Rehabilitation 33, 10 (2011), 811–817.Google ScholarCross Ref
- [67] . 2002. Matching Person and Technology (MPT) assessment process. Technology and Disability 14, 3 (2002), 125–131.Google ScholarCross Ref
- [68] . 2016. MMSE TO EVALUATE ABILITIES IN MATURE WORKERS WITH COGNITIVE IMPAIRMENTS. The Gerontologist 55, 2 (2016), 134–135.Google Scholar
- [69] . 2003. Inclusive design or assistive technology. Inclusive Design (2003), 172–181.Google ScholarCross Ref
- [70] . 2011. User-sensitive inclusive design. Universal Access in the Information Society 10 (2011), 235–243.Google ScholarDigital Library
- [71] . 1998. USERfit - A framework for user centred design in assistive technology. Technology and Disability 9, 3 (1998), 163–171.Google ScholarCross Ref
- [72] . 2018. Effectiveness of technology types in employment-related outcomes for people with intellectual and developmental disabilities: An Extension Meta-analysis. Advances in Neurodevelopmental Disorders 2 (2018), 262–272.Google ScholarCross Ref
- [73] . 2014. Guide to the Software Engineering Body of Knowledge (SWEBOK(R)): Version 3.0. In, IEEE Computer Society Press.Google ScholarDigital Library
- [74] . 2000. Rapid ethnography: Time deepening strategies for HCI field research. In Proceedings of the Conference on Designing Interactive Systems: Processes, Practices, Methods, and Techniques, DIS. 280–286.Google ScholarDigital Library
- [75] . 2016. Ethnographically-informed Design, Development, and Testing of a Mobile Media Collection and Editing Application for Installation and Maintenance of Non-linear Context-aware Prompts. University of Colorado Denver.Google Scholar
- [76] . 2016. A Non-linear Context-aware Prompting System Designed to Assist Working Age Adults with Cognitive Disabilities in Completion of Pre-determined Vocational Tasks in a Warehouse Environment: Development and Preliminary Testing for Error Rate and Efficiency pe. University of Colorado Denver.Google Scholar
- [77] . 1985. Generalized best-first search strategies and the optimality of A*. Journal of the ACM 32, 3 (1985), 505–536.Google ScholarDigital Library
- [78] . 1968. A formal basis for the heuristic determination of minimum cost paths. IEEE Transactions on Systems Science and Cybernetics 4, 2 (1968), 100–107.Google ScholarCross Ref
- [79] . 1995. SUS - A quick and dirty usability scale. Usability Evaluation in Industry (1995), 474–475.Google Scholar
- [80] Developmental Disabilities Assistance and Bill of Rights Act of 2000. Public Law, 106–402.Google Scholar
- [81] . 2009. Determining what individual SUS scores mean: Adding an adjective rating scale. Journal of Usability Studies 4, 3 (2009), 114–123.Google ScholarDigital Library
- [82] . 2015. Why people use and don't use technologies: Introduction to the special issue on assistive technologies for cognition/cognitive support technologies. NeuroRehabilitation 37 (2015), 315–319.Google ScholarCross Ref
- [83] . 2006. Desperately seeking simplicity: How young adults with cognitive disabilities and their families adopt assistive technologies. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 1143–1152.Google ScholarDigital Library
Index Terms
- Helping or Hindering: Inclusive Design of Automated Task Prompting for Workers with Cognitive Disabilities
Recommendations
Toward a More Inclusive Gig Economy: Risks and Opportunities for Workers with Disabilities
CSCWLittle is known about whether and how workers with disabilities participate in the many on-demand labor platforms that make up the growing gig economy. Understanding disabled gig workers' experiences is a vital step toward developing inclusive and ...
Inclusive, Adaptive Design for Students with Learning Disabilities
ICALT '08: Proceedings of the 2008 Eighth IEEE International Conference on Advanced Learning TechnologiesYoung adults with severe disabilities and learning difficulties (SLD) have very limited access to appropriate learning resources. Their unique individual needs and requirements prevent them from accessing traditional methods of online learning, and ...
A kinect-based vocational task prompting system for individuals with cognitive impairments
Difficulties in executing daily living tasks hamper the quality of life of many individuals with cognitive impairments who are otherwise physically mobile. With sufficient and appropriate support on the job, many people with developmental disabilities ...
Comments