Keywords

1 Introduction

Adverse drug events (ADEs) has incurred an additional $3.5 billion in medical costs [1]. Computerized Provider Order Entry (CPOE) was recommended by the Institute of Medicine (IOM) as a way to reduce patient harm caused by medication errors [2]. Health information technology (HIT) is being used extensively in clinical practice and more physicians are adopting CPOEs because of the financial incentives guaranteed by Centers for Medicare and Medicaid (CMS) [3]. CPOEs are as computer-aided medication and laboratory ordering system. It is mandated for any licensed healthcare professional to request laboratory, radiology, and medication orders through CPOE as a part MU stage 2 of the EHR incentive program. One major benefit to computerized medication orders is reduced medication errors created from inaccurate transcription or incomprehensible handwriting. On the other hand, CPOE may cause unintentional penalties, such as, increasing clinicians’ workload, undesirable workflow issues, and generation of new kinds of errors [4–9]. Mediocre CPOE usability of is one of the main factor that leads to consequences, such as decreased efficiency, reduction in the quality of care given to patients, and unsatisfied clinicians [10–13]. Usability is defined in this study as how well users can maneuver a system to effectively and efficiently complete particular tasks [14]. Learnability is defined as the degree to which the system enables users to comprehend how to use the system [15]. In the literature, while there are disparities when describing usability and learnability [15–17], definitions of learnability are intensely associated with usability and proficiency [16, 18, 19]. Learnability is the length of time and effort that is needed for a user to improve proficiency with a system over a period of time and after multiple uses [20].

The objective of this study is to identify differences in learnability in terms of user performance between expert and novice primary care physicians three and seven months after resident’s EHR training. Our null hypothesis is that novice and expert physicians will be more proficient with increased EHR experience. If there is no significant difference between novice and expert physicians then the performance measures identified is not based on novice physicians’ inexperience with the system but that there is room for improvement in the current design of the CPOE.

2 Method

2.1 Study Design

To determine learnability gaps in use of EHR systems between expert and novice physicians, standard lab-based usability tests were conducted incorporating think aloud strategy and video analysis using Morae® (TechSmith, Okemos, MI). Twelve family medicine and four internal medicine resident physicians, completed five artificial, scenarios-based tasks. A mixed methods approach was utilized to identify the learnability gaps between the novice and expert physicians. The mixed methods approach included four types of performance measures, and qualitative debriefing session with participants. This pilot study was approved by the University of Missouri Health Sciences Institutional Review Board.

2.1.1 Organizational Setting

This study was conducted at the University of Missouri Health System (UMHS), which is a 536 bed, tertiary care academic medical hospital situated in Columbia, Missouri. In 2012, UMHS had roughly 553,300 clinic visits and employs more than 70 primary care physicians. The Department of Family and Community Medicine (FCM) oversees six clinics, while the Department of Internal Medicine (IM) manages two primary care clinics [23]. The Healthcare Information and Management Systems Society (HIMSS), a non-profit organization that rates how effectively hospitals are adopting electronic health records (EMR) in the organization, has awarded UMHS with Stage 7 of the EMR Adoption Model [24]. Specifically, UMHS has employed electronic patient charts, inspected clinical data with the use of data warehousing, and distributes health information electronically with authorized health care entities [25]. The CPOE embedded in the EHR gives physicians a safe way to electronically access and place patients’ lab and medication orders, and deliver the orders to the responsible department that is processing the request. UMHS’ EHR database contains all the records from the university’s hospitals and clinics. Measuring learnability between expert and novice physicians in a healthcare system that uses a fully employed EHR system makes the aim of this study achievable.

2.2 Participants

There is presently no evidence-based method to quantity a user’s EHR skill, therefore, novice and expert physicians were categorized based on clinical training level and number of years using the EHR. This verdict was grounded on a conversation with a knowledgeable physician champion (JLB) and two chief resident physicians from both participating departments (FCM, IM). The theory is that after one year resident physicians could achieve necessary skills to be considered an expert. Therefore, ten first year resident physicians were grouped as novice users and six second and third year resident physicians were categorized as expert users. Both FCM and IM departments offer three year residency programs. Convenience sampling methodology was utilized when selecting participants [26]. Both UMHS FCM and IM physicians were selected for the sample because, as primary care residents, they have comparable clinical responsibilities. Twelve FCM and four IM residents was a part of this study. Based on a review of the literature, ten participants were considered appropriate in explorative usability studies to discover the foremost issues to resolve in a product development cycle [27, 28]. Participation was voluntary and subjects were reimbursed for their time.

2.3 Scenario and Tasks

The scenario “a scheduled follow up visit after a hospitalization for pneumonia” was given to resident physicians in round one of the study. For round two, resident physicians received the scenario “a scheduled follow up visit after a hospitalization for heart failure.” Although the residents were given two different scenarios, these two scenarios were equivalent in complexity, workflow, and functionalities used. The purpose of these scenarios were to evaluate physicians’ use of the EHR that comprises of realistic inpatient and outpatient information. Five commonly completed tasks were created for both novice and expert primary care physicians to execute. These tasks met 2014 EHR certification criteria 45 CFR 170.314 for meaningful use stage 2 [20]. The tasks were also a part of the mandatory EHR training conducted at the commencement of physicians’ residency. The tasks had a clear objective that physicians were able to follow without unnecessary clinical cognitive load or ambiguity, which were not a part the study’s goals. The tasks were:

  • Task 1: Place order for chest x-ray

  • Task 2: Place order for Basic metabolic panel (BMP)

  • Task 3: Change a Medication

  • Task 4: Add a medication to your favorites list

  • Task 5: Renew one of the existing medications

Performance Measures.

Four performance measures were used to analyze user performance as follow:

  1. 1.

    Percent task success rate - the percentage of subtasks that participants finished successfully without any errors.

  2. 2.

    Time-on-task - the length of time each participant took to finish each task, starting when participants click “start task” to when “end task” is clicked.

  3. 3.

    Mouse clicks - the sum of clicks on the mouse taken to finish a given task.

  4. 4.

    Mouse movement - the length in pixels of the navigation path to finish each task.

For time on task, mouse clicks, and mouse movements, a lower value usually indicates higher performances. Higher values may portray that the participant had complications with the system when completing tasks.

Data Collection.

Usability data for round one was collected between November 12, 2013 and December 19, 2013 and round two data was collected between February 12, 2014 and April 22, 2014. Round one data was collected at UMHS three months after novice resident physicians concluded their compulsory EHR training. Resident physicians were requested to take part in round two roughly three months after the date they completed round one. Usability testing took twenty minutes and was conducted using a 15 inch laptop with Windows 7 operating system. To maintain constancy and lessen unwelcome disruptions, the participant and the facilitator were the only two individuals in the conference room while the usability session took place. At the beginning of the session, the participant were advised of their rights as a participant. The participant was then given instructions to read that contained a scenario and five tasks. Think aloud strategy was used throughout the session and was recorded using Morae Recorder [30]. We encouraged participants to talk aloud and describe method of completing the tasks. Participants completed the tasks without the assistance of the facilitator who would only mediate if there were any technical difficulties. However, there were no technical difficulties and facilitator was not required to mediate. After participants completed the tasks they filled out the demographic survey. At the end of the session, a debriefing session was conducted where participants were asked to elaborate on tasks they thought were problematic. Observations of interest to the facilitator were deliberated as well.

Data Analysis.

Morae Recorder was used to log audio, video, on-screen activity, as well as inputs from the keyboard and mouse. We established that there were no major EHR interface change between round one data collection and round two that may affect the results of the study. The recorded sessions were observed using Morae Manager by computing performance measures using markers to pinpoint difficulties and errors the participants faced. Video analysis took approximately 1.5 h for each 20 min recorded session. The first phase in analysis was to assess the recorded sessions and tag any tasks that were unmarked during data collection. The second step was to separate each of the five tasks into smaller tasks to compute the task success rate. The statistical test used to compare performance measures was the t-test and geometric mean [31].

3 Results

Percent Task Success Rate.

Geometric mean values of percent task success rates of five tasks were compared between the expert and novice physicians across two rounds (Fig. 1) [31]. There was a 25 percent point increase in novice physicians’ percent task success rate between round one and round two, but it was not statistically significant (50 %, round 1 vs. 62 %, round 2, p = 0.67). Similarly, expert physicians had a 41 percent point increase in percent task success rate between round one and round two, however, there was no statistically significant difference (45 %, round 1 vs. 63 %, round 2, p = 0.58).

Fig. 1.
figure 1

Geometric mean values of percent task success rates of five tasks between the expert and novice physicians.

Full size image

To identify learnability gap, round one and round two task success of both physician groups were compared. In round one, there was no statistically significant difference in the success rate between the two physician groups (45 %, expert group vs. 50 %, novice group, p = 0.91) and in round two, there was no statistically significant difference in the success rate between the two physician groups (63 %, expert group vs. 62 %, novice group, p = 0.98). In round one, novice physicians attained lower success rates in three out of five tasks: 3 – 5 and higher success rate than expert physicians in two out of five tasks: 1 and 2. In round two, novice physicians attained lower success rates in task 1, the same success rate in two tasks: 3 and 4, and lower success rates in two tasks: 2 and 5 (Table 1).

Table 1. Demographics of 15 primary care resident physicians that participated in the usability test presented as percentages. Examined demographics include gender, age, race, and experience other than current EHR.
Full size table

Time on Task (TOT).

Geometric mean values of time-on-task (TOT) were compared between expert and novice physicians across two rounds Fig. 2. There was an 36 % increase in novice physicians’ time on task between round one and round two, but it was not statistically significant (45 s, round 1 vs. 62 s, round 2, p = 0.50). There was a 24 % increase in expert physicians’ time on task between round one and round two, but it was not statistically significant (37 s, round 1 vs. 45 s, round 2, p = 0.66).

Fig. 2.
figure 2

Geometric mean values of time-on-task (TOT) compared between expert and novice physicians.

Full size image

To identify learnability gap, round one and round two time on task for both physician groups were compared. In round one, no substantial difference was observed between the two physician groups (37 s, expert group vs. 45 s, novice group, p = 0.45) and similarly, in round two, no substantial difference (p = 0.23) was observed between the two physician groups (45 s, expert group vs. 62 s, novice group). In round one, novice physicians did not complete any of the five tasks faster than expert physicians, but completed task 4 at the same time. Similarly In round two, novice physicians did not complete any of the five tasks faster than expert physicians, but completed task 5 at the same time.

Mouse Clicks.

Geometric mean values of mouse clicks were compared between the two physician groups across two rounds Fig. 3. There was a 47 % increase in novice physicians’ mouse clicks between round one and round two, but it was not statistically significant (10 clicks, round 1 vs. 14 clicks, round 2, p = 0.51). Similarly, there was a 26 % increase in expert physicians’ mouse clicks between round one and round two, but it was not statistically significant (8 clicks, round 1 vs. 11 clicks, round 2, p = 0.67).

Fig. 3.
figure 3

Geometric mean values of mouse clicks compared between the two physician groups

Full size image

To identify learnability gap, round one and round two mouse clicks for both physician groups were compared. Expert physicians completed the tasks with slightly fewer mouse clicks than novice physicians did in round one (8 clicks, expert group vs. 10 clicks, novice group, p = 0.58) and round two (11 clicks, expert group vs. 14 clicks, novice group, p = 0.38). In round one, novice physicians used more mouse clicks to completed tasks 1–3, less mouse clicks to complete task 4, and the same number of clicks to complete task 5.

Mouse Movements.

Geometric mean values of mouse movement, length of the navigation path to complete a given task, were compared between two physician groups across two rounds (Fig. 4). There was a 68 % increase in in novice physicians’ mouse movements between round one and round two, but it was not statistically significant (8146 pixels, round 1 vs. 13,649 pixels, round 2, p = 0.63). There was a 28 % decrease in expert physicians’ mouse movements between round one and round two, however, there was no statistically significant difference (p = 0.64) (8480 pixels, round 1 vs. 10,817 pixels, round 2).

Fig. 4.
figure 4

Geometric mean values of mouse movement, length of the navigation path to complete a given task, compared between two physician groups.

Full size image

To identify learnability gap, round one and round two mouse movements for both physician groups were compared. Overall, the novice physicians showed slightly longer mouse movements across the five tasks in round one (8480 pixels, expert group vs. 8146 pixels, novice group, p = 0.59) and round two (10,871 pixels, expert group vs. 13,649 pixels, novice group, p = 0.34). In round one, novice physicians used more mouse movement to complete all tasks except to complete task 4, which novice physicians completed using less mouse movements.

4 Discussion and Conclusion

While CPOEs have many benefits, such as, clinical improvement and increased efficiencies, there are multiple challenges created from insufficient software design. In our study, we were unable to discover any statistical difference between expert and novice physicians’ performance measures across round one and round two which means we fail to reject the null hypothesis that physicians will be more proficient with CPOE experience. This study showed that longer exposure levels with CPOE does not correspond to being an expert, competent in using CPOEs [32]. A similar study by Kjeldskov, Skov, and Stage [12] identifying usability issues faced by novice and expert nurses studied the possibility of usability issues vanishing over time. Seven nurses completed fourteen and thirty hours of training before the first usability assessment that included seven tasks and subtasks centralized on the principal feature of the system. The same seven nurses retook the usability assessment involving the same seven tasks after fifteen months of everyday use of the system. Only two novice subjects solved all task (p = 0.01) while the entire expert subjects solved all seven tasks either completely or partially. Similar to the results of our study, no statistically significant difference between novice and expert nurses was found when considering only completely solved tasks (p = 0.08).

When designing CPOEs, disregarding usability issues may be partly responsible for potential human-computer interaction issues that may contribute to loss of productivity and a reduction in quality of patient care [33]. A literature review by Phansalkar et al., found that many basic human factors principles are not followed when designing clinical information systems [34]. Incorporating human-centered design, by involving users’ critiques into the CPOE development process, may productively create a user-friendly system and may contribute to reducing the dissatisfaction that is associated with poor information display [35]. Designing CPOE displays that represent the pertinent information needed by physicians during clinic visits may decrease errors created by cognitive overload [36, 37]. A review written by Khajouei and Jaspers describes multiple studies expressing difficulty when searching for specific information in medication ordering systems because of inferior screen displays [32, 38]. Refining CPOE design based on information needs of clinicians in the electronic progress note by decreasing redundant information that is a part of the display and compressing the information being displayed, could address the human computer interaction problems presented in Khajouei and Jaspers’ study encountered by users of the progress note [37, 39].

Limitations to This Study.

This study was effective in recognizing differences in performance measures between novice and expert physicians but also contained several methodological limitations. This study was restricted to primary care physicians, a small sample size, and only involved one CPOE from one healthcare institution which conveys that results may not transferrable to other healthcare institutions and other specialties. The usability test was also conducted using a small sample of clinical tasks and may not represent other features and functions that may be used in other clinical scenarios. This study took place in a laboratory setting which does not consider distractions faced by physicians during clinical encounters. Although this study contains some methodological limitation, this is a well-controlled study that used triangular evaluation and instructions were straightforward to the physicians which permitted participants to complete the required tasks.