Keywords

1 Introduction

Since its inception, Smartphone use has increased exponentially [1] and following the launch of the iPhone in 2007 [2] and the App Store in 2008 [3] mobile application usage has seen dramatic growth [4]. The iOS App Store recently surpassed one billion downloads with more than two million Apps available [5]. Due to the growth in use, smartphones and mobile applications have become increasingly necessary tools for both clinical practice and education [6, 7]. Examples include the use of innovative digital delivery methods of delivery for Clinical Guidelines; Clinical Decision Support, and Calculations tools [6,7,8].

Some research has suggested that there are potentially negative aspects to smartphone use in clinical settings, most notably relating to patient perception [9] and accuracy of information [10]. However, it is generally accepted that smartphone use to enhance clinical care and healthcare practice is largely positive [6,7,8] with numerous studies providing evidence of the positive impact these devices and their applications have on reducing medical errors [11] improving learning [9] and creating a more efficient process for patients [12, 13].

In a clinical setting, relevant and accurate information is critical, it must be easy and convenient to access, benefiting both clinical practice, and clinical education [6,7,8]. This is especially true for information such as clinical guidelines [14] which are used to support clinicians in making decisions on how to diagnose, treat and care for patients. There is therefore clear potential for research combining methods for the design and development of medical applications and the delivery of medical guidelines.

1.1 Background/Problem Statement

Clinical guidelines are provided to all UK hospitals [14]. Some UK hospitals develop trust level guidelines to deliver more specific and concise information [15]. They are often bespoke, authored by clinical teams ‘in house’ to support patient care.

Local point of care clinical guidelines are generally available as basic web pages, PDFs or documents [14, 15]. Despite widespread availability and use, accessing clinical guidelines and information can be highly inefficient and restrictive [16, 17]. Clinicians require agile access to clinical guidelines and an efficient delivery method.

At present, there are no ‘standards’ (clear methods, designs, or recommendations) relating to clinical guidelines for use on mobile devices. Previous studies have investigated the delivery of clinical guidelines on mobile devices, but rarely implement well known heuristics for design [18,19,20] and often fail to involve users in each aspect of the design and development process, leading to poor usability. Common issues include focussing on navigational design (likely due to the complexity of the information) while continuing to present the guidelines to users in the original format – not optimised for mobile devices (intended for books or larger screens) or limited formats were the information is significantly reduced [21,22,23].

The research described in this paper, therefore, aimed to investigate and develop efficient methods for presenting and authoring clinical guidelines for use on mobile devices. This has been achieved via following a user-centred design (UCD) approach [24, 25]. UCD has been proven to provide positive outcomes when developing software [21, 24]. By producing clinical guidelines specifically developed for mobile devices, we hope to address many of the issues related to efficiency and ease of access, creating a more usable app.

2 Study Design

The ‘Bedside Clinical Guidelines (BCGs)’ have supported care at the bedside since 1996 and are currently utilised across 14 NHS Trusts throughout the UK, and aim to provide “consistent, evidence-based management of patients in acute hospital settings” [15] for ‘in the moment’ bedside use. The 142 guidelines give information on issues faced daily on the ward with breadth from consent to cardiovascular disease, from venous thrombolism to verification of death. Each guideline has a depth from drug dosage through contacting radiology to discharge policy. They are reviewed annually. The BCGs are currently available as an eBook (a pdf of the print edition) on each participating NHS Trust Intranet [15].

Table 1. Summary of methodology and durations for the software development lifecycle
Full size table

Each stage of the study uses aspects from UCD methodology [24, 25], best practice design analysis and evaluation [18,19,20, 24, 25], and software development methodologies [26]. This included observations on clinical technology use, a survey to understand the technology and apps clinicians use, heuristic evaluations to ensure apps meet basic usability standards before testing; focus groups to gather feedback; System usability scales (SUS) [27] to measure any improvements in usability or any aspects that diminish usability.

These methods were used to inform the design of a prototype application which presents the BCGs on a mobile device. This paper discusses stages 1–11. Stages 12-14 are currently in progress.

Ethical approval was granted by Keele University Research Governance in the Faculty of Natural Sciences (ERP2370) and from Research and Development at the University Hospitals of North Midlands NHS Trust.

3 Results and Analysis

3.1 Observations (Study Stage 2: Requirements)

Observations, conducted following published methods [28, 29], were used to identify if (and how) clinical guidelines were being used. They also aimed to establish any current technology utilisation within the hospital, and the clinician’s interactions with technology. This informed requirements for a smartphone application. The ‘jotting note’ method [30] was adopted for recording observations.

Clinicians across multiple departments at the Royal Stoke University Hospital were observed over three months between May and July 2019. Observations were conducted over several sessions in five wards: Respiratory; General Medicine, Accident and Emergency, Paediatric Accident and Emergency, and Resuscitation. Notes taken during each observation were analysed for consistent themes (Table 2).

Table 2. Key observations
Full size table

One key finding from observing clinicians was that some departments embrace technology in all aspects of clinical practice, and some only for information retrieval. Multi-modal technology use was evident, perhaps due to the lack of availability of some systems on mobile devices.

Clinicians were often interrupted during their interaction with technology, normally by colleagues requiring information or patient-specific questions. In many cases, Clinicians repeated steps within software applications due to time-outs or losing their train of thought. While it was visibly frustrating for the clinicians that they had to re-engage with the technology e.g. login or restart the application, it was accepted that this is how the technology behaves. However, there are detrimental effects e.g. loss of time or frustration associated with such less optimal solutions [31].

It was clear during observations that technology plays a key role in ensuring that clinicians have access to a wide range of up to date knowledge. All clinicians utilise the same technology for patient information retrieval. Hospital devices are used for patient information, but personal devices are often used for knowledge retrieval. Clinicians preferred using smartphone apps over web-based services (via an internet browser) when accessing information on their personal devices. This is likely due to the native features of the application in comparison to the web-based versions. An example of this is the British National Formulary (BNF) application, which utilises core-data storage to allow offline access. This mixed-use of technology within this location has been supported by other studies [7, 8, 33].

In addition, junior clinicians use technology to establish and increase their knowledge base, while senior clinicians use it to affirm their knowledge. Junior clinicians use of smartphone applications and web-based services such as the National Institute for Clinical Excellence (NICE) was greater. Other studies support that junior clinicians utilise technology more than their senior counterparts [32]. The observations highlighted the clinical workflow which any design must consider.

3.2 Survey (Study Stage 2: Requirements)

Survey Background.

Previous studies have investigated mobile device and app usage among both clinical students and clinicians. Table 3 shows a summary of the results from previous studies [7, 8, 33] on device and App usage amongst clinicians, and nursing and medical students, categorised by ‘year published’ and where necessary, study limitations. Smartphone usage has become almost universal between 2012 and 2015 in all groups. While App usage has increased in all groups, this appears to be less in nursing students.

Table 3. Summary of the results from 3 key investigations discussed within this study
Full size table

Survey Aims and Objectives.

The aim was to analyse technology use and identify design patterns and functionality in their preferred mobile apps amongst staff in trusts using BCGs.

A questionnaire was developed to answer the following research questions (RQ):

  • RQ1. Is smartphone ownership consistent across all groups surveyed (Consultants, Mid-Level, Junior and Students)?

  • RQ2. Is there a significant difference in the use of iPhone, Android and Other devices by Clinicians/Students?

  • RQ3. Has smartphone use changed significantly since prior research was conducted; do more or fewer clinicians/students now use smartphones on a regular basis to support their practice?

  • RQ4. Is there any consistency regarding which smartphone applications clinicians and students use?

  • RQ5. Is there a relationship between the clinical role and smartphone app use?

  • RQ6. Does age affect the use of smartphone applications for clinical use?

Survey Design, Distribution and Analysis.

The questionnaire collected data relating to the respondents’ device ownership (RQ 1, 2, 3), their role within the hospital (RQ 1, 2, 3, 4, 5), website use (RQ 4, 5); app use (RQ 4), time in role and local guideline use (RQ 4, 5) and respondents age (RQ 6). Specific App use (e.g. App Name) was collected via an open-ended response (RQ 5, 6). No honorarium was offered in exchange for completing the survey.

The survey was distributed via emails from clinical leads to clinicians in three North West UK NHS Trusts (n = ~1400) and medical students (3rd, 4th and 5th years) at Keele University (n = ~300).

Data analysis comprised of coding, frequency analysis, and cross-tabulation. Tests were completed in IBM SPSS Statistics version 24 for Mac. Where appropriate, the Chi-squared (X2) test was used to compare data with results from alternative sources or when comparing between clinical groups, age groups, and devices. A P-level of <0.05 was considered statistically significant.

Survey Results.

The questionnaire received one hundred and forty-six responses (n = 146). Results were analysed by age and role (Medical students 45% (n = 65), Junior/Mid-Level clinicians 23% (n = 34), and Consultants with 32% (n = 47) (Figs. 2 and 4).

Fig. 1.
figure 1

Age range of respondents (RQ 6)

Full size image

Device ownership and manufacturer (RQs 1, 2 and 3).

Table 4 shows the actual number of clinicians; their role, and their preferred smartphone.

Table 4. Mobile device breakdown for clinical role and device type.
Full size table

Only 2 (1.4%) clinicians did not use a Smartphone for clinical practice, both were consultants between the age of 56 and 65.

iPhone ownership was ~72% (n = 106), while android device ownership was 26% (n = 38) (Fig. 1). All roles demonstrate ownership preference for iPhone over android (p = <0.05). This result is significantly different (p = <0.0001) to general smartphone device ownership research showing general ownership of Android and iPhones to be ~49% for each device [34, 35] and supports previous research [8], which found that 75.6% of doctors own iPhones.

Mobile App Usage (RQs 3, 4 and 5).

Survey participants were asked to identify ‘any apps you use on a regular basis to support you in your role’.

9% (n = 13) do not use smartphone apps to support their role of whom 10 were consultant clinicians, representing 15% of the total number of Consultant respondents. Of the 13, eleven accessed the web-based tools provided by their NHS Trust regularly.

Fig. 2.
figure 2

Percentage of respondents using applications ‘per app’

Full size image
Fig. 3.
figure 3

Application use across clinical roles ‘per app’.

Full size image

Survey participants named a variety of apps (Figs. 2 & 3). The most ‘popular’ were Apps supporting prescribing, BNF App (51%: n = 75 of respondents) and Microguide (28% (n = 41) of respondents). The use was greatest amongst more junior clinicians who prescribe most drugs on a ward.

There was a wide range of other Apps with 47% (n = 69) reporting using an app which was not used by others in the survey. The Apps used related to their roles. These Apps included UpToDate (6 of the 7 users were consultants) for management of a wide spectrum of diseases; calculation tools e.g. MDCalc; clinical tools based on a specific clinical discipline; learning tools, and applications for general administration. ‘Geeky Medics’ was used by 60% (n = 28) of students to support their study and clinical practice.

Fig. 4.
figure 4

Respondents who reported not using apps for clinical practice on a regular basis

Full size image

Figure 4 shows that significantly higher percentages (p = <0.0001) of older clinicians (56 to 60 and 61 to 65) do not use Apps. In comparison, relatively few clinicians below the age of 56 reported ‘None’ for using apps on a regular basis to support their practice.

Discussion of Survey Findings.

Smartphone ownership is consistent across all groups surveyed. The early adoption of iPhone app development for web-based clinical service tools such as Medscape, the BNF, and Microguide (launched 2009 [36], 2012 [37], and 2013 [38] respectively) may have influenced the device bias towards the iPhone. Medscape (as an example) was not launched on Android devices until four years after it was made available on iPhone, potentially allowing brand loyalty and user adoption to grow. There is also an element of ‘peer pressure’ [39], potentially leading to higher adoption rate of a particular manufacturer.

Over half of those surveyed regularly use prescribing Apps (BNF and Microguide). A large number of clinicians use Apps which are not widely used by other clinicians.

The pattern use relates to the role of the clinician (Figs. 2, 3, 4).

The ‘App Store’ rankings for the ‘most mentioned’ apps identified in the survey, reinforce the findings of the survey. At the time of writing, the most mentioned app from the survey (BNF) has an Apple ‘App Store’ ranking of 10th in the UK and a Google ‘Play Store’ ranking of 15th. Removing apps for consumer use (such as MyGP or NHS A&E Wait Times), the BNF would rank 1st. Microguide is the next ‘non-consumer’ ranked app in both stores, placed in the top 50 of both stores.

While ‘App Store’ ranking is not significant to design or usability, ‘App Store’ rankings and reported ‘use’ by clinicians/students correlate.

App Analysis.

It is important to establish design patterns to inform the framework of the prototype, this will allow for consistent usability when clinicians adopt new apps for their practice [40].

The most popular apps reported by clinicians in Figs. 2 and 3 were analysed for consistent design features. The analysis investigated the type of menu, information access type for accessing sections, i.e. lists, and if a search function was available all common features which form the framework of the majority of apps. This analysis then informed the design of the prototype app described in Sect. 4.1.

Table 5. Popular app analysis (Basic)
Full size table

As Table 5 shows, the most popular apps all utilise a ‘List View’, either by category or in an alphabetical format. The apps also utilise a filter-based search function, rather than a full search. Finally, these Apps predominantly adopt a tabbed menu system as opposed to allowing users to quickly access other system features e.g. Settings or alternative views.

3.3 Summary

The results and findings during these study stages (1 & 2) have indicated that clinicians utilised a mixture of technologies and a cross-platform approach will, therefore, need to be considered. App design should allow clinicians to utilise features during clinical workflow, avoiding any design that will require the clinicians to engage for a long period of time e.g. manual calculations. This can be addressed by implementing the design aspects discussed in the App Analysis, integrating features such as a filter for efficiency, and easy access to the features any new app will offer. These findings informed the design and evaluation of a prototype application discussed in the next section.

4 Design and Evaluation

A review of the BCGs shows that the authored word versions already contain different types of information within a formal structure which need remodelling, plus new requirements, identified in Sect. 3, for presentation as an App.

4.1 Prototype Version 1 (Study Stages 3–5)

Technology Selection.

This study (Table 4) supports a cross-platform development approach. Hybrid Application Development methods [41, 42] produce an application which employs web technologies such as HTML, CSS and JavaScript. The hybrid application files are then integrated within the native platform technologies. This produces an application that can be distributed across multiple platforms, whilst still having access to the fundamental technologies offered within the native system. This enables conversion to various platforms, offering a multimodal approach when distributing future versions of the app. Any future development can be integrated into other healthcare systems e.g. electronic health records (EHRs) which are often web-based.

Design Overview.

Results from the review of BCGs in word format, the observation and survey studies inform the design of the initial BCG prototype application.

Figure 5 shows the initial prototype design of the application. Note the menu button in the top right, implemented during this prototype stage as the app functions were limited and did not require a ‘tabbed’ menu as the survey and app analysis suggested. Several design aspects were considered, these included how Warnings/Alerts were presented; Filtering/Highlights search text; Algorithms for diagnosis; Diagnostic Aids; Calculations; Evidence for each guideline; and the main menu to access individual guidelines.

A heuristic evaluation [18,19,20] of the prototype refined several aspects, these included: Guideline sections requiring more distinction; warnings required more prominent colours; sections and headers also required more distinction; guideline information was not presented similarly to what clinicians were used to.

Fig. 5.
figure 5

The initial prototype of the BCG app.

Full size image

A second prototype was then developed shown in Fig. 6. The sections were more distinguishable, and colours were utilised to ensure menu and guideline sections were more obvious to the user. Warnings were made more prominent by utilising red for the background and text.

Fig. 6.
figure 6

The second prototype of the BCG app after a basic heuristic evaluation

Full size image

Flowchart Prototype.

The BCGs contain a number of decision algorithms for use during clinical practice. Figure 7 shows a standard decision algorithm to determine if a patient should be referred to the on-call respiratory physiotherapist. Decision algorithms are key components of guidelines and due to their size and complexity, pose a usability issue (highlighted in Fig. 7) when designing for mobile.

Fig. 7.
figure 7

Example of a decision algorithm

Full size image

Figure 8 shows the apps prototype decision algorithm designed for displaying on a smartphone. The prototype version was developed using JavaScript, HTML 5 and CSS3. The design displays the selection or path the clinician has followed, and therefore limits the algorithm to only the required information.

Fig. 8.
figure 8

Designs for the smartphone algorithm, right was an iteration of the initial design (left).

Full size image

Focus Group Evaluation of Prototype Version 1 (Study Stage 6)

The prototype in Figs. 6 and 8 was demonstrated to clinicians in a focus group of 21 clinicians in a single session (student, junior and senior) at the Royal Stoke University Hospital. The main aim being to obtain functionality and design feedback for the prototype application from target users. The focus group was conducted utilising open discussion [43, 44]. These open discussion sessions were audio-recorded and transcribed. The transcripts were then analysed using thematic analysis [45].

It was apparent during the initial focus group that another method would have to be adopted for large group feedback. Sessions were time-sensitive (scheduling constraints inherent in clinical roles) and individual sessions or smaller groups, though preferred, were not possible. Idea writing [46, 47] was therefore adopted for the second focus group of 17 clinicians, which allowed all participants to contribute in a structured manner within the time constraints. During this session, clinicians interacted with a prototype of the application and were asked to feedback on each aspect of the design which was presented as a ‘concept’. Although this limited open discussion (by design), it allowed for more specific feedback regarding the design of the BCG app.

Table 6 shows an example of feedback provided by clinicians during the idea writing session.

Table 6. Example of outcomes from an idea writing session conducted with clinicians
Full size table

The feedback from both focus groups was analysed for consistent themes. The key themes identified from the focus groups are that clinicians appreciate the clean, clear layouts that do not impede workflow. An example of this is the flowchart design within the prototype application. Clinicians provided positive feedback regarding the prototype Q&A style format (Fig. 8), but also suggested retaining the original flowchart design to give a gestalt view. Clinician’s feedback also suggested the use of acronyms (e.g. PE for Pulmonary Embolism) when searching or filtering guidelines. This is in contrast to standard usability guidelines [48, 49] and reflects the challenges faced when designing for experts. Clinicians suggested that warnings require a hierarchy based on their severity with the use of more noticeable colours

Thus, changes that would be required in the next iteration of the prototype BCG app:

  1. 1.

    Decision algorithms to be displayed in-line with the guideline information.

  2. 2.

    The original ‘flowchart’ decision algorithm is provided.

  3. 3.

    Acronym use is prevalent in medicine, but not all clinicians have knowledge of acronyms. Methods to address both experts and novices should be adopted.

  4. 4.

    Guideline decision tools such as calculations should be automated.

  5. 5.

    Warnings should be clearer and adopt better salience for the user.

  6. 6.

    Guideline length would need reducing.

Usability Testing of Initial Prototype (Study Stage 8)

The System Usability Scale (SUS) [27] was used to establish the usability level of the prototype application (Version 1 created during study stage 5) from the clinicians’ viewpoint. It also provided a baseline to measure future changes in the design and how they impact the usability. During 2 sessions, 26 clinicians were asked to complete information retrieval scenarios, developed in collaboration with senior clinicians at the Royal Stoke University (example shown in Fig. 9) and then complete the SUS.

Fig. 9.
figure 9

Example information retrieval scenario used in testing.

Full size image

The app was shown to have a high usability score, with an overall score of 81 out of 100 (calculated utilising the methods described in [27]). Question 5 ‘how integrated features of the system are’ showed the widest gap between ideal and current usability scores. This agrees with the focus groups.

This SUS score indicates an initial high level of usability; however, the focus groups identified several specific areas of improvement which are described in the following section.

4.2 Prototype Version 2 (Study Stage 10)

Design Overview.

It was evident through feedback from the Focus Groups that the guideline length would need to be reduced. Research agrees with this feedback, as it helps to avoid unnecessary scrolling and prevents potential impact on clinical workflow, especially in regard to memorability and usability [50]. Design aspects including accordions were utilised to support this. Design patterns such as accordions [51] were utilised to support this (Fig. 10) which greatly reduced the length of some guidelines.

Fig. 10.
figure 10

Left image shows the closed format of the BCG accordion, right image shows how the accordion displays the contained information.

Full size image

The BCGs contain tables for easy presentation in the book format, however these can be problematic on mobile devices due to constraints inherent in their design and size [52]. Figure 11 shows a guideline table converted to a diagnostic tool. The table requires clinicians to manually complete calculations. The BCG app version calculates the outcome and provides clinicians with clear and precise recommendations.

Fig. 11.
figure 11

Left image shows the original table format of the BCG classification tool (Dukes classification for infective endocarditis), the right image shows the BCG App version which allows users to select criteria and display a single recommendation.

Full size image

Acronyms are not understood by some clinical staff [53, 54]. Figure 12 shows acronyms displayed on popovers to potentially reduce errors due to misunderstandings [53, 54].

Fig. 12.
figure 12

Concept for displaying acronym details.

Full size image

Clinical Guideline Warnings.

The BCG Medical Guidelines contain over three-hundred warnings in a black box design. The focus groups, expert clinicians and authors were consulted on the design of a simple method of displaying a reduced number of warnings to avoid alert fatigue [55,56,57]. Figure 13 shows the original and new warning designs. The use of colour and icons improves the impact of the warnings [58].

Fig. 13.
figure 13

Top images show the original warnings in the acute heart failure guideline. Bottom images show the BCG App versions, one with less text.

Full size image

4.3 Summary

The user feedback has led to the design of the BCG App. Usability testing has shown promising results. Focus group participants described the app as “a much more efficient approach to presenting this information”, “clear and easy to navigate”, “easy to understand”, “clean” and “Familiar”. Usability testing using cognitive walkthroughs will inform further improvements before the app is used in live pilot testing.

5 Conclusion

This study has reaffirmed that smartphone ownership is consistent across all clinical roles (with iPhone ownership being dominant). Medical app usage in a clinical setting is becoming ubiquitous. This has implications for not only Doctors and app developers but also for Hospitals, Trusts and their patients as the majority of the applications reported in this study were not officially authorised by the NHS.

It is clear from the observations, survey and app analysis, there is a need to consider the wide variety of tools needed by a clinician when developing applications. Clinicians use several tools which would benefit from being integrated into a simple, easy to use system, which presents the information in line with elements such as calculators or decision algorithms. Mobile Medical apps like this require ease of use at point of care and integration into clinical workflow.

There is a real need for further investigation in this area and for doctors and app developers to work more closely to align needs and to develop standards. Applying a user centred design to the information presented to clinicians can yield improvements to usability results and this research shows that co-designing applications of this nature help to maintain accuracy and produce a usable system. When designing for mobile, it is important to design not only for the inherent strengths and weaknesses of the device but also for the context of use. Designing for “in the moment use” in a Hospital means designing for interruption and designing for users with specific expertise means including functionality that is counter-intuitive to standard design guidelines e.g. using acronyms. Reflecting on the use of UCD itself in this domain, there are severe constraints related to limited access to clinicians and so traditional methods have required adaption. Future work, therefore, will consider the use of implicit feedback (usage logs) to gather feedback to inform user modelling and interface adaptation.

Study Limitations.

Although this survey was conducted across multiple locations, it was limited geographically (NW England) and to single locations within the trust. Increasing the study’s reach; having multiple sites in multiple trusts, would enable a thorough analysis across each trust and enable comparisons at both single-site and trust level. This survey limitation may be affected by recommended apps dominating within that area. It may also be affected by bias, clinicians stating what they ‘should’ say compared what they actually use for clinical practice. Focus groups based on local information and not from further trusts, although focus groups at further NHS Trusts are planned.

Funding.

This study was funded in part by the Keele University Acorn Fund and the University Hospital North Midlands NHS Trust Charity.