Keywords

1 Introduction

Telehealth initiatives have often focused on connecting patients with medical providers in order to improve access to specialized knowledge and to provide medical services to remote users [1,2,3]. A smaller portion have been geared at connecting physicians with other physicians or particular medical specialists (e.g., [4]). This current study focuses on such physician-physician consultation, specifically in the area of developmental-behavioral pediatrics.

Developmental-behavioral pediatricians (DBP) are pediatricians with subspecialty training and experience that allows them to consider, in their assessments and treatments, the medical and psychosocial aspects of children’s and adolescents’ developmental and behavioral problems, such as autism spectrum disorders (e.g., autism disorder, Asperger’s syndrome, Rett syndrome), attention and behavioral disorders (e.g., ADHD, depression, anxiety disorder) and learning disorders. DBPs work closely with family members and advocate for their patients by working with schools, preschools, and other agents, such as social workers or relevant agencies involved with developmental care and education [5]. This pediatric subspecialty is marked by substantial personnel shortages [6], turning primary care physicians (PCPs) into key players in the management of autism spectrum disorders. For primary care physicians caring for patients with Autism Spectrum Disorders (ASD), there is a critical need for just-in-time guidance by DBP specialists, suggesting telehealth approaches as one possible solution to alleviate access issues.

Our user experience evaluation team (Michigan State University (MSU) Usability/Accessibility Research and Consulting) consulted on a research project with the Department of Pediatrics and Human Development (in the College of Human Medicine at MSU) to conduct a usability evaluation that assessed the potential of a teleconsultation process to improve communication between PCPs and a DBP with expertise in ASDs. The research focused on understanding the extent to which autism-related information needs of PCPs can be met by consulting an autism specialist or DBP at a distance, and given the complexity of cases involving ASD patients, allow for high quality communication between primary care providers and the specialist and provide relevant and actionable support to the PCP.

While the details of virtual provider-specialist consultations can vary based on local contexts, for the purposes of this research, teleconsultations consisted of pre-arranged, approximately 15-minute long interactions via HIPAA-compliant Zoom teleconferencing software and were offered to PCP participants wishing to consult a DBP with expertise in the diagnosis and management/treatment of ASD. Our pilot study to determine the usability of teleconsultations is one element of a multifaceted initiative to enhance support services for Michigan children with ASD and their families.

2 Methods

Our usability study concentrated on obtaining performance, observational, and subjective satisfaction data from representative users performing typical tasks using the teleconsultation process. In addition to this investigation of user-system interaction, user-user interaction (i.e., interaction between a PCP and a DBP) was analyzed. While effectiveness, efficiency, and satisfaction remain relevant parameters, this combination introduces other elements of interest, such as communication quality and interactivity between users.

We examined the extent to which actual PCPs consulting with the DBP reported feeling that their concerns and questions were being heard, understood, and adequately responded to by the remotely located specialist to allow for the provision of quality care to the patient. Given the pilot function of the current version of the process, this study also aimed to understand PCPs’ perceived value of the teleconsultation process and their needs, preferences, perceived obstacles, and concerns. Finally, the study aimed to capture observations regarding the communicative aspects of teleconferencing in order to make recommendations for enhancements of this process.

2.1 Participants

Six practicing primary care physicians (PCPs) were recruited via an email requesting their participation in a 1-hour session including a “mock teleconsultation and an exit interview” geared at “testing the feasibility of mobile consultations.” Participants did not receive any incentive or compensation for their participation. All six participants were pediatricians practicing in either East Lansing, Lansing, Flint, or Grand Rapids. Participants ranged in age from 38 to 62 and had between 9 to 33 years of practice. They reported seeing autistic children in their practice between “less than once per week” (but no less than once per month) to “2–5 times per week.” All of them used electronic health records (Centricity, EPIC, CPS 11, New Gen, or eMDs) and reported using a desktop computer and the Internet daily. In contrast, exposure to videoconferencing technology varied widely between “2–5 times per week” and “less than once per month.” With one exception, participants approached the teleconsultation session with interest in teleconferencing-based access to a developmental behavioral pediatrician, and had positive expectations regarding effects of such access on their knowledge and ability to provide quality care.

A developmental-behavioral pediatrician served as the consultant for the sessions. This DBP, who practices medicine in Flint, Michigan and specializes in the diagnosis and management of autism, had prior experience with an electronic health record, used a laptop and the Internet daily, and had significant experience with videoconferencing technology.

2.2 Procedure and Metrics

We conducted 60-minute sessions with six users (primary care physicians) and a DBP to determine the effectiveness of the teleconsultation process. PCPs were recruited from the surrounding areas of southern Michigan by the MSU Department of Pediatrics and Human Development. For two of the primary care physicians, the timing of the scheduled consultations enabled a member of the department to be present to assist with setting up Zoom teleconferencing software in person. The remaining PCPs were instructed on how to set up Zoom through emailed instructions.

After the PCP, DBP, and the session moderator met virtually in the designated 3-party Zoom session, the moderator greeted the participant, made introductions, and read a description of the study. For each teleconsultation task, participants were asked to first describe a case scenario, which had been provided to them at random and in advance of the session. The case scenarios consisted of a set of case notes describing a young patient with ASD-relevant symptoms and his or her medical, social, family, and medication history, results of previous assessments, and a list of possible questions. These cases were developed by a pediatrician with expertise in autism spectrum disorders in conjunction with Usability/Accessibility Research and Consulting.

After providing the overview of a case, PCPs were asked to pose specific questions from their scenario to the DBP to guide her input in the case (and the DBP did not have exposure to the cases or questions in advance). PCPs were encouraged to flesh out details as needed, especially if the specialist were to ask follow-up or clarification questions that were not answered within the stimulus materials but that they would likely be able to answer if the child in question were an actual patient of theirs.

The DBP responded with thoughts on additional evaluations or, if sufficient diagnostic clarity had been established previously, prioritized suggestions of treatment options. If sufficient diagnostic clarity had not yet been established previously, she occasionally provided hypothetical treatment options contingent on the outcomes of the additional evaluations, as well as other relevant advice on navigating insurance constraints and identifying and connecting with locally available resources.

To increase the naturalness of the consultation, the moderator turned off their video component (and still observed and listened to the consultation) while the PCP consulted with the DBP, and re-enabled their video portion only after the conclusion of each 10–15 min long consultation in order to administer the post-task satisfaction survey while the specialist temporarily left the meeting to fill out her own post-task satisfaction form. For this survey, the PCPs and the DBP were asked to rate the effectiveness of the consultation and to identify the main questions(s) that had been asked during the consultation to allow for comparison across the two parties.

After the two rounds of case consultation and post-task questionnaires, a post-study questionnaire (consisting of 25 items) was administered to participants to assess the effectiveness of the teleconsultation process, physicians’ satisfaction with the technical aspects of the process, the perceived quality of the communication that occurred, and professional usefulness of the process. The DBP completed only the communication quality portion of this questionnaire to examine the user-user interaction, and she provided qualitative feedback on the overall process.

The System Usability Scale (SUS) was used for the first portion of the post-study questionnaire with PCPs rating their level of agreement to ten statements (e.g., I thought the teleconsultation process was easy to use) to assess the usability of the overall process. The SUS, created by John Brooke, is a quick and reliable tool for measuring usability that can be used for small sample sizes [7]. Participant responses are calculated and averaged to find the overall SUS score (which ranges between 0–100).

To assess the perceived quality of the video-based interaction within the teleconsultation, participants were asked to complete a portion of the Communication Quality Questionnaire [8] as part of the post-study questionnaire. This measures the degree to which interactions are smooth, efficient, yet personal and overall satisfying to the two parties involved and has been used in both face-to-face and computer-mediated contexts (e.g., [9]). This portion of the questionnaire consisted of ten five-point semantic differential items, such as “Please rate the quality of the communication that occurred in the teleconsultation process on the following dimensions: In-depth (5)–Superficial (1).”

Finally, PCPs were asked to rate their level of agreement to statements aimed at gathering their attitudes towards the teleconsultation process (e.g., I felt comfortable using the teleconsultation process), and open-ended questions were asked verbally to gain feedback on how the teleconsultation process could be enhanced (e.g., Would you like the option to send notes, video, etc. before and/or during the teleconsultation?).

3 Results

At the conclusion of each of the two consultation tasks, the PCPs and the DBP were asked to rate their level of agreement to the following statement on a scale of 1–5 (Strongly Disagree to Strongly Agree): “The consultation provided answers to the question asked and would assist me (the PCP) in providing care to the patient.” Generally, the DBP tended to rate the consultations more critically (averaging 3.7) than the participating PCPs (averaging 4.8). In addition, the PCPs and the DBP were asked separately what the main question(s) was that was posed to the specialist, and comparison showed that each user understood the focus of each consultation in these sessions (which is an important factor for this process).

The average scores from the post-study questionnaire are included in Table 1. The average SUS score for the teleconsultation process was 77.5 for the PCPs, which is in the acceptable range (above 70 is considered acceptable). Two participants had lower scores that fall in the marginally acceptable range (between 50–70), and these participants in particular had major difficulties when attempting to set up Zoom and when entering the teleconsultation meeting, which likely contributed to the lower ratings they gave for the SUS and the statements related to their attitudes toward the consultation process; in contrast, they gave high post-task ratings and high ratings for the Communication Quality Scale.

Table 1. Quantitative post-study results: System Usability Scale (SUS), Communication Quality (CQ), and attitudes toward the teleconsultation process
Full size table

The average Communication Quality score for the teleconsultation process was 4.7 for the PCPs, and 4.2 for the DBP. In the scale’s original validation work [8], respondents rated their average satisfaction with routine interactions with others as about 7 on the 9 point scales (9 being high, 1 being low), which is equivalent to a score of 4 on the 5-point scale used in this study. As such, especially the PCPs’ scores are on the high end. Furthermore, one particular item that tended to lower the score was the “Formal (1)–Informal (5)” item. Given the professional context of these interactions, some formality is not necessarily as indicative of lack of immediacy/warmth as it is in the context of everyday interactions on which the scale is designed. Therefore, the positive ratings can be seen as conservative estimates of the PCPs’ perceived communication quality. On the other hand, a concern about social desirability and politeness in responding may operate more strongly on the PCPs’ side than on the side of the DBP as participant-researcher, which may explain the DBP’s slightly less positive ratings.

In addition to the Communication Quality questionnaire, observations regarding the communication and interactivity of participants and the overall process setup were examined to determine considerations for a successful teleconsultation process, which are overviewed in the next section. These observations (based on previous work such as [3, 10]) included both nonverbal and verbal communicative behaviors (including facial expression, eye contact, gestures, and interactivity in terms of conversational back-and-forth, clarification questions from both parties, mirroring, and expressions of mutual understanding/engagement/rapport); the situational contexts in which doctors used the technology (including work interruptions, phone calls, settings); and the performance of the specific teleconferencing software used, different ways in which the relevant hardware (monitor, keyboard, webcam, mic, speakers) was set up across users, and challenges related to its use. We found a variety of communicative and interaction behaviors, some of which is more conducive to the consultation’s quality. For example, a pattern of presenting the case description, asking a question, listening to the advice, asking another question(s) and listening, and finally thanking the DBP could be noted in participants that tended to be less expressive during the consultation (i.e., using less back channeling such as nodding). With other participants, we observed repeated turn-taking between both parties that became more conversational with back channeling throughout (e.g., nodding, smiling, saying “okay,” “makes sense,” etc.) to indicate understanding. Several participants also mentioned that this type of session was not necessarily typical because they would likely be multi-tasking during a consultation and would not be entirely focused on the conversation. We also noted variation in terms of participants’ positioning in relation to their camera and screen, with some appearing back-lit with hard to discern facial expressions.

Process enhancements are discussed in the next section, but overall the teleconsultations were perceived as valuable by the PCPs because of the following:

  • validating the PCPs’ ideas for patient plans and strengthening their confidence in talking with families;

  • providing new ideas and redirecting their thinking;

  • connecting PCPs with additional resources while considering their location;

  • and providing a high level of specificity and sequential ordering of advice.

4 Tips for Enhancing Teleconsultation Process

Based on the overall results and feedback, recommendations for enhancements of the teleconsultation process include the following areas:

  • Support PCPs in setup of video conferencing tool and include the option to use a telephone for flexibility and simplicity

  • Establish a procedure for sharing patient information and dealing with interruptions during a teleconsultation

  • Foster communication quality and interaction in a mediated environment

  • Provide a summary of the teleconsultation through use of a template

  • Consider additional implementation factors including HIPAA compliance, billing and costs, scheduling logistics, and the participation of third parties

Flowcharts to illustrate examples of the setup process before a teleconsultation (Fig. 1) and the teleconsultation process using a video conferencing tool (Fig. 2) are also included.

Fig. 1.
figure 1

Example of setup process before first teleconsultation using video conferencing tool

Full size image
Fig. 2.
figure 2

Example of teleconsultation process (Note: Telephone would be used instead if there was an insufficient connection for video conferencing.)

Full size image

4.1 Support PCPs in Setup of Video Conferencing Tool and Include the Option to Use a Telephone for Flexibility and Simplicity

Three of the four participants who did not have in-person technical support for downloading and using the Zoom software for the first time encountered difficulties, and one of the two participants who did receive support stated a lack of confidence in his ability to tackle unfamiliar computer-related tasks. Therefore, successful adoption of offered teleconsultations is likely to hinge on easing these initial technical challenges for users, although participants also thought the process would become much easier after they became familiar.

Possibly more effective than very specific set up instructions for Zoom would be efforts to provide a contact person who can walk interested PCPs through the process and test the teleconferencing tool with them for the first or second time. Similarly, participating institutions could eliminate the downloading and installation part of the process by having IT support preinstall the program on PCPs’ preferred computers for this purpose.

To maximize the potential of video conferencing, PCPs need digestible information or just-in-time support for checking and meeting computer requirements including the webcam and microphone, connection, as well as Zoom-specific functionality such as screen sharing or the chat function for the sharing of links. Additionally, it seems ideal to pick a platform like Zoom and not change to a different tool, to the extent possible, so that users do not have to go through the initial learning curve repeatedly.

The option to consult via telephone in addition to the option to use videoconferencing is also needed to meet the needs of the users by allowing for flexibility and simplicity (e.g., not requiring Zoom meetings only). Although most of the PCPs and the DBP felt that using videoconferencing for consultations was ideal, most also mentioned that this medium of communication may not always be feasible or practical when they have limited time available to schedule and prepare for a consultation and likely will be multitasking. After having some difficulty with the Zoom setup or mentioning a lack of computer usage, the option to use telephone in some instances was preferred. For example, an initial video consultation could help establish the relationship and rapport between the PCP and DBP, and then the option to schedule telephone consultations could be available as needed.

4.2 Establish a Procedure for Sharing Patient Information and Dealing with Interruptions During a Teleconsultation

A procedure needs to be established to allow PCPs to share patient information with the DBP before, during, and after consultations (e.g., if a follow-up is scheduled). For example, PCPs mentioned the desire for this option and a variety of methods for sharing information, including: using fax or email to send patient information and reports from other providers (e.g., from psychologist, etc.), providing access to a patient’s entire chart, holding up textual/visual information to the camera while videoconferencing, and sharing video of the patient (e.g., live or pre-recorded). However, determinations are needed to ensure any methods used are HIPAA-compliant, as well as what types of information will be useful for the DBP to help avoid overwhelming amounts of content being shared.

A template and/or checklist is needed which indicates the key types of information that are useful for the DBP to have before or during a consultation (e.g., any previous evaluations/screenings/referrals and the results, past medications used, developmental milestones and behaviors, etc.). Instructions could indicate to the PCP that they should be prepared to present these types of patient information and also what they feel is important, or to send the template to the DBP before with the requested content (e.g., provide a means for the PCP to organize the patient history and their thoughts). In addition, a procedure is then needed to establish how the patient information will be shared with the DBP to ensure privacy, and (if possible) how additional types of content can be shared, such as videos.

A few important ground rules should be established in advance, such as how to deal with common interruptions during a teleconsultation, especially if potentially private information may be mentioned during an incoming phone call or when a medical staff member stops by in-person. Users should therefore be instructed on how to briefly mute one’s microphone for the duration of such an interruption. PCPs and the DBP should consider whether or not recording of a session or parts thereof is permitted and what to do if the Zoom connection is lost or technical disruptions interfere with the timeline. This could be as simple as sharing a phone number for back-up communication.

4.3 Foster Communication Quality and Interaction in a Mediated Environment

As the party more familiar with the teleconferencing model, the DBP specialist should continue the helpful practice of using orienting statements that address the fact that the approach may be initially unfamiliar but that interactivity is possible and desirable.

Given that the DBP frequently reported not being quite clear on whether her advice was sufficient or addressed the PCP’s needs exactly (as observed in her post-task ratings and comments), she should continue to encourage the PCP to interrupt as needed, to ask follow-up and clarification questions, and to not feel any need to politely hold back or “save the expert’s face” when something is not quite clear to them. Continuing the conversational tone of the meeting and monitoring non-verbal expressions can provide clues on the level of understanding reached as well, although consulting DBPs will have to be aware that some participants are unlikely to be quite as expressive and less prone to back channeling as others in this regard. And even though the video component likely reduces multitasking, in the hectic reality of the medical field some of this may still occur along with more substantial interruptions by third parties.

Other orienting statements could include whether or not notes or a session report will be provided, whether or not parties are expected to take notes (and hence will look occupied either with their screen or a notepad at times), and whether multiple monitors are being used (which would explain why the other party appears to not be looking at the participant).

To improve the richness of the mediated communication, participants can avoid back lighting which makes facial expressions hard to see and position their webcam near the top of their screen to improve eye contact and turn-taking regulation. They can position themselves close enough to the computer to ensure sufficient audio quality but not so close that their gestures and body language are cut out from view. Additionally, if audio quality was insufficient in a test run, they can use an external microphone/headphones to minimize the detrimental effects that having to strain to hear or repeatedly talking over each other can have on the flow of the communication and the establishing of rapport.

4.4 Provide a Summary of the Teleconsultation Through Use of a Template

A summary report or overview of the consultation would be very useful for PCPs to ensure they have a record of the advice and plan provided by the specialist. This report would likely become part of a patient’s record, and therefore a template for this type of summary is needed to ensure consistent reporting and to establish a procedure for the type of information that should be included (and what should not be included for this record). A template would also ensure that the DBP is aware of the types of the information they are expected to summarize for the PCP, allowing for productive and efficient note-taking during the consultation. A “toolkit” should also be compiled that includes information on possible resources, evaluation tools, information packets, diagnostic requirements for insurance, and so on, providing the DBP with an easily accessible pool of information to pull from as needed to include with a report for a PCP.

4.5 Consider Additional Implementation Factors Including HIPAA Compliance, Billing and Costs, Scheduling Logistics, and the Participation of Third Parties

HIPAA Compliance Related to Information Sharing.

Determinations need to be made on how information can be shared with the DBP for teleconsultations to ensure HIPAA compliance. For example, the overall teleconsultation procedure needs to include instructions on whether a release from a patient or the patient’s parents is required (and for which types of information), or whether these teleconsultations should avoid the sharing of identifying information and be considered part of necessary medical care (e.g., similar to a referral and thereby not require a release from a patient). Privacy and security requirements also need to be considered when establishing a procedure for sharing patient information with the DBP. Additionally, the locations that are available for PCPs and the DBP to communicate should be considered, and if a consultation occurs in a (semi-)public place, whether and how this could affect compliance concerns.

Costs and Preference of PCPs to Bill for Their Time.

Billing procedures should be considered, such as: Can PCPs bill for consultation time and if yes, how so? Or would consultations be considered a professional courtesy in their current conception? Would the documentation that several PCPs requested contribute to turning the consultations into a more official/formal and therefore billable event?

Scheduling Logistics.

Scheduling issues include two of the main concerns mentioned by PCPs: How to find time for these additional consultations and how to minimize possible interferences? For example, the times that the DBP would be available need to be determined (e.g. certain days per month, during what times, after hours or day time only). Billable consultations may alleviate the concern about finding time in the day, and scheduled times rather than random call backs would address concerns about interruptions. Times determined in advance may or may not allow the PCP to schedule certain patients strategically around that time as well. Having a strategy for handling various levels of consultation requests may be beneficial, and considerations should be made regarding possible support for follow-up communication (e.g., consider how consultation can be arranged on PCP side). Existing consulting models (e.g. psychiatry) may provide insights. If additional DBPs are added to the pool of consulting experts, scheduling should take into account the PCPs’ preference for continuity by providing access to the same expert across teleconsultations.

Including Third Parties.

Suggestions by the DBP to connect the PCP with a familiar social worker who would assist in navigating local resources for the patient were repeatedly greeted with strong interest. This could take the shape of three-way video conferencing meetings unless the DBP and the social worker are collocated during some of the set-aside consultation times. Similarly, it may or may not be possible to involve the patient and/or his or her parents directly in interactions with the DBP, as suggested by another PCP. Therefore, additional scheduling/logistical strategies related to such third-party involvement should be considered.

5 Conclusion

Overall, the teleconsultation process was well received by both the PCPs and DBP in this study. The high post-task and post-study ratings (including System Usability Scale, Communication Quality, and attitudes toward the teleconsultation process scores), as well as the positive feedback from participants, indicated the effectiveness of the teleconsultations and satisfaction with this process. In this pilot study, efficiency was also observed through consultations being completed within the expected time range of 15 min or less. Based on qualitative feedback from the participants, the teleconsultations:

  • validated the PCPs’ ideas for patient plans and strengthened their confidence in talking with families;

  • provided new ideas and redirected their thinking;

  • connected PCPs with additional resources while considering their location;

  • and provided a high level of specificity and sequential ordering of advice.

PCPs especially expressed their interest in the opportunity to connect with an ASD specialist in a timely and efficient manner, and most felt that the video element of the consultation was ideal, and that the teleconsultations enhanced clarity and understanding, attention, engagement, and collegiality.

Enhancement considerations offered above can be used to determine clear procedures and implementation factors and to foster communication quality and interaction for teleconsultations. More broadly, these insights can be used for mediating communication and improving the process of video conferencing between professionals.