Keywords

1 Introduction

The world is steadily growing more urban than rural, yet the gulf between the quality and availability of medical care between the two remains wide. A major contributor is the difficulty in recruiting and retaining rural health workers. While the imbalance of health workforce is common to almost all countries, it has the most severe impact in low-income countries (LICs) [1]. For example, the urban to rural disparity of practicing physicians per capita in Nepal has been estimated at more than 40 to 1 [2]. There is no one single explanation towards this disparity, but one key element is the isolation and lack of professional development opportunities, such as Continuing Medical Education (CME) for rural physicians.

CME is defined as educational activities that serve to maintain, develop, or increase the knowledge, skills, and performance of medical practitioners in the medical field [3]. The ultimate goal of CME is to improve the health of individuals through services provided by the medical practitioners. CME is not only essential to improve physician performance and patient health outcomes, but also critical to maintaining a sense of professional pride, distinction, and identity [4]. Studies have shown effectiveness of CME in areas such as the acquisition and retention of medical knowledge, improving attitudes, skills, physician behaviors and clinical outcomes [5]. However, CME represents an overwhelming challenge in rural areas of LICs due to the complexity of socioeconomic and infrastructural problems, including limited access to context-relevant CME content from conferences, textbooks, journals and the internet, lack of mandated CME standards, as well as financial and technical challenges. There exist few useful tools to effectively bridge the paucity of resources and access to CME, and little knowledge about how CME might be used to improve rural medical care and population health.

This research serves two objectives. First, we build, deploy, and evaluate an ICT4D (Information and Communication Technology for Development) solution called CMES (CME on a Stick) for the delivery and sharing of affordable and high-quality CME content for rural medical practitioners, focusing on Nepal as a test-bed. We believe that a free and open platform for CME knowledge sharing and distribution in LICs can catalyze significant reductions in urban-rural health disparities. It is important to note that ICT solutions such as Internet-in-a-Box and Raspberry Pi have been used to deliver educational contents to offline communities. However, none addresses the unique characteristics of effective CME in LICs, such as providing interactive and multi-modal methods of learning, gaining administrative support, offering incentives for practice change, and conducting needs assessment of health professionals.

Second, in the process of building CME capacity among rural medical practitioners, we continue the application and refinement of the Citizen-centric Capacity Development (CCD) framework for ICT4D [6]. As an inter-disciplinary research domain, ICT4D integrates computer science, information systems, and development studies [7]. While ICT includes the full range of computer hardware, software, and telecommunication facilities, ICT4D research is envisioned as an enabling tool with the capability to improve quality of life, empowerment, and economic development for the communities where it is introduced. Despite the huge investment from government, non-government organizations and private sectors in ICT4D projects, the impact of ICT in LICs remains minimal [8]. The struggle of linking ICT deployment and achievement of development goals is usually a result of focusing more on technology and neglecting the complex social, economic, and political factors in LICs [9]. Although literature calls for exploring a broader context for ICT4D project [8, 10], existing research in ICT4D mostly focuses on either technical solutionsor impact evaluation. Because the broader development context often influences the design choices of the ICT solution within which it is set [11], we believe that it should be considered throughout the ICT solution design process rather than explored in isolation. The purpose of CCD framework is to assist the design, implementation, and evaluation of ICT4D project with the consideration of the unique and complex context for development in resource constrained communities (e.g. LICs, rural communities).

The rest of the paper is organized as follows. First, we review CME challenges in LICs Sect. 2, followed by a description of proposed CCD framework and project background in Sects. 3 and 4. Section 5 discusses the CMES design process for CME capacity development, and Sect. 6 concludes the paper by examining long term project goals and future research directions.

2 CME and Its Challenges in LICs

The instructional design of CME activities draws insights from different academic disciplines such as adult learning, practice-based learning, continuing professional education, organizational change, development and behavior, and health services research [12]. Most of the CME methods and activities are based on theories on adult learning, such as lifelong learning, theories of motivation, self-directed learning, reflection, and adult learning [13]. Different CME methods include conferences and workshops, printed or recorded materials, mentoring and opinion leaders, clinical practice guidelines, interactive education, case-based training, audit and peer group discussions, educational outreach visits, reminders, and online education [5, 14]. Among these, attending conferences and reading printed materials are CME activities most commonly undertaken by medical practitioners.

While CME is identified as a key element for maintaining good quality clinical care and a priority for health care providers in many countries [15], the effectiveness of CME remains debatable. A recent synthesis of systematic reviews [4] on CME effectiveness concluded positive impact on physician performance and patient health outcomes. In addition, literature identifies key attributes of successful CME: (1) use multiple media and methods of learning; (2) provide adequate time to digest and incorporate knowledge; (3) incorporate needs assessments to ensure that the activity is controlled by and meets the needs of health professionals; (4) be interactive; (5) administrative support, and (6) provide policy incentives for practice change [4, 16]. In our solution design, we take into consideration the first four attributes, while fully aware that attributes five and six represent social and political aspects of CME and may not be directly addressed by our ICT-enabled solutions.

Additionally, literature highlights many unique challenges to CME in LICs. First, in many developed countries such as in Europe and America, CME is mandatory and its quality and participation are often warranted by a national board. For example, the Accreditation Council for Continuing Medical Education (ACCME) enforces standards and quality of CME, and requires recertification of medical professionals to maintain their licenses. On the contrary, in many LICs, there is no external enforcement of CME activities. As a result, there is no incentive for CME, or sanction for non-participation, except self-motivation. Other challenges include lack of time to participate due to pressures of work, lack of funded study leaves, insufficient financial support for external CME activities, limited access to CME content especially for rural medical professionals, lack of basic ICT knowledge required for accessing information online, and shortage of CME activities tailored towards individual needs [17]. These challenges are even greater in rural areas. For example, most rural medical professionals do not have access to up-to-date CME content due to unavailable or unreliable internet access. Medical practitioners who work in rural hospital and clinics generally prefer local access to CME to reduce travel time and avoid absence from work. When many have access to computers, they would need support to use it for CME [2].

Considerable effort and research have focused on promoting CME content sharing in LICs. Research4lifeFootnote 1 provides free or low cost online access of scientific research content for LICs. One of the Research4life programs, the Health InterNetwork Access to Research Initiatives (HINARI)Footnote 2 offers free access to up-to-date biomedical and health literature for health and medical institutions in LICs. While it postulates a way of supporting online access to CME content, it does not address the issues of limited internet connectivity and lack of basic ICT knowledge among medical professionals in LICs. For example, there are often difficulties in logging into HINARI and many participating organizations experience problems of organizing password distributions [18] even when there is internet connectivity. Our CMES design not only presents a novel solution towards offline CME knowledge sharing and distribution, but also ensures CME content update when internet is available.

3 Citizen-centric Capacity Development (CCD) Framework

In this research, we adapt the initial citizen-centric framework for ICTE capacity development [6] to guide our solution design and evaluation. Two key strengths highlight the rationale for its adaptation: citizen-centric requirement engineering and a pluralist research design. The revised CCD framework for ICT4D (Fig. 1) involves two similar phases. Phase 1 adapts the citizen-centric approach [19] for ICT capacity development. Phase 2 is the evaluation and interpretation of the effective use and benefit of ICT. Below, we highlight and justify the main changes in the revised CCD framework, while interested readers can refer to the original framework (Thomas and Li 2015) for detailed definitions and steps.

Fig. 1.
figure 1

Citizen-centric Capacity Development (CCD) framework

Full size image

First, we consider that the citizen-centric approach for capacity development from the original framework is essential for linking ICT with development. In the context of development literature, capacity development can be defined in a wider context as “…the process through which individuals, organizations and societies obtain, strengthen and maintain the capabilities to set and achieve their own development objectives over time” [20]. In ICT4D where many diverse and competing actors are involved in development, it is critical for an engaged outsider (i.e. ICT solution designers and developers) to understand and support local people to determine their own societal goals and challenges, build the capacity through ICT-enabled solution implementation that help the community achieve positive societal changes, and organize community members to act upon and sustain these changes [21]. Thus, it is desired to involve local people from the start of ICT solution development through citizen-centric requirement engineering (Phase 1 of Fig. 1). Second, we believe that innovative design and effective evaluation of ICT-based intervention are two integral parts of ICT4D research. The ICT-enabled development goal can only be achieved through dynamic integration of social, psychological, and technical mechanisms in the ICT-solution design and evaluation. Such integration requires different research methods to address different tasks and research problems. The initial framework provides a strong foundation by systematically combining qualitative, quantitative, and design science research methods.

We further revised the framework to address some of its limitations. First, it does not explicitly capture the ultimate goal of the ICT-based intervention in LICs, which is the social-economic development of local community. An ICT4D project should start with a clearly defined societal goal that can be potentially addressed by an ICT solution. Furthermore, while the complex socio-economic, political, cultural, and financial factors should be examined to achieve the potential benefit of the ICT-based intervention, an ambitious yet achievable ICT4D project goal needs to be described. An achievable project goal serves two purposes. One is to provide the basis for searching existing information and knowledge base that would inform the ICT solution design. The other purpose it to guide the research design towards evaluating the effective use of societally-beneficial ICT-based intervention. This goal-oriented design and evaluation process is incorporated in the improved CCD framework. Second, the original framework assumes the evaluation and interpretation of effective use and benefit of ICT (Phase 2 of Fig. 1) to be quantitative. Often, ICT4D projects present intangible outcomes that can be difficult to define and measure quantitatively. Further, even when a project has clearly defined short-term goals, it is hard to articulate well-defined metrics to measure its long-term benefits. Thus, a good project evaluation should be resilient to accommodate multiple methods, capture changes over time, and be scientifically sound [22]. The updated CCD framework reflects the multi-method approach towards evaluation and interpretation for ICT capacity building.

4 CMES Project Background

In this section, we describe the background of CMESFootnote 3 project. The project was initiated by a group of medical doctors in Nepal, who advocate improving rural health care by focusing on training rural health care workers, especially those in hard to reach villages in the Everest region. As an underdeveloped country (145th of 188 countries in the Human Development Index) [23], Nepal faces great challenges as medical resources are extremely limited and infrastructure for the delivery of health care is fragile. As a landlocked country between India and China, 83% of Nepal’s terrain is mountainous and about 80% of the population lives in rural and remote areas. Despite years of efforts to expand Nepal’s telecom market, infrastructure development in the rugged high altitudes of the Himalayas is extremely difficult. Furthermore, the political instability (25 prime ministers in just 26 years of democracy) has had an adverse effect on Nepal’s already struggling economic situation.

In the context of health care, World Health Organization (WHO) identifies Nepal with a critical shortage of health care providers with only seven providers per 10,000 capita. There is also a huge gap between urban and rural health care, where rural population has very limited access to doctors and hospital services. Lack of professional development opportunities is one of the main reasons for reluctance among medical practitioners to serve in the rural areas of Nepal (Butterworth et al. 2008). Based on interviews with Nepal medical practitioners, one of main challenges to CME is that it is not mandated by the medical standards body (i.e., the Nepal Medical Council). This results in localized medical practices with great disparities between rural and urban areas.

The inception of the CMES project was inspired by the Computer-on-a-Stick (COS) solution that we developed for schools in Haiti [6]. One of the doctors who came across the COS said:

“The wisdom of the Haiti COS project is providing learning resources for students without the need for an Internet connection. In countries where Internet connection and electricity are unreliable…this leaps over the limiting step to access CME content. This is a common problem in Nepal where load shedding occurs daily for 12+ h and for days in the rural areas. I started thinking about using the same model for medical education.”

While CME covers many areas, CME in Emergency Medicine (EM) was identified as an immediate need because Nepal has a large ethnic population living in the mountainous regions and is a popular destination for adventure travelers. EM is one of the newest and fastest developing medical specialties in Nepal. Yet, national standards in EM training and certification have not been developed or mandated. While major urban hospitals start to adopt EM practices and training, the emergency care is entirely lacking in rural areas [24]. The excessive mortality in the recent 2015 earthquake is a direct testament to this dire situation. One of the doctors described the needs for EM training in Nepal as:

“From remote villages to larger cities, my experience has been a lack of quality and affordable EM CME for all health providers including nurses, rural community health workers (CHW), midwives, and physicians. I realized my EM training and wilderness medicine background provided pertinent topics for Nepal’s austere environments…the villages and even the cities… Kathmandu is an austere environment considering strikes, earthquakes, the petro embargo and rush hour traffic.”

Initial target participants were also identified, as described by one of the doctors:

“In Nepal, the target audience will be rural health providers, nurses, midwives, first-year interns, residents, or interns that are unable to secure one of the few residency spots in a specialty. The only EM residency program in Nepal that I am aware of is at Patan Hospital and it only has four positions open to GPs (General Practitioners) who have EM experience. Residency programs in all specialties are scarce in Nepal and residents pay tuition level fees during their training. Often, it is only the wealthy or well-connected that can afford to continue their studies. The city and town Emergency Departments are staffed by GPs and interns. The village health posts are staffed by nurses and CHW. These should be the target practitioners.”

After obtaining the support from a world-class CME content provider, and multiple medical administrators at hospitals and clinics in Nepal, we assembled a project team to investigate potential ICT-enabled solutions for CME in LICs with the goal of using Nepal as a test bed. In the following section, we describe the CMES project that is guided by the first phase of the CCD framework. The project has gone through multiple iterations of ICT artifacts design and testing, and initial CMES-based capacity building. At the time of this paper submission, we have initiated the second phase of the CCD framework, which is the qualitative and quantitative evaluation of CMES use and benefit. The second release that includes an improved CMES software targets launch in nine rural and urban locations in January, 2017.

5 CMES Platform Development

5.1 CMES Project Goals (Phase I)

The CMES project started with a clear societal goal from local medical practitioners in Nepal, which is to improve the rural health care by providing affordable and quality CME content for rural health care workers. The local doctors also suggested a potential ICT solution that is similar to COS to solve the problem of accessing CME content with limited or no Internet connectivity. In order to translate this societal goal to an ambitious, achievable and measurable ICT goal, we first consulted the knowledge base related to offline content delivery and CME challenges in LICs and rural areas. Among existing offline content delivery solutions, Internet-in-a-Box and RACHEL-PiFootnote 4 are two alternatives for the delivery of educational contents without the need for Internet. There is also E-Pustakalaya solutionFootnote 5 that provides education-focused free and open digital library for rural schools and can be installed on low power servers with limited or no Internet connectivity requirements. However, none of these solutions include CME content. Furthermore, all of them offer pre-loaded static content, and content update is quite cumbersome. The previous COS solution is also not desirable, as it requires the Linux computing environment and focuses on K-12 education rather than professionals in the medical field.

In addition to literature review as highlighted in Sect. 2, we also consulted stakeholders (e.g., medical administrators, NGO leaders who share similar interests, government officials, and medical policy makers) and citizens (e.g., medical doctors). Interviews with medical administrators (including senior medical doctors) and NGO leaders confirmed the literature findings. For example, the most common challenge for CME in Nepal is the lack of regulatory requirement for CME, as highlighted by the founder of a community-based, not-for-profit hospital:

“Like many developing countries, CME in Nepal is not mandated by the authorization body (Nepal Medical Council) and there is no CME standard. It becomes a voluntary effort of the director of each department to implement CME… The CME goal will be (even) different from department to department in the same hospital. This results in the localized medical practices (knowledge sharing usually happens at the individual department level) with great disparities between Urban and Rural areas.”

The interviewees also identified many social, political, and financial factors critical to the success of CME, such as administrative support, mentoring and encouragement, access to relevant and update-to-date CME content, policy incentives, and intrinsic motivation of medical practitioners. For example, a NGO Leader describes the issue of self-motivation as:

“The medical practitioners would be more encouraged to work in rural areas if they feel connected with the main/urban hospitals. The practitioners must be motivated from their heart to adopt and use CME. Otherwise it would be too difficult to implement it. We can encourage them for sure telling that it will help for their career but that is not good enough. If the practitioners are interested to work just for money, we can’t motivate them.”

Although not all of these identified factors can be addressed by ICT-based intervention alone, they should nevertheless be considered in a broader context for CME capacity development. For example, we met with the Secretary of Health at Nepal Central Government and the Director of the Medical council as it is the government’s responsibility and interest to improve the rural health. Keeping them aware and receiving their support are essential for the long-term development and sustenance of a project of this nature.

Our interviewees included not only initial targeted participants, but also doctors who work in the participating urban hospitals and medical clinics. All interviewees provided valuable input in shaping our project goals. For example, one western doctor who volunteers at an advanced travel clinic in Kathmandu mentioned that access to medical content was really not an issue there, as they had high speed internet and subscription to content services. However, she knew many different groups and people who are interested in promoting rural health services in Nepal. she suggested the need for a platform to brings those players together and distribute CME with various entities, groups, and organizations that share the same goal.

Informed by existing knowledge and voice of the local people, we defined our project goals as: “provide free and open access to CME content in LICs, and promote standardized CME knowledge sharing and distribution”. Based on these goals, we targeted the development of a low-cost, adaptable CMES platform that include:

  • A CME content storage solution that capitalizes on Internet-based modalities for provisioning CME content;

  • A CMES software application that enables CME content dissemination and synchronization with minimal need for Internet connectivity;

  • An offline CME content server based on raspberry-pi where content can be accessed when Internet connectivity is unavailable or insufficient for most practical purposes;

  • and a LISTSERV for participants to discuss CME related topics.

The team also acknowledged that promoting standardized and localized CME content sharing requires a long-term commitment from multi-functional and multi-cultural teams. Many key stakeholders were identified and brought into the project as partners and advisers.

5.2 Citizen-Centric Requirement Engineering (Phase 1)

Once the project goals were articulated and a set of potential ICT solutions were identified, we started the citizen-centric requirement engineering for the CMES software application development. It included iterations of citizen and stakeholder interviews, analysis, user requirement, low-fidelity prototype development, and citizen walkthroughs (Phase 1 of the CCD framework). The citizen-centric approach ensured the design and construction of CMES application met the vocalized needs of local medical practitioners. Interviews with stakeholders identified three design principles for our application:

  • Free: the CME content is delivered for free to the users;

  • Adaptive: the CME application is adaptive to the user’s device modality;

  • Ease-of-use: the application should look and feel like something they already know how to use and require minimal training.

Citizen interviews provided insights into key features that were desired for the CMES application:

  • Content on CMES should be updatable when internet is available (e.g. when a rural doctor goes to a location with internet connectivity);

  • A search function that enables the quick identification of content by tags or keywords;

  • Display storage information of the USB drive;

  • A personalizable and customizable local CME library structure that enables the user to retain CME topics of interest and delete irrelevant or unused ones.

Because the CMES application was envisioned to require a computer to operate, it was critical to understand the types of computers (including hardware and software) that users might have. Having conducted a survey and visited the hospital sites in Nepal, we found out that 100% of targeted users were using Windows machines, among which 60% were older than 4 years and 20% were running Windows XP operating systems. This brought some system constraints in the application development, such as:

  • How to ensure version management across different operating systems?

  • How to ensure the application runs the same and effortlessly across different machine?

  • How to make sure that the application will retain the same user experience in different types of operational systems?

Figure 2 depicts our CMES architectural design. The CMES application was written in Java. For content storage and management, we used Microsoft Azure, a scalable cloud-based computing platform with integrated tools and managed services. Our project takes advantage of its built-in server management, security, availability, and scalability features.

Fig. 2.
figure 2

Initial CMES architectural design

Full size image

Soon after the initial architectural design was finalized, a low-fidelity prototype including initial application screen mock-ups was presented to the citizens for feedback. For example, three different content viewing structures were presented to the citizens and they all chose the tree-view (commonly seen in windows operating environment) which they felt most comfortable with. The citizen walkthrough also inspired additional design features for the CMES application:

  • A launch screen with links to tutorials in pdf and video format;

  • Focus on simplicity of use (e.g., a landing page following the launch screen with two options: Search and Sync);

  • Sync feature activated only when Internet is available;

  • USB write-protection to prevent accidental content deletion and safe guard intellectual property of CME content provider;

  • Packaged Java Runtime Environment (JRE) within the application (during the low-fidelity prototype testing, it was found that many computers in Nepal hospitals did not have JRE).

Figure 3 presents a snippet of our application launch screen and interested readers can explore more about the CMES by viewing the video tutorialFootnote 6.

Fig. 3.
figure 3

Application launch screen

Full size image

Based on the feedback from citizen walk through and analysis, many changes were made to the initial prototype. For example, the look and feel of the initial screen was refined and tailored as shown in Fig. 4, which was well received during the citizen walkthroughs. Some additional features were designed for software usability. For example, Nepal does not have reliable Internet connections, especially in rural areas. An Internet connectivity indicator was therefore added at the bottom of the screen.

Fig. 4.
figure 4

Application landing screen

Full size image

As the CMES project continues to evolve, the citizen’s needs and preferences have become more articulated, and the supporting software infrastructure and application design have expanded. For example, all screens automatically resize to match the display resolution of the viewing device and tooltips added to guide user interactions. Each CMES content type included a file type logo and a simple click of the file would invoke the default application registered with the computer’s operating system. Because Internet speed is a challenge in Nepal, a status bar was added to indicate both percentage and number of files downloaded or deleted. In the event of Internet failure during download or delete, the application will roll back to the previous state.

5.3 CME Capacity Development (Phase 1)

After multiple iterations of software development and two field visits (two weeks each) by the researchers and CMES developers to Nepal, the CMES program was officially launched during Fall 2016. A medical doctor dispensed USB drives to the participants and provided CMES training. The training of doctors was conducted at hospitals in Kathmandu. It consisted of either one to one training session or a Power Point group presentation, which included instructional documents and video, and hands-on demonstration of the CMES application. This was followed by a question and answer session, during which the participants were given time to practice using the software. For participants who were unable to attend training in Kathmandu, their organizational administrator was trained to demonstrate and train the medical staff on site. Troubleshooting and follow-ups were encouraged via phone, Skype and email. Six organizations participated in the initial launch with a total of 25 hospitals and/or clinics sites and 39 participating medical practitioners. All participants were physicians, among whom more than half were working in remote and rural areas of Nepal. Since the completion of onsite training, 80 additional medical practitioners have expressed interest in the CMES thumb drives. Two rural hospitals, one in Lukla and one in Chitwan, have submitted requests to participate in the CMES project. We anticipate approximately 100 thumb drives to be distributed among medical practitioners in Nepal by January 2017 to meet the demand.

As described preciously, ICT-enabled capacity building in LICs has to be deeply embedded in its social, economic, and political environment [21]. Successful ICT-enabled capacity building requires the balance of ambitious yet achievable ICT goals with the long term societal goals. Slower than expected pace of measurable outcomes is expected when developing ICT solutions for challenging environments [25]. The capacity development in Phase 1 (Fig. 1) of the CCD framework is not just about delivering the ICT solution to end users. Rather, it requires strategic consideration of social and psychological mechanisms that need to be addressed to achieve the effectiveness use of ICT. The CMES project is no exception.

The success of the initial CMES launch is attributable to a couple of factors. First, the CMES project is a multi-functional, multi-cultural collaboration effort that has engaged many different stakeholders from the start. This allowed us to identify voluntary participants and seek administrative support from partnering hospitals. Second, meetings with the Nepal Medical Council and several senior doctors confirm impending policy changes related to CME in the near future. According the director of Nepal Medical council, the council is committed to introducing CME mandate in Nepal, and has already created a committee to initiate the necessary steps. Although the CMES program will not directly influence such a policy change, the director viewed the proposed solution as a valuable asset to deliver customized, updatable and personalizable CME knowledge library. Third, the application design benefited greatly from involving citizens in the process. The functionality and simplicity of the software was very well received. All participants commented on the ease of use, navigation, and click-and-go features of the software. Fourth, we were able to engage several key stakeholders who are well respected senior doctors or NGO leaders in Nepal. They served as local champions for the project and helped us connect with many medical intuitions. For example, a chief medical doctor in a rural hospital sent the following email:

“I am super excited to hear about your CMES program, exactly something that is greatly lacking in this remote part of the world. Thank you so much for your kind interest in including us in this program. I am sure it will give very useful medical information for our team up here and truly will be well used in this remote hospital.”

Last but not least, long before the design and building of the CMES application, we opened dialog with various stakeholders regarding CME challenges and factors to be considered to ensure the success of CMES program. The level of commitment and belief in the potential societal benefits of CMES varied significantly among those stakeholders. One of the NGO leaders with enormous experience in training rural health workers noted“…mentors and administrative support are an important component of any training and CME program.” As such, we carefully evaluated the needs of participating organizations, which include regional and district hospitals, medical clinics, and health centers. One of the hospitals initially expressed interest in participating in the CMES program, but later showed little engagement and administrative support. Subsequently, although keeping them in our participants list, we only provided one USB drive to their chief EM officer. On the other hand, the experience at another hospital was awe inspiring. The hospital’s director is passionate about improving rural health and the hospital has an aggressive in-house CME program for their nurses, plastic surgery residents, staff doctors, and house officers (those who have graduated from medical schools but not completed residency). The lead doctor not only engaged all available staff in the initial CMES program launch, but also suggested co-developing CME content. We consider the capacity development at their site a success, as described by the medical doctor who provided CMES training at their site:

“…arranged a conference for all available staff so I presented the CMES program not only to ED [emergency department] doctors but the critical care, surgery and medical teams. They are all interested in CMES. They consider the thumb drives as property of the hospital and are treating them like a library book to be borrowed. The ED Director and the CEO will each have a thumb drive. Any other doctor is welcome to use one for a two-week period of time so they will circulate between the departments. Then they have to present one article at their CME conference.”

5.4 Effectiveness and Benefit of CMES (Phase 2)

The evaluation of CMES program towards achieving its ICT4D goals has been initiated. We plan to include both qualitative and quantitative methods to evaluate the adoption, effective usage, and benefits of our CME platform. The CMES application includes extensive logging features to track usage patterns of each device, such as frequency of application launch, synchronization, content viewed, and search behaviors. The logs are uploaded to remote server any time the CMES sync operation is triggered, which will enable us to conduct varying levels of assessment related to the ICT usage. A follow-up questionnaire will be sent to each participant. They will be encouraged to provide comments and criticisms on how the program can be improved to meet their CME needs. Qualitative in-person interviews are planned in May 2017 to determine whether and how CME access has impacted practice patterns, physician identity, and professional networking. We expect a longer period of time for participants to see the societal-benefit of the CMES. We also plan to have post implementation meetings with stakeholders for future recommendations. Moreover, we are currently in the process of reviewing theoretical foundations of CME, such as experiential learning theory [26] and transformative learning theory [27], to help us better understand the CMES program adoption and usage.

6 Conclusion and Future Research

We developed a low-cost, adaptable platform for the delivery of CME content for medical practitioners in LICs. The initial implementation of CME content server and the USB-based CMES software application for capacity development was conducted in Nepal. While planning the second iteration of application release and continuing CME capacity building, the evaluation of capacity building effort has also started. This research contributes not only to the theoretical knowledge of linking ICT and development in ICT4D solution design and evaluation, but also the practical knowledge of building ICT-based CME capacity for rural areas in LICs.

For the future research, we have started prototyping two other components of the CMES platform: a CMES-pi offline CME content server based on raspberry pi and a LISTSERV. The CMES-pi solution will include a content server that can be updated remotely, and a mobile application that will allow medical practitioners view CME content on their mobile devices. A needs assessment is planned to identify CME content needs of the rural healthcare workers. It will provide us insights on how to expand CME to other medical specialty areas, such as nursing and pediatrics. As suggested by the secretary of Nepal Health Services, we plan to engage Nepal Medical Association and Nepal Nursing Association for participation from those specialties. We also plan to provide a web-based, Internet dependent CMES version for urban doctors in LICs, who are also often constrained by limited CME resources. To encourage participation, we are in the process of designing feedback loops, such as certificates of completion when CME content is reviewed and related questions are answered. As described on our website, the long-term project goals include engaging a global community to contribute core CME content for free in many languages, and benefit the public health in a global context. We are currently reaching out to agencies in other LICs for CMES program participation.