Usability evaluation of selected picture archiving and communication systems at the national level: Analysis of users’ viewpoints

doi:10.1016/j.ijmedinf.2020.104372

International Journal of Medical Informatics

Volume 147, March 2021, 104372

https://doi.org/10.1016/j.ijmedinf.2020.104372 Get rights and content

Highlights

•
This research can reveal some of the usability issues of the PACS at the national level from the users’ point of view.
•
The results of this study can aid designers and developers of PACS to identify weaknesses of these systems.
•
Identifying all of the problems can help optimizing future PACS or redesigning existing PACS and make them more usable.

Abstract

Background and objectives

Picture Archiving and Communication System (PACS) can be considered as one of the most important information systems used in healthcare facilities and its usability problems can lead to delays in the acceptance of information systems, increased medical errors and also user dissatisfaction. The primary purpose of this study was to evaluate the usability of PACS from the perspective of its main users.

Methods

This study used a combination of qualitative and quantitative methods and was carried out in 2019 where the research community consisted of PACS in five selected companies in Iran. The statistical sample included 200 individuals using the PACS in several hospitals across the country. Moreover, the sample was selected using the multistage random method. The data were then collected using the standard Computer System Usability Questionnaire (CSUQ) consisting of 5 sections and 19 items. Finally, the data were analyzed by SPSS software, version 18 using both descriptive and inferential methods. Content analysis was done for the qualitative data sets.

Results

It was indicated that ease of use, as a category, was not significantly different from the perspective of various PACS users. However, the ANOVA test revealed that there were significant differences in terms of information quality, user interface quality, overall user satisfaction and usability of PACS from the users’ perspectives. Furthermore, content analysis of users' comments showed that speeding up the image processing and frequent system failures were amongst the most positive and negative aspects of the PACS, respectively.

Conclusions

According to the perspective of the users of the investigated PACS in Iran, the usability of these PACS had a favorable status regarding ease of use while provided lower information quality. Generally, based on the users’ viewpoints, the PACS from Company B were the most usable while the PACS provided by Company D were the least usable. It is suggested that the information quality and user interface of systems be improved by using appropriate analysis and needs assessment of the end users.

Introduction

The International Organization for Standardization (ISO) defines usability as “the extent to which a system, product or service is used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context” [1]. Based on this definition, the healthcare information systems, currently being used worldwide, have various usability drawbacks and issues [2]. Numerous studies have indicated that the usability characteristics of an information system include ease of use, system efficiency, and user satisfaction. These aspects should be clearly addressed and resolved in order to improve the system adoption by the users [[2], [3], [4], [5]]. Moreover, usability problems can be regarded as one of the main factors in delaying the adoption of information systems and are, in fact, the primary reasons for a rise in medical errors and user dissatisfaction [5,6]. Furthermore, previous findings have proposed that usability problems can cause a number of medication errors, such as making mistakes while entering medication dosages into patients’ healthcare information records [7,8].

Picture Archiving and Communication System (PACS) can be considered as a widely used electronic information system [9] where previous usability evaluation studies have indicated a number of its usability problems. For instance, Goldermann et al. [10] revealed a number of such problems including image resolution, data integrity, system efficiency, and computational capacity of the system after evaluating several PACS being used in Germany. In addition, Jorritsma et al. [11] identified well-organized information, providing user-friendly experience by using better icons and toolbars, besides a more desirable user interface, ease of use, and a faster feedback as important factors to improve the usability of PACS.

An improvement in the usability of information systems can in part lead to productive users, reduction in the number of errors, increase in the efficiency and effectiveness of systems, rise in user acceptance, and ultimately, a significant improvement in user satisfaction with PACS, which is a highly important aspect of the usability [12,13]. The usability evaluation of information systems can be carried out based on two main methods, namely, the expert-based and user-based methods [14]. Moreover, Computer System Usability Questionnaire (CSUQ) is one of the widely-used and standard tools, which evaluates the system usability in terms of ease of use, the quality of information and user interface based on the user’s perspective and feedback [15]. It should be noted that previous PACS usability evaluation studies in Iran have been mainly focused either on the expert-based method or on tools other than the current questionnaire [[16], [17], [18]]. Moreover, in previous studies conducted in Iran, the users’ viewpoints on such aspects as ease of use, information quality, user interface quality, and overall satisfaction with PACS were not evaluated via the present questionnaire and content analysis method. Since radiology technicians and radiologists are two of the most common users of PACS [19], this study evaluated the usability of selected PACS from the perspective of these users in terms of ease of use, information quality, user interface quality, and overall satisfaction. In this study, first, a list of PACS vendor companies that had a certificate of information exchange with SEPAS awarded by the Iranian Ministry of Health and Medical Education, was prepared and then their usability was determined using the CSUQ standard questionnaire.

Section snippets

Study design and setting

This study was conducted by both quantitative and qualitative methods. Quantitative research was performed by analytical method and qualitative research was done by content analysis in 2019. The research population included PACS at healthcare institutions that had won certification of information exchange with SEPAS by the Iranian Ministry of Health and Medical Education. SEPAS can be briefly described as an electronic health record system in all teaching hospitals of Iran that collects and

Results

In this study, 65 % of the participating users were women. The mean age of all users was 35.53 ± 7.67 while half of the users (50 %) were in the 30−39 age bracket. Furthermore, over half of the users (58.4 %) had a bachelor's degree while 89.3 % had previous working experience with computer.

Table 1 shows the overall usability scores for users based on their demographic and background information. According to this table, the highest usability score belonged to the 30−39 age group while the

Discussion

The purpose of this study was to evaluate the usability of PACS from the users’ perspective. The results indicated that the overall usability of the PACS provided by Company B received the highest score. In contrast, the one from company D had the lowest scores among the five companies. Moreover, the usability scores of the PACS provided by companies A and B were rated as “good” while the other providers were rated as “moderate”. The PACS offered by Company B was the only one rated as “good”

Conclusions

Based on the findings of this study, the PACS provided by Company B held the highest usability scores in all aspects, including ease of use, information quality, user interface, and overall user satisfaction. The PACS provided by company D received the lowest scores among the five providers under study. According to users’ viewpoints, although PACS speeds up the imaging process and facilitates access to the patient's images, frequent system failure and low system performance are inevitable if

Authors’ contributions

MF made substantial contributions to the conception, drafting and design as well as acquisition of funding. HT, MSJ and AMN participated in data collection and HT performed the statistical analysis. HT and MSJ contributed to manuscript drafting, revision and approval and MF supervised the research group. The author(s) read and approved the final manuscript.

Statements of ethical approval

This study was approved by the Research Ethics Committee of Kashan University of Medical Sciences Research Council (Number: IR.KAUMS.NUHEPM.REC.1398.049) and conducted following the guidelines of the Declaration of Helsinki. In accordance with the opinion of the Ethics Committee and given the fact that no information about participants is provided in this paper, the subjects who participated in this study gave verbal consent to voluntarily take part in this research.

Statement

I have the pleasure of sending you the manuscript entitled “Usability Evaluation of Selected Picture Archiving and Communication Systems at the National Level: Analysis of Users’ Viewpoints” authored by “Mehrdad Farzandipour, Monireh Sadeqi Jabali, Ali Mohammad Nickfarjam, Hamidreza Tadayon” to be considered for publication as a research article in your prestigious journal. This study is concerned with one of the areas of human-computer interaction in medical sciences. This research has been

Funding

This study was supported by a grant from the Research Council of Kashan University of Medical Sciences [grant number: 98135]. The authors did not receive any grants from nonprofit organizations or funding agencies either in public or commercial sectors.

Declaration of Competing Interest

The authors report no declarations of competing interest.

Acknowledgements

Special thanks to Research Center for Health Information Management and to the Vice Chancellor for Research at Kashan University of Medical Sciences for funding and their supports. We would like to offer our special thanks to all PACS companies and participants from the teaching hospitals in Iran for their contributions to this study.

References (45)

J. Viitanen et al.
National questionnaire study on clinical ICT systems proofs: physicians suffer from poor usability
Int. J. Med. Inform.
(2011)
P.H. Lilholt et al.
Heuristic evaluation of a telehealth system from the Danish TeleCare North Trial
Int. J. Med. Inform.
(2015)
T.P. Thyvalikakath et al.
A usability evaluation of four commercial dental computer-based patient record systems
J. Am. Dent. Assoc.
(2008)
A.W. Kushniruk et al.
Cognitive and usability engineering methods for the evaluation of clinical information systems
J. Biomed. Inform.
(2004)
A.W. Kushniruk et al.
Technology induced error and usability: the relationship between usability problems and prescription errors when using a handheld application
Int. J. Med. Inform.
(2005)
W. Jorritsma et al.
Merits of usability testing for PACS selection
Int. J. Med. Inform.
(2014)
M.W.M. Jaspers
A comparison of usability methods for testing interactive health technologies: methodological aspects and empirical evidence
Int. J. Med. Inform.
(2009)
R. Khajouei et al.
Errors and causes of communication failures from hospital information systems to electronic health record: a record-review study
Int. J. Med. Inform.
(2018)
S.L. Tan et al.
Picture archiving and communication systems: a multicentre survey of users experience and satisfaction
Eur. J. Radiol.
(2010)
M.T. Britto et al.
Usability testing finds problems for novice users of pediatric portals
J. Am. Med. Inform. Assoc.
(2009)

K. Chalil Madathil et al.

An investigation of the efficacy of electronic consenting interfaces of research permissions management system in a hospital setting

Int. J. Med. Inform.

(2013)

G. Pare et al.

Knowledge barriers to PACS adoption and implementation in hospitals

Int. J. Med. Inform.

(2007)

International Organization for Standardization

Ergonomic Requirements for Office Work With Visual Display Terminals (VDTs): Requirements for Non-keyboard Input Devices

(2000)

E. Nabovati et al.

Usability evaluation of Laboratory and Radiology Information Systems integrated into a hospital information system

J. Med. Syst.

(2014)

W.-S. Tzeng et al.

A Socio-technical assessment of the success of picture archiving and communication systems: the radiology technologist’s perspective

BMC Med. Inform. Decis. Mak.

(2013)

I. Geldermann et al.

Black box integration of computer-aided diagnosis into PACS deserves a second chance: results of a usability study concerning bone age assessment

J. Digit. Imaging

(2013)

R. Khajouei et al.

Evaluating nurses’ satisfaction with two nursing information systems

Cin Comput. Inform. Nurs.

(2017)

N. Staggers et al.

Debunking health IT usability myths

Appl. Clin. Inform.

(2013)

J.R. Lewis

IBM computer usability satisfaction questionnaires: psychometric evaluation and instructions for use

Int. J. Hum. Interact.

(1995)

G. Deimazar et al.

Evaluation of USABILITY of picture archiving and communication system (PACS) in clinical settings based on ISO 9241/10 from the perspective of end users

Pharmacophore

(2017)

M. Zahiri Esfahani

Usability Evaluation of User Interface of Picture Archiving and Communication Systems (PACSs) and the Key Factors Influencing the Selection of This System: School of Health Information and Management

(2017)

Cited by (11)

Usability evaluation of electronic prescribing systems from physicians' perspective: A case study from southern Iran
2024, Informatics in Medicine Unlocked
The evaluation of e-prescribing systems' usability is crucial as they are integral to the quality of healthcare services. This study evaluates the usability of three e-prescribing systems and examines the impact of individual factors on system usability.
The objective of this descriptive study was to assess the usability of e-prescribing systems (EP, Dinad, and Shafa) as perceived by 105 physicians from three clinics at Hormozgan University of Medical Sciences in Bandar Abbas, Iran. The data was collected using the 2020 edition of the Isometric Questionnaire 9241/110, which comprises of seven axes and 66 questions. The participants were asked to rate their opinions on a 5-point Likert scale, with options ranging from completely disagree [1] to completely agree [5].
EP, Dinad, and Shafa received average scores of 3.45, 3.32, and 3.24, respectively. Self-descriptiveness and User Error Tolerance axes were rated the highest ratings, with average scores of 3.60 and 3.48. Conversely, conformity and suitability axes received the lowest ratings, with average scores of 3.19 and 3.22, respectively. Upon evaluating the usability axes, the EP significantly improved controllability and user engagement compared to other systems. The usability of Dinad and Shafa in the Gynecology clinic was significantly higher than the two other clinics. Also, older physicians with more work experience rated the Shafa significantly higher than two other systems.
The evaluated systems had average usability. although there was no statistically significant difference in the usability of these systems, the evaluation of dimensions revealed unique strengths in each system.
Evaluation of Picture Archiving and Communication System (PACS): Radiologists’ perspective
2023, Informatics in Medicine Unlocked
Evaluating Picture Archiving and Communication Systems (PACS) from the perspective of radiologists can provide valuable insights into the system's strengths and weaknesses, enabling designers to receive feedback and make improvements accordingly. As user acceptance is critical for the successful implementation of information systems, it is important to identify the views of radiologists on PACS. Discovering the users' perspectives can help identify possible system weaknesses and provide feedback to designers.
This cross-sectional survey study involved radiologists and radiology residents working in two teaching hospitals that used different PACS. A total of 138 participants were included in the study using a census method. A standard questionnaire comprising 40 items based on a three-option Likert scale (disagree, no idea, and agree) was used to collect data, which was divided into four sections. After translation, the content validity of the questionnaire was confirmed (CVI>0.9), and its reliability was confirmed by a Cronbach's alpha coefficient of 0.71. The data were analyzed using SPSS software, and the mean score for each item was calculated. The independent sample t-test was used to compare the mean scores of each section between the two PACS.
Radiologists identified better education for medical students as the most significant advantage of PACS. Almost 90% of radiologists believed that PACS met their expectations for patient care and improved care delivery. The biggest challenge of PACS was the continued dependence on paper after implementation, rather than being entirely paperless. According to radiologists, PACS B had a more significant impact on patient care and posed fewer challenges than PACS A (P value < 0.01).
Based on radiologists' perspectives, PACS improves care delivery and patients' satisfaction compared to traditional film-based systems, while also reducing patient care costs by eliminating the need for film and paper. However, the system faces several challenges, such as poor image quality and slow processing speeds. The use of appropriate infrastructure to implement the system could be one solution to these challenges.
A systematic review of effects of exchanging and sharing medical images systems in a sociotechnical context: Evaluation perspectives
2023, Informatics in Medicine Unlocked
The social perspectives of the system on how users treat the system, their performance, and their behaviors with the system evaluation are essential for program managers seeking to augment the quality of their programs and improve outcomes. This study reviewed the sociotechnical effects of exchanging and sharing medical images.
In March 2018 (updated September 2019), a literature review was conducted on PubMed/MEDLINE, Scopus, and Web of Science databases. In the final stage of the study, the articles that were examined in detail were those that focused on evaluating and assessing Picture Archiving and Communication Systems, Radiology Information Systems, and Teleradiology, with a view to enhancing sociotechnical knowledge. The changes and effects on healthcare settings, particularly within hospital environments, were also considered.
The study extracted a total of 55 relevant articles, which covered three main themes: comparing methods or instruments, economic evaluation, and technical evaluation. These evaluations focused significantly on financial issues, such as the cost-effectiveness of these systems, their impact on clinicians' workload, and comparisons between teleradiology and conventional picture-sharing methods. The evaluations primarily compared Picture Archiving and Communication Systems and Radiology Information Systems with Teleradiology systems.
The results indicated that most of the articles published on teleradiology often compared the latest methods of teleradiology with conventional methods to evaluate the quality of transmitted images, compression types, and alternative tools in a radiology department. Settings of any healthcare system are essential; thus, we are encouraged to evaluate systems from a sociotechnical perspective.
PACS, RIS, and teleradiology can be helpful for clinicians and consumers of health information technology. The disadvantages of such systems are mainly concerned with managerial problems or technical issues.
Decision support for healthcare cyber security
2022, Computers and Security
Citation Excerpt :
This flexibility allows the configuration to be suitable for both large scale and medium scale hospitals. One other important property of this configuration is having Picture Archiving and Communication Systems (PACS), Fig. 5, (Farzandipour et al., 2020) systems for patients’ image data storage which is the universal standard for modern hospitals, Fig. 3 has three main parts. The elements shown on the left correspond to the main system room elements and the networking structure.
The pandemic has demonstrated that healthcare systems are prime targets for attackers. Finding an optimal security control set is a constant challenge for health organizations, where cost is a major consideration. The purpose of this paper is to demonstrate a healthcare cost optimization system as well as a case study based on two IT setup configurations that have been evaluated by medical experts as well as IT experts. These configurations would aid in conveying the complexity of the decision parameters and demonstrating how CySecTool handles this difficulty. In the study, 64 different security controls were linked to 70 vulnerabilities that could occur at any level of a hospital system dealing with both internal and external attacks/risks. The study also includes a novel visualization scheme that allows for the observation of vulnerabilities and also their subcategories based on Microsoft's STRIDE categorization.
Information Asymmetry Between Physicians and Patients Undergoing Laparoscopic Cholecystectomy: Analysis of Patients' Awareness Level
2022, Journal of the American Medical Directors Association
Usability Evaluation of Electronic Prescribing Systems from Physician' Perspective: A Case Study from Southern Iran
2024, SSRN

View all citing articles on Scopus

View full text

Usability evaluation of selected picture archiving and communication systems at the national level: Analysis of users’ viewpoints

Highlights

Abstract

Background and objectives

Methods

Results

Conclusions

Introduction

Section snippets

Study design and setting

Results

Discussion

Conclusions

Authors’ contributions

Statements of ethical approval

Statement

Funding

Declaration of Competing Interest

Acknowledgements

Int. J. Med. Inform.

Int. J. Med. Inform.

J. Am. Dent. Assoc.

J. Biomed. Inform.

Int. J. Med. Inform.

Int. J. Med. Inform.

Int. J. Med. Inform.

Int. J. Med. Inform.

Eur. J. Radiol.

J. Am. Med. Inform. Assoc.

Int. J. Med. Inform.

Int. J. Med. Inform.

Ergonomic Requirements for Office Work With Visual Display Terminals (VDTs): Requirements for Non-keyboard Input Devices

Usability evaluation of Laboratory and Radiology Information Systems integrated into a hospital information system

J. Med. Syst.

A Socio-technical assessment of the success of picture archiving and communication systems: the radiology technologist’s perspective

BMC Med. Inform. Decis. Mak.

Black box integration of computer-aided diagnosis into PACS deserves a second chance: results of a usability study concerning bone age assessment

J. Digit. Imaging

Evaluating nurses’ satisfaction with two nursing information systems

Cin Comput. Inform. Nurs.

Debunking health IT usability myths

Appl. Clin. Inform.

IBM computer usability satisfaction questionnaires: psychometric evaluation and instructions for use

Int. J. Hum. Interact.

Evaluation of USABILITY of picture archiving and communication system (PACS) in clinical settings based on ISO 9241/10 from the perspective of end users

Pharmacophore

Usability Evaluation of User Interface of Picture Archiving and Communication Systems (PACSs) and the Key Factors Influencing the Selection of This System: School of Health Information and Management