Stefano Guidi
ABSTRACT
Direct observational time and motion studies, in which one or more observers continuously observe and record an individual's activities over a certain period of time, can provide valuable insights into work and communication patterns and assist in identification of areas to intervene to support safe and effective work. These studies have increased in number and sophistication, also due to the availability of software tools for PDA to support the observers. A key methodological issue for these studies is the assessment of Inter-Observer Reliability (IOR). IOR is crucial for ensuring reliability of data collection in which multiple observers are involved. In workflow time studies the multivariate, time-stamped and ordered nature of the data limits the applicability of traditional inter-rater reliability measures, and makes this assessment challenging. Measures such as Cohen's Kappa, in fact, are only applicable to one variable at the time, so that high K scores for one aspect can be achieved even if two observers disagree substantially on other variables which are the object of their observation. Secondly, computing these measures first requires either matching pairs of datapoints from different observers viewing the same entity, a problem that cannot be done with perfect certainty, or restructuring the data into fixed-length time window sequences. No single method can address all the different aspects on which observers in time and motion studies can disagree, so one should adopt a composite method. We developed a set of functions and a Shiny app to assist this process for data collected with the Work Observation Method By Activity Timing (WOMBAT) method. The app allows the loading of data from pairs/groups of observers and computes a wide set of agreement measures on both matched data and on time window data. These measures are presented in a dashboard along with interactive visualizations of the observers’ data, and can be saved and plotted over time in a different section.
- Marcelo Lopetegui, Po-Yin Yen, Albert Lai, 2014. Time motion studies in healthcare: what are we talking about? J. Biomed. Inform. 49 (2014) 292–299. DOI: https://doi.org/10.1016/j.jbi.2014.02.017Google Scholar
Digital Library
- Scott R Walter, Magdalena Z Raban, William T M Dunsmuir, 2017. Emergency doctors' strategies to manage competing workload demands in an interruptive environment: An observational workflow time study. Appl Ergon. 2017 Jan; 58:454-460. DOI: https://doi.org/10.1016/j.apergo.2016.07.020Google Scholar
- Johanna I Westbrook, Magdalena Z Raban, Scott R Walter, Heather E Douglas. 2018. Task errors by emergency physicians are associated with interruptions, multitasking, fatigue and working memory capacity: a prospective, direct observation study. BMJ Qual Saf. 27:655-663. DOI: https://doi.org/10.1136/bmjqs-2017-007333Google ScholarCross Ref
- Johanna I Westbrook, Enrico Coiera, William T M Dunsmuir, 2010. The impact of interruptions on clinical task completion. Qual Saf Health Care 2010;19: 284–9. DOI: https://doi.org/10.1136/qshc.2009.039255Google Scholar
- Johanna I Westbrook, Amanda Ampt, Leanne Kearney, Marilyn I Rob. 2008. All in a day's work: an observational study to quantify how and with whom doctors on hospital wards spend their time. Med J Aust 2008; 188:506–9. DOI: https://doi.org/10.5694/j.1326-5377.2008.tb01762.xGoogle Scholar
- Johanna I Westbrook, Christine Duffield, Ling Li, 2011. How much time do nurses have for patients? A longitudinal study of hospital nurses’ patterns of task time distribution and interactions with other health professionals. BMC Health Serv Res. 2011; 11:319. DOI: https://doi.org/10.1186/1472-6963-11-319Google Scholar
- Tommaso Bellandi, Alessandro Cerri, Giulia Carreras, 2018. Interruptions and multitasking in surgery: a multicentre observational study of the daily work patterns of doctors and nurses. Ergonomics 61(1): 40-47. DOI: https://doi.org/10.1080/00140139.2017.1349934Google ScholarCross Ref
- Danny Tzu-Yu Wu. 2019. Computer-Based Tools for Recording Time and Motion Data for Assessing Clinical Workflow. In: Zheng K (eds) Cognitive Informatics. Health Informatics. Springer, Cham. DOI: https://doi.org/10.1007/978-3-030-16916-9_11Google Scholar
- Johanna I Westbrook, Amanda Ampt. 2009. Design, application and testing of the Work Observation Method by Activity Timing (WOMBAT) to measure clinicians’ patterns of work and communication. Int J Med Inform. 78S: S25-S33. DOI: https://doi.org/10.1016/j.ijmedinf.2008.09.003Google ScholarCross Ref
- Marcelo Lopetegui, Po-Yin Yen, Albert M Lai, et el. 2012. Time Capture Tool (TimeCaT): development of a comprehensive application to support data capture for Time Motion Studies. AMIA Annu Symp Proc. 2012;2012:596–605. DOI:Google Scholar
- Kai, Zheng, Elizabeth L Ciemins, Holly H Lanham, Curt Lindberg. 2015. Examining the relationship between health IT and ambulatory care workflow redesign. AHRQ Publication No. 15-0058-EF. Rockville: Agency for Healthcare Research and Quality.Google Scholar
- Scott R Walter, William T M Dunsmuir, Johanna I Westbrook. 2019. Inter-observer agreement and reliability assessment for observational studies of clinical work. J Biomed Inform. 100, 103317. DOI: https://doi.org/10.1016/j.jbi.2019.103317Google ScholarDigital Library
- Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educ. Psychol. Meas., 20, 37-46. DOI: https://doi.org/10.1177/001316446002000104Google ScholarCross Ref
- Anthony J Conger. 1980. Integration and generalisation of Kappa for multiple raters. Psychol. Bull., 88, 322-328. DOI: https://psycnet.apa.org/doi/10.1037/0033-2909.88.2.322Google ScholarCross Ref
- Joseph L Fleiss.1971. Measuring nominal scale agreement among many raters. Psychol. Bull., 76, 378-382. DOI: https://psycnet.apa.org/doi/10.1037/h0031619Google ScholarCross Ref
- Marcelo A Lopetegui, Shasha Bai, Po-Yin Yen, 2013. Inter-observer reliability assessments in time motion studies: the foundation for meaningful clinical workflow analysis. AMIA Annu Symp Proc 2013; 889-96.Google Scholar
- Marcelo Lopetegui. 2014. Inter-Observer Reliability Assessments in Continuous Observation Time Motion Studies. PhD dissertation.Google Scholar
- Harald Janson, Ulf Olsson. 2001. A measure of agreement for interval or nominal multivariate observations. Educ. Psychol. Meas., 61, 2001:277-89. DOI: https://doi.org/10.1177%2F00131640121971239Google Scholar
- Lawrence I-Kuei Lin. 1989. A concordance correlation coefficient to evaluate reproducibility. Biometrics.;45(1):255–68. DOI: 10.2307/2532051Google Scholar
- Scott R. Walter, William TM Dunsmuir, Magdalena Z Raban, 2019. Understanding Clinical Workflow Through Direct Continuous Observation: Addressing the Unique Statistical Challenges. In: Zheng K. (eds.) Cognitive Informatics. Springer, Cham. DOI: 10.2307/2532051Google Scholar
Recommendations
Motion control and trajectory tracking control for a mobile robot via disturbance observer
This paper investigates the tracking control of a wheeled mobile robot in the unknown environment. A disturbance observer is developed with utilization the integral filter. The proposed control scheme employs the disturbance observer control approach to ...
Observer-based fuzzy adaptive robust control of nonlinear systems with time delays and unmodeled dynamics
In this paper, an adaptive fuzzy output feedback backstepping control approach is developed for a class of nonlinear time-delay systems with unmeasured states and unmodeled dynamics. Using fuzzy logic systems to approximate unknown nonlinear functions, ...
Reduced-order observer-based output feedback control of nonlinear time-delay systems with prescribed performance
This paper studies the problem of output feedback control for a class of nonlinear time-delay systems with prescribed performance. The system is in the form of triangular structure with unmodelled dynamics. First, we introduce a reduced-order observer ...
Comments