Keywords

1 Introduction

Nuclear power plant (NPP) outages involve maintenance and repair activities of a large number of workers in limited workspaces, while having tight schedules and zero-tolerance for accidents [1]. During a typical outage, more than 2,000 workers will be working around the NPP [2]. Expensive delays in maintenance projects (around $1.5 million per day ode delay) require tight outage schedules (~20 days) [3]. Any accidents or incidents can cause risks in collaborative field operations and unwanted wastes and delays.

Human factors play critical roles in busy workspaces that have high safety and productivity requirements. Improper design of site layouts and workspaces can force the workers to waste time on acquiring materials and tools for completing their work [1, 4]. Improper arrangements of workplaces can cause awkward positions of human bodies that reduce the productivity and increase the risks of occupational diseases [5, 6]. In addition, cluttered site conditions and occlusions can influence the capabilities of workers in recognizing potential risks on job sites [7–9]. When workers work simultaneously across multiple areas of a job site, their activities can rely on each other, or compete for limited workspaces and resources. Human errors, such as miscommunications between workers in crowded job sites, can cause unnecessary waiting of workers for collaboration activities or resources, or unexpected sharing of spaces and resources that reduces the productivity of scheduled tasks [10]. Comprehending and diagnosing human factors issues in outage processes and workspaces is thus important for proactive control of outage operations through real-time adjustment of resource allocations, and for improving the design of outage workspaces and processes as long-term solutions.

Existing methods and technologies used for outage control and human-factor analysis could hardly achieve real-time monitoring and analysis of outage processes. Such methodological limitations impede engineers from understanding certain details about how human factors contribute to the inefficiencies, accidents, and incidents observed during outages. In current practice, an Outage Control Center (OCC) has human individuals who observe multiple places across an outage jobsite and who exchange data and information with multiple groups of outage participants for coordination purposes [1]. Manual field data collection and analysis, including video data collection and interpretation, highly rely on the experiences, intuition, and cognitive capabilities of human individuals to achieve reliable and timely outage process monitoring and control. For busy and crowded jobsites of outages, a manual approach could hardly guarantee zero-errors and observation of all critical spatiotemporal details that are relevant to waste of time and resources, as well as risks that involve correlated operations at multiple locations.

This research proposes an automated multi-sensor-based outage control framework that enables effective detection and diagnosis of human behaviors that cause inefficiencies and risks in outage processes. In this framework, detection algorithms automatically detect objects that include workers, building elements, equipment, materials, equipment, and building structures from image sequences collected in the fields of views of cameras. Object tracking algorithms then monitor motion and changes of the detected objects for deriving construction progress and spatiotemporal relationships between workers, objects, and construction activities. A spatiotemporal comparison algorithm then automatically compares the as-designed construction progress and expected spatiotemporal relationships among workers, objects, and activities against the object recognition and tracking results for identifying discrepancies and dangerous spatiotemporal relationships that can cause accidents. Finally, human behavior modeling methods and anomaly detection algorithms will collectively diagnose human behaviors (e.g., very long waiting, frequent travels without much progresses, activities out of the order as specified in the schedules) that cause a majority of risks of delays and accidents. The authors expect that this integrated outage control framework will not only improve the safety, productivity, and quality of current outage project, but also provide sufficient data for future outages to reduce excessively long planning stage of an outage project.

2 Background Research and Literature Review

Figure 1 shows an overview of the envisioned outage control process that integrates advanced technologies in order to enable proactive control of an outage composed of large number of rapidly executed processes interwoven in both spatial and temporal domains. The overall process has three stages: (1) the inspection stage “discovers” parts of the nuclear plant that need maintenance before and during the outage, and generates a to-do list; (2) in the following scheduling stage, the to-do list generated by the inspection stage triggers the development of an outage project schedule (before the outage) and the update of that schedule (during the outage); (3) the execution stage will carry out the scheduled maintenance activities, under continuous monitoring and adjustment of engineers for ensuring the productivity and safety of the outage processes. This overall process is a loop of “outage control” during the outage, because information and data gathered from the field will guide outage participants to adjust their activities and resource allocation for minimizing the risks of delays and accidents/incidents, while maximizing the productivity.

Fig. 1.
figure 1

Overview of the loops of outage control and technologies potentially useful

Full size image

Within this loop of outage control, human factors issues play critical roles. As shown in Fig. 2, three major elements interact with each other and form various human-related problems in outage control. Quality of communication and collaboration between outage participants determines the performance of teamwork in terms of the timeliness, comprehensiveness, and responsiveness to changes in schedule, field conditions, and availability of resources. Communication protocol designs and team organizations involve considerations about human behaviors related to expressing their needs, perceiving messages from others, and psychological mechanisms related to collaboration behaviors within and between groups of outage participants. Interactions between project engineers and coordinators with field tasks generated based on maintenance needs are critical for improving the schedule updating process according to the field needs. Cognitive capability and behaviors of field engineers, coordinators, and people in the OCC determine the timeliness and effectiveness of schedule updates. On the other hand, various outage participants interact with the workspaces of tasks within outage processes while completing these tasks. The cognitive and physical conditions of human individuals can vary in different environments, because different environments pose different physical and environmental stresses to human individuals.

Fig. 2.
figure 2

Overview of human-related factors in nuclear plant outage control

Full size image

Previous studies in the domains of civil and construction engineering, computer vision, cognitive science, and geospatial analysis have demonstrated the potential for automating the monitoring of human behaviors within engineering workflows, though serious barriers still exist. Computer vision algorithms can extract features from videos in order to detect and track objects across multiple videos taken at different locations [11]. However, when the number of objects increases, the computational complexity and reliability of tracking objects having interwoven trajectories become unacceptable [12]. Some civil engineering applications share the same needs and job site environments as nuclear plants, and computer vision technologies tested in civil engineering applications have the potential to support NPP operation and maintenance activities. For example, construction engineering researchers have examined automatic spatial data collection techniques, such as 3D laser scanning, for automated workspace modeling and management [13, 14]. The authors believe those methods can also assist in the management of NPPs. To date, such civil engineering studies have mostly focused on deriving spatial information for progress monitoring and quality control [13, 14]. Limited studies were devoted to comprehending how behaviors of workers produce bottlenecks and risks in construction workflows in packed workspaces, such as the ones in outage control.

3 Scientific Objectives

The scientific objectives of the presented study include: (1) Establish real-time object/human tracking and spatiotemporal analysis methods for automatically comparing the productivity and expected arrangements of field activities against as-planned schedules and field operation rules related to safety and productivity; (2) Establish human factors modeling methods for automatically detecting anomalous human behaviors that cause risks and inefficiencies in the field, such as unexpected trajectories of workers that cause inefficient collaborations between the OCC, satellite outage centers, NPP workers, and maintenance service providers. The authors expect that addressing these scientific objectives will assist automated and proactive outage control based on more timely and detailed spatiotemporal information from the field related to field activities, environments, and human behaviors. The ultimate goal is to support automated work status analysis and automatic pending support notifications in NPP maintenance.

4 Research Methodology

4.1 Overview of the Proposed Approach

The proposed project will overcome the challenges described above. Figure 3 shows the overall approach. We will develop automatic image analysis algorithms that can process 3D imageries collected by depth cameras, such as the Kinect sensor, to track 3D motions of human and equipment in packed workspaces, as well as process 2D imageries collected by electro-optical cameras to track the changes across large and open job sites. Image analysis algorithms would enable reliable workflow surveillance, as well as detailed analysis of interactions among outage participants. The outputs of the imagery analysis algorithms will represent the real-time status of various elements (human, objects, and spatiotemporal relationship among them) within outage workflows. Comparing real-time status against the as-planned status of workflow will reveal inefficiencies and risks. Such a comparison has two aspects:

  1. 1.

    “Workflow Performance Analysis” focuses on assessing the productivity of various activities and spatiotemporal relationships among these activities. Anomaly in certain activities can cause delays which may propagate into stoppages in workflows. Comparing the real-time schedule against the as-planned schedule can help decision makers to update the schedule and avoid waste. In some cases, decision makers also have safety concerns related to improper spatiotemporal relationships between activities (lifting using a crane would prohibit having a human below the crane).

  2. 2.

    “Human Performance Analysis” focuses on assessing the trajectories of people and interactions among them. Unexpected long interactions and trajectories can cause inefficiencies. Improper design and arrangement of equipment and workspaces can also cause inefficiencies and risks related to human factors. Thus, an awareness of anomalies related to human factors will guide decision makers in improving the design of the OCC.

Fig. 3.
figure 3

Overview of the automatic imagery analysis framework for NPP outage control

Full size image

4.2 Real-Time Object/Human Tracking and Spatiotemporal Analysis

This part of the computational framework involves the following elements: (1) Real-time and robust algorithms that reliably track human, equipment and environmental objects across multiple videos and monitor their changes (e.g., motions, addition, and removal); (2) Real-time assessment of the progress and productivity of individual activities, based on object/human tracking results. The approach is to match both the features of human/objects/activities (colors, shapes), and their spatiotemporal relationships, with contextual human/objects/activities (objects or human individuals having similar spatiotemporal contexts) in order to identify corresponding human/objects in multiple imageries. The purpose is to reduce the search space of corresponding human individuals or objects in multiple imageries, while improving the reliability of human/object tracking. The paragraphs below describe these methods in detail.

Tracking location changes, additions, and removal of human or objects relevant to maintenance activities is important for understanding the progress of the work. For example, movements of construction equipment and tools indicate active workspaces and additions or removals of building components indicate the progress of installations. Frequent motions of people in certain areas indicate how active workspaces are in terms of human activities. The challenge here is how to ensure the comprehensiveness, robustness, and computational efficiency of human/object tracking in cluttered environments while determining changes of objects. A typical NPP outage site could have numerous moving people and objects within cluttered environments. Tracking large numbers of people or objects while considering occlusions and clutters in busy workspaces could be unreliable and thus require large amounts of computing time and memory with the use of conventional object tracking methods [6]. When multiple objects in the workspaces have similar appearances (e.g., similar machines, similar ductworks, etc.), reliable object tracking becomes even more difficult.

In this research, we use depth cameras, such as Kinect sensors, for capturing 3D objects and their motion in smaller spaces (e.g., imaging range < 10 m), and using high-resolution cameras for tracking objects in large and open spaces, such as on roofs of turbine buildings. Given 2D and 3D imagery data, spatiotemporal relationships among objects and activities will be utilized in order to narrow down the search spaces of object tracking, and to correct ambiguities caused by similar objects; we will then be able to identify added and removed objects. Specifically, two levels of object tracking and change analysis will be employed: local level, which focuses on object detection of certain appearances, and global level, which focuses on detecting objects with certain spatiotemporal relationships to contextual objects. At the local level, we are examining feature-based methods for detecting workers, equipment, materials, and environmental objects from 2D/3D imagery data. Comparative performance analysis of multiple object tracking methods are expected to reveal the suitability of these methods for different environments encountered during NPP outages [14]. At the global level, the authors are exploring hierarchical spatiotemporal pattern analysis algorithms that group spatiotemporal relationships among objects at multiple levels of details (LODs), and match spatiotemporal relationships at multiple LODs to constrain the object tracking algorithms for reducing the search space. Figure 4 shows spatial relationships matched across multiple imagery data sets in order to identify corresponding lines extracted from different images.

Fig. 4.
figure 4

Matching spatial relationships between objects for associating lines from different imagery data sets when dislocations, additions, and removals of ducts occur.

Full size image

4.3 Automatic Workflow and Human Performance Diagnosis Through Comparing Field Observations and as-Planned Activities

This part of the computational framework involves the following elements: (1) Algorithms that automatically compare the observed maintenance activities and their spatiotemporal contexts against the expected ones for revealing anomalous activities that indicate inefficiencies and risks in outage processes; (2) Algorithms that automatically compare the observed human behaviors against the expected ones for identifying anomalous interactions between human individuals and environmental conditions that indicate how human performance influences the productivity and safety of an outage. Outputs of these algorithms collectively provide observations that assist the OCC in detecting and addressing possibly improper team arrangements and behaviors, as well as observations that assist safety engineers in detecting unsafe human motions and time-wasting human activities, such as abnormally long conversations between human individuals in areas where progress lags behind, or unexpected trajectories of workers that are long and not following the safe and productive paths as expected by the OCC.

Given the human/object tracking results, the algorithm can then assess the status of individual activities. The objective is to associate object tracking and change detection results with specific activities in the outage schedule, and then determine the “resource consumption status” (number of workers and equipment used) of those activities and real-time progress. Specifically, to assess the status of an activity (e.g., lifting a girder to the roof), the authors acquired the location, needed resources, and expected progress information of the assessed activity from the as-planned schedules. Detailed motions of construction activities in a 4D (3D geometries plus time dimension) simulation environment further reveal how spatiotemporal arrangements of maintenance activities within outage workspaces influence the productivity and safety of field operations. Figure 5 shows a conceptual 4D simulation: two jobsites working in parallel are sharing one safety inspector and one material/tool storage place; the time needed for a cycle of checking out materials/tools, safety checking, working, and checking out materials/tools again can fluctuate as uncertainties exist in human behaviors, field conditions, and availability of the safety inspector and storage place shared by two job sites.

Fig. 5.
figure 5

A 4D simulation example: one safety inspector shared by two job sites

Full size image

A 4D simulation will generate random numbers about the amounts of time needed for each step in those cyclic processes using the average and variance information of productivity based on historical productivity data, and execute workflows with steps that have uncertain durations for predicting the most likely productivity of the whole process. Simulation models can help engineers predict the “as-planned” waiting time of workers for the safety inspector and the material/tool storage area, considering uncertainties in the durations of traveling and working.

Assuming that the real-time schedule would have similar activities and relationships among them as the as-planned schedule, the authors are examining an algorithm that compares the location information of activities in the as-planned schedule against the object tracking results from imaging sensors (depth cameras and CCD cameras) that are pointing to those locations in the field. Specifically, the algorithm will match the detected objects, including workers, equipment, building elements, and materials, against the resource information in the as-planned schedule. Such matching between imagery data and as-planned schedules associates objects in the as-planned schedule with objects detected in imagery data. Given such data-schedule associations, the authors are testing an algorithm that automatically uses the “addition” events of building elements for progress monitoring (e.g., counting the number of elements installed against the as-planned progress), and then use the activity durations derived from imagery data to calculate the productivity.

In order to achieve real-time 2D/3D imagery data analysis for detecting anomalous interactions between people and anomalous trajectories, the authors are using object-tracking algorithms in recognizing roles of people on outage job sites, and analyzing motions of these people that indicate interactions between them. Specifically, this research focuses on lifting operations around turbine buildings and cooling towers. The authors are designing a field experiment that has crane operators, helpers, NPP safety personnel, and field supervisors wear different vests and hard hats to indicate their roles, and uses the spatial contexts of people and communication frequencies between certain people for checking the tracking results. Figure 6 shows a test case of tracking multiple people and inferring their social relationships based on the time periods shared by them in a scene observed by a camera. The research team is using this tested algorithm for recognizing roles of outage participants and individuals who have close collaboration relationships with each other in outage processes. Such information can help the OCC diagnose how interactions among people influence the productivity of outage.

Fig. 6.
figure 6

Tracking multiple people in an outdoor scene (left); inferred relationships (right)

Full size image

Given the tracked motions and gestures of people, the authors will be applying a pattern classification approach that uses example images having behaviors labeled (e.g., talking, nodding) for “training” the classifier. The trained classifier will then take new images for automatically classifying motions and gestures as various interactions. Comparing the frequency, time and durations of those interactions against the baseline model will reveal anomalous interactions.

Another part of the studied computational framework is to analyze the trajectories of individuals in identifying unexpected motions. The motion tracking results capture the motions of workers staying at a place to complete a task, and the trajectories and velocities of movements of workers across workspaces. The authors plan to compare the trajectories’ information (paths, speeds, frequencies of travels) against the expected trajectories based on the literature review and interviews. Anomalous motions will involve trajectories significantly different from expected trajectories, unexpectedly frequent travels, and movements having speeds significantly slower or faster than as-planned speeds. This information will assist human factors scientists in understanding how the arrangement of working environments influences worker efficiency and safety. In addition, the human motions and posture analysis information from the OCC will also assist human factors engineers to design better control centers that improve the efficiency and effectiveness of OCC personnel (e.g., better table size, computer and control panel design for improved human performance).

The last part of the computational framework will be conducting spatiotemporal relationship analysis among multiple activities and objects in order to assess improper spatiotemporal arrangements that could lead to incidents or accidents. Currently, this part of research is in the planning stage, the authors plan to transfer the object tracking and change detection results into simulation models of job sites. The real-time productivity information derived by human/object tracking algorithms would be the input for updating the as-planned schedule into the real-time schedule. The movements of equipment, workers, and other objects would produce a safety simulation model for predicting likely clashes between objects, and alert us risky relationships (e.g., crane above a human). Safety manuals of outage list all types of improper spatial arrangements in the field as “risky relationships”.

5 Results of Experimental Design and Workflow Simulation

The authors have designed an indoor experiment to test the proposed computational framework described above for automatically identifying inefficiencies in field workflows and anomalous human behaviors. Figure 5 shows the layout of the room for testing the human/object tracking and imagery analysis algorithms. This figure also shows the locations of two Kinect sensors and one camera for observing the scene. Figure 7 shows the simulation model designed by us for serving as the as-designed process. The representation of this simulation model follows the specification defined in [15]. A comparison of the automatic imagery data analysis results and the simulation results will validate the technical feasibility of the proposed computational framework.

Fig. 7.
figure 7

Process simulation model for testing the proposed computational framework

Full size image

6 Conclusion

This paper presents a computational framework that enables automatic imagery data analysis for diagnosing human factors in the outage of a nuclear plant. The literature review and preliminary studies conducted by the authors revealed the technical feasibility and potential of this computational framework. Future studies will be carrying out the experiments and field data collection during real outages for improving and validating this computational framework.