Vision-based method for semantic information extraction in construction by integrating deep learning object detection and image captioning

Highlights

• Proposes a novel method for semantic information extraction from images in construction engineering.

• Integrates deep learning object detection and image captioning.

• Achieves the Consensus-based Image Description Evaluation (CIDEr) of 1.84 in experiments.

• Develops an algorithm for visualizing construction scene graph.

Abstract

Recently, vision-based monitoring has been widely adopted in construction management to improve crew productivity, reduce safety risks, and facilitate site planning. However, automated retrieval of semantic information (e.g., objects, activities, and interactions between objects) from construction images remains challenging due to the complex nature of construction sites. This paper proposes a novel semantic information extraction method by integrating deep learning object detection and image captioning, which aims to explore salient information from construction images or videos. In the proposed method, object detection has been employed as an encoder to extract the feature maps of construction object zones and the holistic image. The image captioning has been selected as the decoder to extract the semantic information. A post-processing method has been proposed to parse the semantic information into a graph format for better accessibility and visualization. In experiments, the proposed method has achieved the Consensus-based Image Description Evaluation (CIDEr) of 1.84. By adopting the proposed method, semantic information behind construction images can be presented to construction managers to assist their decision-making.

Introduction

The construction industry is one of the largest industry sectors in North America, which contributes to 4.3% of the Gross Domestic Product (GDP) of the United States in 2021 [1]. Cameras have recently become standard equipment in construction engineering, allowing construction professionals to monitor their sites remotely [[2], [3], [4]]. Analyzing construction images/videos by vision-based methods is beneficial to construction management in terms of improving crew productivities [5], machine's well-being [6], and project environmental performance [7], monitoring progress [8], [9], reducing safety risks [10], and enhancing construction logistics [11]. Extracting semantic information (e.g., objects, activities, and interactions between objects) from construction images is the fundamental step for many vision-based applications in construction management [2], [12], [13], [14], [15].

Object detection is a vision-based technology that can extract pre-defined classes of objects and their location information from construction images, which has been applied to defect detection [16], [17], equipment classification [18], and safety monitoring [19] in construction engineering. However, object detection can only provide information of object category and localization, which may not be sufficient for advanced construction applications (e.g., activity recognition and interaction analysis) [15]. Therefore, other vision-based technologies have been employed for extracting semantic information from images or videos. For example, Kim and Chi [5] utilized an additional model based on the Recurrent Neural Network (RNN) upon the object detector to recognize the activity of excavators. Liu et al. [20] proposed a method to extract the semantic information from the site image as natural language descriptions. Tang et al. [9] combined the object detection and human-object-interactions recognition module to ground the interaction information of workers onto the image. Moreover, Natural Language Processing (NLP) technologies can play an important role in conducting similar roles. When current techniques could not directly extract target information from the images, NLP techniques extract information from human observation reports. For example, researcher utilized Named Entity Recognition techniques to extract information about safety accident [21], [22], equipment and labor information [23], [24], and relations [25], [26].

Currently, executing separate dedicated models on the site image could extract the object, activity, and interaction information, achieving the goal of semantic information extraction [27]. However, executing separate models is time-consuming, and separate models may lack the consistency of the entity label since they are trained on different datasets. Also, extracted semantic information by separate models lacks visual connections between the recognized labels with image regions. Visual connection is vital because the semantic information extracted in labels cannot provide enough information for analysis or decision-making [10], [28], [29], [30]. For example, to track the activity of equipment or labor, it is required to know the object's location so that its trajectory can be generated and analyzed [31], [32], [33]. Sometimes, an activity cannot be identified as unsafe for safety management unless this activity happens in some restricted areas [27]. This means providing the location of the object and its activity is required [13], [14]. Therefore, combining object detection and semantic information extraction into an integrated model is a promising way to provide richer information for downstream vision-based analysis and decision making.

In this research, the authors proposed a novel vision-based method by integrating object detection, image captioning, and data post-processing to extract semantic information for the construction machine images with the visual connection. This method contains a novel process to integrate object detection and image captioning to extract information about object categorization and location, object activity, and interactions between objects. A novel attention mechanism has been added to the integrated model to obtain the visual connection between object detection results and extracted semantic information. This study provides a novel integrated model that will extract semantic information with visual connection for construction images and videos. The extracted information could enhance the visualization ability of current methods by providing object categorization, location, and activity information. It also has potential to facilitate object tracking and safety management. For example, provided the location and activity information, the activity trajectory of the equipment could be generated, and the unsafety behavior could also be analyzed and determined.