Deep Learning Approach for Enhanced Object Recognition and Assembly Guidance with Augmented Reality

Abstract

In an effort to enhance the efficiency and precision of manual part assembly in industrial settings, the development of software for assembly guidance becomes imperative. Augmented reality (AR) technology offers a means to provide visual instructions for assembly tasks, rendering the guidance more comprehensible. Nevertheless, a significant challenge lies in the technology’s limited object detection capabilities, especially when distinguishing between similar assembled parts. This project proposes the utilization of deep learning neural networks to enhance the accuracy of object recognition within the AR guided assembly application. To achieve this objective, a dataset of assembly parts, known as the Visual Object Classes (VOC) dataset, was created. Data augmentation techniques were employed to expand this dataset, incorporating scale HSV (hue saturation value) transformations. Subsequently, deep learning models for the recognition of assembly parts were developed which were based on the Single Shot Multibox Detector (SSD) and the YOLOv7 detector. The models were trained and fine-tuned, targeting on the variations of the positions of detected parts. The effectiveness of this approach was evaluated using a case study involving an educational electronic blocks circuit science kit. The results demonstrated a high assembly part recognition accuracy of over 99% in mean average precision (MAP), along with favorable user testing outcomes. Consequently, the AR application was capable of offering high-quality guidance to users which holds promise for application in diverse scenarios and the resolution of real-world challenges.

This work was supported in part by the Zhejiang Provincial Natural Science Foundation of China under Grant LQ21F020024, and in part by the Ningbo Science and Technology (S&T) Bureau through the Major S &T Program under Grant 2021Z037 and 2022Z080.