Framework for a closed-loop cooperative human Cyber-Physical System for the mining industry driven by VR and AR: MHCPS

https://doi.org/10.1016/j.cie.2022.108050Get rights and content

Highlights

  • Human-factors must still dominate CPS systems in complex working conditions.

  • Computing, communication and control can conduct two-way closed-loop flow.

  • A stable and reliable human–computer interaction control realized by integration of VR and AR.

  • Fusion of vision, simulation and sensors information produces more reliable information.

  • Edge computing and distributed computing accelerate the flow of information.

Abstract

Underground mining systems face numerous challenges, such as limited intelligence equipment, lack of accurate maps, difficult maintenance, and poor integration between humans and equipment. Here, based on the basic architecture of CPS (Cyber-Physical Systems), a closed-loop collaborative human CPS operational system for the mining industry is proposed (MHCPS). The MHCPS uses an information system based on VR (Virtual Reality) and AR (Augmented Reality) which serve to integrate humans with intelligent physical systems, the overall mining process, and closed-loop circulation of computing, communication, and control (3C). The 3C operation allowed centralized control center operators and inspection operators to interact with physical equipment and jointly complete tasks. Communication in the system is made dependable by a two-way information flow network. To ease computing load, a distributed computing mode integrating human and edge computing is proposed. In the system, fused VR simulation data, AR point cloud information, and sensor information are accurately returned to the human–machine interface in real-time. Based on complex computations, an operation method that integrates traditional operations with a VR/AR human–machine interface resolved the conflicts involved when fusing multiple interfaces. Finally, the prototype system was developed and assessed in the laboratory. The experimental results showed that the two-way closed-loop communications transmission between human, VR system, AR system, and physical system were stable. The computations based on imaging, sensing, and simulation information accurately reconstructed a precise virtual image of the physical equipment. The VR/AR system allowed for reliable control while also facilitating remote interactions between human operators and the other three systems, effectively optimizing the physical system. This system demonstrated the feasibility of a CPS system under complex working conditions dominated by humans. It also improved on synchronizing manual controls and machine operations, which maintained efficient cooperation.

Section snippets

Sources of problems and literature review

CPS are multi-dimensional and complex systems that integrate computing, networks, and the physical environment (Sonkoly et al., 2020). Using organic integration and deep collaboration of 3C (computing, communication, and control) technologies, the real-time perception, dynamic control, and information services of some large-scale engineering systems has been realized (Shahin et al., 2020). Proposed as early as 2005, CPS did not have a significant effect at the time due to technical limitations (

Introduction of mining work-face

Automatic mining work-faces, located in underground coal seams 200–1000 m below the ground, are mainly composed of shearers, scraper conveyors, and hydraulic supports. They operate under conditions where coal seams exhibit frequent and irregular fluctuations. The shearer cuts coal onto the scraper conveyor which transports the falling coal out of the mine. The hydraulic support groups support the roof of the mine as a whole to ensure the safety of operators. Limited by cost and technological

Overall operating framework

We adopted the concept of a human–machine integrated system to construct CPS operating framework. The goal was to form a new intelligent mining complex with high levels of integration among human beings, computing systems, and physical system in the network environment utilizing the HCPS concept. Mining systems are made more dependable and efficient through the integrated design of computing, communication, and physical systems.

The traditional HCPS includes human beings, information systems,

Three key technologies and system implementation

The information space includes the VR and AR systems and represents the precise synchronization and modeling of all elements and individuals in physical space as the basis of the HCPS. To achieve this, there are four basic requirements. (1) The physical equipment should be contain a certain intelligence level; (2) the VR system needs to construct high-precision simulated virtual image, and its intrinsic knowledge model must be enriched; (3) the AR system must obtain additional sensory

3C flow directions and fusion methods

After explaining the key technologies and establishing the relationship between operators and the three systems, the issues present among the 3C components must then be explained. These interconnected relationships are shown by the connecting arrows in Fig. 3.

The goal of a 3C operation is the seamless integration of humans with 3C, the achievement of which has three main requirements. (1) A communication platform must be used to ensure perfect information interactions between modules. (2) Based

Prototype system design

The prototype system was designed in the laboratory and the functions of the system were assessed based on the communication network layout. The attitude measurement sensors and Ultra-Wide Band (UWB) positioning base station were installed on hydraulic support groups. The inertial navigation system (INS) was arranged on the shearer and the attitude data was transmitted to the industrial computer of the control center via wireless transmission. The distributed VR monitoring system was installed

Conclusion

  • (1)

    The MHCPS is proposed for complex underground mining working conditions to meet the demand for automatic operations and serve as the foundation for the design of a CPS operation framework.

  • (2)

    In this large framework, AR and VR technologies are fused in real time to make up an information system that drives the 3C system and forms a closed information loop. The system facilitates the perfect fusion of virtual simulation information, real-time sensing information, and visual information, all to

Funding

This work was supported by the National Natural Science Foundation of China [grant number 52004174], the Fund for ShanXi “1331” Project, Key project of the Chinese Society of Academic Degrees and Graduate Education [grant number 2020ZDA12], Key Research and Development Program of Shanxi [grant number 201903D121141], Natural Science Foundation of Shanxi Province [grant number 201901D211022] and Scientific and Technological Innovation Programs of Higher Education Institutions in ShanXi [grant

CRediT authorship contribution statement

Jiacheng Xie: Writing – original draft, Methodology, Software. Shuguang Liu: Data curation, Visualization, Software, Investigation. Xuewen Wang: Conceptualization, Supervision, Validation.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

References (48)

  • CSIRO. (2019) Mining Equipment, Technology and Services Roadmap[EB/OL]....
  • I. D’Adamo et al.

    Sustainable Revolution: Let’s Go Sustainable to Get Our Globe Cleaner

    Sustainability

    (2020)
  • Egger J , Masood T. (2020). Augmented reality in support of intelligent manufacturing - A systematic literature review....
  • T.M. Fernández-Caramés et al.

    A Fog Computing and Cloudlet Based Augmented Reality System for the Industry 4.0 Shipyard

    Sensors.

    (2018)
  • S. Ge et al.

    (2020) Digital twin for smart coal mining working working: Technological frame and construction

    Journal of China Coal Society

    (2020)
  • Hietanen A , Pieters R , Lanz M, Latokartano J, Kämäräinen, JK. (2020) AR-based interaction for human-robot...
  • W. Hou et al.

    Green Survivable Collaborative Edge Computing in Smart Cities

    IEEE Transactions on Industrial Informatics

    (2018)
  • Lp A , Ed A , Ru B , et al. Pérez L, Diez E. Usamentiag R, F.Garcíab D.(2019) Industrial robot control and operator...
  • Z. Kowalczuk et al.

    Sphere Drive and Control System for Haptic Interaction With Physical, Virtual, and Augmented Reality

    IEEE Transactions on Control Systems Technology

    (2018)
  • M.Kostoláni, J. Murín and Š. Kozák. (2019). Intelligent predictive maintenance control using augmented reality,2019...
  • T. Mikkonen et al.

    Exploring Virtual Reality as an Integrated Development Environment for Cyber-Physical Systems

    2019 August 45th Euromicro Conference on Software Engineering and Advanced Applications (SEAA)

    (2019)
  • Minerva R, Lee GM, Crespi N. (2020) Digital Twin in the IoT Context: A Survey on Technical Features, Scenarios, and...
  • Minoufekr M, Schug P, Zenker P, Plapper P. (2019) Modelling of CNC Machine Tools for Augmented Reality Assistance...
  • S. Mukhopadhyay et al.

    Information modeling for cyber-physical production system based on digital twin and AutomationML

    The International Journal of Advanced Manufacturing Technology

    (2020)
  • Cited by (0)

    View full text