1 Introduction

Future approaches for improving production systems have to consider a holistic view of production to avoid an isolated optimization of subsections at the expense of the overall system. To manage the resulting complexity in the analysis of all dependencies in manufacturing, computer-aided methods in simulation and visualization are suitable [1, 2]. These methods allow detecting weaknesses and bottlenecks without impairing the real production processes. Especially at the material flow simulation, a conflict arises inevitably between the need for models being as realistic as possible on the one hand, while not unnecessarily being too complex on the other hand, in order to keep the amount of required work for model preparation reasonable [3]. To resolve this conflict, useful assumptions have to be made, which simplify the models but do not weaken their significance [4]. The same applies to the visualization of procedures and facilities, which should be designed merely as realistically as necessary so that the user comprehension improvement is in adequate proportion to the modelling effort [5].

After developing a simulation-capable model, which is as close to reality as required, the simulation results must first be processed to allow an interpretation. Usually, this is realized by selecting data, for example by analysing workload-diagrams and hence identifying possible bottlenecks. The data is normally aggregated to indexes and parameters using methods of statistics [6]. Furthermore, the processing of data includes a representation, which is suitable for interpretation, in order to provide a problem-specific and targeted decision support for the user. Here, one particular challenge is to provide simulation parameters and highly aggregated results for the user in a way that the user is not overburdened and the relevant results remain obvious. Thus, it is appropriate to merge the abstracted results from the material flow simulation by means of the established visualization methods based on the technology Virtual Reality (VR) with the aim to increase system comprehension of both, planners and decision makers.

Conventional simulation tools visualize the material flow simulation results based on 2D symbols, pictograms, and plausible 3D representations. This paper introduces a procedure, which enables users from the process layout planning to discuss questions concerning the planned layout (placement of machines, design of storage area, etc.) of a production system in the course of a Virtual Design Review of the layout on a large-screen visualization facility. For this purpose, an interactive visualization of the layout is provided for Virtual Design Review sessions. The interactive visualization of the layout illustrates the spatial localization of the flow of materials at a complex 3D model of a production system and facilitates the understanding of the complex dependencies. The procedure supports the assessment of different material flow concepts and consequently plays a part in contributing to increase planning certainty significantly.

In this paper, we focus on combining the material flow simulation with a 3D visualization in order to facilitate comprehension of the complex flow of materials. This requires building a 3D model of the production system and an interface for connecting the visualization system with the material flow simulation. In the following, we next describe the overall procedure for integrating VR-based visualizations into material flow simulations in Sect. 2, before we address specific aspects of the model acquisition in Sect. 3 and the design of the interface between the material flow simulation and the 3D visualization in Sect. 4, as well as, some implementation aspects in Sect. 5, respectively.

2 From Production Data to Virtual Design Review

To create an interactive visualization of the production layout, a lot of information is required. On the one hand, the dimensions of the production area, the machines and the material flow system are needed. On the other hand, data of the products, e.g. processing steps and time and required transport needs to be given. Figure 1 gives a rough overview of the processing of all this information and what basic elements are involved. The OMEGA model [7] serves as a detailed and transparent representation of the production processes. From there, the information is transferred to the material flow model in the tool Plant Simulation [8]. This material flow model maps the production processes in the form of executable code. The third element is the Virtual Design Review, which creates the three-dimensional illustration of the material flow model. This process comprises a few more sub steps, which are described in the following in some more detail.

Fig. 1.
figure 1

From production data to a Virtual Design Review

Full size image

Data Collection of Existing Production: The reference system data is put together and analyzed. It is advisable to select representative product variants and map them in the form of OMEGA process models. These models have to include all information, which is required for the material flow model and the Virtual Design Review. Thus, they serve as an important basis for the interactive visualization. The OMEGA models are based on large amounts of production data, for example transaction data, stock quantities, product keys, etc. The data is transferred to the model in expert discussions and simultaneously translated to the material flow model. This process includes several cyclic repetitions so that the level of detail of the OMEGA model and the material flow model grows gradually. An advantage of this procedure is that changes and extensions can easily be added.

For the visualization of the material flow simulation, the production machines have to be modelled, but usually CAD data of the machines, a company applies, are not available. One solution for this problem is to use a 3D scanner to digitize all important parts and machinery of the production system. This method leads to a 3D model of the entire production system within a reasonable amount of time. The result of this phase is a database for the modeling and simulation of the material flow regarding the current situation.

Modeling and Simulation of an Existing Production System: By means of the coordinated database it is possible to build up a simulation model of the current production system, for example with the software tool Plant Simulation. First, this includes the modularization of the production resources and then, with the help of components from the Plant Simulation library, their creation and parameterization. Additionally, mobile objects like workers, forklifts, semi-finished and finished products are integrated. Finally, the production control is implemented and optimization potentials are identified.

Development of Alternative Concepts for the Production System: On the basis of the optimization potentials, different concepts of material flow optimization are developed. The concepts could for example be oriented towards principles like KANBAN or Just-in-Time. Based on the material flow model of the current situation they are created in the form of Plant Simulation models.

Evaluation of the Alternative Concepts: In accordance with the simulation results, the concept with the best outcome concerning economical and technical criteria is selected. This concept is the input data for the visualization in VR.

Virtual Design Review: Then, a VR model of the production system has to be built to illustrate the favored concept. While the material flow is implemented and simulated in Plant Simulation, the representation should happen in a visualization- or game-engine like Unity3D [9]. To achieve a collaboration between two different platforms, a synchronization of Plant Simulation and Unity3D is required. This synchronization is realized via a small interface, which transfers only a minimum of crucial high-level data from the material flow simulation to the interactive visualization. This is feasible by shifting parts of the application intelligence from the material flow simulation to the interactive visualization. That way, a lot of interface transfer traffic can be avoided. Thus, more complex simulation models can be connected and simulated in real time.

Virtual Inspection: Finally, the connection of the material flow simulation to the VR visualization can be presented in the course of a virtual inspection. Thereby, the advantages of the collaboration between material flow simulation and VR visualization become apparent. For example, the recognition value for the employees is higher and it is possible to identify problems which occur because of structural obstacles and which cannot be captured in Plant Simulation.

Fig. 2.
figure 2

Procedure of the simulation environment

Full size image

Figure 2 shows, which elements participate in the simulation environment and how the data is transferred between the different elements. While the data analysis and implementation in Plant Simulation has to be accomplished to a high extent manually, the synchronization needs to work automatically and fast to achieve a high-performance real time visualization. In respect to such a high-performance visualization, the subsequent sections describe in detail an easy and simple model acquisition, as well as, the communication interface between the material flow model and the Virtual Design Review.

3 Model Acquisition

A problem, that occurs when building up a complete factory floor, is to prepare a suitable 3D model of the production system and its components. Usually, CAD data of the production machines, a company applies, are not available. If CAD data exists, the CAD models are usually way too detailed to be used in an interactive virtual environment with an acceptable performance. In addition, optimizing high-polygon meshes is far too complex for achieving a reasonable result. But then, remodelling all machines as a low-polygon model suitable for interactive frame rates by hand is a time-consuming and cost-intensive task. One solution for this problem is to use a 3D scanner. In our case, a tablet-based 3D scanner was used to digitize all important machinery and components of the production system. Since the applied 3D scanner not only scans the 3D geometry but also generates a decent texturing of the model, the scanned production machines are suitable for an interactive visualization. With this method, we capture most of the machines of a specific production system within one day and place them inside a basic 3D model of the factory floor leading to a suitable 3D model of the entire production system (see Fig. 3).

Fig. 3.
figure 3

Scanned 3D model of a production machine (left) and scanned 3D models placed on a factory floor of a production system suitable for an interactive visualization.

Full size image

Due to the nature of the scanning process, the scanned 3D-models of the production machines and facilities are geometrically not as exact as the corresponding CAD-models. But this was not crucial in our application context. The scanned 3D-models solely support the spatial orientation and localization of the material flow within the production facility and have no further function. In particular, the scanned 3D-models are not animated, in order to illustrate further details of the material flow. Since they merely make up the setting of the production facility their limited geometrical quality is sufficient in this application context.

4 Interfacing Material Flow Simulation and 3D Visualization

Material flow simulation is a typical example for a discrete event based simulation. That means, the simulation model is updated in discrete time steps. Typical data, which is updated in each time step is, e.g., actual position, attributes and states of production machines, material, devices, and workers, etc. If we want to visualize the simulation model in an external application outside of the material flow simulation system, we need to update our visualization with all the data from the material flow simulation model at every single time step. With complex simulation models and small time steps this leads to high data traffic on the network connection between the material flow simulation system and the visualization system. This is especially a problem when both systems run on different computers, which are connected via a network with limited bandwidth.

In general, material flow simulation, e.g. within the tool Plant Simulation, provides just an abstract model of the flow of materials, which is not illustrative and easy-to-understand by the user. The abstract model of the production system only provides a spatially idealized placement of the individual components of the production system. The pathways of the material between individual production components and machines are just modeled in an idealized way. Details within the material flow can only be modeled roughly, e.g. the time material needs to be transported along a path between production machines is specified by a single time value representing the necessary time for the material to be transported between two production machines.

Material flow simulation can simulate complex production processes over a long time period, in order to demonstrate a buffer workload over time. But this leads to large simulation time steps, which results in a material flow simulation being too fast to be illustrative and comprehensive for the user. Often, there is a need for parts of a material flow simulation to be visualized in real time, e.g. transportation processes along defined pathways, in order to be illustrative and comprehensive for the user.

In contrast to material flow models, a 3D model of the production system contains exact geometric properties of each component and its localization and thus provides spatially exact information about the pathways of the material flow within the production system. That way, it is more suitable to illustrate and visualize the material flow in a graphical and comprehensive way. The intrinsic geometric properties of a 3D model facilitate a more exact representation of the actual pathways of the material flow and thus a more exact simulation.

Furthermore, a 3D visualization system is also capable of simulating the real time aspects of a material flow simulation in more detail and thus more precise. E.g., in a 3D visualization system, a forklift transporting material between production machines on the before mentioned pathways, can be simulated with its real acceleration and speed profile instead of a single constant time value. This allows to model the real motion times of a forklift or human workers as part of a production system more precisely.

In the 3D visualization, the forklift motion can be modeled based on a physics model, which considers collisions between objects, such as a forklift and infrastructure or material. That way, we were able to model congestions due to geometric bottle necks, which occur e.g. when material is stored on an aisle or near doorways, due to storage space overflow at a production machine. By this, transport pathways can dynamically change or be closed over time, due to congestion etc., which facilitates a dynamic routing the material flow. All these aspects cannot be modeled in detail in a conventional material flow model.

5 Implementation of the Synchronization Interface

In order to utilizes the best of both worlds and to provide a graphical visualization, which is more intuitive and comprehensive, we connected the material flow simulation from Plant Simulation with a corresponding 3D visualization in Unity3D via an interface for data exchange. Both models run asynchronously and are synchronized only via a few synchronization points.

The material flow simulation provides simulation data over long time periods of production resulting into a rather abstracted or high-level overview of the whole process, while the 3D visualization simulates and illustrates single small time periods in real time, i.e. especially transportation processes, which are sparsely distributed over the complete simulation time period.

For these real time transportation processes, which are triggered by a synchronization event, the simulation is handed over from the material flow simulation to the 3D visualization. The 3D visualization enhances the material flow simulation by providing a more detailed simulation of the transportation action, based on a physics- and collision-based simulation and a geometric 3D-model covering the geometrical aspects of the actual routing of the transportation pathways. When the transportation simulation is finished, a synchronization event again triggers to hand back the results from the 3D visualization to the material flow simulation as feedback. That way, variable transportation times of forklifts due to changing pathways, which are based on a detailed motion simulation, are considered in the material flow simulation, instead of roughly estimated, constant time values.

For a better handling of the combined simulation, a fast-forward and a real-time mode were implemented. The material flow simulation basically covers long production time periods in fast forward mode, while the 3D visualization covers the actual transportation processes based on a physics simulation in real time mode, in order to enhance comprehension and a graphical illustration. The fast-forward mode demonstrated, how buffer workloads develop over longer production time periods, while the real-time mode illustrated real-time transportation processes, that are only fully comprehensible when shown in real-time. The switching between these two modes is done automatically via discrete synchronization events.

In a conventional material flow simulation a huge amount of data is generated while running. Sending all this information to the external visualization would cause flooding the interface, especially when running both programs on different computers. However, to be able to run both synchronously, we had to shift parts of the material flow simulation logic, such as routing via defined pathways, from the material flow simulation to the interactive visualization. Most of the data generated by the material flow simulation is updating the actual position of materials and workers. However, this information can be generated in the 3D visualization, since we know the start and destination and actual pathways of materials and workers due to the geometric 3D-model of the production system. Furthermore, we are also able to simulate collision avoidance for workers and a realistic walking speed within the 3D visualization, which is not considered by the material flow simulation at all.

Hence, the interface between the material flow simulation and the visualization only includes a few high-level commands, such as e.g. sending a signal from the material flow simulation to the visualization in order to trigger a motion and for sending a signal back from the 3D visualization back to the material flow when the destination is reached. That way, a lot of interface transfer traffic can be avoided and more complex simulation models can be connected and controlled in real time.

The technical implementation of the synchronization is as follows. For the communication with other programs, Plant Simulation provides a so-called socket module, which can operate as a host or a client. The communication interface consists of a subnetwork including 10 methods, the sockets ‘receiver’ and ‘sender’ and different tables. The elements are separated into four different application areas. The first area is the initialization and contains four methods to read information about the stationary and mobile entities and to initialize them. In the second area, named control, there are four methods, which direct the mobile entities and tell, if a part is ready for collection at a stationary entity. The third area is named communication and has two methods for receiving and sending messages. Furthermore, it contains the sockets receiver and sender and is responsible for the communication between Plant Simulation and Unity3D. The last area is called information and contains two tables, in which data about the entities is stored. The procedure of the synchronization using the described interface is as follows.

At the beginning, the initialization is executed. It sends the ID and x- and y-position of all existing stationary entities (e.g. machines) from Plant Simulation to the 3D visualization in Unity3D. In the 3D visualization the stationary entries are then instantiated and placed according to the submitted location. Additionally, the mobile entities (e.g. workers) with their ID and type are transferred. In the 3D visualization all mobile entries are placed inside their defined rest area waiting for incoming tasks. Beyond that, it is possible to add or remove both stationary and mobile entities to Plant Simulation during the simulation so that they can be sent to the 3D visualization, which then expands or reduces the already existing entities dynamically. During the simulation the 3D visualization is informed of what mobile entity should move from a starting entity to a destination entity. After the dispatch of this information, the mobile entity is unavailable for the simulation until the 3D visualization communicates, that the entity has reached its destination. When receiving such a motion command the 3D visualization sends a mobile entity from the actual position to the desired destination. The mobile entity then walks (if it is a worker) or drives (if it is a forklift) to that destination using a path finding algorithm, which takes the actual route situation into account. For example, if a path is dynamically blocked by some material, the mobile entity will take an alternative route to the destination. Also, if two mobile entities cross its paths, both try to avoid a collision, e.g. by an evasion maneuver or by waiting for the other to pass by. When reaching the destination, the 3D visualization informs Plant Simulation, which then unblocks the mobile entity in the material flow simulation.

Technically, the interface between Plant Simulation and Unity3D is realized using a UDP sender and receiver on both sides. Using the network device for communication allows the material flow simulation and the 3D visualization to run on different hardware. Based on the Open Sound Control (OSC) protocol [10], a set of communication commands was defined and implemented on both sides. The advantage of using OSC as protocol is that it is well defined, human readable and straightforward expandable, so that realizing new commands extending the protocol is very easy. Another advantage, especially while developing the communication protocol, is the use of already existing OSC tools (e.g. Line Lemur, Touch OSC, etc.), which enable the simulation of the communication counterpart.

Fig. 4.
figure 4

Process planers utilizing an illustrative 3D visualization of the material flow within a planned production system during a virtual inspection at a large screen visualization facility.

Full size image

6 Conclusion

The main goal of the proposed approach in this paper is to optimize the material flow of a production system. It extends the standard approach of material flow optimization with interactive, VR-based 3D visualizations of the optimized concepts. To realize this, a 6-stage approach was introduced. The new approach is based on applying a conventional material flow simulation system in combination with an VR-based 3D visualization, in order to facilitate system comprehension via an illustrative visualization of the material flow within the production system. For this, a synchronization interface between the material flow simulation and the 3D visualization was introduced.

The advantages of enhancing the simulation with visualization are obvious. Figure 4 shows an application scenario of a joint virtual inspection, where production system planers and decision makers inspect alternative material flow concepts for a production system based on an illustrative 3D visualization. The approach facilitates the animation of planned situations and demonstrates the results as a 3D visualization on a large screen visualization facility. For instance, the approach enables to obtain an impression of the planned production system without impairing the real production processes. Thus, it is easier to detect problems that may occur because of the planned changeover. The developed procedure supports the assessment of different material flow concepts and consequently plays a part in contributing to increase planning certainty significantly.