1 Introduction

Many areas of Artificial Intelligence have made a significant progress lately; methods and techniques of this field are applicable in industry and participate in the so-called Industry 4.0. On the information technology side, this trend is accelerated in areas such as cyber-physical systems (CPS), cloud computing (CC) and Internet of Things (IoT). A complete review [1] over the period 2014–2017 on the three approaches and their influence on manufacturing sector indicates: (1) a continuous doubling of the number of publications in ISI journals, (2) a large number of conceptual methods and few case studies, simulations and experiments, (3) a lack of researches on the human-machine interface and the interaction between industrial equipment, and (4) there are no studies presenting the cost/effort of implementation versus the advantages presented in literature. Adoption of these methods and techniques has a slow pace in robotic production systems due to the difficult way for this technology to penetrate such heterogeneous, complex environments, which are highly dependent on manufacturers of industrial solutions.

By applying the IoT principles and solving some aspects of integration and resource interaction, today's industry can overcome certain challenges. Among them, there are: real-time data collection from existing equipment, expanding sensory systems, smooth integration with new devices and functionalities, improving coordination and collaboration methods by applying techniques from distributed artificial intelligence. From a technological and socio-economical point of view, manufacturing systems will benefit, through simple integration into cloud computing, of optimization and prediction mechanisms at production, maintenance, energy consumption and human staff levels.

As highlighted, the lack of case studies or experiments in literature [1] makes it hard to anticipate the implementation effort; some elements of difficulty are: current physical resources offer limited support for the IoT-specific communication protocol, a production system based on cloud introduces overloading of the communication network and delays in data transmission (concerning this, systems for fog and edge computing can be considered), and, moreover, coordination and collaboration between resources under IoT and determining false or inaccurate states are difficult, too.

The paper is organized as follows. Related work is presented in Sect. 2. Then, in Sect. 3 the considered manufacturing cell is described. The architecture of the IoT solution and the developed Node-RED application, facilitating communication between IoT devices, are detailed in Sect. 4. A way of analyzing communication delays is revealed in the following Sect. 5. The paper ends with some results and conclusions.

2 Related Work

A preliminary study on the IoT reference architecture and its major components appears in a European FP7 project [2], which has influenced the analysis and application in many areas (transport and logistics, medicine, social activities and so-called intelligent environments). IoT concept finds a distinct notion in manufacturing as Industrial IoT (IIoT). The overview presented in [3] highlights the issues regarding connectivity in IIoT and the challenges for this solution, too. Security and privacy requirements are usually brought into discussion where a breach may have a negative impact in production ecosystems. On the other hand, IIOT can be the needed glue so that many technologies should be able to extend and improve even the closed manufacturing systems at the cell level.

The authors present in [4] a complex IoT system consisting of two micro-assembly manufacturing cells, a full suite of sensors and cameras, software modules and monitoring applications, a cloud platform and a virtual assembly environment (a virtual reality simulator). Inter-connectivity between IoT and cloud manufacturing is demonstrated in [5], where the dynamics of a production system can be determined in real-time and analyzed in cloud to achieve flexible resource management. More specifically, in [6] there is presented an IoT architecture with five levels (resources, perception, network, services and applications) to expose in real time the states of interconnected resources, similar to an SCADA solution. In [7], the underlined advantages of IIoT regard the way precision machining can be achieved using embedded sensors and intelligent algorithms that can detect in real time deviations from nominal values during the manufacturing processes. A method to integrate an industrial robot into an IoT system is detailed in [8]. Authors show that the integration process gets simpler when the manufacturing system is developed as an IoT one.

The most common connectivity frameworks considered in IoT applications are Distributed Device Service (DDS), OPC Unified Architecture (OPC UA), Robot Operating System (ROS), and Message Queuing Telemetry Transport (MQTT). An overview of these protocols and their performance are detailed in [9]. The last one, MQTT, was designed for low bandwidth environments and low power consumption, particularly for sensors and mobile devices in unreliable networks. System latency depends on data-transfer over the network and on computing at the edge device and cloud application [10]. Cloud solutions may have latency issues when an increased volume of data is transferred through the cloud. An advantage is obtained with fog and edge computing by going as close as possible to devices and sensors. These techniques facilitate IoT communication without cloud and only preprocessed information is sent towards cloud services. Several benefits of fog computing for manufacturing are presented in [11]. The study described in [12] highlights the strong points of using edge computing at the robot level. Lately, the digital twin concept is requested to mirror a physical process with one or more simulated environments, leading to a strong event-based connectivity with sensors and actuators. The development of digital twin for a shop floor conveyor is presented in [13].

In all mentioned projects [4,5,6,7] the basic mechanism of IoT systems is adopted to facilitate the transfer of information from sensors and embedded devices to cloud areas (for analysis and decision making) and the transfer of commands from cloud manufacturing to resources. Furthermore, the design of IoT concepts considers a collaborative work among resources and between workers and resources, too. To the authors’ knowledge, the issue of collaboration in a robotic cell under the IoT principles was still not addressed.

3 The Considered Classical Manufacturing System

To be closer to the needed transition of present manufacturing systems towards IIoT paradigm, a research on a current production environment including industrial devices is a mandatory issue. The aim is to preserve the functionality of an existing manufacturing cell when disabling the local IO communication among devices where it is possible, and to integrate devices through an IoT platform. This approach is near a fully disconnected system that should be interconnected only through several communication protocols via cloud services.

Figure 1 shows the experimental environment; the considered laboratory system contains two industrial robots (6 DoF), a machine-tool, a computer vision system, a conveyor and more storage devices with inductive and optical sensors. This system was integrated to solve manufacturing goals using two working areas for producing, transferring, inspecting, and assembling a customer product with several types of parts. Through a centralized application, a human operator adds manufacturing commands and monitors the production. It is to notice that these devices are integrated through I/O local connections, which define several strong constraints. For example, the conveyor is directly linked only to the robot R1, meaning that the second robot, namely R2, has to communicate with robot R1 to work with the conveyor. Moreover, the communication protocol is limited by the IO interface; namely, one digital output of a robot is linked to a digital input of the other robot. States of sensors/devices are directly exposed to the digital/analogical inputs of dependent devices; for instance, this is the case for the interface between robot R2 and CNC machine 5. In another cases, states are obtained via serial, TCP/IP, and other protocols; the vision inspection system (marked 4 in Fig. 1) gives the state of working table for the robot R1 through a serial communication, meaning the operator application (10 in Fig. 1) gets this vision state via robot R1 - no direct link is supported. Pneumatic actuators of robot grippers (labeled with 8 and 9) are directly commanded by robots through local connections. Extending the sensorial system and/or the actuators for a robot involves to solve first the local integration constraints. Commands are also sent by means of I/O outputs and communication protocols. These kinds of dependences limit the fast adaptability and define a rigid system.

Fig. 1.
figure 1

Manufacturing cell to be enhanced according to the IoT concept

Full size image

Some other remarks are important to consider. Common industrial devices have specialized and limited programming capabilities which do not allow developing an IoT protocol. As the authors pointed out in [8] and [12], an industrial robot has its own private protocol provided by the manufacturer to facilitate the communication with a computer through a custom API. Other equipment, such as the visual system and CNC Mill, expose services only by the serial ports for limited clients. The diversity of protocols requires support of an integrator, and usually the number of possible new devices and sensors that can be used in the system is limited by the old equipment; for example, the cost of integration of a new camera is higher than the device itself.

Some parts of the above presented environment can be simulated in RobotStudio (both robots are ABB). In this case, no interconnectivity is possible with the physical process. Our IoT–based solution enables a new way to develop the digital twin concept.

4 IoT-Based Solution

IoT is based on the publish/subscribe pattern, where the sender and receiver are loosely coupled through a broker server. A message published on a specific topic by a client is routed by the server to all clients that are subscribed to that topic. MQTT is the most used IoT messaging solution over TCP/IP protocol. IoT clients are separated into devices and applications. IoT devices are able to publish their states/events and receive commands. IoT applications are developed to receive states/events and publish commands toward devices. At this point, the direct interaction among devices is not supported. Without changing the IoT approach, the proposed solution includes an IoT application with the role of changing the type of message from event to command. Thus, a published event can be also a command, and the received command can be an event, too; the difference is made when the content of messages is interpreted.

The classical manufacturing systems already have some means of external communication. None of our manufacturing devices supports implementation of an MQTT client. Thus, we have to develop IoT adapters where it is possible, considering the external interaction supported by manufacturer. It is to notice that a general architecture for an industrial IoT adapter follows the scheme from Fig. 2. An industrial device enables external communication through services and/or digital/analog channels and has an application program interface (API) or protocol to interact with. Implementation of an adapter is restricted by the client interface: the operation system and API supported by the programming language. For those devices that have only digital/analog interactions, some types of embedded devices with WiFi options can be used. An IoT adapter for an ABB industrial robot is detailed in [8]. Other implemented adapters for real/virtual controllers, visual system and CNC machine are sketched in Fig. 3.

Fig. 2.
figure 2

General scheme of an Industrial IoT Device

Full size image
Fig. 3.
figure 3

Developed adaptors for robots, visual system, and CNC device

Full size image

Figure 4 illustrates the resulting IoT architecture for the considered production system (see Fig. 1) based on an IBM Watson IoT broker [14]. In this stage, some sensors and equipment are indirectly connected to the cloud through their dependent device; for instance, the conveyor (marked with 3) is controlled by the robot.

As mentioned earlier, two or more IoT devices can talk one to another by a developed Node-RED [15] cloud application (see Fig. 5). The received events are converted in commands by the EventToCmd function node. This mechanism broadcasts every event to all devices, excepting the sender. A limitation can be made by filtering an event according to its content. In this case, two rules are applied: (1) if the event con-

Fig. 4.
figure 4

IoT architecture for a classical manufacturing system

Full size image

tains the attribute “to” equal with *, then all devices receive the command, (2) when a name of device is specified in attribute, the indicated device receives the information. Not all events must be shared among devices; in such a case, they are ignored by not using the attribute “to”.

Monitoring or debugging the manufacturing process at the cloud level is possible too. Each device’s task can be individually tested with the insertion of commands from Node-RED, by linking inject, function, and ibmiot nodes. Collecting, analyzing, and visualizing data may be easily made by taking information from published events. For example, measuring the time between sending a command and receiving the result is carried out by two function nodes, start and stop, in Fig. 5.

Two issues can be observed: (1) the manufacturing system produces a lot of events and data that are transported to cloud; (2) the delay of messages can be a drawback in some manufacturing scenarios.

Fig. 5.
figure 5

Node RED IoT application for supporting device to device communication

Full size image

5 Measurement of IoT-based Communication

Sharing data and/or states among manufacturing devices through an IoT system will expand collaboration inside the production system. Time-related performance of communication is the key aspect in the adoption of this technology, and it depends on the private protocols provided by manufacturers, as one can see in this section.

The proposed experimental test bench includes two IoT devices, with the first one publishing a number that will be increased by the second device. Figure 6 illustrates the entire interaction protocol between devices, considering the Node-RED cloud application, too. Increasing a number is the cheapest operation and easy to achieve on any manufacturing device. Measurement of round-trip delay (RTD) involves developing a program, at the device level, to measure and save the time between publishing an event and receiving the command. Another option is to measure, at the cloud level, the time from receiving an event from the IoT device till sending a command to it (see the double headed arrow with label time). This method does not contain the cost of publishing an event, receiving a command, and of measuring instructions. The count-based method is also used to check the loss of messages in the network or in the IoT system.

Fig. 6.
figure 6

Interaction protocol to measure delays

Full size image

Several RTD measurements have been made by considering pairs of IoT devices developed according to Fig. 3. Table 1 summarizes the results obtained with different types of IoT devices: a simple software application (labeled with software in Table 1), a robot controller, and a small embedded device with WiFi connection (Wemos D1 WiFi mini). For the robot IoT device, we had the possibility to use several types of controllers from ABB: S4Cplus (Robot 1), S4C (Robot 2), IRC5 (Robot 3), and virtual IRC5 controllers from RobotStudio (Robot 4). Network latency with the IoT platform located in another country was between 52–55 ms, measured over TCP/IP protocol; Quality of Service (QoS) for MQTT was set to 0, meaning fast delivering without guarantees of message reachability. The QoS level and the network latency have given us a good overview of RTD.

The first measurement from Table 1, which is for two applications developed in python with the module ibmiotf, shows an average time around 111 ms. In this case, the device1 measures the time in accordance with the protocol presented in Fig. 6. One can observe that RTD is near the latency of network (2 round trips * 52 ms: device1 → cloud → device2 → cloud → device1). In the second experiment, the device1 only sends messages and the measure is made at the cloud level. We expected to be around 55 ms as in the first experiment, but the average is a bit higher, namely with 10 ms. A simple explication is that the effort of both devices is different. Another experiment was to use a 4G network (see line 3 in Table 1). As we anticipated, the average time increased due to the latency of the network. So far, the delay of IoT platform was directly dependent on the network, meaning that if one brings the IoT system closer to the manufacturing network, the RTD can be relatively acceptable.

Table 1. Time measurement – round trip delay
Full size table

Knowing the average time delay between two simple software applications without physical parts, the developed IoT adapters for ABB controllers can be analyzed. The experiments 4–6 contain as device2 an S4Cplus controller. The polling rate of adapter is set to 200 ms, representing the minimum value offered by manufacturer for RAP protocol (see Fig. 3). This parameter influences the notification time when something has changed for the controller (e.g., when the controller updates the received counter – see Fig. 6). Another aspect of the developed experiment is the running program on the controller that must be maintained in a big loop which should contain a waiting time. This temporizing instruction allows external connections to modify some variables of the program through the RAP protocol. In our case, the adaptor updates the received counter in one persistent variable and waits the updated value in another variable (see Appendix). Thus, in experiment 4, the waiting time is set as the polling rate (200 ms); using a waiting time having the value 0, the controller limits the external interaction and the delay gets bigger (see case 5); by increasing the waiting time to 300 ms, the difference is found in the round trip delay (case 6). Thus, one can see that the best waiting time is equal to the polling rate. By comparing experiment 1 with 4, one can see that the average RTD is increased in the last case with 241 ms, which can be explained by the limitation of the RAP solution. Another comparison for S4C controller (Robot 2) which is older than S4Cplus shows that the round-trip delay is higher (see rows 4 and 7 in Table 1).

In the presented manufacturing process, robots 1 and 2 interact through an I/O digital interface. Switching communication through IoT, the average delay is around 1400 ms, which is more than we expected. Thus, we must admit that a 20 years old controller is not designed for external interactions. By contrast, the new version of controller, namely IRC5 (Robot 3), was used in experiment 9. The entire architecture for external interaction was changed (no polling rate was used), and the results are better than in the first experiment. Similar results were obtained using a virtual controller from RobotStudio (see row 10 in Table 1).

The last two lines of Table 1 contain results for a small device (Wemos D1 WiFi mini) supporting an MQTT client. The two measures at device1 and cloud level are showed in Fig. 7, according to the interaction protocol presented in Fig. 6. Similar plots were obtained with different devices. One can observe two aspects: the delays are mainly produced by the communication supported by of the industrial equipment and this limit can be overcome only with local integration. In conclusion, the proposed approach opens an easy way to develop interaction among old and new devices without a high cost of implementation.

Fig. 7.
figure 7

Comparison of cloud based (blue) and device based (red) measures using Wemos D1 as device2

Full size image

6 Conclusions

The presented approach strengthens the vision of IoT concept by allowing device to device sharing information. This improves coordination and collaboration by endorsing solutions from distributed artificial intelligence. Time-related performance of round-trip message depends on the external communication supported by each manufacturer. However, this issue can be overcome by locally connecting external IoT devices with I/O capabilities and by using the fast MQTT protocol for devices with considerable delays; for example, a robot can publish events through its I/O system.

Without changing the current manufacturing equipment, a production system can be closer to Industry 4.0 by adopting/developing IoT adapters. Generally speaking, it will not be the case to entirely remove the local I/O interaction in order to obtain a full IoT system, as we did in this research. The local connections existing in the classical manufacturing systems can be kept, but when there is a need for trying new hardware and software solutions, without disrupting the system configuration, the proposed IoT approach can be of great help. Preventive maintenance, context-aware systems, digital twin and digital product are challenges that can be integrated into classical manufacturing systems. Our experiments revealed that a digital twin is achievable by introducing IoT digital twin devices in the system.

Moving the entire communication through cloud is a complication for the proposed solution. About this, fog computing and IoT gateway have to be further considered.