Human–robot collisions detection for safe human–robot interaction using one multi-input–output neural network

Abstract

In this paper, a multilayer feedforward neural network-based approach is proposed for human–robot collision detection taking safety standards into consideration. One multi-output neural network is designed and trained using data from the coupled dynamics of the manipulator with and without external contacts to detect unwanted collisions and to identify the collided link using only the intrinsic joint position and torque sensors of the manipulator. The proposed method is applied to the collaborative robots, which will be very popular in the near future, and is implemented and evaluated in 3D space motion taking into account the effect of the gravity. KUKA LWR manipulator is an example of the collaborative robots, and it is used for doing the experiments. The experimental results prove that the developed system is considerably efficient and very fast in detecting the collisions in the safe region and identifying the collided link along the entire workspace of the three-joint motion of the manipulator. Separate/uncoupled neural networks, one for each joint, are also designed and trained using the same data, and their performance is compared with the coupled one.

Appendices

Appendix 1: Discussion of the dynamic coupling

1.1 First part: dynamic parameters of the manipulator

The inertia matrix, Coriolis and centrifugal matrix, and gravity vector of the three-link manipulator from (1) are calculated in Murray et al. (1994) as

$$ M\left( \theta \right) = \left[ {\begin{array}{*{20}c} {M_{11} } \\ 0 \\ 0 \\ \end{array} \begin{array}{*{20}c} 0 \\ {M_{22} } \\ {M_{32} } \\ \end{array} \begin{array}{*{20}c} { 0} \\ { M_{23} } \\ { M_{33} } \\ \end{array} } \right] $$

(3)

where

$$ \begin{aligned} M_{11} & = I_{y2} s_{2}^{2} + I_{y3} s_{23}^{2} + I_{z1} + I_{z2} c_{2}^{2} + I_{z3} c_{23}^{2} + m_{2} r_{2}^{2} c_{2}^{2} + m_{3} \left( {l_{2} c_{2} + r_{3} c_{23} } \right)^{2} \\ M_{22} & = I_{x2} + I_{x3} + m_{3} l_{2}^{2} + m_{2} r_{2}^{2} + m_{3} r_{3}^{2} + 2m_{3} l_{2} r_{3} c_{3} \\ M_{23} & = I_{x3} + m_{3} r_{3}^{2} + m_{3} l_{2} r_{3} c_{3} \\ M_{32} & = I_{x3} + m_{3} r_{3}^{2} + m_{3} l_{2} r_{3} c_{3} \\ M_{33} & = I_{x3} + m_{3} r_{3}^{2} \\ \end{aligned} $$

$$ C\left( {\theta ,\dot{\theta }} \right) = \left[ {\begin{array}{*{20}c} {C_{11} } \\ {C_{21} } \\ {C_{31} } \\ \end{array} \begin{array}{*{20}c} {C_{12} } \\ {C_{22} } \\ {C_{32} } \\ \end{array} \begin{array}{*{20}c} { C_{13} } \\ { C_{23} } \\ { 0} \\ \end{array} } \right] $$

(4)

where

$$ \begin{aligned} C_{11} & = \left( {\left( {I_{y2} - I_{z2} - m_{2} r_{2}^{2} } \right)c_{2} s_{2} + \left( {I_{y3} - I_{z3} } \right)c_{23} s_{23} - m_{3} \left( {l_{2} c_{2} + r_{3} c_{23} } \right)\left( {l_{2} s_{2} + r_{3} s_{23} } \right)} \right)\dot{\theta }_{2} + \left( {\left( {I_{y3} - I_{z3} } \right)c_{23} s_{23} - m_{3} r_{3} s_{23} \left( {l_{2} c_{2} + r_{3} c_{23} } \right)} \right)\dot{\theta }_{3} \\ C_{12} & = \left( {\left( {I_{y2} - I_{z2} - m_{2} r_{2}^{2} } \right)c_{2} s_{2} + \left( {I_{y3} - I_{z3} } \right)c_{23} s_{23} - m_{3} \left( {l_{2} c_{2} + r_{3} c_{23} } \right)\left( {l_{2} s_{2} + r_{3} s_{23} } \right)} \right)\dot{\theta }_{1} \\ C_{13} & = \left( {\left( {I_{y3} - I_{z3} } \right)c_{23} s_{23} - m_{3} r_{3} s_{23} \left( {l_{2} c_{2} + r_{3} c_{23} } \right)} \right)\dot{\theta }_{1} \\ C_{21} & = \left( {\left( {I_{z2} - I_{y2} + m_{2} r_{2}^{2} } \right)c_{2} s_{2} + \left( {I_{z3} - I_{y3} } \right)c_{23} s_{23} + m_{3} \left( {l_{2} c_{2} + r_{3} c_{23} } \right)\left( {l_{2} s_{2} + r_{3} s_{23} } \right)} \right)\dot{\theta }_{1} \\ C_{22} & = - l_{2} m_{3} r_{3} s_{3} \dot{\theta }_{3} \\ C_{23} & = - l_{2} m_{3} r_{3} s_{3} \dot{\theta }_{2} - l_{2} m_{3} r_{3} s_{3} \dot{\theta }_{3} \\ C_{31} & = \left( {\left( {I_{z3} - I_{y3} } \right)c_{23} s_{23} + m_{3} r_{3} s_{23} \left( {l_{2} c_{2} + r_{3} c_{23} } \right)} \right)\dot{\theta }_{1} \\ C_{32} & = l_{2} m_{3} r_{3} s_{3} \dot{\theta }_{2} \\ \end{aligned} $$

$$ G\left( \theta \right) = \left[ {\begin{array}{*{20}c} 0 \\ { - \left( {m_{2} gr_{2} + m_{3} gl_{2} } \right)\cos \theta_{2} - m_{3} gr_{3} \cos \left( {\theta_{2} + \theta_{3} } \right)} \\ { - m_{3} gr_{3} \cos \left( {\theta_{2} + \theta_{3} } \right)} \\ \end{array} } \right] $$

(5)

Equations (1) and from (3) to (5) show the coupling that occurs between the joints. From (3), it is noted that the inertial force of link 3 affects the measured joint torques and the external torques of the joints 2 and 3. The measured joint torque and the external torque of joint 1 are dependent on the masses of the link 2 and the link 3 “end-effector” ($m_{2} , m_{3}$), but do not depend on their accelerations. It should be noted that this case is a special selection of the joints, but in other cases, there might be effect on the measured/external torques of joint 1 due to the inertial forces of the links 2 and 3. The inertial force of the link 2 affects the measured joint torques and the external torques of joint 2 and not joint 3, whereas the inertial force of the link 1 affects only the measured joint torque and the external torque of joint 1 and not joints 2 and 3.

The Coriolis and centrifugal terms presented in (4) show the coupling occurs between the three joints. The Coriolis forces of link 1 or 2 affect the measured joint torques and external torques of the three joints, whereas the Coriolis forces of link 3 affect only the measured joint torques and external torques of the joints 1 and 2 and not joint 3. From (5), it is clear that the gravitational force of link 3 affects the measured joint torques and external torques of both joints 2 and 3 which prove the coupling between these two joints. The gravitational force of link 2 affects only the measured joint torque and external torque of joint 2, whereas there is no gravitational force of link 1.

1.2 Second part: results of the experiments that investigate the coupling

The results of the 18 experiments done to prove the dynamic coupling between the joints, as shown in Fig. 2, are presented in Table 7 where the maximum amplitudes of the measured joint torques $\left( {\tau_{{{\text{msr}}1}} , \tau_{{{\text{msr}}2}} , \tau_{{{\text{msr}}3}} } \right)$ and the external torques $\left( {\tau_{{{\text{ext}}1}} , \tau_{{{\text{ext}}2}} , \tau_{{{\text{ext}}3}} } \right)$ at the time when the obstacle collides with the manipulator are included to summarize the effect on the torques of the joints 1, 2, and 3.

Table 7 The results of the 18 experiments to investigate the joint coupling

In Table 7, “Yes” means that the effect on the measured torque and the external torque of the joint $i$ ($i = 1, 2, 3$) is high or clear, when the collision between the obstacle and the manipulator occurs, whereas “No” means that it is very small and negligible. For the effect of the collision on the measured/external torques of the joints 1 and 2, “Yes” is chosen for the all cases/configurations, since it is obvious that this effect is high. For the effect of the collision on the measured/external torques of the joint 3, “Yes” is chosen only for the first configuration where the effect of the inertial forces is high. This configuration is taken as a reference configuration to choose “Yes” or “No” for the other, the second and the third, configurations where there is very small effect from the inertial forces. In the second and the third configurations, the percentage of the external torque of this joint (joint 3) to the external torque of joint 1, $\frac{{\tau_{{{\text{ext}}3}} }}{{\tau_{{{\text{ext}}1}} }}$ or the percentage of the measured torque of joint 3 to the measured torque of joint 1, $\frac{{\tau_{msr3} }}{{\tau_{msr1} }}$, is compared with a threshold and if this percentage is higher than or equal to this threshold then “Yes” is chosen otherwise “No.” This threshold is defined as 80% from $\frac{{\tau_{{{\text{ext}}3}} }}{{\tau_{{{\text{ext}}1}} }}$ or 80% from $\frac{{\tau_{{{\text{msr}}3}} }}{{\tau_{{{\text{msr}}1}} }}$ which is calculated from the first/reference configuration. This threshold is calculated for both weights (1 kg and 2 kg), and the lowest value is chosen. The threshold also is changing with the increase/change in the speed. It should be noted that the measured or the external torque of the joint 1 is taken as a reference torque because it has the highest value or in another meaning it is the most affected one by the collision. The selected threshold is presented in Table 8.

Table 8 The threshold value (%) to choose “Yes” or “No” for the effect of the collision, between the obstacle and the manipulator, on the torques of joint 3 for the second and the third configurations

Appendix 2: Protocol of the experiments in the paper

The protocol for the entire experiments executed in this paper is presented and discussed in Table 9. The all experiments, except experiment 6, are used with the proposed coupled NN. Experiments 1, 2, and 3 are used also for training and evaluation the uncoupled NNs, and experiments 4 and 6 are used for their verification as discussed in Sect. 7. The desired sinusoidal motion $\theta_{\text{des}} \left( t \right)$ for the three joints simultaneously is chosen as

$$ \theta_{\text{des}} \left( t \right) = \frac{\pi }{4} - \frac{\pi }{4}\cos \left( {2\pi ft} \right) $$

(6)

where $f$ is the frequency of the sinusoidal motion.

Table 9 The steps and protocol for all the discussed experimental work in the paper

Appendix 3: The training process of the NN

After trying different weights’ initializations with different hidden neurons (40, 50, 70, 90, 120, 150, and 170), the best performance for the coupled NN and uncoupled NNs that give us the minimum squared error (MSE) and the proper collision thresholds is shown in Figs. 20 and 21, respectively. For training whether the coupled NN or uncoupled ones, the best performance occurs after using 150 hidden neurons and 1000 iterations.

Appendix 4: Latencies from the sensors

As discussed in the paper, MATLAB Toolbox is used for the training process of the NN. The training process of the coupled NN is implemented offline on AMD Ryzen 5 1600 Six-Core processor since too many data require good hardware with high RAM to make the training fast. After the training process is finished, the coupled trained NN is implemented on KUKA LWR robot to check online its efficiency and generalization. The collision detection time, presented in Table 2, comes/arises from the following time:

• 800 µs is required to read from the internal joints torque, position, and velocity sensors as well as to send motion commands to the robot.

• 145 µs is required to read from the external force sensor (its latency 144 µs).

• 6 µs is required for the NN calculations.

• 1 ms (sampling rate) − (800 + 145 + 6) µs = 49 µs is required to other calculations and algorithmic commands.

This process is illustrated in Fig. 22. The required time for reading from the sensors of KUKA, as presented, is less than 1 ms (the sampling rate), so the collision detection time, which is presented in Table 2, is enough for reaction. Therefore, safe human–robot interaction is achieved.

Fig. 22

The time of the occurring loop in KRC