Multimodal information fusion for urban scene understanding

Abstract

This paper addresses the problem of scene understanding for driver assistance systems. To recognize the large number of objects that may be found on the road, several sensors and decision algorithms have to be used. The proposed approach is based on the representation of all available information in over-segmented image regions. The main novelty of the framework is its capability to incorporate new classes of objects and to include new sensors or detection methods while remaining robust to sensor failures. Several classes such as ground, vegetation or sky are considered, as well as three different sensors. The approach was evaluated on real publicly available urban driving scene data.

Appendix A: Decision making

In our case of study, several arguments can be stated in favor of the optimistic strategy. It is often more conclusive than the pessimistic strategy, coherent with frame refinement and computationally efficient. As shown by Barnett [4], given $k$ plausibility functions, finding the singleton with maximum plausibility of the combined function only needs $O(k|\varOmega |)$ operations, while for the belief it is necessary to do $O(|\varOmega |^k)$ operations. We indeed have the following property:

$$\begin{aligned} pl_{1,2}(\{\omega \})&= \frac{1}{1-\kappa }pl_1(\{\omega \})pl_2(\{\omega \}) \end{aligned}$$

(34a)

$$\begin{aligned}&\propto pl_1(\{\omega \})pl_2(\{\omega \}), \quad \forall \omega \in \varOmega . \end{aligned}$$

(34b)

To compute the pignistic probabilities, the combined mass functions need to be explicitly computed, which requires a number of operations exponential in $|\varOmega |$.

To show the differences between different decision-making strategies, let us consider the following mass function defined on :

$$\begin{aligned} m^\varOmega (\{\text {grass, road}\}) = 0.2, \end{aligned}$$

(35a)

(35b)

(35c)

Table 7 shows the beliefs, plausibilities and pignistic probabilities on the singletons. Here, the pessimistic strategy cannot lead to any decision: actually, in the worst case scenario, any decision could be wrong given the current mass function. Choosing {grass} instead of would be wrong if the masses and were actually entirely related to . Inversely, the other decision would also be wrong if the same masses were now related respectively to {grass} and {road}. On the other hand, both $pl^\varOmega$ and Bet$P^\varOmega$ would lead to , which seems quite reasonable.

Table 7 bel$^\varOmega$, $pl^\varOmega$ and Bet$P^\varOmega$ from mass function (35)

Now, if the singleton is refined into {tree, obstacle, sky}, the mass function (35) will simply be rewritten as:

$$\begin{aligned}&m^\varTheta (\{\text {grass, road}\}) = 0.2,\end{aligned}$$

(36a)

$$\begin{aligned}&m^\varTheta (\{\text {grass, tree, obstacle, sky}\}) = 0.3, \end{aligned}$$

(36b)

$$\begin{aligned}&m^\varTheta (\{\text {road, tree, obstacle, sky}\}) = 0.5. \end{aligned}$$

(36c)

Table 8 shows the measures induced by this new mass function. Following Bet$P^\varTheta$, the decision is changed and now leads to {road}. In contrast, the plausibility criterion does not discriminate between {tree}, {obstacle} and {sky}, which are still more plausible than {grass} and {road}. The optimistic strategy thus remains coherent with its previous decision.

Table 8 bel$^\varTheta$, $pl^\varTheta$ and Bet$P^\varTheta$ from mass function (36)