A hierarchical parallel fusion framework for egocentric ADL recognition based on discernment frame partitioning and belief coarsening

Abstract

Recently, egocentric activity recognition has become a major research area in pattern recognition and artificial intelligence due to its high significance in potential applications in medical care, rehabilitation, smart home/office, etc. In this study, we develop a hierarchical parallel multimodal fusion framework for the recognition of egocentric activities in daily living (ADL). This framework uses the Dezert–Smarandache theory and is constructed around three modalities: location, motion and vision data from a wearable hybrid sensor system. The reciprocal distance and a trained support vector machine classifier are used to form the basic belief assignments (BBA) of location and motion. For vision data composed of egocentric photo streams, a well-trained convolutional neural network is utilized to produce a set of textual tags and the entropy-based statistics for these tags are used to construct the vision BBA. Discernment partitioning and belief coarsening theory are adopted for the hierarchical fusion of the three BBA functions from different ADL levels. Experimental results show that the recognition accuracy of the proposed fusion method was significantly higher than that of the methods based on single modality or modality combinations when our method was applied to real-life multimodal egocentric activity datasets. In addition, our method also achieved higher adaptability and scalability.

Appendix

In this appendix, we will show how the vision basic belief assignment (BBA) function $m_{V} (\bullet)$ is transformed by BBA coarsening to the location and motion level to obtain the coarsened BBA functions $m_{V}^{L} (\bullet)$ and $m_{V}^{M} (\bullet)$.

In Sect. 4.4, we have given their discernment frames based on the most appropriate activity of daily living (ADL) classification modes for each of the three modalities, i.e., location, motion and vision. We rewrite them as

$$\left\{ \begin{gathered} \Theta_{L} = \{ {\text{"HM",}}\;{{\text{"WP",}}}\;{\text{"OP"}}\} \hfill \\ \Theta_{M} = \;\{ {\text{"LY",}}\;{\text{"SD",}}\;{{\text{"ST",}}}\;{{\text{"WK"}}}\} \hfill \\ \Theta_{V} = \,\{ {\text{"}}CN{\text{"}},\;{\text{"}}CU{\text{"}},\;{\text{"}}ET{\text{"}},\;{\text{"EM"}},\;{\text{"}}MT{\text{"}},\;{\text{"}}RD{\text{"}},\;{\text{"}}SP{\text{"}},\;{\text{"SL"}}, \hfill \\ \;\quad \quad \;\,{\text{"}}TK{\text{"}},\;{\text{"}}TU{\text{"}},\,{\text{"}}TP{\text{"}},\;{\text{"}}WO{\text{"}},\;{\text{"}}WU{\text{"}},\;{\text{"}}TV{\text{"}},\;{\text{"}}WT{\text{"}}\} \hfill \\ \end{gathered} \right.$$

(13)

where $\Theta_{L}$, $\Theta_{M}$, and $\Theta_{V}$ represent the location discernment frame, motion frame, and vision frame, respectively. In terms of hierarchy, the activities in $\Theta_{L}$ and $\Theta_{M}$ are high-level activities, and the activities in $\Theta_{V}$ can be regarded as low-level refinements of the activities in $\Theta_{L}$ and $\Theta_{M}$.

As some of the activities in $\Theta_{V}$ are ambiguous when they are divided into high-level frames, they cannot meet the frame partitioning requirement in Eq. (4). For example, “computer use” can occur both at home and in the workplace, and “reading” can be performed both when sitting and standing. Therefore, when $\Theta_{V}$ maps to $\Theta_{L}$ or $\Theta_{M}$, the ambiguous activities are further refined. The refinements $\Omega_{V}^{L}$ and $\Omega_{V}^{M}$ corresponding to $\Theta_{L}$ and $\Theta_{M}$ can be written as

$$\begin{gathered} \Omega_{V}^{L} = \{ {\text{"}}CN{{\text{",}}}\;{{\text{"}}}CU_{{{\text{HM}}}} {{\text{",}}}\;{{\text{"CU}}}_{{{\text{WP}}}} {{\text{",}}}\;{{\text{"}}}ET_{{{\text{HM}}}} {{\text{",}}}\;{{\text{"}}}ET_{{{\text{OP}}}} {\text{",}}\;{\text{"EM"}},\;{\text{"}}MT{\text{"}},\;{\text{"}}RD_{{{\text{HM}}}} {{\text{"}}},\;{\text{"}}RD_{{{\text{WP}}}} {{\text{"}}},\;{\text{"}}SP{\text{"}},\;{\text{"SL",}} \hfill \\ \quad \quad \quad {\text{"}}TK_{{{\text{HM}}}} {\text{",}}\;{\text{"}}TK_{{{\text{WP}}}} {\text{",}}\;{\text{"}}TU_{{{\text{HM}}}} {\text{",}}\;{\text{"}}TU_{{{\text{WP}}}} {\text{",}}\;{\text{"}}TU_{{{\text{OP}}}} {\text{",}}\;{\text{"}}TP{\text{",}}\;{\text{"}}WO{\text{",}}\;{\text{"}}WU_{{{\text{HM}}}} {\text{",}}\;{\text{"}}WU_{{{\text{OP}}}} {\text{",}}\;{\text{"}}TV{\text{",}} \hfill \\ \quad \quad \quad {\text{"}}WT_{{{\text{HM}}}} {{\text{"}}},\;{\text{"}}WT_{{{\text{WP}}}} {\text{"}}\} \hfill \\ \end{gathered}$$

(14)

$$\begin{gathered} \Omega_{V}^{M} = \{ {\text{"}}CN{\text{",}}\;{\text{"}}CU{\text{",}}\;{\text{"}}ET_{{{\text{SD}}}} {\text{",}}\;{\text{"}}ET_{{{\text{ST}}}} {\text{",}}\;{\text{"EM",}}\;{\text{"}}MT{\text{",}}\;{\text{"}}RD_{{{\text{SD}}}} {\text{",}}\;{\text{"}}RD_{{{\text{ST}}}} {\text{",}}\;{\text{"}}SP{\text{",}}\;{\text{"S}}L{\text{",}}\;{\text{"}}TK_{{{\text{SD}}}} {\text{",}}\; \hfill \\ \quad \quad \;\;\;{\text{"}}TK_{{{\text{ST}}}} {\text{",}}\;{\text{"}}TU_{{{\text{SD}}}} {\text{",}}\;{\text{"}}TU_{{{\text{ST}}}} {\text{",}}\;{\text{"}}TU_{{{\text{LY}}}} {\text{",}}\;{\text{"}}TP{\text{",}}\;{\text{"}}WO{\text{",}}\;{\text{"}}WU{\text{",}}\;{\text{"}}TV_{{{\text{SD}}}} {\text{",}}\;{\text{"}}TV_{{{\text{ST}}}} {\text{",}}\;{\text{"}}WT{\text{"}}\} \hfill \\ \end{gathered}$$

(15)

where the subscripts denote the refined activity; for example, CU_HM and CU_WP denote computer use at home and in the workplace, respectively, and RD_SD and RD_ST denote reading when sitting and standing, respectively. After the refinement sets have been acquired, we can define the mappings $\omega_{L \to V} :D^{{\Theta_{L} }} \to D^{{\Omega_{V}^{L} }}$ and $\omega_{M \to V} :D^{{\Theta_{M} }} \to D^{{\Omega_{V}^{M} }}$ for partitioning the refinement frame into the high-level frame:

$$\left\{ \begin{gathered} \omega_{L \to V} (\{ {\text{"HM"}}\} ) = \{ {\text{"}}CN{\text{",}}\;{\text{"}}CU_{{\text{HM}}} {{\text{",}}}\;{\text{"}}ET_{\text{HM}} {\text{",}}\;{\text{"EM",}}\;{\text{"}}RD_{{\text{HM}}} {{\text{",}}}\;{\text{"SL",}}\;{\text{"}}TK_{{\text{HM}}} {{\text{",}}}\;{\text{"}}TU_{{\text{HM}}} {{\text{",}}}\;{\text{"WO}}_{{\text{HM}}} {{\text{",}}} \hfill \\ \quad \quad \quad \quad \quad \quad \quad \;\;{\text{"}}WU_{{{\text{HM}}}} {{\text{",}}}\;{\text{"}}TV_{\text{HM}} {{\text{"}}}\} \hfill \\ \omega_{L \to V} (\{ {\text{"WP"}}\} ) = \{ {\text{"CU}}_{{\text{WP}}} {{\text{",}}}\;{\text{"ET}}_{{\text{WP}}} {{\text{",}}}\;{\text{"}}MT{\text{",}}\;{\text{"}}RD_{{{\text{WP}}}} {\text{",}}\;{\text{"}}TK_{{{\text{WP}}}} {\text{",}}\;{\text{"}}TU_{{{\text{WP}}}} {\text{",}}\;{\text{"WO}}_{{{\text{WP}}}} {\text{",}}\;{\text{"}}WT{\text{"}}\} \hfill \\ \omega_{L \to V} (\{ {\text{"OP"}}\} ) = \{ {\text{"}}ET_{{{\text{OP}}}} {\text{",}}\;{\text{"}}SP{\text{",}}\;{\text{"TK}}_{{{\text{OP}}}} {\text{",}}\;{\text{"}}TU_{{{\text{OP}}}} {\text{",}}\;{\text{"}}TP{\text{",}}\;{\text{"}}WO_{{{\text{OP}}}} {\text{",}}\;{\text{"}}WU_{{{\text{OP}}}} {\text{",}}\;{\text{"TV}}_{{{\text{OP}}}} {\text{"}}\} \hfill \\ \end{gathered} \right.$$

(16)

$$\left\{{\begin{array}{*{20}l}{\omega _{{M \to V}} (\{ {\text{"LY"}}\} ) = \{ {\text{"SL"}},\;{\text{"TU}}_{{{\text{LY}}}}{{\text{"}}} \} } \\ {\omega _{{M \to V}} (\{{\text{"SD"}}\} ) = \{ {\text{"}}CU{\text{"}},\;{\text{"}}ET_{{{\text{SD}}}}{\text{"}},\; {\text{"EM}}_{{{\text{SD}}}}{\text{"}},\; {\text{"MT"}},\;{\text{"}}RD_{{{\text{SD}}}}{\text{"}},\;{\text{"}}TK_{{{\text{SD}}}}{\text{"}},\;{\text{"}}TU_{{{\text{SD}}}}{\text{"}},\;{\text{"TP}}_{{{\text{SD}}}}{\text{"}} ,\;{\text{"}}TV{\text{"}},\;{\text{"}}WT{\text{"}}\} } \\ {\omega _{{M \to V}} (\{{\text{"ST"}}\} ) = \{ {\text{"CN"}},\; {\text{"ET}}_{{{\text{ST}}}}{\text{"}},\;{\text{"EM}}_{{{\text{ST}}}}{\text{"}},\;{\text{"}}RD_{{{\text{ST}}}}{\text{"}},\; {\text{"SP}}_{{{\text{ST}}}}{\text{"}}\;{\text{"}}TK_{{{\text{ST}}}}{\text{"}},\;{\text{"}}TU_{{{\text{ST}}}}{\text{"}},\;{\text{"TP}}_{{{\text{ST}}}}{\text{"}},\;{\text{"WU"}}\} } \\ {\omega _{{M \to V}} (\{ {\text{"WK"}}\} ) = \{ {\text{"SP}}_{{{\text{WK}}}}{\text{"}},\;{\text{"}}WO{\text{"}}\} } \\ \end{array} } \right.$$

(17)

According to (4), BBA coarsening can be performed on the basis of the frame partition mapping $\omega$. Let $m_{V}^{L} (\bullet)$ be a BBA function defined on frame $\Omega_{V}^{L}$. Taking $m_{V}^{L} ({\text{"}}HM{\text{"}})$ as an example, we substitute the frame partition mapping $\omega_{L \to V} (\{ {\text{"HM"}}\} )$ into (4) to obtain

$${m}_{V}^{L}\left(\text{"HM"}\right)={\mathcal{m}}_{{\Theta }_{L}}({\omega }_{L \to V}(\{"\text{HM"}\}))=$$

$$\begin{gathered} \max \{ m_{V}^{L} {(}\beta {)}\;{|}\;\beta \subseteq \omega_{L \to V} (\{ {\text{"HM"}}\} )\;{\text{and}}\;\beta \in \Theta_{L}^{*} \} = \hfill \\ \max \{ m_{V}^{L} ({\text{"}}CN{\text{"}}),\;m_{V}^{L} ({\text{"}}CU_{{{\text{HM}}}} {\text{"}}),\;m_{V}^{L} ({\text{"}}ET_{{{\text{HM}}}} {\text{"}}),\;m_{V}^{L} ({\text{"EM"),}}\;m_{V}^{L} ({\text{"}}RD_{{{\text{HM}}}} {\text{"}}),\;m_{V}^{L} ({\text{"SL"}}),\; \hfill \\ \quad \quad m_{V}^{L} ({\text{"}}TK_{{{\text{HM}}}} {\text{"}}),\;m_{V}^{L} ({\text{"}}TU_{{{\text{HM}}}} {\text{"}}),\;m_{V}^{L} ({\text{"WO}}_{{{\text{HM}}}} {\text{"}}),\;m_{V}^{L} ({\text{"}}WU_{{{\text{HM}}}} {\text{"}}),\;m_{V}^{L} ({\text{"}}TV_{{{\text{HM}}}} {\text{"}})\} \hfill \\ \end{gathered}$$

(18)

As $\Omega_{V}^{L}$ is obtained by refining some activities in $\Theta_{V}$, $\Theta_{V}$ should be a coarsening set of $\Omega_{V}^{L}$; therefore, $\Theta_{V}$ is compatible with $\Omega_{V}^{L}$. According to the description in Sect. 3, their BBA functions are consistent, and therefore, unrefined activities such as CN have the following relations:

$$m_{V}^{L} ({\text{"CN"}}) = m_{V} ({\text{"CN"}})$$

(19)

Furthermore, for refined activities such as CU_HM and CU_WP, the coarsening properties of the BBA function mean that the following expression should be satisfied:

$${m}_{V}({\text{"CU"}})={m}_{V}^{L}\left({\text{"CU"}}\right)={\mathcal{m}}_{{\Theta }_{V}}(\{{\text{"CU}}_{{\text{HM}}}{{\text{"}}}, \, {\text{"CU}}_{{\text{WP}}}{{\text{"}}}\})=\text{max}\{{m}_{V}^{L}({\text{"CU}}_{{\text{HM}}}{{\text{"}}}), {m}_{V}^{L} ({\text{"CU}}_{{\text{WP}}}{{\text{"}}})\}$$

(20)

From the discussion in Sect. 4.3, we can see that the proposed vision-based activity recognition method does not distinguish the locations at which activities occur. In other words, the same activity occurring at different locations has the same belief assignment, that is,

$$m_{V}^{L} ({\text{"CU}}_{{{\text{HM}}}}{\text{"}}) = m_{V}^{L} ({\text{"CU}}_{{{\text{WP}}}}{\text{"}})$$

(21)

From (20), we obtain

$$m_{V}^{L} ({\text{"CU}}_{{{\text{HM}}}} {\text{"}}) = m_{V}^{L} ({\text{"CU}}_{{{\text{WP}}}} {\text{"}}) = m_{V} ({\text{"CU"}})$$

(22)

By treating other refined and unrefined activities similarly, (22) can be rewritten as

$$\begin{gathered} m_{V}^{L} ({\text{"HM"}}) = \max \{ m_{V} ({\text{"}}CN{\text{"}}),\;m_{V} ({\text{"}}CU{\text{"}}),\;m_{V} ({\text{"}}ET{\text{"}}),\;m_{V} ({\text{"EM"),}}\;m_{V} ({\text{"}}RD{\text{"}}),\;m_{V} ({\text{"SL"}}),\;m_{V} ({\text{"}}TK{\text{"}}),\; \hfill \\ \quad \quad \quad \quad \quad \quad \quad \;\;m_{V} ({\text{"}}TU{\text{"}}),\;m_{V} ({\text{"WO"}}),\;m_{V} ({\text{"WU"}}),\;m_{V} ({\text{"TV"}})\} \hfill \\ \end{gathered}$$

(23)

Similarly, $m_{V}^{L} ({\text{"WP"}})$ and $m_{V}^{L} ({\text{"OP"}})$ are given by

$$\begin{gathered} m_{V}^{L} ({\text{"WP"}}) = \max \{ m_{V} ({\text{"CU"}}),\;m_{V} ({\text{"ET"}}),\;m_{V} ({\text{"}}MT{\text{"}}),\;m_{V} ({\text{"}}RD{\text{"}}),\;m_{V} ({\text{"}}TK{\text{"}}),\;m_{V} ({\text{"}}TU{\text{"}}), \hfill \\ \quad \quad \quad \quad \quad \quad \quad \;m_{V} ({\text{"WO"}}),\;m_{V} ({\text{"}}WT{\text{"}})\} \hfill \\ m_{V}^{L} ({\text{"OP")}} = \max \{ m_{V} ({\text{"}}ET{\text{"}}),\;m_{V} ({\text{"}}SP{\text{"}}),\;m_{V} ({\text{"TK"}}),\;m_{V} ({\text{"}}TU{\text{"}}),\;m_{V} ({\text{"}}TP{\text{"}}),\;m_{V} ({\text{"}}WO{\text{"}}), \hfill \\ \quad \quad \quad \quad \quad \quad \quad \;m_{V} ({\text{"}}WU{\text{"}}),\;m_{V} ({\text{"TV"}})\} \hfill \\ \end{gathered}$$

(24)

Equations (23) and (24) are the results of the coarsening of $m_{V}^{L} ( \bullet )$. They indicate that by ignoring the difference among the belief assignments of the same activity at different locations, the belief assignment of the element of $\Theta_{L}$ in $\Omega_{V}^{L}$ is equal to the maximum belief assignment of all elements of $\omega_{L \to V} ( \bullet )$ in $m_{V} ( \bullet )$.

Let $m_{V}^{M} ( \bullet )$ be the BBA function defined on frame $\Omega_{V}^{M}$. By using almost the same method as for the coarsening of $m_{V}^{L} ( \bullet )$, a similar conclusion can be drawn: the belief assignment of the element of $\Theta_{V}$ in $\Omega_{V}^{M}$ is equal to the maximum belief assignment of all the elements of $\omega_{M \to V} ( \bullet )$ in $m_{V} ( \bullet )$, that is,

$$\left\{ \begin{gathered} m_{V}^{M} ({\text{"LY"}}) = \max \{ m_{V} ({\text{"SL"}}),\;m_{V} ({\text{"}}TU{\text{"}})\} \hfill \\ m_{V}^{M} ({\text{"SD"}}) = \max \{ m_{V} ({\text{"}}CU{\text{"}}),\;m_{V} ({\text{"}}ET{\text{"}}),\;m_{V} ({\text{"EM"),}}\;m_{V} ({\text{"}}MT{\text{"}}),\;m_{V} ({\text{"}}RD{\text{"}}),\;m_{V} ({\text{"}}TK{\text{"}}),\; \hfill \\ \quad \quad \quad \quad \quad \quad \quad \;\;m_{V} ({\text{"}}TU{\text{"}}),\;m_{V} ({\text{"}}TP{\text{"}}),\;m_{V} ({\text{"}}TV{\text{"}}),\;m_{V} ({\text{"}}WT{\text{"}})\} \hfill \\ m_{V}^{M} ({\text{"ST"}}) = \max \{ m_{V} ({\text{"}}CN{\text{"}}),\;m_{V} ({\text{"}}ET{\text{"}}),\;m_{V} ({\text{"EM"),}}\;m_{V} ({\text{"RD"}}),\;m_{V} ({\text{"}}SP{\text{"}}),\;m_{V} ({\text{"}}TK{\text{"}}), \hfill \\ \quad \quad \quad \quad \quad \quad \quad \;\,m_{V} ({\text{"}}TU{\text{"}}),\;m_{V} ({\text{"TP"}}),\;m_{V} ({\text{"}}WU{\text{"}})\} \hfill \\ m_{V}^{M} ({\text{"WK"}}) = \max \{ m_{V} ({\text{"}}SP{\text{"}}),\;m_{V} ({\text{"WO"}})\} \hfill \\ \end{gathered} \right.$$

(25)