Data Association for Semantic World Modeling from Partial Views

Abstract

Autonomous mobile-manipulation robots need to sense and interact with objects to accomplish high-level tasks such as preparing meals and searching for objects. To achieve such tasks, robots need semantic world models, defined as object-based representations of the world involving task-level attributes. In this work, we address the problem of estimating world models from semantic perception modules that provide noisy observations of attributes. Because attribute detections are sparse, ambiguous, and are aggregated across different viewpoints, it is unclear which attribute measurements are produced by the same object, so data association issues are prevalent. We present novel clustering-based approaches to this problem, which are more efficient and require less severe approximations compared to existing tracking-based approaches. These approaches are applied to data containing object type-and-pose detections from multiple viewpoints, and demonstrate comparable quality to the existing approach using a fraction of the computation time.

Appendix: Posterior and Predictive Distributions for a Single Light

In this appendix, we verify the claim from Sect. 2 that finding the posterior and predictive distributions on color and location for a single light is straightforward, given that we know which observations were generated by that light. Let $\left\{ (o, x) \right\}$ denote the set of light color-location detections that correspond to a light with unknown parameters (c, l). Color and location measurements are assumed to be independent given (c, l) and will be considered separately. We assume a known discrete prior distribution $\pi \in \varDelta ^{(T-1)}$ on colors, reflecting their relative prevalence. Using the color noise model (Eq. 1), the posterior and predictive distributions on c are:

$$\begin{aligned} \mathbb {P}\left( c \,\big |\, \left\{ o \right\} \right) \propto \left[ \prod _o \phi ^c_o \right] \times \pi _c ; \quad \mathbb {P}\left( o' \,\big |\, \left\{ o \right\} \right)&= \sum _{c=1}^T \mathbb {P}\left( o' \big | c \right) \; \mathbb {P}\left( c \big | \left\{ o \right\} \right) \nonumber \\&= \sum _{c=1}^T \phi ^c_{o'} \; \mathbb {P}\left( c \,\big |\, \left\{ o \right\} \right) . \end{aligned}$$

(15)

We can use this to find the light’s probability of detection:

$$\begin{aligned} p_\text {D} \triangleq 1 - \mathbb {P}\left( o'=0 \,\big |\, \left\{ o \right\} \right) = 1 - \sum _{c=1}^T \phi ^{c}_0 \; \mathbb {P}\left( c \,\big |\, \left\{ o \right\} \right) . \end{aligned}$$

(16)

Unlike the constant false positive rate $p_{\text {FP}}$, the detection (and false negative) rate is dependent on the light’s color posterior.

For location measurements, we emphasize that both the mean $\mu$ and precision $\tau = \frac{1}{\sigma ^2}$ of the Gaussian noise model is unknown. Modeling the variance as unknown allows us to attain a better representation of the location estimate’s empirical uncertainty, and not naïvely assume that repeated measurements give a known fixed reduction in uncertainty each time. We use a standard conjugate prior, the distribution $\text {NormalGamma} (\mu , \tau ; \lambda , \nu , \alpha , \beta )$.⁴ It is well known (e.g., [5]) that after observing n observations with sample mean $\hat{\mu }$ and sample variance $\hat{s}^2$, the posterior is a normal-gamma distribution with parameters:

$$\begin{aligned} \lambda '&= \lambda + n ; \; \nu ' = \frac{\lambda }{\lambda +n} \nu + \frac{n}{\lambda +n} \hat{\mu } ; \; \alpha ' = \alpha + \frac{n}{2} ; \; \beta ' \nonumber \\&= \beta + \frac{1}{2} \left( n\hat{s}^2 + \frac{\lambda n}{\lambda +n} \left( \hat{\mu } - \nu \right) ^2 \right) . \end{aligned}$$

(17)

The upshot of using a conjugate prior for location measurements is that the marginal likelihood of location observations has a closed-form expression. The posterior predictive distribution for the next location observation $x'$ is obtained by integrating out the latent parameters $\mu , \tau$, and has the following expression:

$$\begin{aligned} \mathbb {P}\left( x' \,\big |\, \left\{ x \right\} \,;\, \lambda , \nu , \alpha , \beta \right)&= \int _{(\mu , \tau )} \mathbb {P}\left( x \,\big |\, \mu , \tau \right) \mathbb {P}\left( \mu , \tau \,\big |\, \left\{ x \right\} \, \right) \nonumber \\&= \frac{1}{\sqrt{2 \pi }} \frac{{\beta '}^{\alpha '}}{{\beta ^+}^{\alpha ^+}} \frac{\sqrt{\lambda '}}{\sqrt{\lambda ^+}} \frac{\varGamma (\alpha ^+)}{\varGamma (\alpha ')} , \end{aligned}$$

(18)

where the hyperparameters with ‘$'$’ superscripts are updated according to Eq. 17 using the empirical statistics of $\left\{ x \right\}$ only (excluding $x'$), and the ones with ‘$+$’ superscripts are likewise updated but including $x'$. The ratio in Eq. 18 assesses the fit of $x'$ with the existing observations $\left\{ x \right\}$ associated with the light.