Abstract
It is a well known result in the vision literature that the motion of independently moving objects viewed by an affine camera lie on affine subspaces of dimension four or less. As a result a large number of the recently proposed motion segmentation algorithms model the problem as one of clustering the trajectory data to its corresponding affine subspace. While these algorithms are elegant in formulation and achieve near perfect results on benchmark datasets, they fail to address certain very key real-world challenges, including perspective effects and motion degeneracies. Within a robotics and autonomous vehicle setting, the relative configuration of the robot and moving object will frequently be degenerate leading to a failure of subspace clustering algorithms. On the other hand, while gestalt-inspired motion similarity algorithms have been used for motion segmentation, in the moving camera case, they tend to over-segment or under-segment the scene based on their parameter values. In this paper we present a principled approach that incorporates the strengths of both approaches into a cohesive motion segmentation algorithm capable of dealing with the degenerate cases, where camera motion follows that of the moving object. We first generate a set of prospective motion models for the various moving and stationary objects in the video sequence by a RANSAC-like procedure. Then, we incorporate affine and long-term gestalt-inspired motion similarity constraints, into a multi-label Markov Random Field (MRF). Its inference leads to an over-segmentation, where each label belongs to a particular moving object or the background. This is followed by a model selection step where we merge clusters based on a novel motion coherence constraint, we call in-frame shear, that tracks the in-frame change in orientation and distance between the clusters, leading to the final segmentation. This oversegmentation is deliberate and necessary, allowing us to assess the relative motion between the motion models which we believe to be essential in dealing with degenerate motion scenarios.We present results on the Hopkins-155 benchmark motion segmentation dataset [27], as well as several on-road scenes where camera and object motion are near identical. We show that our algorithm is competitive with the state-of-the-art algorithms on [27] and exceeds them substantially on the more realistic on-road sequences.
Similar content being viewed by others
References
Namdev, R.K., Kundu, A., Krishna, K.M., Jawahar, C.V.: Motion segmentation of multiple objects from a freely moving monocular camera. In: 2012 IEEE International Conference on Robotics and Automation (ICRA), pp. 4092–4099, IEEE (2012)
Lezama, J., Alahari, K., Sivic, J., Laptev, I.: Track to the future: Spatio-temporal video segmentation with long-range motion cues. In: 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE (2011)
Elhamifar, E., Vidal, R.: Sparse subspace clustering. In: IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2009, pp. 2790–2797. IEEE (2009)
Sugaya, Y., Kanatani, K.: Geometric structure of degeneracy for multi-body motion segmentation. In: Statistical Methods in Video Processing, pp. 13–25. Springer, Berlin Heidelberg (2004)
Flores-Mangas, F., Jepson, A.D.: Fast rigid motion segmentation via incrementally-complex local models. In: 2013 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2259–2266. IEEE (2013)
Yan, J., Pollefeys, M.: A general framework for motion segmentation: Independent, articulated, rigid, non-rigid, degenerate and non-degenerate. In: Computer Vision ECCV 2006, pp. 94–106, Springer, Berlin Heidelberg (2006)
Kanatani, K.: Motion segmentation by subspace separation and model selection. Image 1, 1 (2001)
Rao, S.R., Tron, R., Vidal, R., Ma, Y.: Motion segmentation via robust subspace separation in the presence of outlying, incomplete, or corrupted trajectories. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2008. pp. 1–8. IEEE (2008)
Felzenszwalb, P.F., Girshick, R.B., McAllester, D., Ramanan, D.: Object detection with discriminatively trained part-based models. IEEE Transactions on Pattern Analysis and Machine Intelligence 32(9), 1627–1645 (2010)
Fragkiadaki, K., Shi, J.: Figure-ground image segmentation helps weakly-supervised learning of objects. In: Computer Vision ECCV 2010, pp. 561–574. Springer, Berlin Heidelberg (2010)
Isack, H., Boykov, Y.: Energy-based geometric multi-model fitting. Int. J. Comput. Vis. 97(2), 123–147 (2012)
Delong, A., Osokin, A., Isack, H.N., Boykov, Y.: Fast approximate energy minimization with label costs. Int. J. Comput. Vis. 96(1), 1–27 (2012)
Lee, Y.J., Kim, J., Grauman, K.: Key-segments for video object segmentation. In: 2011 IEEE International Conference on Computer Vision (ICCV), pp. 1995–2002. IEEE (2011)
Shi, J., Malik, J.: Normalized cuts and image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence 22(8), 888–905 (2011)
Hartley, R., Zisserman, A.: Multiple view geometry in computer vision. Cambridge university press (2003)
Kolmogorov, V.: Convergent tree-reweighted message passing for energy minimization. IEEE Transactions on Pattern Analysis and Machine Intelligence 28(10), 1568–1583 (2006)
Comport, A.I., Malis, E., Rives, P.: Real-time quadrifocal visual odometry. Int. J. Robot. Res. 29(2-3), 245–266 (2010)
Wertheimer, M.M.: Laws of organization in perceptual forms (1938)
Wong, H.S., Chin, T.J., Yu, J., Suter, D.: Efficient multi-structure robust fitting with incremental top-k lists comparison. In: Computer VisionACCV 2010, pp. 553–564. Springer, Berlin Heidelberg (2011)
Romero-Cano, V., Nieto, J.I.: Stereo-based motion detection and tracking from a moving platform. In: Intelligent Vehicles Symposium (IV), 2013 IEEE pp. 499–504, IEEE (2013)
Geiger, A., Lenz, P., Urtasun, R.: Are we ready for autonomous driving? The KITTI vision benchmark suite. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3354–3361. IEEE (2012)
Brox, T., Malik, J.: Object segmentation by long term analysis of point trajectories. In: Computer Vision ECCV 2010, pp. 282–295. Springer, Berlin Heidelberg (2010)
Tomasi, C., Kanade, T.: Shape and motion from image streams under orthography: a factorization method. Int. J. Comput. Vis. 9(2), 137–154 (1992)
Kundu, A., Krishna, K.M., Sivaswamy, J.: Moving object detection by multi-view geometric techniques from a single camera mounted robot. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, 2009. IROS 2009, pp. 4306–4312. IEEE (2009)
Tsai, D., Flagg, M., Nakazawa, A., Rehg, J.M.: Motion coherent tracking using multi-label mrf optimization. Int. J. Comput. Vis. 100(2), 190–202 (2012)
Jain, S., Govindu, V.M.: Efficient Higher-Order Clustering on the Grassmann Manifold, ICCV, 2013 (2013)
Tron, R., Vidal, R.: A benchmark for the comparison of 3-d motion segmentation algorithms. In: IEEE Conference on Computer Vision and Pattern Recognition, 2007, CVPR’07. pp. 1–8. IEEE (2007)
Vidal, R., Ma, Y.: A unified algebraic approach to 2-D and 3-D motion segmentation. In: Computer Vision ECCV 2004, pp. 1–15. Springer, Berlin Heidelberg (2004)
Fragkiadaki, K., Zhang, G., Shi, J.: Video segmentation by tracing discontinuities in a trajectory embedding. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1846-1853. IEEE (2012)
Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24(6), 381–395 (1981)
Li, Z., Guo, J., Cheong, L.-F., Zhou, S.Z.: Perspective Motion Segmentation via Collaborative Clustering, ICCV (2013)
Lenz, P., Ziegler, J., Geiger, A., Roser, M.: Sparse scene flow segmentation for moving object detection in urban environments. In: 2011 IEEE Intelligent Vehicles Symposium (IV), pp. 926–932. IEEE (2011)
Zografos, V., Nordberg, K.: Fast and accurate motion segmentation using Linear Combination of Views. In: BMVC, pp. 1–11 (2011)
Weiss, Y., Adelson, E.H.: A unified mixture framework for motion segmentation: Incorporating spatial coherence and estimating the number of models. In: 1996 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 1996. Proceedings CVPR’96, pp. 321–326. IEEE (1996)
Black, M.J., Anandan, P.: The robust estimation of multiple motions: Parametric and piecewise-smooth flow fields. Comp. Vis. Image Underst. 63(1), 75–104 (1996)
Li, T., Kallem, V., Singaraju, D., Vidal, R.: Projective factorization of multiple rigid-body motions. In: IEEE Conference on Computer Vision and Pattern Recognition, 2007, CVPR’07. pp. 1–6. IEEE (2007)
Weiss, Y.: Smoothness in layers: Motion segmentation using nonparametric mixture estimation. In: 1997 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 1997, Proceedings. pp. 520–526. IEEE (1997)
Weiss, Y.: Segmentation using eigenvectors: a unifying view. In: The proceedings of the Seventh IEEE International Conference on Computer Vision, 1999, vol. 2, pp. 975–982. IEEE (1999)
Torr, P.H.S.: Geometric motion segmentation and model selection. Philosophical Transactions of the Royal Society of London. Series A: Mathematical, Phys. Eng. Sci. 356(1740), 1321–1340 (1998)
Vidal, R., Soatto, S., Sastry, S.S.: A factorization method for 3D multi-body motion estimation and segmentation. In: Proceesings of the Annual Allerton Conference on Communication Control and Computing, vol. 140, no. 13, pp. 11626-1635. The University; 1998 (2002)
Vidal, R., Sastry, S.: Optimal segmentation of dynamic scenes from two perspective views. In: Proceedings IEEE Computer Society Conference on Computer Vision and Pattern Recognition 2003, vol. 2, pp. II–281. IEEE (2003)
Shakernia, O., Vidal, R., Sastry, S.: Multibody motion estimation and segmentation from multiple central panoramic views. In: IEEE International Conference on Robotics and Automation, 2003, Proceedings, ICRA’03, vol. 1, pp. 571–576. IEEE (2003)
Chen, G., Lerman, G.: Spectral curvature clustering (SCC). Int. J. Comput. Vis. 81(3), 317–330 (2009)
Chin, T.-J., Wang, H., Suter, D.: The ordered residual kernel for robust motion subspace clustering. In: NIPS, vol. 9, pp. 333–341 (2009)
Shi, J., Tomasi, C.: Good features to track. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 1994, Proceedings CVPR’94, pp. 593–600. IEEE (1994)
Zappella, L., Provenzi, E., Llad, X., Salvi, J.: Adaptive motion segmentation algorithm based on the principal angles configuration. In: Computer Vision-ACCV 2010, pp. 15–26. Springer (2011)
Vidal R., Hartley, R.: Motion segmentation with missing data by PowerFactorization and Generalized PCA. In: CVPR (2004)
Zappella, L. et al.: Enhanced local subspace affinity for feature-based motion segmentation. Pattern Recogn. 44.2, 454–470 (2011)
Ochs, P., Brox, T.: Higher order motion models and spectral clustering. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE (2012)
Yuan, C., Medioni, G., Kang, J., Cohen, I.: Detecting motion regions in the presence of a strong parallax from a moving camera by multiview geometric constraints. IEEE Transactions on Pattern Analysis and Machine Intelligence 29(9), 1627–1641 (2007)
Kang, J., Cohen, I., Medioni, G., Yuan, C.: Detection and tracking of moving objects from a moving platform in presence of strong parallax. In: Tenth IEEE International Conference on Computer Vision, 2005, ICCV 2005, vol. 1, pp. 10–17 (2005)
Lourakis, M.I., Argyros, A.A., Orphanoudakis, S.C.: Independent 3D motion detection using residual parallax normal flow fields? In: Proceedings of the International Conference on Computer Vision, pp. 1012–1017 (1998)
Shashua, A., Wolf, L.: Homography tensors: on algebraic entities that represent three views of static or moving planar points. In: Proceedings of the European Conference on Computer Vision, vol. 1, pp. 507–521 (2000)
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Tourani, S., Krishna, K.M. Using In-frame Shear Constraints for Monocular Motion Segmentation of Rigid Bodies. J Intell Robot Syst 82, 237–255 (2016). https://doi.org/10.1007/s10846-015-0195-1
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10846-015-0195-1