Abstract
The optical flow within a scene can be an arbitrarily complex composition of motion patterns that typically differ regarding their scale. Hence, using a single algorithm with a single set of parameters is often not sufficient to capture the variety of these motion patterns. In particular, the estimation of large displacements of small objects poses a problem. In order to cope with this problem, many recent methods estimate the optical flow by a fusion of flow candidates obtained either from different algorithms or from the same algorithm using different parameters. This, however, typically results in a pipeline of methods for estimating and fusing the candidate flows, each requiring an individual model with a dedicated solution strategy. In this paper, we investigate what results can be achieved with a pure variational approach based on a standard coarse-to-fine optimization. To this end, we propose a novel variational method for the simultaneous estimation and fusion of flow candidates. By jointly using multiple smoothness weights within a single energy functional, we are able to capture different motion patterns and hence to estimate large displacements even without additional feature matches. In the same functional, an intrinsic model-based fusion allows to integrate all these candidates into a single flow field, combining sufficiently smooth overall motion with locally large displacements. Experiments on large displacement sequences and the Sintel benchmark demonstrate the feasibility of our approach and show improved results compared to a single-smoothness baseline method.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Adams, L., Ortega, J.: A multi-color sor method for parallel computation. In: Proceedings of International Conference on Parallel Processing, pp. 53–56 (1982)
Baker, S., Scharstein, D., Lewis, J.P., Roth, S., Black, M.J., Szeliski, R.: A database and evaluation methodology for optical flow. Int. J. Comput. Vis. 92(1), 1–31 (2010)
Barnes, C., Shechtman, E., Goldman, D.B., Finkelstein, A.: The generalized PatchMatch correspondence algorithm. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6313, pp. 29–43. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15558-1_3
Black, M.J., Anandan, P.: Robust dynamic motion estimation over time. In: Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 292–302 (1991)
Black, M.J., Anandan, P.: The robust estimation of multiple motions: parametric and piecewise smooth flow fields. Comput. Vis. Image Underst. 63(1), 75–104 (1996)
Bredies, K., Kunisch, K., Pock, T.: Total generalized variation. SIAM J. Imaging Sci. 3(3), 492–526 (2010)
Brox, T., Bruhn, A., Papenberg, N., Weickert, J.: High accuracy optical flow estimation based on a theory for warping. In: Pajdla, T., Matas, J. (eds.) ECCV 2004. LNCS, vol. 3024, pp. 25–36. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-24673-2_3
Brox, T., Malik, J.: Large displacement optical flow: descriptor matching in variational motion estimation. IEEE Trans. Pattern Anal. Mach. Intell. 33(3), 500–513 (2011)
Butler, D.J., Wulff, J., Stanley, G.B., Black, M.J.: A naturalistic open source movie for optical flow evaluation. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7577, pp. 611–625. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33783-3_44
Charbonnier, P., Blanc-Féraud, L., Aubert, G., Barlaud, M.: Two deterministic half-quadratic regularization algorithms for computed imaging. In: Proceedings of IEEE International Conference on Image Processing, pp. 168–172 (1994)
Demetz, O., Stoll, M., Volz, S., Weickert, J., Bruhn, A.: Learning brightness transfer functions for the joint recovery of illumination changes and optical flow. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8689, pp. 455–471. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10590-1_30
Drayer, B., Brox, T.: Combinatorial regularization of descriptor matching for optical flow estimation. In: Proceedings of British Machine Vision Conference, pp. 42.1–42.12 (2015)
Förstner, W., Gülch, E.: A fast operator for detection and precise location of distinct points, corners and centres of circular features. In: Proceedings of ISPRS Intercommission Conference on Fast Processing of Photogrammetric Data, pp. 281–305 (1987)
Horn, B., Schunck, B.: Determining optical flow. Artif. Intell. 17, 185–203 (1981)
Lempitsky, V., Roth, S., Rother, C.: FusionFlow: discrete-continuous optimization for optical flow estimation. In: Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 1–8 (2008)
Nagel, H.H., Enkelmann, W.: An investigation of smoothness constraints for the estimation of displacement vector fields from image sequences. IEEE Trans. Pattern Anal. Mach. Intell. 8, 565–593 (1986)
Papenberg, N., Bruhn, A., Brox, T., Didas, S., Weickert, J.: Highly accurate optic flow computation with theoretically justified warping. Int. J. Comput. Vis. 67(2), 141–158 (2006)
Sevilla-Lara, L., Sun, D., Learned-Miller, E.G., Black, M.J.: Optical flow estimation with channel constancy. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8689, pp. 423–438. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10590-1_28
Sigal, L., Balan, A.O., Black, M.J.: HumanEva: synchronized video and motion capture dataset for evaluation of articulated human motion. Int. J. Comput. Vis. 87(1–2), 4–27 (2010)
Simoncelli, E.P., Adelson, E.H., Heeger, D.J.: Probability distributions of optical flow. In: Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 310–315 (1991)
Stoll, M., Volz, S., Bruhn, A.: Adaptive integration of feature matches into variational optical flow methods. In: Lee, K.M., Matsushita, Y., Rehg, J.M., Hu, Z. (eds.) ACCV 2012. LNCS, vol. 7726, pp. 1–14. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-37431-9_1
Stoll, M., Volz, S., Maurer, D., Bruhn, A.: A time-efficient optimisation framework for parameters of optical flow methods. In: Sharma, P., Bianchi, F.M. (eds.) SCIA 2017. LNCS, vol. 10269, pp. 41–53. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-59126-1_4
Tu, Z., Poppe, R., Veltkamp, R.C.: Weighted local intensity fusion method for variational optical flow estimation. Pattern Recogn. 50, 223–232 (2016)
Volz, S., Bruhn, A., Valgaerts, L., Zimmer, H.: Modeling temporal coherence for optical flow. In: Proceedings of International Conference on Computer Vision, pp. 1116–1123 (2011)
Weinzaepfel, P., Revaud, J., Harchaoui, Z., Schmid, C.: Deepflow: large displacement optical flow with deep matching. In: Proceedings of International Conference on Computer Vision, pp. 1385–1392 (2013)
Xu, L., Jia, J., Matsushita, Y.: Motion detail preserving optical flow estimation. IEEE Trans. Pattern Anal. Mach. Intell. 34, 1744–1757 (2012)
Zimmer, H., Bruhn, A., Weickert, J., Valgaerts, L., Salgado, A., Rosenhahn, B., Seidel, H.-P.: Complementary optic flow. In: Cremers, D., Boykov, Y., Blake, A., Schmidt, F.R. (eds.) EMMCVPR 2009. LNCS, vol. 5681, pp. 207–220. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-642-03641-5_16
Acknowledgements
We thank the German Research Foundation (DFG) for financial support within project B04 of SFB/Transregio 161.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG, part of Springer Nature
About this paper
Cite this paper
Stoll, M., Maurer, D., Bruhn, A. (2018). Variational Large Displacement Optical Flow Without Feature Matches. In: Pelillo, M., Hancock, E. (eds) Energy Minimization Methods in Computer Vision and Pattern Recognition. EMMCVPR 2017. Lecture Notes in Computer Science(), vol 10746. Springer, Cham. https://doi.org/10.1007/978-3-319-78199-0_6
Download citation
DOI: https://doi.org/10.1007/978-3-319-78199-0_6
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-78198-3
Online ISBN: 978-3-319-78199-0
eBook Packages: Computer ScienceComputer Science (R0)