Timo Stich
Skip Abstract Section
Abstract
We present a method for image interpolation that is able to create high-quality, perceptually convincing transitions between recorded images. By implementing concepts derived from human vision, the problem of a physically correct image interpolation is relaxed to that of image interpolation which is perceived as visually correct by human observers. We find that it suffices to focus on exact edge correspondences, homogeneous regions and coherent motion to compute convincing results. A user study confirms the visual quality of the proposed image interpolation approach. We show how each aspect of our approach increases perceived quality of the result. We compare the results to other methods and assess achievable quality for different types of scenes.
- Adelson, E. and Bergen, J. 1991. The plenoptic function and the elements of early vision. In Computational Models of Visual Processing, M. Landy and J. Movshon Eds., MIT Press, Cambridge, MA, 3--20.Google Scholar
- Baker, S. and Matthews, I. 2004. Lucas-Kanade 20 years on: A unifying framework. Int. J. Comput. Vision 56, 3, 221--255. Google ScholarDigital Library
- Baker, S., Scharstein, D., Lewis, J., th, S. R., Black, M., and Szeliski, R. 2007. A database and evaluation methodology for optical flow. In Proceedings of the International Conference on Computer Vision (ICCV '07). IEEE, Los Alamitos, CA, 1--8.Google Scholar
- Barron, J., Fleet, D., and Beauchemin, S. 1994. Performance of optical flow techniques. Int. J. Comput. Vision 12, 1, 43--77. Google ScholarDigital Library
- Beier, T. and Neely, S. 1992. Feature-based image metamorphosis. In Proceedings of the Conference on Computer Graphics and Interactive Techniques (SIGGRAPH). ACM, New York, 35--42. Google ScholarDigital Library
- Belongie, S., Malik, J., and Puzicha, J. 2001. Matching shapes. In Proceedings of the International Conference on Computer Vision (ICCV'01). IEEE, Los Alamitos, CA, 454--461.Google Scholar
- Bertsekas, D. 1992. Auction algorithms for network flow problems: A tutorial introduction. Comput. Optim. Appl. 1, 7--66.Google ScholarCross Ref
- Black, M. J. and Anandan, P. 1996. The robust estimation of multiple motions: Parametric and piecewise-smooth flow fields. Comput. Vision Image Understand. 63, 1, 75--104. Google ScholarDigital Library
- Bouguet, J. Y. 2000. Pyramidal implementation of the lucas-kanade feature tracker: Description of the algorithm. Tech. rep., Intel Microprocessor Research Labs.Google Scholar
- Brox, T., Bruhn, A., Papenberg, N., and Weickert, J. 2004. High accuracy optical flow estimation based on a theory for warping. In Proceedings of the European Conference on Computer Vision (ECCV'04). Springer-Verlag, Berlin, 25--36.Google Scholar
- Bruhn, A., Weickert, J., and Schnörr, C. 2005. Lucas/kanade meets horn/schunck: Combining local and global optic flow methods. Int. J. Comput. Vision 61, 3, 211--231. Google ScholarDigital Library
- Buehler, C., Bosse, M., McMillan, L., Gortler, S., and Cohen, M. 2001. Unstructured Lumigraph Rendering. In Proceedings of the Conference on Computer Graphics and Interactive Techniques (SIGGRAPH'01). ACM, New York, 425--432. Google ScholarDigital Library
- Canny, J. 1986. A computational approach to edge detection. Trans. Patt. Anal. Mach. Intell. 8, 679--714. Google ScholarDigital Library
- Carranza, J., Theobalt, C., Magnor, M., and Seidel, H. P. 2003. Free-viewpoint video Of human actors. In Proceedings of the Conference on Computer Graphics and Interactive Techniques (SIGGRAPH'03). ACM, New York, 569--577. Google ScholarDigital Library
- Chen, S. and Williams, L. 1993. View interpolation for image synthesis. In Proceedings of the Conference on Computer Graphics and Interactive Techniques (SIGGRAPH'93). ACM, New York, 279--288. Google ScholarDigital Library
- Cooke, T., Cunningham, D., and Bülthoff, H. 2004. The perceptual influence of spatiotemporal noise on the reconstruction of shape from dynamic occlusion. In Proceedings of the 26th DAGM Symposium on Pattern Recognition. Springer-Verlag, Berlin, 407--414.Google Scholar
- Cunningham, D., Shipley, T., and Kellman, P. 1998. The dynamic specification of surfaces and boundaries. Perception 27, 4, 403--416.Google ScholarCross Ref
- Debevec, P., Borshukov, G., and Yu, Y. 1998. Efficient view-dependent image-based rendering with projective texture-mapping. In Proceedings of the Eurographics Rendering Workshop (EGRW '98). Springer-Verlag, Berlin, 105--116.Google Scholar
- Exner, S. 1894. Entwurf zu einer physiologischen Erklärung der psychischen Erscheinungen. F. Deuticke, Leipzig Wien.Google Scholar
- Felzenszwalb, P. and Huttenlocher, D. 2004. Efficient graph-based image segmentation. Int. J. Comput. Vision 59, 167--181. Google ScholarDigital Library
- Fu, M. F., Au, O., and Chan, W. C. 2002. Temporal interpolation using wavelet domain motion estimation and motion compensation. In Proceedings of the International Conference on Image Processing (ICIP ‘02). IEEE, Los Alamtios, CA, 393--396.Google Scholar
- Gibson, J. 1955. The Perception of the Visual World. Cambridge University Press, Cambridg, UK.Google Scholar
- Gibson, J., Kaplan, G., Jr., H. R., and Wheeler, K. 1969. The change from visible to invisible: a study of optic transitions. Percept. Psychophys. 5, 2, 116--131.Google ScholarCross Ref
- Gortler, S., Grzeszczuk, R., Szeliski, R., and Cohen, M. 1996. The Lumigraph. In Proceedings Conference on Computer Graphics and Interactive Techniques (SIGGRAPH'96). ACM, New York, 43--54. Google ScholarDigital Library
- Hartley, R. and Zisserman, H. 2000. Multiple View Geometry in Computer Vision. Cambridge University Press, Cambridge, UK. Google ScholarDigital Library
- Heeger, D., Boynton, G., Demb, J., Seidemann, E., and Newsome, W. 1999. Motion opponency in visual cortex. J. Neurosci. 19, 7162--7174.Google ScholarCross Ref
- Horn, B. and Schunck, B. 1981. Determining optical flow. Artif. Intell. 17, 185--203.Google ScholarDigital Library
- Hubel, D. 1995. Eye, Brain, and Vision 2nd Ed. W. H. Freeman, New York.Google Scholar
- Isaksen, A., McMillan, L., and Gortler, S. 2000. Dynamically reparameterized light fields. In Proceedings of the Conference on Computer Graphics and Interactive Techniques (SIGGRAPH'00). ACM, New York, 297--306. Google ScholarDigital Library
- Jain, J. and Jain, A. 1981. Displacement measurement and its application in interframe image coding. IEEE Trans. Comm. 29, 12, 1799--1808.Google ScholarCross Ref
- Kellman, P. J. and Shipley, T. F. 1992. Perceiving objects across gaps in space and time. Curr. Directions Psych. Science 1, 6, 193--199.Google ScholarCross Ref
- Levoy, M. and Hanrahan, P. 1996. Light Field Rendering. In Proceedings Conference on Computer Graphics and Interactive Techniques (SIGGRAPH). 31--42. Google ScholarDigital Library
- Lucas, B. and Kanade, T. 1981. “an iterative image registration technique with an application to stereo vision”. In Proceedings of the International Joint Conference on Artificial Intelligence. 674--679. Google ScholarDigital Library
- Mark, W., McMillan, L., and Bishop, G. 1997. Post-Rendering 3D Warping. In Proceedings of the Symposium on Interactive 3D Graphics. 7--16. Google ScholarDigital Library
- Marr, D. 1983. Vision: A Computational Investigation into the Human Representation and Processing of Visual Information. W. H. Freeman. Google ScholarDigital Library
- Matusik, W., Buehler, C., Raskar, R., Gortler, S., and McMillan, L. 2000. Image-Based Visual Hulls. In Proceedings Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '00). 369--374. Google ScholarDigital Library
- Matusik, W. and Pfister, H. 2004. 3D TV: A scalable system for real-time acquisition, transmission, and autostereoscopic display of dynamic scenes. In Proceedings Conference on Computer Graphics and Interactive Techniques (SIGGRAPH'04). ACM, New York 814--824. Google ScholarDigital Library
- McMillan, L. and Bishop, G. 1995. Plenoptic modeling: an image-based rendering system. In Proceedings Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '95). ACM, New York, 39--46. Google ScholarDigital Library
- Microsoft Corporation. Media player 9 video quality demos. http://www.microsoft.com/windows/windowsmedia/demos/video quality demos.aspx.Google Scholar
- O'Sullivan, C., Howlett., S., Mcdonnell, R., Morvan, Y., and O'Conor, K. 2004. Perceptually adaptive graphics. InEurographics, State-of-the-art-Report 6.Google Scholar
- Perona, P. and Malik, J. 1990. Scale-space and edge detection using anisotropic diffusion. Trans. Patt. Anal. Mach. Intell. 12, 7, 629--639. Google ScholarDigital Library
- Qian, N. and Andersen, R. 1997. A physiological model for motion-stereo integration and a unified explanation of Pulfrich-like phenomena. Vision Res. 37, 1683--1698.Google ScholarCross Ref
- Ramanarayanan, G., Bala, K., and Ferwerda, J. 2008. Perception of complex aggregates. In Proceedings Conference on Computer Graphics and Interactive Techniques (SIGGRAPH'08). ACM, New York, 1--10. Google ScholarDigital Library
- Ramanarayanan, G., Ferwerda, J., Walter, B., and Bala, K. 2007. Visual equivalence: Towards a new standard for image fidelity. In Proceedings of the Conference on Computer Graphics and Interactive Techniques (SIGGRAPH'07). ACM, New York, 654--663. Google ScholarDigital Library
- Reichardt, W. 1961. Autocorrelation, A principle for the evaluation of sensory information by the central nervous system. In Sensory communication, W. Rosenblith Ed., MIT Press-Wiley, New York, 303--317.Google Scholar
- Ruzon, M. and Tomasi, C. 1999. Color edge detection with the compass operator. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR'99). IEEE, Los Alamitos, 160--166.Google Scholar
- Schaefer, S., McPhail, T., and Warren, J. 2006. Image deformation using moving least squares. In Proceedings of the Conference on Computer Graphics and Interactive Techniques (SIGGRAPH'06). ACM, New York, 533--540. Google ScholarDigital Library
- Seitz, S. and Dyer, C. 1996. View morphing. In Proceedings of the Conference on Computer Graphics and Interactive Techniques (SIGGRAPH'96). ACM, New York, 21--30. Google ScholarDigital Library
- Shipley, T. and Cunningham, D. 2001. Perception of occluding and occluded objects over time: Spatiotemporal segmentation and unit formation. In From Fragments to Object: Segmentation and Grouping in Vision. Elsevier Science Ltd., Oxford, UK, 557--585.Google Scholar
- Snavely, N., Seitz, S., and Szeliski, R. 2006. Photo tourism: Exploring photo collections in 3D. In Proceedings of the Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '06). ACM, New York, 835--846. Google ScholarDigital Library
- Stich, T., Linz, C., Albuquerque, G., and Magnor, M. 2008a. View and time interpolation in image space. Comput. Graphics Forum 27, 7, 1781--1787.Google ScholarCross Ref
- Stich, T., Linz, C., Wallraven, C., Cunningham, D., and Magnor, M. 2008b. Perceptionmotivated interpolation of image sequences. In Proceedings of the Symposium on Applied Perception in Graphics and Visualization (APGV'08). ACM, New York, 97--106. Google ScholarDigital Library
- Tourapis, A., Cheong, H., Liou, M., and Au, O. 2001. Temporal interpolation of video sequences using zonal based algorithms. In Proceedings of the International Conference on Image Processing (ICIP '01). IEEE, Los Alamitos, CA, 895--898.Google Scholar
- van Santen, J. P. H. and Sperling, G. 1985. Elaborated reichardt detectors. J. Opt. Soc. Am. A, 2, 300--320.Google ScholarCross Ref
- Vangorp, P. and Dutré, P. 2008. Shape-dependent gloss correction. In Proceedings of the Symposium on Applied Perception in Graphics and Visualization (APGV'08). ACM, New York, 123--130. Google ScholarDigital Library
- Vangorp, P., Laurijssen, J., and Dutré, P. 2007. The influence of shape on the perception of material reflectance. In Proceedings of the Conference on Computer Graphics and Interactive Techniques (SIGGRAPH'07). ACM, New York, 1--9. Google ScholarDigital Library
- Vedula, S., Baker, S., and Kanade, T. 2005. Image based spatio-temporal modeling and view interpolation of dynamic events. ACM Trans. Graphics 24, 2, 240--261. Google ScholarDigital Library
- Wallach, H. 1935. Ueber visuell wahrgenommene Bewegungsrichtung. Psychologische Forschung 20, 325--380.Google ScholarCross Ref
- Wallraven, C., Bülthoff, H. H., Fischer, J., Cunningham, D. W., and Bartz, D. 2007. The evaluation of real-world and computer-generated stylized facial expressions. ACM Trans. Appl. Percept. 4, 3, 1--24. Google ScholarDigital Library
- Wertheimer, M. 1938. Laws of organization in perceptual forms. In A Source Book of Gestalt Psychology, W. Ellis, Ed. Kegan Paul, Trench, Trubner & Co. Ltd., London, 71--88.Google Scholar
- Wolberg, G. 1998. Image warping: a survey. Visual Comput. 14, 360--372.Google ScholarCross Ref
- Wood, D., Azuma, D., Aldinger, K., Curless, B., Duchamp, T., Salesin, D., and Stuetzle, W. 2000. Surface light fields for 3D photography. In Proceedings of the Conference on Computer Graphics and Interactive Techniques (SIGGRAPH'00). ACM, New York, 287--296. Google ScholarDigital Library
- Yuille, A. L. and Poggio, T. A. 1986. Scaling theorems for zero crossings. IEEE Trans. Pattern Anal. Mach. Intell. 8, 1, 15--25. Google ScholarDigital Library
- Zitnick, C., Jojic, N., and Kang, S. B. 2005. Consistent segmentation for optical flow estimation. In Proceedings of the International Conference on Computer Vision (ICCV '05) 2, 1308--1315. Google ScholarDigital Library
- Zitnick, C., Kang, S., Uyttendaele, M., Winder, S., and Szeliski, R. 2004. High-quality video view interpolation using a layered representation. In Proceedings of the Conference on Computer Graphics and Interactive Techniques (SIGGRAPH'04). ACM, New York, 600--608. Google ScholarDigital Library
Index Terms
- Perception-motivated interpolation of image sequences
Recommendations
Perception-motivated interpolation of image sequences
APGV '08: Proceedings of the 5th symposium on Applied perception in graphics and visualizationWe present a method for image interpolation which is able to create high-quality, perceptually convincing transitions between recorded images. By implementing concepts derived from human vision, the problem of a physically correct image interpolation is ...
Rational Interpolation Splines with Region Control and Image Interpolation Application
AbstractIn this work, we deal with the region control of interpolation curves and surfaces using a class of rational interpolation splines in one and two dimensions. Simple sufficient data-dependent constraints are derived on the local control ...
An Isophote-Oriented Image Interpolation Method
ISCSCT '08: Proceedings of the 2008 International Symposium on Computer Science and Computational Technology - Volume 01This paper presents an isophote-oriented image interpolation method that attempts to produce smooth reconstructions of the image’s level curves while still maintaining image fidelity. The proposed method employs an interpolation kernel that is obtained ...
Comments