skip to main content
research-article

4D imaging through spray-on optics

Published: 20 July 2017 Publication History

Abstract

Light fields are a powerful concept in computational imaging and a mainstay in image-based rendering; however, so far their acquisition required either carefully designed and calibrated optical systems (micro-lens arrays), or multi-camera/multi-shot settings. Here, we show that fully calibrated light field data can be obtained from a single ordinary photograph taken through a partially wetted window. Each drop of water produces a distorted view on the scene, and the challenge of recovering the unknown mapping from pixel coordinates to refracted rays in space is a severely underconstrained problem. The key idea behind our solution is to combine ray tracing and low-level image analysis techniques (extraction of 2D drop contours and locations of scene features seen through drops) with state-of-the-art drop shape simulation and an iterative refinement scheme to enforce photo-consistency across features that are seen in multiple views. This novel approach not only recovers a dense pixel-to-ray mapping, but also the refractive geometry through which the scene is observed, to high accuracy. We therefore anticipate that our inherently self-calibrating scheme might also find applications in other fields, for instance in materials science where the wetting properties of liquids on surfaces are investigated.

Supplementary Material

ZIP File (a35-iseringhausen.zip)
Supplemental files.
MP4 File (papers-0008.mp4)

References

[1]
Adamson, A. and A. Gast. 1997. Physical Chemistry of Surfaces. Wiley.
[2]
Adelson, E. H. and J. R. Bergen. 1991. The plenoptic function and the elements of early vision. Computational models of visual processing 1, 2 (1991).
[3]
Antipa, N., S. Necula, R. Ng, and L. Waller. 2016. Single-Shot Diffuser-Encoded Light Field Imaging. In IEEE International Conference on Computational Photography (ICCP).
[4]
Barnum, P. C., S. G. Narasimhan, and T. Kanade. 2010. A Multi-layered Display with Water Drops. ACM Trans. Graph. 29, 4, Article 76 (2010), 7 pages.
[5]
Bouguet, J.-Y. 2004. Camera calibration toolbox for MATLAB. (2004).
[6]
Brakke, K. A. 1992. The surface evolver. Experimental mathematics 1, 2 (1992), 141--165.
[7]
Brakke, K. A. 2013. Surface Evolver 2.70. (2013). http://facstaff.susqu.edu/brakke/evolver/evolver.html.
[8]
Chai, J.-X., X. Tong, S.-C. Chan, and H.-Y. Shum. 2000. Plenoptic Sampling. In Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '00). 307--318.
[9]
Chronis, N., G. L. Liu, K.-H. Jeong, and L. P. Lee. 2003. Tunable liquid-filled microlens array integrated with microfluidic network. Opt. Express 11, 19 (Sep 2003), 2370--2378.
[10]
Cimpoi, M., S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. 2014. Describing Textures in the Wild. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR).
[11]
Davis, A., M. Levoy, and F. Durand. 2012. Unstructured Light Fields. Comp. Graph. Forum 31, 2 (May 2012), 305--314.
[12]
De Gennes, P.-G., F. Brochard-Wyart, and D. Quéré. 2004. Capillarity and wetting phenomena: drops, bubbles, pearls, waves. Springer Science & Business Media.
[13]
Eigen, D., D. Krishnan, and R. Fergus. 2013. Restoring an Image Taken through a Window Covered with Dirt or Rain. In IEEE International Conference on Computer Vision (ICCV). IEEE, 633--640.
[14]
Fergus, R., A. Torralba, and W. T. Freeman. 2006. Random lens imaging. Technical Report MIT-CSAIL-TR-2006--058.
[15]
Fuchs, M., M. Kächele, and S. Rusinkiewicz. 2013. Design and Fabrication of Faceted Mirror Arrays for Light Field Capture. Computer Graphics Forum 32, 8 (2013), 246--257.
[16]
Georgiev, T., K. C. Zheng, B. Curless, D. Salesin, S. Nayar, and C. Intwala. 2006. Spatio-angular Resolution Tradeoffs in Integral Photography. In Proceedings of the 17th Eurographics Conference on Rendering Techniques (EGSR '06). Eurographics Association, 263--272.
[17]
Goldlücke, B., E. Strekalovskiy, and D. Cremers. 2012. The natural vectorial total variation which arises from geometric measure theory. SIAM Journal on Imaging Sciences 5, 2 (2012), 537--563.
[18]
Gortler, S. J., R. Grzeszczuk, R. Szeliski, and M. F. Cohen. 1996. The Lumigraph. In Proc. 23rd Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '96). ACM, New York, NY, USA, 43--54.
[19]
Gulshan, V., C. Rother, A. Criminisi, A. Blake, and A. Zisserman. 2010. Geodesic star convexity for interactive image segmentation. In Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on. IEEE, 3129--3136.
[20]
Han, J. Y. and K. Perlin. 2003. Measuring Bidirectional Texture Reflectance with a Kaleidoscope. ACM Trans. Graph. (Proc. SIGGRAPH 2003) (2003), 741--748.
[21]
Hartley, R. I. and A. Zisserman. 2004. Multiple View Geometry in Computer Vision (2nd ed.). Cambridge University Press.
[22]
Hickson, P., E. F. Borra, R. Cabanac, S. C. Chapman, V. De Lapparent, M. Mulrooney, and G. A. Walker. 1998. Large Zenith Telescope project: a 6-m mercury-mirror telescope. In Astronomical Telescopes & Instrumentation. International Society for Optics and Photonics, 226--232.
[23]
Honauer, K., O. Johannsen, D. Kondermann, and B. Goldluecke. 2016. A dataset and evaluation methodology for depth estimation on 4D light fields. In Asian Conference on Computer Vision. Springer.
[24]
Hullin, M. B., M. Fuchs, I. Ihrke, H.-P. Seidel, and H. P. A. Lensch. 2008. Fluorescent Immersion Range Scanning. ACM Trans. Graph. (Proc. SIGGRAPH 2008) 27, 3 (Aug. 2008), 87:1--87:10.
[25]
Hullin, M. B., H. P. A. Lensch, R. Raskar, H.-P. Seidel, and I. Ihrke. 2011. Dynamic Display of BRDFs. In Computer Graphics Forum (Proc. EUROGRAPHICS), Oliver Deussen and Min Chen (Eds.). Eurographics, Blackwell, Llandudno, UK, 475--483.
[26]
Ihrke, I., K. Kutulakos, H. Lensch, M. Magnor, and W. Heidrich. 2008. State of the art in transparent and specular object reconstruction. In EUROGRAPHICS 2008 STAR.
[27]
Ihrke, I., G. Wetzstein, D. Lanman, and W. Heidrich. 2011. State of the art in computational plenoptic imaging. In EUROGRAPHICS 2011 STAR.
[28]
Iliev, S. 1995. Iterative method for the shape of static drops. Computer Methods in Applied Mechanics and Engineering 126, 3 (1995), 251--265.
[29]
Iliev, S. 1997. Static drops on an inclined plane: equilibrium modeling and numerical analysis. Journal of colloid and interface science 194, 2 (1997), 287--300.
[30]
Iliev, S. and N. Pesheva. 2003. Wetting properties of well-structured heterogeneous substrates. Langmuir 19, 23 (2003), 9923--9931.
[31]
Iliev, S. and N. Pesheva. 2006. Nonaxisymmetric drop shape analysis and its application for determination of the local contact angles. Journal of colloid and interface science 301, 2 (2006), 677--684.
[32]
Jakob, W. 2010. Mitsuba renderer. (2010). http://www.mitsuba-renderer.org.
[33]
Kim, C., H. Zimmer, Y. Pritch, A. Sorkine-Hornung, and M. Gross. 2013. Scene Reconstruction from High Spatio-Angular Resolution Light Fields. ACM Trans. Graph. (Proc. SIGGRAPH 2013) 32, 4 (2013), 73:1--73:12.
[34]
Kuiper, S. and B. Hendriks. 2004. Variable-focus liquid lens for miniature cameras. Applied Physics Petters 85, 7 (2004), 1128--1130.
[35]
Kutulakos, K. N. and E. Steger. 2008. A Theory of Refractive and Specular 3D Shape by Light-Path Triangulation. International Journal of Computer Vision 76, 1 (2008), 13--29.
[36]
Levoy, M. and P. Hanrahan. 1996. Light Field Rendering. In Proc. 23rd Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '96). ACM, New York, NY, USA, 31--42.
[37]
Levoy, M., R. Ng, A. Adams, M. Footer, and M. Horowitz. 2006. Light Field Microscopy. ACM Trans. Graph. (Proc. SIGGRAPH 2013) (2006), 924--934.
[38]
Lippmann, G. 1908. La photographie intégrale. CR Acad. Sci. 146 (1908), 446--451.
[39]
Lowe, D. G. 1999. Object recognition from local scale-invariant features. In IEEE International Conference on Computer Vision (ICCV). 1150--1157.
[40]
Manakov, A., J. F. Restrepo, O. Klehm, R. Hegedüs, E. Eisemann, H.-P. Seidel, and I. Ihrke. 2013. A Reconfigurable Camera Add-on for High Dynamic Range, Multi-Spectral, Polarization, and Light-Field Imaging. ACM Trans. Graph. (Proc. SIGGRAPH 2013) 32, 4, Article 47 (July 2013), 14 pages.
[41]
Mukaigawa, Y., S. Tagawa, J. Kim, R. Raskar, Y. Matsushita, and Y. Yagi. 2011. Hemispherical Confocal Imaging Using Turtleback Reflector. In Computer Vision - ACCV 2010. Springer, 336--349.
[42]
Ng, R. 2005. Fourier Slice Photography. ACM Trans. Graph. (Proc. SIGGRAPH 2005) (2005), 735--744.
[43]
O'Neill, F. T. and J. T. Sheridan. 2002. Photoresist reflow method of microlens production Part I: Background and experiments. Optik-International Journal for Light and Electron Optics 113, 9 (2002), 391--404.
[44]
Pock, T., D. Cremers, H. Bischof, and A. Chambolle. 2010. Global Solutions of Variational Models with Convex Regularization. SIAM Journal on Imaging Sciences (2010).
[45]
Raskar, R., A. Agrawal, C. A. Wilson, and A. Veeraraghavan. 2008. Glare Aware Photography: 4D Ray Sampling for Reducing Glare Effects of Camera Lenses. ACM Trans. Graph. (Proc. SIGGRAPH 2008) 27, 3, Article 56 (Aug. 2008), 10 pages.
[46]
Shan, Q., B. Curless, and T. Kohno. 2010. Seeing Through Obscure Glass. In Proceedings of the 11th European Conference on Computer Vision: Part VI (ECCV'10). Springer-Verlag, Berlin, Heidelberg, 364--378. http://dl.acm.org/citation.cfm?id=1888212.1888241
[47]
Taguchi, Y., A. Agrawal, A. Veeraraghavan, S. Ramalingam, and R. Raskar. 2010. Axial-Cones: Modeling Spherical Catadioptric Cameras for Wide-Angle Light Field Rendering. ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia 2010) 29, 6 (Dec 2010), 172:1--172:8.
[48]
Tao, M. W., S. Hadap, J. Malik, and R. Ramamoorthi. 2013. Depth from Combining Defocus and Correspondence Using Light-Field Cameras. In IEEE International Conference on Computer Vision (ICCV). 673--680.
[49]
Tarini, M., H. P. A. Lensch, M. Goesele, and H.-P. Seidel. 2005. 3D acquisition of mirroring objects using striped patterns. Graphical Models 67, 4 (2005), 233--259.
[50]
Torralba, A. and W. Freeman. 2014. Accidental Pinhole and Pinspeck Cameras. International Journal of Computer Vision 110, 2 (2014), 92--112.
[51]
Vaish, V. and others. 2008. The (New) Stanford Light Field Archive. (2008). http://lightfield.stanford.edu/lfs.html.
[52]
Veeraraghavan, A., R. Raskar, A. Agrawal, A. Mohan, and J. Tumblin. 2007. Dappled Photography: Mask Enhanced Cameras for Heterodyned Light Fields and Coded Aperture Refocusing. ACM Trans. Graph. (Proc. SIGGRAPH 2007) 26, 3, Article 69 (2007).
[53]
Wang, T.-C., A. Efros, and R. Ramamoorthi. 2016. Depth estimation with occlusion modeling using light-field cameras. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) (2016).
[54]
Wanner, S. and B. Goldlücke. 2014. Variational Light Field Analysis for Disparity Estimation and Super-Resolution. IEEE Transactions on Pattern Analysis and Machine Intelligence 36, 3 (2014), 606--619.
[55]
Wei, L.-Y., C.-K. Liang, G. Myhre, C. Pitts, and K. Akeley. 2015. Improving Light Field Camera Sample Design with Irregularity and Aberration. ACM Trans. Graph. 34, 4, Article 152 (2015), 11 pages.
[56]
Weinmann, M., A. Osep, R. Ruiters, and R. Klein. 2013. Multi-View Normal Field Integration for 3D Reconstruction of Mirroring Objects. Proceedings of the International Conference on Computer Vision (Dec. 2013), 2504--2511.
[57]
Wender, A., J. Iseringhausen, B. Goldlücke, M. Fuchs, and M. B. Hullin. 2015. Light Field Imaging through Household Optics. In Vision, Modeling & Visualization, David Bommes, Tobias Ritschel, and Thomas Schultz (Eds.). Eurographics Association, 159--166.
[58]
Wetzstein, G., I. Ihrke, and W. Heidrich. 2013. On Plenoptic Multiplexing and Reconstruction. International Journal of Computer Vision 101, 2 (2013), 384--400.
[59]
Wilburn, B., N. Joshi, V. Vaish, E.-V. Talvala, E. Antunez, A. Barth, A. Adams, M. Horowitz, and M. Levoy. 2005. High Performance Imaging Using Large Camera Arrays. ACM Trans. Graph. (Proc. SIGGRAPH 2005) (2005), 765--776.
[60]
You, S., R. T. Tan, R. Kawakami, Y. Mukaigawa, and K. Ikeuchi. 2016. Waterdrop Stereo. CoRR (2016). arXiv:1604.00730v1

Cited By

View all

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics
ACM Transactions on Graphics  Volume 36, Issue 4
August 2017
2155 pages
ISSN:0730-0301
EISSN:1557-7368
DOI:10.1145/3072959
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 July 2017
Published in TOG Volume 36, Issue 4

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. analysis by synthesis
  2. inverse rendering
  3. plenoptic imaging

Qualifiers

  • Research-article

Funding Sources

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)21
  • Downloads (Last 6 weeks)1
Reflects downloads up to 17 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2023)Acquisition of light field images & videosImmersive Video Technologies10.1016/B978-0-32-391755-1.00012-2(163-171)Online publication date: 2023
  • (2022)Seeing through obstructions with diffractive cloakingACM Transactions on Graphics10.1145/3528223.353018541:4(1-15)Online publication date: 22-Jul-2022
  • (2022)Totems: Physical Objects for Verifying Visual IntegrityComputer Vision – ECCV 202210.1007/978-3-031-19781-9_10(164-180)Online publication date: 23-Oct-2022
  • (2021)Multi-Mask Camera Model for Compressed Acquisition of Light FieldsIEEE Transactions on Computational Imaging10.1109/TCI.2021.30537027(191-208)Online publication date: 2021
  • (2021)In-the-Wild Single Camera 3D Reconstruction Through Moving Water Surfaces2021 IEEE/CVF International Conference on Computer Vision (ICCV)10.1109/ICCV48922.2021.01233(12538-12547)Online publication date: Oct-2021
  • (2021)Compressively sampled light field reconstruction using orthogonal frequency selection and refinementSignal Processing: Image Communication10.1016/j.image.2020.11608792(116087)Online publication date: Mar-2021
  • (2019)Rain Wiper: An Incremental Randomly Wired Network for Single Image DerainingComputer Graphics Forum10.1111/cgf.1382538:7(159-169)Online publication date: 14-Nov-2019
  • (2018)Megapixel adaptive opticsACM Transactions on Graphics10.1145/3197517.320129937:4(1-12)Online publication date: 30-Jul-2018
  • (2018)Full 3D reconstruction of transparent objectsACM Transactions on Graphics10.1145/3197517.320128637:4(1-11)Online publication date: 30-Jul-2018
  • (2017)Perceptually-guided foveation for light field displaysACM Transactions on Graphics10.1145/3130800.313080736:6(1-13)Online publication date: 20-Nov-2017

View Options

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media