skip to main content
research-article
Open access

Split-Aperture 2-in-1 Computational Cameras

Published: 19 July 2024 Publication History

Abstract

While conventional cameras offer versatility for applications ranging from amateur photography to autonomous driving, computational cameras allow for domain-specific adaption. Cameras with co-designed optics and image processing algorithms enable high-dynamic-range image recovery, depth estimation, and hyperspectral imaging through optically encoding scene information that is otherwise undetected by conventional cameras. However, this optical encoding creates a challenging inverse reconstruction problem for conventional image recovery, and often lowers the overall photographic quality. Thus computational cameras with domain-specific optics have only been adopted in a few specialized applications where the captured information cannot be acquired in other ways. In this work, we investigate a method that combines two optical systems into one to tackle this challenge. We split the aperture of a conventional camera into two halves: one which applies an application-specific modulation to the incident light via a diffractive optical element to produce a coded image capture, and one which applies no modulation to produce a conventional image capture. Co-designing the phase modulation of the split aperture with a dual-pixel sensor allows us to simultaneously capture these coded and uncoded images without increasing physical or computational footprint. With an uncoded conventional image alongside the optically coded image in hand, we investigate image reconstruction methods that are conditioned on the conventional image, making it possible to eliminate artifacts and compute costs that existing methods struggle with. We assess the proposed method with 2-in-1 cameras for optical high-dynamic-range reconstruction, monocular depth estimation, and hyperspectral imaging, comparing favorably to all tested methods in all applications.

Supplementary Material

ZIP File (papers_1042.zip)
supplemental

References

[1]
Abdullah Abuolaim and Michael S Brown. 2020. Defocus deblurring using dual-pixel data. In European Conference on Computer Vision. Springer, 111--126.
[2]
Abdullah Abuolaim, Radu Timofte, and Michael S Brown. 2021. NTIRE 2021 challenge for defocus deblurring using dual-pixel images: Methods and results. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 578--587.
[3]
Manoj Aggarwal and Narendra Ahuja. 2004. Split aperture imaging for high dynamic range. International Journal of Computer Vision 58 (2004), 7--17.
[4]
Boaz Arad and Ohad Ben-Shahar. 2016. Sparse Recovery of Hyperspectral Signal from Natural RGB Images. In European Conference on Computer Vision. Springer, 19--34.
[5]
Boaz Arad, Radu Timofte, Ohad Ben-Shahar, Yi-Tun Lin, and Graham D Finlayson. 2020. Ntire 2020 challenge on spectral reconstruction from an rgb image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. 446--447.
[6]
Seung-Hwan Baek, Hayato Ikoma, Daniel S Jeon, Yuqi Li, Wolfgang Heidrich, Gordon Wetzstein, and Min H Kim. 2021. Single-shot Hyperspectral-Depth Imaging with Learned Diffractive Optics. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 2651--2660.
[7]
Seung-Hwan Baek, Incheol Kim, Diego Gutierrez, and Min H Kim. 2017. Compact single-shot hyperspectral imaging using a prism. ACM Transactions on Graphics (TOG) 36, 6 (2017), 1--12.
[8]
Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias Müller. 2023. Zoedepth: Zero-shot transfer by combining relative and metric depth. arXiv preprint arXiv:2302.12288 (2023).
[9]
X Briottet, Y Boucher, A Dimmeler, A Malaplate, A Cini, Marco Diani, HHPT Bekman, P Schwering, T Skauli, I Kasen, et al. 2006. Military applications of hyperspectral imagery. In Targets and backgrounds XII: Characterization and representation, Vol. 6239. SPIE, 82--89.
[10]
Nicola Brusco, S Capeleto, M Fedel, Anna Paviotti, Luca Poletto, Guido Maria Cortelazzo, and G Tondello. 2006. A system for 3D modeling frescoed historical buildings with multispectral texture information. Machine Vision and Applications 17, 6 (2006), 373--393.
[11]
Marcela Carvalho, Bertrand Le Saux, Pauline Trouvé-Peloux, Andrés Almansa, and Frédéric Champagnat. 2018. Deep Depth from Defocus: how can defocus blur improve 3D estimation using dense neural networks?. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops. 0--0.
[12]
A. Chakrabarti and T. Zickler. 2011. Statistics of Real-World Hyperspectral Images. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR). 193--200.
[13]
Julie Chang and Gordon Wetzstein. 2019. Deep optics for monocular depth estimation and 3d object detection. (2019), 10193--10202.
[14]
Su-Kai Chen, Hung-Lin Yen, Yu-Lun Liu, Min-Hung Chen, Hou-Ning Hu, Wen-Hsiao Peng, and Yen-Yu Lin. 2023. Learning continuous exposure value representations for single-image hdr reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 12990--13000.
[15]
Paul E Debevec and Jitendra Malik. 2008. Recovering high dynamic range radiance maps from photographs. In ACM SIGGRAPH 2008 classes. 1--10.
[16]
Jiangxin Dong, Stefan Roth, and Bernt Schiele. 2020. Deep wiener deconvolution: Wiener meets deep learning for image deblurring. Advances in Neural Information Processing Systems 33 (2020), 1048--1059.
[17]
Liang Gao and Lihong V Wang. 2016. A review of snapshot multidimensional optical imaging: measuring photon tags in parallel. Physics reports 616 (2016), 1--37.
[18]
Rahul Garg, Neal Wadhwa, Sameer Ansari, and Jonathan T Barron. 2019. Learning single camera depth estimation using dual-pixels. In Proceedings of the IEEE/CVF international conference on computer vision. 7628--7637.
[19]
Carl Friedrich Gauss. 1843. Dioptric examinations by CF Gauss. in the Dieterich bookstore.
[20]
Bhargav Ghanekar, Vishwanath Saragadam, Dushyant Mehran, Anna-Karin Gustavsson, Aswin C. Sankaranarayanan, and Ashok Veeraraghavan. 2022. PS2F: Polarized Spiral Point Spread Function for Single-Shot 3D Sensing. IEEE Trans. Pattern Analysis and Machine Intelligence (TPAMI) / Special Issue of ICCP (August 2022).
[21]
Paul Green, Wenyang Sun, Wojciech Matusik, and Fredo Durand. 2007. Multi-aperture photography. In Acm Siggraph 2007 Papers. 68--es.
[22]
Shir Gur and Lior Wolf. 2019. Single Image Depth Estimation Trained via Depth From Defocus Cues. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
[23]
Nathan A Hagen, Liang S Gao, Tomasz S Tkaczyk, and Robert T Kester. 2012. Snapshot advantage: a review of the light collection improvement for parallel high-dimensional measurement systems. Optical Engineering 51, 11 (2012), 111702.
[24]
Harel Haim, Shay Elmalem, Raja Giryes, Alex Bronstein, and Emanuel Marom. 2018. Depth Estimation From a Single Image Using Deep Learned Phase Coded Mask. IEEE Transactions on Computational Imaging 4 (2018), 298--310.
[25]
Samuel W Hasinoff, Dillon Sharlet, Ryan Geiss, Andrew Adams, Jonathan T Barron, Florian Kainz, Jiawen Chen, and Marc Levoy. 2016. Burst photography for high dynamic range and low-light imaging on mobile cameras. ACM Transactions on Graphics (ToG) 35, 6 (2016), 1--12.
[26]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770--778.
[27]
Hayato Ikoma, Cindy M Nguyen, Christopher A Metzler, Yifan Peng, and Gordon Wetzstein. 2021. Depth from defocus with learned optics for imaging and occlusion-aware depth estimation. In 2021 IEEE International Conference on Computational Photography (ICCP). IEEE, 1--12.
[28]
Daniel S Jeon, Seung-Hwan Baek, Shinyoung Yi, Qiang Fu, Xiong Dun, Wolfgang Heidrich, and Min H Kim. 2019. Compact snapshot hyperspectral imaging with diffracted rotation. (2019).
[29]
Zeeshan Khan, Mukul Khanna, and Shanmuganathan Raman. 2019. Fhdr: Hdr image reconstruction from a single ldr image using feedback network. In 2019 IEEE Global Conference on Signal and Information Processing (GlobalSIP). IEEE, 1--5.
[30]
Masahiro Kobayashi, Michiko Johnson, Yoichi Wada, Hiromasa Tsuboi, Hideaki Takada, Kenji Togo, Takafumi Kishi, Hidekazu Takahashi, Takeshi Ichikawa, and Shunsuke Inoue. 2016. A low noise and high sensitivity image sensor with imaging and phase-difference detection AF in all pixels. ITE Transactions on Media Technology and Applications 4, 2 (2016), 123--128.
[31]
Sarawak Kuching. 2007. The performance of maximum likelihood, spectral angle mapper, neural network and decision tree classifiers in hyperspectral image analysis. Journal of Computer Science 3, 6 (2007), 419--423.
[32]
Anat Levin, Rob Fergus, Frédo Durand, and William T Freeman. 2007. Image and depth from a conventional camera with a coded aperture. ACM transactions on graphics (TOG) 26, 3 (2007), 70--es.
[33]
Lingen Li, Lizhi Wang, Weitao Song, Lei Zhang, Zhiwei Xiong, and Hua Huang. 2022. Quantization-Aware Deep Optics for Diffractive Snapshot Hyperspectral Imaging. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 19780--19789.
[34]
Guoxuan Liu, Ning Xu, Huaidong Yang, Qiaofeng Tan, and Guofan Jin. 2022. Miniaturized structured illumination microscopy with diffractive optics. Photonics Research 10, 5 (2022), 1317--1324.
[35]
Yu-Lun Liu, Wei-Sheng Lai, Yu-Sheng Chen, Yi-Lung Kao, Ming-Hsuan Yang, Yung-Yu Chuang, and Jia-Bin Huang. 2020. Single-image HDR reconstruction by learning to reverse the camera pipeline. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1651--1660.
[36]
Kris Malkiewicz and M David Mullen. 2009. Cinematography. Simon and Schuster.
[37]
N. Mayer, E. Ilg, P. Häusser, P. Fischer, D. Cremers, A. Dosovitskiy, and T. Brox. 2016. A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation. In IEEE International Conference on Computer Vision and Pattern Recognition (CVPR). http://lmb.informatik.uni-freiburg.de/Publications/2016/MIFDB16 arXiv:1512.02134.
[38]
Christopher A Metzler, Hayato Ikoma, Yifan Peng, and Gordon Wetzstein. 2020. Deep optics for single-shot high-dynamic-range imaging. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1375--1385.
[39]
Roope Näsi, Eija Honkavaara, Päivi Lyytikäinen-Saarenmaa, Minna Blomqvist, Paula Litkey, Teemu Hakala, Niko Viljanen, Tuula Kantola, Topi Tanhuanpää, and Markus Holopainen. 2015. Using UAV-based photogrammetry and hyperspectral imaging for mapping bark beetle damage at tree-level. Remote Sensing 7, 11 (2015), 15467--15493.
[40]
Shree K Nayar. 2006. Computational cameras: Redefining the image. Computer 39, 8 (2006), 30--38.
[41]
Elias Nehme, Daniel Freedman, Racheli Gordon, Boris Ferdman, Lucien E Weiss, Onit Alalouf, Tal Naor, Reut Orange, Tomer Michaeli, and Yoav Shechtman. 2020. Deep-STORM3D: dense 3D localization microscopy and PSF design by deep learning. Nature methods 17, 7 (2020), 734--740.
[42]
Liyuan Pan, Shah Chowdhury, Richard Hartley, Miaomiao Liu, Hongguang Zhang, and Hongdong Li. 2021. Dual pixel exploration: Simultaneous depth estimation and image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 4340--4349.
[43]
Sri Rama Prasanna Pavani and Rafael Piestun. 2009. 3D microscopy with a double-helix point spread function. In Three-Dimensional and Multidimensional Microscopy: Image Acquisition and Processing XVI, Vol. 7184. SPIE, 65--71.
[44]
Yifan Peng, Qilin Sun, Xiong Dun, Gordon Wetzstein, Wolfgang Heidrich, and Felix Heide. 2019. Learned large field-of-view imaging with thin-plate optics. ACM Trans. Graph. 38, 6 (2019), 219--1.
[45]
Abhijith Punnappurath and Michael S. Brown. 2019. Reflection Removal Using a Dual-Pixel Sensor. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
[46]
René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. 2021. Vision transformers for dense prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 12179--12188.
[47]
René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. 2022. Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-Shot Cross-Dataset Transfer. IEEE Transactions on Pattern Analysis and Machine Intelligence 44, 3 (2022).
[48]
Erik Reinhard, Wolfgang Heidrich, Paul Debevec, Sumanta Pattanaik, Greg Ward, and Karol Myszkowski. 2010. High dynamic range imaging: acquisition, display, and image-based lighting. Morgan Kaufmann.
[49]
Mushfiqur Rouf, Rafał Mantiuk, Wolfgang Heidrich, Matthew Trentacoste, and Cheryl Lau. 2011. Glare encoding of high dynamic range images. In CVPR 2011. IEEE, 289--296.
[50]
Marcel Santana Santos, Tsang Ing Ren, and Nima Khademi Kalantari. 2020. Single image HDR reconstruction using a CNN with masked features and perceptual loss. arXiv preprint arXiv:2005.07335 (2020).
[51]
Yoav Shechtman, Steffen J Sahl, Adam S Backer, and William E Moerner. 2014. Optimal point spread function design for 3D imaging. Physical review letters 113, 13 (2014), 133902.
[52]
Zheng Shi, Yuval Bahat, Seung-Hwan Baek, Qiang Fu, Hadi Amata, Xiao Li, Praneeth Chakravarthula, Wolfgang Heidrich, and Felix Heide. 2022. Seeing through obstructions with diffractive cloaking. ACM Transactions on Graphics (TOG) 41, 4 (2022), 1--15.
[53]
Vincent Sitzmann, Steven Diamond, Yifan Peng, Xiong Dun, Stephen Boyd, Wolfgang Heidrich, Felix Heide, and Gordon Wetzstein. 2018. End-to-end optimization of optics and image processing for achromatic extended depth of field and super-resolution imaging. ACM Trans. Graph. (TOG) 37, 4 (2018), 114.
[54]
Qilin Sun, Ethan Tseng, Qiang Fu, Wolfgang Heidrich, and Felix Heide. 2020. Learning Rank-1 Diffractive Optics for Single-shot High Dynamic Range Imaging. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
[55]
Ethan Tseng, Ali Mosleh, Fahim Mannan, Karl St-Arnaud, Avinash Sharma, Yifan Peng, Alexander Braun, Derek Nowrouzezahrai, Jean-Francois Lalonde, and Felix Heide. 2021. Differentiable Compound Optics and Processing Pipeline Optimization for End-to-end Camera Design. ACM Transactions on Graphics (TOG) 40, 2, Article 18 (2021).
[56]
Congli Wang, Ni Chen, and Wolfgang Heidrich. 2022. dO: A differentiable engine for deep lens design of computational imaging systems. IEEE Transactions on Computational Imaging 8 (2022), 905--916.
[57]
Hongcheng Wang, Ramesh Raskar, and Narendra Ahuja. 2005. High dynamic range video using split aperture camera. In IEEE 6th Workshop on Omnidirectional Vision, Camera Networks and Non-classical Cameras, Washington, DC, USA. Citeseer.
[58]
Yicheng Wu, Vivek Boominathan, Huaijin Chen, Aswin Sankaranarayanan, and Ashok Veeraraghavan. 2019. PhaseCam3D --- Learning Phase Masks for Passive Single View Depth Estimation. 2019 IEEE International Conference on Computational Photography (ICCP) (2019), 1--12.
[59]
Shumian Xin, Neal Wadhwa, Tianfan Xue, Jonathan T Barron, Pratul P Srinivasan, Jiawen Chen, Ioannis Gkioulekas, and Rahul Garg. 2021. Defocus map estimation and deblurring from a single dual-pixel image. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 2228--2238.
[60]
F. Yasuma, T. Mitsunaga, D. Iso, and S.K. Nayar. 2008. Generalized Assorted Pixel Camera: Post-Capture Control of Resolution, Dynamic Range and Spectrum. Technical Report.
[61]
Jonghee Yoon, James Joseph, Dale J Waterhouse, A Siri Luthman, George SD Gordon, Massimiliano Di Pietro, Wladyslaw Januszewicz, Rebecca C Fitzgerald, and Sarah E Bohndiek. 2019. A clinically translatable hyperspectral endoscopy (HySE) system for imaging the gastrointestinal tract. Nature communications 10, 1 (2019), 1--13.
[62]
Yuzhi Zhao, Lai-Man Po, Qiong Yan, Wei Liu, and Tingyu Lin. 2020. Hierarchical regression network for spectral reconstruction from RGB images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. 422--423.

Cited By

View all
  • (2025)Point spread function modeling and engineering in black-box lens systemsOptics Express10.1364/OE.54676733:3(4211)Online publication date: 28-Jan-2025
  • (2025)The Inverse Problems You Carry in Your PocketXRDS: Crossroads, The ACM Magazine for Students10.1145/370340531:2(44-49)Online publication date: 8-Jan-2025

Index Terms

  1. Split-Aperture 2-in-1 Computational Cameras

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Transactions on Graphics
    ACM Transactions on Graphics  Volume 43, Issue 4
    July 2024
    1774 pages
    EISSN:1557-7368
    DOI:10.1145/3675116
    Issue’s Table of Contents
    This work is licensed under a Creative Commons Attribution International 4.0 License.

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 19 July 2024
    Published in TOG Volume 43, Issue 4

    Check for updates

    Author Tags

    1. computational imaging
    2. co-designed optics
    3. dual-pixel sensor
    4. HDR imaging
    5. hyperspectral imaging
    6. monocular depth estimation

    Qualifiers

    • Research-article

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)900
    • Downloads (Last 6 weeks)113
    Reflects downloads up to 25 Jan 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2025)Point spread function modeling and engineering in black-box lens systemsOptics Express10.1364/OE.54676733:3(4211)Online publication date: 28-Jan-2025
    • (2025)The Inverse Problems You Carry in Your PocketXRDS: Crossroads, The ACM Magazine for Students10.1145/370340531:2(44-49)Online publication date: 8-Jan-2025

    View Options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Login options

    Full Access

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media