research-article

Differentiable Compound Optics and Processing Pipeline Optimization for End-to-end Camera Design

Authors:

Karl St-Arnaud,

Avinash Sharma,

Alexander Braun,

Derek Nowrouzezahrai,

Jean-François Lalonde,

Felix HeideAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 40, Issue 2

Article No.: 18, Pages 1 - 19

https://doi.org/10.1145/3446791

Published: 21 June 2021 Publication History

Abstract

Most modern commodity imaging systems we use directly for photography—or indirectly rely on for downstream applications—employ optical systems of multiple lenses that must balance deviations from perfect optics, manufacturing constraints, tolerances, cost, and footprint. Although optical designs often have complex interactions with downstream image processing or analysis tasks, today’s compound optics are designed in isolation from these interactions. Existing optical design tools aim to minimize optical aberrations, such as deviations from Gauss’ linear model of optics, instead of application-specific losses, precluding joint optimization with hardware image signal processing (ISP) and highly parameterized neural network processing. In this article, we propose an optimization method for compound optics that lifts these limitations. We optimize entire lens systems jointly with hardware and software image processing pipelines, downstream neural network processing, and application-specific end-to-end losses. To this end, we propose a learned, differentiable forward model for compound optics and an alternating proximal optimization method that handles function compositions with highly varying parameter dimensions for optics, hardware ISP, and neural nets. Our method integrates seamlessly atop existing optical design tools, such as Zemax. We can thus assess our method across many camera system designs and end-to-end applications. We validate our approach in an automotive camera optics setting—together with hardware ISP post processing and detection—outperforming classical optics designs for automotive object detection and traffic light state detection. For human viewing tasks, we optimize optics and processing pipelines for dynamic outdoor scenarios and dynamic low-light imaging. We outperform existing compartmentalized design or fine-tuning methods qualitatively and quantitatively, across all domain-specific applications tested.

References

[1]

Donald Baxter, Frederic Cao, Henrik Eliasson, and Jonathan Phillips. 2012. Development of the I3A CPIQ spatial metrics. Proceedings of SPIE 8293 (2012), 1.

[2]

Vladimir Bychkovsky, Sylvain Paris, Eric Chan, and Frédo Durand. 2011. Learning photographic global tonal adjustment with a database of input/output image pairs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.

Digital Library

[3]

Julie Chang, Vincent Sitzmann, Xiong Dun, Wolfgang Heidrich, and Gordon Wetzstein. 2018. Hybrid optical-electronic convolutional neural networks with optimized diffractive optics for image classification. Scientific Reports 8, 1 (2018), 12324.

[4]

Julie Chang and Gordon Wetzstein. 2019. Deep optics for monocular depth estimation and 3D object detection. In Proceedings of the IEEE International Conference on Computer Vision (ICCV).

[5]

Chen Chen, Qifeng Chen, Jia Xu, and Vladlen Koltun. 2018. Learning to see in the dark. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.

[6]

Qifeng Chen, Jia Xu, and Vladlen Koltun. 2017. Fast image processing with fully-convolutional networks. In Proceedings of the IEEE International Conference on Computer Vision (ICCV).

[7]

Jung-Min Choi, Sung-Joon Jang, Sang-Seol Lee, Youngbae Hwang, and Byeong Ho Choi. 2014. Memory optimization of bilateral filter and its hardware implementation. In Proceedings of the 18th IEEE International Symposium on Consumer Electronics (ISCE). 1–2.

[8]

Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. 2007. Image denoising by sparse 3-D transform-domain collaborative filtering. IEEE Transactions on Image Processing 16, 8 (2007), 2080–2095.

Digital Library

[9]

Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. 2019. Neural architecture search: A survey. Journal of Machine Learning Research 20, 55 (2019), 1–21.

[10]

Qingnan Fan, Jiaolong Yang, David Wipf, Baoquan Chen, and Xin Tong. 2018. Image smoothing via unsupervised learning. ACM Transactions on Graphics 37, 6 (2018), Article 259.

Digital Library

[11]

Fengzhou Fang, Xiaodong Zhang, Albert Weckenmann, Guoxiong Zhang, and Chris Evans. 2013. Manufacturing and measurement of freeform optics. CIRP Annals 62, 2 (2013), 823–846.

[12]

Grant R. Fowles. 1989. Introduction to Modern Optics. Courier Corporation.

[13]

Andreas Fregin, Julian Müller, Ulrich Kre***el, and Klaus Dietmayer. 2018. The DriveU traffic light dataset: Introduction and comparison with existing datasets. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA).

[14]

Galvoptics. 2020. Photopic Eye Response Filter. Retrieved April 2, 2021 from https://www.galvoptics.co.uk/optical-components/optical-filters/photopic-eye-response-filter/

[15]

Kenneth Garrard, Thomas Bruegge, Jeff Hoffman, Thomas Dow, and Alex Sohn. 2005. Design tools for freeform optics. In Current Developments in Lens Design and Optical Engineering VI, Vol. 5874. International Society for Optics and Photonics, 58740A.

[16]

Carl Friedrich Gauss. 1841. Dioptrische Untersuchungen. Dieterich.

[17]

Joseph M. Geary. 2002. Introduction to Lens Design: With Practical ZEMAX Examples. Willmann-Bell Richmond.

[18]

Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. 2013. Vision meets robotics: The KITTI dataset. International Journal of Robotics Research 32, 11 (2013), 1231–1237.

Digital Library

[19]

Michaël Gharbi, Gaurav Chaurasia, Sylvain Paris, and Frédo Durand. 2016. Deep joint demosaicking and denoising. ACM Transactions on Graphics 35, 6 (2016), 191.

Digital Library

[20]

Michaël Gharbi, Jiawen Chen, Jon Barron, Samuel W. Hasinoff, and Frédo Durand. 2017. Deep bilateral learning for real-time image enhancement. ACM Transactions on Graphics 36, 4 (2017), Article 118.

Digital Library

[21]

Radek Grzeszczuk, Demetri Terzopoulos, and Geoffrey Hinton. 1998. NeuroAnimator: Fast neural network emulation and control of physics-based models. In Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH). ACM, New York, NY.

Digital Library

[22]

Johannes Hanika and Carsten Dachsbacher. 2014. Efficient Monte Carlo rendering with realistic lenses. Computer Graphics Forum 33 (2014), 323–332.

Digital Library

[23]

James E. Harvey, Ryan G. Irvin, and Richard N. Pfisterer. 2015. Modeling physical optics phenomena by complex ray tracing. Optical Engineering 54, 3 (2015), 035105.

[24]

Samuel W. Hasinoff, Dillon Sharlet, Ryan Geiss, Andrew Adams, Jon Barron, Florian Kainz, Jiawen Chen, and Marc Levoy. 2016. Burst photography for high dynamic range and low-light imaging on mobile cameras. ACM Transactions on Graphics 35, 6 (2016), Article 192, 12 pages.

Digital Library

[25]

James Hegarty, John Brunhaver, Zachary DeVito, Jonathan Ragan-Kelley, Noy Cohen, Steven Bell, Artem Vasilyev, Mark Horowitz, and Pat Hanrahan. 2014. Darkroom: Compiling high-level image processing code into hardware pipelines. ACM Transactions on Graphics 33, 4 (2014), Article 144.

Digital Library

[26]

Felix Heide, Markus Steinberger, Yun-Ta Tsai, Mushfiqur Rouf, Dawid Pająk, Dikpal Reddy, Orazio Gallo, et al. 2014. FlexISP: A flexible camera image processing framework. ACM Transactions on Graphics 33, 6 (2014), Article 231, 13 pages.

Digital Library

[27]

Michael Hirsch, Suvrit Sra, Bernhard Schölkopf, and Stefan Harmeling. 2010. Efficient filter flow for space-variant multiframe blind deconvolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, Los Alamitos, CA, 607–614.

[28]

ISO. 2017. ISO 71696. Photography - Electronic still picture imaging – Resolution and spatial frequency responses. 1–49. Retrieved January 5, 2019 from https://www.iso.org/standard/71696.html

[29]

Michael J. Kidger. 2002. Fundamental Optical Design. SPIE Press.

[30]

Rudolf Kingslake and Roger B. Johnson. 2009. Lens Design Fundamentals. Academic Press.

[31]

Craig E. Kolb, Don P. Mitchell, and Pat Hanrahan. 1995. A realistic camera model for computer graphics. In Proceedings of the 22nd Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH).

Digital Library

[32]

Kowa. 2020. LM6NCL. Retrieved April 2, 2021 from https://lenses.kowa-usa.com/ncl-series/490-lm6ncl.html

[33]

Tzu-Mao Li, Michaël Gharbi, Andrew Adams, Frédo Durand, and Jonathan Ragan-Kelley. 2018. Differentiable programming for image processing and deep learning in Halide. ACM Transactions on Graphics 37, 4 (2018), Article 139, 13 pages.

Digital Library

[34]

Daniel Malacara-Hernández and Zacarías Malacara-Hernández. 2016. Handbook of Optical Design. CRC Press, Boca Raton, FL.

[35]

Christopher A. Metzler, Hayato Ikoma, Yifan Peng, and Gordon Wetzstein. 2020. Deep optics for single-shot high-dynamic-range imaging. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.

[36]

Ali Mosleh, Avinash Sharma, Emmanuel Onzon, Fahim Mannan, Nicolas Robidoux, and Felix Heide. 2020. Hardware-in-the-loop end-to-end optimization of camera image processing pipelines. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[37]

ON Semiconductor. 2015. MT9P001: 1/2.5-Inch 5 Mp CMOS Digital Image Sensor. Retreived on 03 May, 2021 from https://www.onsemi.com/pdf/datasheet/mt9p001-d.pdf.

[38]

Jun Nishimura, Timo Gerasimow, Rao Sushma, Alexsandar Sutic, Chyuan-Tyng Wu, and Gilad Michael. 2018. Automatic ISP image quality tuning using nonlinear optimization. In Proceedings of the International Conference on Image Processing (ICIP).

[39]

Yifan Peng, Qilin Sun, Xiong Dun, Gordon Wetzstein, Wolfgang Heidrich, and Felix Heide. 2019. Learned large field-of-view imaging with thin-plate optics. ACM Transactions on Graphics 38, 6 (2019), 219.

Digital Library

[40]

Jonathan B. Phillips and Henrik Eliasson. 2018. Camera Image Quality Benchmarking.Wiley Publishing.

Digital Library

[41]

Rajeev Ramanath, Wesley E. Snyder, Youngjun Yoo, and Mark S. Drew. 2005. Color image processing pipeline. IEEE Signal Processing Magazine 22, 1 (2005), 34–43.

[42]

Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster R-CNN: Towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems.

Digital Library

[43]

Max-Gerd Retzlaff, Johannes Hanika, Jürgen Beyerer, and Carsten Dachsbacher. 2016. Potential and challenges of using computer graphics for the simulation of optical measurement systems.GMA/ITG Fachtagung: Sensoren und Messsysteme 18 (2016), 322–329.

[44]

Emanuel Schrade, Johannes Hanika, and Carsten Dachsbacher. 2016. Sparse high-degree polynomials for wide-angle lenses. Computer Graphics Forum 35 (2016), 89–97.

Digital Library

[45]

Ling Shao, Ruomei Yan, Xuelong Li, and Yan Liu. 2014. From heuristic optimization to dictionary learning: A review and comprehensive comparison of image denoising algorithms. IEEE Transactions on Cybernetics 44, 7 (2014), 1001–1013.

[46]

Yichang Shih, Brian Guenter, and Neel Joshi. 2012. Image enhancement using calibrated lens simulations. In Proceedings of the European Conference on Computer Vision.

Digital Library

[47]

Vincent Sitzmann, Steven Diamond, Yifan Peng, Xiong Dun, Stephen Boyd, Wolfgang Heidrich, Felix Heide, and Gordon Wetzstein. 2018. End-to-end optimization of optics and image processing for achromatic extended depth of field and super-resolution imaging. ACM Transactions on Graphics 37, 4 (2018), 114.

Digital Library

[48]

Georgii Georgievich Sliusarev. 1984. Abberation and Optical Design Theory (2nd ed.). Adam Higler Ltd., Bristol, England.

[49]

Warren J. Smith. 2005. Modern Lens Design (2nd ed.). McGraw-Hill, New York, NY.

[50]

EMVA. 2010. Standard for Characterization of image Sensors and Cameras. Release 3.https://www.emva.org/wp-content/uploads/EMVA1288-3.1a.pdf.

[51]

Benjamin Steinert, Holger Dammertz, Johannes Hanika, and Hendrik P. A. Lensch. 2011. General spectral camera lens simulation. Computer Graphics Forum 30 (2011), 1643–1654.

[52]

David G. Stork and Patrick R. Gill. 2014. Optical, mathematical, and computational foundations of lensless ultra-miniature diffractive imagers and sensors. International Journal on Advances in Systems and Measurements 7, 3 (2014), 4.

[53]

Haiyin Sun. 2016. Lens Design: A Practical Guide. CRC Press, Boca Raton, FL.

[54]

Libin Sun, Neel Joshi, Brian Guenter, and James Hays. 2015. Lens factory: Automatic lens generation using off-the-shelf components. arXiv:1506.08956

[55]

Qilin Sun, Ethan Tseng, Qiang Fu, Wolfgang Heidrich, and Felix Heide. 2020. Learning rank-1 diffractive optics for single-shot high dynamic range imaging. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.

[56]

Carlo Tomasi and Roberto Manduchi. 1998. Bilateral filtering for gray and color images. In Proceedings of the 6th International Conference on Computer Vision. IEEE, Los Alamitos, CA, 839–846.

Digital Library

[57]

Ethan Tseng, Felix Yu, Yuting Yang, Fahim Mannan, Karl S. T. Arnaud, Derek Nowrouzezahrai, Jean-François Lalonde, and Felix Heide. 2019. Hyperparameter optimization in black-box image processing using differentiable proxies. ACM Transactions on Graphics 38, 4 (2019), 27.

Digital Library

[58]

Bruce H. Walker. 2008. Optical Engineering Fundamentals. Vol. 82. SPIE Press, Bellingham, WA.

[59]

Li Xu, Jimmy Ren, Qiong Yan, Renjie Liao, and Jiaya Jia. 2015. Deep edge-aware filters. In Proceedings of the 32nd International Conference on Machine Learning (ICML). 1669–1678.

Digital Library

[60]

Yangyang Xu and Wotao Yin. 2013. A block coordinate descent method for regularized multiconvex optimization with applications to nonnegative tensor factorization and completion. SIAM Journal on Imaging Sciences 6, 3 (2013), 1758–1789.

Digital Library

[61]

Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. 2020. BDD100K: A diverse driving dataset for heterogeneous multitask learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.

[62]

Hao Zhang, Wenjiang Liu, Ruolin Wang, Tao Liu, and Mengtian Rong. 2016. Hardware architecture design of block-matching and 3D-filtering denoising algorithm. Journal of Shanghai Jiaotong University (Science) 21, 2 (2016), 173–183.

[63]

Lei Zhang, Xiaolin Wu, Antoni Buades, and Xin Li. 2011. Color demosaicking by local directional interpolation and nonlocal adaptive thresholding. Journal of Electronic Imaging 20, 2 (2011), 023016.

[64]

Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. 2018. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition.

Cited By

Shi RZhang TZhou YShao YZhang HWei RBai J(2025)End-to-end hybrid infrared imaging system design with thermal analysisOptics Express10.1364/OE.55030933:3(4011)Online publication date: 27-Jan-2025
https://doi.org/10.1364/OE.550309
Seger TMenke CSonntag MUrban K(2025)Efficient evaluation of the Jacobian in the damped least-squares method for optical design problems using algorithmic differentiationOptics Express10.1364/OE.54604933:2(3054)Online publication date: 21-Jan-2025
https://doi.org/10.1364/OE.546049
Dong ZLing YLi YSu Y(2025)Motion Hologram: Jointly optimized hologram generation and motion planning for photorealistic 3D displays via reinforcement learningScience Advances10.1126/sciadv.ads987611:5Online publication date: 31-Jan-2025
https://doi.org/10.1126/sciadv.ads9876
Show More Cited By

Index Terms

Differentiable Compound Optics and Processing Pipeline Optimization for End-to-end Camera Design
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Image and video acquisition
        Computational photography

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 40, Issue 2

April 2021

174 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/3454118

Issue’s Table of Contents

Copyright © 2021 Association for Computing Machinery.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 June 2021

Accepted: 01 January 2021

Revised: 01 December 2020

Received: 01 August 2020

Published in TOG Volume 40, Issue 2

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

66
Total Citations
View Citations
2,103
Total Downloads

Downloads (Last 12 months)461
Downloads (Last 6 weeks)58

Reflects downloads up to 18 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Shi RZhang TZhou YShao YZhang HWei RBai J(2025)End-to-end hybrid infrared imaging system design with thermal analysisOptics Express10.1364/OE.55030933:3(4011)Online publication date: 27-Jan-2025
https://doi.org/10.1364/OE.550309
Seger TMenke CSonntag MUrban K(2025)Efficient evaluation of the Jacobian in the damped least-squares method for optical design problems using algorithmic differentiationOptics Express10.1364/OE.54604933:2(3054)Online publication date: 21-Jan-2025
https://doi.org/10.1364/OE.546049
Dong ZLing YLi YSu Y(2025)Motion Hologram: Jointly optimized hologram generation and motion planning for photorealistic 3D displays via reinforcement learningScience Advances10.1126/sciadv.ads987611:5Online publication date: 31-Jan-2025
https://doi.org/10.1126/sciadv.ads9876
Chan PWei CHuggett ADonzella V(2025)Raw Camera Data Object Detectors: An Optimisation for Automotive Video Processing and TransmissionIEEE Access10.1109/ACCESS.2025.352928713(21695-21706)Online publication date: 2025
https://doi.org/10.1109/ACCESS.2025.3529287
Jiang LZhang YTian GZhang HChen YGao STu Z(2025)Optical aberration correction empowering micro-nano satellite for adaptive-sharpening and wide-parallax imagingOptics and Lasers in Engineering10.1016/j.optlaseng.2024.108761186(108761)Online publication date: Mar-2025
https://doi.org/10.1016/j.optlaseng.2024.108761
Nie YSu RZhang JOttevaere H(2025)End-to-end aberration correction network for enhancing miniature microscope resolutionOptics and Lasers in Engineering10.1016/j.optlaseng.2024.108558184(108558)Online publication date: Jan-2025
https://doi.org/10.1016/j.optlaseng.2024.108558
Zhang WRen ZZhou JChen SFeng HLi QXu ZChen Y(2024)End-to-end automatic lens design with a differentiable diffraction modelOptics Express10.1364/OE.54059032:25(44328)Online publication date: 20-Nov-2024
https://doi.org/10.1364/OE.540590
Rouxel AMonmayrant ALacroix SCamon HLopez S(2024)Accurate ray-tracing optical model for coded aperture spectral snapshot imagersApplied Optics10.1364/AO.51577563:7(1828)Online publication date: 22-Feb-2024
https://doi.org/10.1364/AO.515775
Shi ZDun XWei HDong SWang ZCheng XHeide FPeng Y(2024)Learned Multi-aperture Color-coded Optics for Snapshot Hyperspectral ImagingACM Transactions on Graphics10.1145/368797643:6(1-11)Online publication date: 19-Dec-2024
https://dl.acm.org/doi/10.1145/3687976
Shi ZChugunov IBijelic MCôté GYeom JFu QAmata HHeidrich WHeide F(2024)Split-Aperture 2-in-1 Computational CamerasACM Transactions on Graphics10.1145/365822543:4(1-19)Online publication date: 19-Jul-2024
https://dl.acm.org/doi/10.1145/3658225
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Issue’s Table of Contents