Abstract
Inspired by the compound eyes of insects, many multi-aperture optical imaging systems have been proposed to improve the imaging quality, e.g., to yield a high-resolution image or an image with a large field-of-view. Previous research has reviewed existing multi-aperture optical imaging systems, but few papers emphasize the light field acquisition model which is essential to bridge the gap between configuration design and application. In this paper, we review typical multi-aperture optical imaging systems (i.e., artificial compound eye, light field camera, and camera array), and then summarize general mathematical light field acquisition models for different configurations. These mathematical models provide methods for calculating the key indexes of a specific multi-aperture optical imaging system, such as the field-of-view and sub-image overlap ratio. The mathematical tools simplify the quantitative design and evaluation of imaging systems for researchers.
摘要
受昆虫复眼启发, 为提高光学成像质量, 如获得高分辨率图像或大视场图像, 研究者提出了许多多孔径光学成像系统。光场采集数学模型是联系多孔径光学成像系统结构设计与应用的纽带, 但光场采集数学模型较少被关注。本文系统梳理了典型多孔径光学成像系统(仿生复眼、光场相机、相机阵列), 总结了不同结构下多孔径光学成像系统的一般性光场采集数学模型。列出的数学模型既可用于计算特定多孔径光学成像系统的关键指标, 如视场大小和子图像重叠比等, 也可作为数学工具, 便于研究者完成对成像系统的定量设计与评估。
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Adelson EH, Bergen JR, 1991. The plenoptic function and the elements of early vision. In: Landy MS, Movshon JA (Eds.), Computational Models of Visual Processing. MIT Press, Cambridge, USA, p.3–20.
Afshari H, Jacques L, Bagnato L, et al., 2013. The PANOPTIC camera: a plenoptic sensor with real-time omnidirectional capability. J Signal Process Syst, 70(3):305–328. https://doi.org/10.1007/s11265-012-0668-4
Aurenhammer F, 1991. Voronoi diagrams—a survey of a fundamental geometric data structure. ACM Comput Surv, 23(3):345–405. https://doi.org/10.1145/116873.116880
Barsky BA, Horn DR, Klein SA, et al., 2003. Camera models and optical systems used in computer graphics: Part I, object-based techniques. Int Conf on Computational Science and Its Applications, p.246–255. https://doi.org/10.1007/3-540-44842-X_26
Bishop TE, Zanetti S, Favaro P, 2009. Light field superresolution. IEEE Int Conf on Computational Photography, p.1–9. https://doi.org/10.1109/ICCPHOT.2009.5559010
Bolles RC, Baker HH, Marimont DH, 1987. Epipolar-plane image analysis: an approach to determining structure from motion. Int J Comput Vis, 1(1):7–55. https://doi.org/10.1007/BF00128525
Born M, Wolf E, Bhatia A, et al., 1999. Principles of Optics: Electromagnetic Theory of Propagation, Interference and Diffraction of Light. Cambridge University Press, Cambridge, UK. https://doi.org/10.1017/CBO9781139644181
Brady DJ, Gehm ME, Stack RA, et al., 2012. Multiscale gigapixel photography. Nature, 486(7403):386–389. https://doi.org/10.1038/nature11150
Cao AX, Shi LF, Shi RY, et al., 2014. Image processing algorithm study of large FOV compound eye structure. Acta Photon Sin, 43(5):510005 (in Chinese). https://doi.org/10.3788/gzxb20144305.0510005
Cao AX, Shi LF, Deng QL, et al., 2015. Structural design and image processing of a spherical artificial compound eye. Optik, 126(21):3099–3103. https://doi.org/10.1016/j.ijleo.2015.07.094
Cao AX, Wang JZ, Pang H, et al., 2018. Design and fabrication of a multifocal bionic compound eye for imaging. Bioinspir Biomim, 13(2):026012. https://doi.org/10.1088/1748-3190/aaa901
Carles G, Downing J, Harvey AR, 2014. Super-resolution imaging using a camera array. Opt Lett, 39(7):1889–1892. https://doi.org/10.1364/OL.39.001889
Chai JX, Tong X, Chan SC, et al., 2000. Plenoptic sampling. Proc 27th Annual Conf on Computer Graphics and Interactive Techniques, p.307–318. https://doi.org/10.1145/344779.344932
Cheng Y, Cao J, Zhang YK, et al., 2019. Review of state-of-the-art artificial compound eye imaging systems. Bioinspir Biomim, 14(3):031002. https://doi.org/10.1088/1748-3190/aaffb5
de Berg M, van Kreveld M, Overmars M, et al., 2001. Computational Geometry: Algorithms and Applications (2nd Ed.). Springer-Verlag.
Deng ZF, Chen F, Yang Q, et al., 2016. Dragonfly-eye-inspired artificial compound eyes with sophisticated imaging. Adv Funct Mater, 26(12):1995–2001. https://doi.org/10.1002/adfm.201504941
Duparré J, Dannberg P, Schreiber P, et al., 2005. Thin compound-eye camera. Appl Opt, 44(15):2949–2956. https://doi.org/10.1364/Ao.44.002949
Duparré J, Radtke D, Tünnermann A, 2007. Spherical artificial compound eye captures real images. Proc SPIE 6466, MOEMS and Miniaturized Systems VI, Article 64660K. https://doi.org/10.1117/12.696258
Durand F, Holzschuch N, Soler C, et al., 2005. A frequency analysis of light transport. ACM Trans Graph, 24(3):1115–1126. https://doi.org/10.1145/1073204.1073320
Georgiev T, Chunev G, Lumsdaine A, 2011. Superresolution with the focused plenoptic camera. Proc SPIE, Computational Imaging IX, Article 7873. https://doi.org/10.1117/12.872666
Geyer C, Daniilidis K, 2000. A unifying theory for central panoramic systems and practical implications. Proc 6th European Conf on Computer Vision, p.445–461. https://doi.org/10.1007/3-540-45053-X_29
Golish DR, Vera EM, Kelly KJ, et al., 2012. Development of a scalable image formation pipeline for multiscale gigapixel photography. Opt Expr, 20(20):22048–22062. https://doi.org/10.1364/OE.20.022048
Gong XW, Yu WX, Zhang HX, et al., 2013. Progress in design and fabrication of artificial compound eye optical systems. Chin J Opt, 6(1):34–45 (in Chinese). https://doi.org/10.3788/CO.20130601.0034
Gortler SJ, Grzeszczuk R, Szeliski R, et al., 1996. The lumigraph. Proc 23rd Annual Conf on Computer Graphics and Interactive Techniques, p.43–54. https://doi.org/10.1145/237170.237200
Guo F, Zheng YP, Wang KY, 2012. Lenses matching of compound eye for target positioning. Proc SPIE 8420, 6th Int Symp on Advanced Optical Manufacturing and Testing Technologies: Optical System Technologies for Manufacturing and Testing, Article 84200B. https://doi.org/10.1117/12.979925
Hahne C, Aggoun A, Haxha S, et al., 2014a. Baseline of virtual cameras acquired by a standard plenoptic camera setup. 3DTV-Conf: the True Vision-Capture, Transmission and Display of 3D Video, p.1–3. https://doi.org/10.1109/3DTV.2014.6874734
Hahne C, Aggoun A, Haxha S, et al., 2014b. Light field geometry of a standard plenoptic camera. Opt Expr, 22(22):26659–26673. https://doi.org/10.1364/OE.22.026659
Hao YP, Li L, 2015. New progress in structure design and imaging systems of artificial compound eye. Laser Infr, 45(12):1407–1412 (in Chinese). https://doi.org/10.3969/j.issn.1001-5078.2015.12.001
Hartley R, Zisserman A, 2004. Multiple View Geometry in Computer Vision (2nd Ed.). Cambridge University Press, UK. https://doi.org/10.1017/CBO9780511811685
Heikkila J, Silven O, 1997. A four-step camera calibration procedure with implicit image correction. Proc IEEE Computer Society Conf on Computer Vision and Pattern Recognition, p.1106–1112.
Isaksen A, McMillan L, Gortler SJ, 2000. Dynamically reparameterized light fields. Proc 27th Annual Conf on Computer Graphics and Interactive Techniques, p.297–306. https://doi.org/10.1145/344779.344929
Johannsen O, Sulc A, Goldluecke B, 2016. What sparse light field coding reveals about scene structure. IEEE Conf on Computer Vision and Pattern Recognition, p.3262–3270. https://doi.org/10.1109/CVPR.2016.355
Joshi N, Avidan S, Matusik W, et al., 2007. Synthetic aperture tracking tracking through occlusions. IEEE 11th Int Conf on Computer Vision, p.1–8. https://doi.org/10.1109/ICCV.2007.4409032
Kannala J, Brandt SS, 2006. A generic camera model and calibration method for conventional, wide-angle, and fish-eye lenses. IEEE Trans Patt Anal Mach Intell, 28(8):1335–1340. https://doi.org/10.1109/tpami.2006.153
Kim C, Zimmer H, Pritch Y, et al., 2013. Scene reconstruction from high spatio-angular resolution light fields. ACM Trans Graph, 32(4):1–12. https://doi.org/10.1145/2461912.2461926
Land MF, 1989. Variations in the structure and design of compound eyes. In Stavenga DG, Hardie RC (Eds.), Facets of Vision. Springer, Berlin, p.90–111. https://doi.org/10.1007/978-3-642-74082-4_5
Leitel R, Brückner A, Buß W, et al., 2014. Curved artificial compound-eyes for autonomous navigation. Proc SPIE 9130, Micro-Optics, Article 91300H. https://doi.org/10.1117/12.2052710
Levoy M, 2006. Light fields and computational imaging. Computer, 39(8) 46–55. https://doi.org/10.1109/mc.2006.270
Levoy M, Hanrahan P, 1996. Light field rendering. Proc 23rd Annual Conf on Computer Graphics and Interactive Techniques, p.31–42. https://doi.org/10.1145/237170.237199
Levoy M, Ng R, Adams A, et al., 2006. Light field microscopy. ACM Trans Graph, 25(3):924–934. https://doi.org/10.1145/1141911.1141976
Li L, Yi AY, 2012. Design and fabrication of a freeform microlens array for a compact large-field-of-view compound-eye camera. Appl Opt, 51(12):1843–1852. https://doi.org/10.1364/AO.51.001843
Liang CK, Shih YC, Chen HH, 2011. Light field analysis for modeling image formation. IEEE Trans Image Process, 20(2):446–460. https://doi.org/10.1109/TIP.2010.2063036
Lim JG, Ok HW, Park BK, et al., 2009. Improving the spatial resolution based on 4D light field data. Proc 16th IEEE Int Conf on Image Processing, p.1169–1172. https://doi.org/10.1109/ICIP.2009.5413719
Lin HT, Chen C, Kang SB, et al., 2015. Depth recovery from light field using focal stack symmetry. IEEE Int Conf on Computer Vision, p.3451–3459. https://doi.org/10.1109/ICCV.2015.394
Lin ZC, Shum HY, 2004. A geometric analysis of light field rendering. Int J Comput Vis, 58(2):121–138. https://doi.org/10.1023/B:VISI.0000015916.91741.27
Lindlein N, Leuchs G, 2012. Geometrical optics. In: Träger F (Ed.), Springer Handbook of Lasers and Optics. Springer, Berlin, p.35–87.
Lumsdaine A, Georgiev T, 2009. The focused plenoptic camera. IEEE Int Conf on Computational Photography, p.1–8. https://doi.org/10.1109/ICCPHOT.2009.5559008
Luo JS, Guo YC, Wang X, 2015. Development of a multi-focusing artificial compound eye with decreasing focal length. Acta Photon Sin, 44(10):1022002 (in Chinese). https://doi.org/10.3788/gzxb20154410.1022002
McMillan L, Bishop G, 1995. Plenoptic modeling: an image-based rendering system. Proc 22nd Annual Conf on Computer Graphics and Interactive Techniques, p.39–46. https://doi.org/10.1145/218380.218398
Moon P, Spencer DE, 1953. Theory of the photic field. J Franklin Inst, 255(1):33–50. https://doi.org/10.1016/0016-0032(53)90727-3
Ng R, 2005. Fourier slice photography. ACM Trans Graph, 24(3):735–744. https://doi.org/10.1145/1073204.1073256
Ng R, Hanrahan P, 2005. Light field photography with a hand-held plenoptic camera. Proc SPIE 6342, Int Optical Design Conf, Article 63421E. https://doi.org/10.1117/12.692290
O’Shea DC, Zajac A, 1986. Elements of modern optical design. Phys Today, 39(5):87–88. https://doi.org/10.1063/1.2815008
Pang K, Fang FZ, Song L, et al., 2017. Bionic compound eye for 3D motion detection using an optical freeform surface. J Opt Soc Am B, 34(5):B28–B35. https://doi.org/10.1364/JOSAB.34.000B28
Popovic V, Seyid K, Akin A, et al., 2014. Image blending in a high frame rate FPGA-based multi-camera system. J Signal Process Syst, 76(2):169–184. https://doi.org/10.1007/s11265-013-0858-8
Shi CY, Wang YY, Liu CY, et al., 2017. SCECam: a spherical compound eye camera for fast location and recognition of objects at a large field of view. Opt Expr, 25(26):32333–32345. https://doi.org/10.1364/OE.25.032333
Soler C, Subr K, Durand F, et al., 2009. Fourier depth of field. ACM Trans Graph, 28(2):1–12. https://doi.org/10.1145/1516522.1516529
Song YM, Xie YZ, Malyarchuk V, et al., 2013. Digital cameras with designs inspired by the arthropod eye. Nature, 497(7447):95–99. https://doi.org/10.1038/nature12083
Suo JL, Ji XY, Dai QH, 2012. An overview of computational photography. Sci China Inform Sci, 55(6):1229–1248. https://doi.org/10.1007/s11432-012-4587-6
Tanida J, Kumagai T, Yamada K, et al., 2000. Thin Observation Module by Bound Optics (TOMBO): an optoelectronic image capturing system. Proc SPIE 4089, Optics in Computing, Article 2000. https://doi.org/10.1117/12.386797
Tardif JP, Sturm P, Roy S, 2006. Self-calibration of a general radially symmetric distortion model. Proc 9th European Conf on Computer Vision, p.186–199. https://doi.org/10.1007/11744085_15
Vaish V, Wilburn B, Joshi N, et al., 2004. Using plane + parallax for calibrating dense camera arrays. Proc IEEE Computer Society Conf on Computer Vision and Pattern Recognition, p.2–9. https://doi.org/10.1109/CVPR.2004.257
Vaish V, Levoy M, Szeliski R, et al., 2006. Reconstructing occluded surfaces using synthetic apertures: stereo, focus and robust measures. IEEE Computer Society Conf on Computer Vision and Pattern Recognition, p.2331–2338. https://doi.org/10.1109/CVPR.2006.244
Venkataraman K, Lelescu D, Duparré J, et al., 2013. Picam: an ultra-thin high performance monolithic camera array. ACM Trans Graph, 32(6):1–13. https://doi.org/10.1145/2508363.2508390
Wang TC, Chandraker M, Efros AA, et al., 2018. SVBRDF-invariant shape and reflectance estimation from a light-field camera. IEEE Trans Patt Anal Mach Intell, 40(3):740–754. https://doi.org/10.1109/TPAMI.2017.2680442
Wang YQ, Yang JG, Guo YL, et al., 2019. Selective light field refocusing for camera arrays using bokeh rendering and superresolution. IEEE Signal Process Lett, 26(1):204–208. https://doi.org/10.1109/LSP.2018.2885213
Wang YW, Cai BL, Lu Y, et al., 2017. Optical system design of artificial compound eye based on field stitching. Microw Opt Technol Lett, 59(6):1277–1279. https://doi.org/10.1002/mop.30525
Wanner S, Goldluecke B, 2012a. Globally consistent depth labeling of 4D light fields. IEEE Conf on Computer Vision and Pattern Recognition, p.41–48. https://doi.org/10.1109/CVPR.2012.6247656
Wanner S, Goldluecke B, 2012b. Spatial and angular variational super-resolution of 4D light fields. Proc 12th European Conf on Computer Vision, p.608–621. https://doi.org/10.1007/978-3-642-33715-4_44
Wanner S, Goldluecke B, 2014. Variational light field analysis for disparity estimation and super-resolution. IEEE Trans Patt Anal Mach Intell, 36(3):606–619. https://doi.org/10.1109/TPAMI.2013.147
Wen C, Ma T, Wang C, et al., 2019. Progress in research on the compound eye structure and visual navigation of insects. Chin J Appl Entomol, 56(1):28–36 (in Chinese). https://doi.org/10.7679/j.issn.2095-1353.2019.004
Weng J, Cohen P, Herniou M, 1992. Camera calibration with distortion models and accuracy evaluation. IEEE Trans Patt Anal Mach Intell, 14(10):965–980. https://doi.org/10.1109/34.159901
Wilburn B, Joshi N, Vaish V, et al., 2005. High performance imaging using large camera arrays. ACM Trans Graph, 24(3):765–776. https://doi.org/10.1145/1073204.1073259
Williem W, Park IK, 2016. Robust light field depth estimation for noisy scene with occlusion. IEEE Conf on Computer Vision and Pattern Recognition, p.4396–4404. https://doi.org/10.1109/CVPR.2016.476
Wu G, Masia B, Jarabo A, et al., 2017. Light field image processing: an overview. IEEE J Sel Top Signal Process, 11(7):926–954. https://doi.org/10.1109/jstsp.2017.2747126
Wu SD, Jiang T, Zhang GX, et al., 2017. Artificial compound eye: a survey of the state-of-the-art. Artif Intell Rev, 48(4):573–603. https://doi.org/10.1007/s10462-016-9513-7
Wu SD, Zhang GX, Zhu M, et al., 2018a. Geometry based three-dimensional image processing method for electronic cluster eye. Integr Comput-Aided Eng, 25(3):213–228. https://doi.org/10.3233/ICA-180564
Wu SD, Zhang GX, Jiang T, et al., 2018b. Multi-aperture stereo reconstruction for artificial compound eye with cross image belief propagation. Appl Opt, 57(7):B160–B169. https://doi.org/10.1364/AO.57.00B160
Wu SD, Zhang GX, Neri F, et al., 2019. A multi-aperture optical flow estimation method for an artificial compound eye. Integr Comput-Aided Eng, 26(2):139–157. https://doi.org/10.3233/ICA-180593
Yang JC, Everett M, Buehler C, et al., 2002. A real-time distributed light field camera. Proc 13th Eurographics Workshop on Rendering, p.77–86.
Yang T, Zhang YN, Yu JY, et al., 2014. All-in-focus synthetic aperture imaging. Proc 13th European Conf on Computer Vision, p.1–15.
Yu XD, Zhang YJ, Wang YY, et al., 2019. Optical design of a compound eye camera with a large-field of view for unmanned aerial vehicles. Acta Photon Sin, 48(7):0722003 (in Chinese). https://doi.org/10.3788/gzxb20194807.0722003
Yuan W, Li LH, Lee WB, et al., 2018. Fabrication of microlens array and its application: a review. Chin J Mech Eng, 31(1):16. https://doi.org/10.1186/s10033-018-0204-y
Zhang C, Chen T, 2003. Spectral analysis for sampling image-based rendering data. IEEE Trans Circ Syst Video Technol, 13(11):1038–1050. https://doi.org/10.1109/tcsvt.2003.817350
Zhang JM, Chen Y, Tan HQ, et al., 2020. Optical system of bionic compound eye with large field of view. Opt Prec Eng, 28(5):1012–1020 (in Chinese). https://doi.org/10.3788/OPE.20202805.1012
Zhang YK, Du JL, Shi LF, et al., 2010. Artificial compound-eye imaging system with a large field of view based on a convex solid substrate. Proc SPIE 7848, Holography, Diffractive Optics, and Applications IV, Article 78480U. https://doi.org/10.1117/12.869482
Zhang ZY, 2000. A flexible new technique for camera calibration. IEEE Trans Patt Anal Mach Intell, 22(11):1330–1334. https://doi.org/10.1109/34.888718
Zhang ZZ, Qiu S, Jin WQ, et al., 2018. Image mosaic of bionic compound eye imaging system based on image overlap rate prior. Proc SPIE 10846, Optical Sensing and Imaging Technologies and Applications, Article 108462C. https://doi.org/10.1117/12.2505433
Zhou PL, Yu HB, Zhong Y, et al., 2020. Fabrication of waterproof artificial compound eyes with variable field of view based on the bioinspiration from natural hierarchical micro-nanostructures. Nano-Micro Lett, 12(1):166. https://doi.org/10.1007/s40820-020-00499-x
Zhu H, Wang Q, Yu JY, 2017. Light field imaging: models, calibrations, reconstructions, and applications. Front Inform Technol Electron Eng, 18(9):1236–1249. https://doi.org/10.1631/FITEE.1601727
Zhu L, Zhang YL, Sun HB, 2019. Miniaturising artificial compound eyes based on advanced micronanofabrication techniques. Light Adv Manuf, 2(1):84–100. https://doi.org/10.37188/lam.2021.007
Author information
Authors and Affiliations
Contributions
Qiming QI and Hongqi FAN structured the outline of the paper. Qiming QI drafted the paper. Zhengzheng SHAO and Ping WANG helped check the first two sections. Ruigang FU helped organize the paper. Qiming QI and Hongqi FAN revised and finalized the paper.
Corresponding author
Additional information
Compliance with ethics guidelines
Qiming QI, Ruigang FU, Zhengzheng SHAO, Ping WANG, and Hongqi FAN declare that they have no conflict of interest.
Project supported by the National Natural Science Foundation of China (No. 62001482) and the Hunan Provincial Natural Science Foundation of China (No. 2021JJ40676)
Qiming QI was born in 1994. He received his MS degree in information and communication engineering from National University of Defense Technology, Changsha, China in 2021. His research interests are bionic vision application and optical automatic target recognition.
Hongqi FAN, corresponding author of this invited paper, received his BS degree in mechanical engineering and automation from Tsinghua University, Beijing, China in 2001, and his PhD degree in information and communication engineering from National University of Defense Technology, Changsha, China in 2008. He is currently a professor at National University of Defense Technology. His research interests include information fusion, target tracking, signal processing, and intelligent guidance systems.
Rights and permissions
About this article
Cite this article
Qi, Q., Fu, R., Shao, Z. et al. Multi-aperture optical imaging systems and their mathematical light field acquisition models. Front Inform Technol Electron Eng 23, 823–844 (2022). https://doi.org/10.1631/FITEE.2100058
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1631/FITEE.2100058
Key words
- Multi-aperture optical imaging system
- Artificial compound eye
- Light field camera
- Camera array
- Light field acquisition model