Skip to main content
Log in

On the Mathematical Foundations of Computational Photography

Does the Flutter Shutter Work Better at Night?

  • Published:
Journal of Mathematical Imaging and Vision Aims and scope Submit manuscript

Abstract

Flutter shutter (coded exposure) cameras allow to extend indefinitely the exposure time for uniform motion blurs. Recently, Tendero et al. (SIAM J Imaging Sci 6(2):813–847, 2013) proved that for a fixed known velocity v the gain of a flutter shutter in terms of root means square error (RMSE) cannot exceeds a 1.1717 factor compared to an optimal snapshot. The aforementioned bound is optimal in the sense that this 1.1717 factor can be attained. However, this disheartening bound is in direct contradiction with the recent results by Cossairt et al. (IEEE Trans Image Process 22(2), 447–458, 2013). Our first goal in this paper is to resolve mathematically this discrepancy. An interesting question was raised by the authors of reference (IEEE Trans Image Process 22(2), 447–458, 2013). They state that the “gain for computational imaging is significant only when the average signal level J is considerably smaller than the read noise variance \(\sigma _r^2\)” (Cossairt et al., IEEE Trans Image Process 22(2), 447–458, 2013, p. 5). In other words, according to Cossairt et al. (IEEE Trans Image Process 22(2), 447–458, 2013) a flutter shutter would be more efficient when used in low light conditions e.g., indoor scenes or at night. Our second goal is to prove that this statement is based on an incomplete camera model and that a complete mathematical model disproves it. To do so we propose a general flutter shutter camera model that includes photonic, thermal (The amplifier noise may also be mentioned as another source of background noise, which can be included w.l.o.g. in the thermal noise) and additive [The additive (sensor readout) noise may contain other components such as reset noise and quantization noise. We include them w.l.o.g. in the readout.] (sensor readout, quantification) noises of finite variances. Our analysis provides exact formulae for the mean square error of the final deconvolved image. It also allows us to confirm that the gain in terms of RMSE of any flutter shutter camera is bounded from above by 1.1776 when compared to an optimal snapshot. The bound is uniform with respect to the observation conditions and applies for any fixed known velocity. Incidentally, the proposed formalism and its consequences also apply to e.g., the Levin et al. motion-invariant photography (ACM Trans Graphics (TOG), 27(3):71:1–71:9, 2008; Method and apparatus for motion invariant imag- ing, October 1 2009. US Patent 20,090,244,300, 2009) and variant (Cho et al. Motion blur removal with orthogonal parabolic exposures, 2010). In short, we bring mathematical proofs to the effect of contradicting the claims of Cossairt et al. (IEEE Trans Image Process 22(2), 447–458, 2013). Lastly, this paper permits to point out the kind of optimization needed if one wants to turn the flutter shutter into a useful imaging tool.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Similar content being viewed by others

References

  1. Agrawal, A., Raskar, R.: Resolving objects at higher resolution from a single motion-blurred image. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1–8 (2007)

  2. Agrawal, A., Raskar, R.: Optimal single image capture for motion deblurring. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2560–2567 (2009)

  3. Agrawal, A., Xu, Y.: Coded exposure deblurring: optimized codes for PSF estimation and invertibility. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2066–2073 (2009)

  4. Boracchi, G., Foi, A.: Uniform motion blur in Poissonian noise: blur/noise tradeoff. IEEE Trans Image Process 20(2), 592–598 (2011)

    Article  MathSciNet  Google Scholar 

  5. Buades, T., Lou, Y., Morel, J.-M., Tang, Z.: A note on multi-image denoising. In: Proceedings of the IEEE International Workshop on Local and Non-Local Approximation in Image Processing (LNLA), pp. 1–15 (2009)

  6. Cho, T.S., Levin, A., Durand, F., Freeman, W.T.: Motion blur removal with orthogonal parabolic exposures. In: Proceedings of the IEEE International Conference on Computational Photography (ICCP), pp. 1–8 (2010)

  7. Cossairt, O., Gupta, M., Mitra, K., Veeraraghavan, A.: Performance bounds for computational imaging. In: Computational Optical Sensing and Imaging. Optical Society of America (2013)

  8. Cossairt, O., Gupta, M., Nayar, S.K.: When does computational imaging improve performance? IEEE Trans Image Process 22, 447–458 (2013)

  9. Ding, Y., McCloskey, S., Yu, J.: Analysis of motion blur with a flutter shutter camera for non-linear motion. In: Proceedings of the Springer-Verlag European Conference on Computer Vision (ECCV), pp. 15–30 (2010)

  10. Foi, A., Alenius, S., Trimeche, M., Katkovnik, V., Egiazarian, K.: A spatially adaptive Poissonian image deblurring. In: Proceedings of the IEEE International Conference on Image Processing (ICIP), 1:I–925 (2005)

  11. Gehm, M.E., John, R., Brady, D.J., Willett, R.M., Schulz, T.J.: Single-shot compressive spectral imaging with a dual-disperser architecture. Opt. Express 15(21), 14013–14027 (2007)

    Article  Google Scholar 

  12. Hitomi, Y., Gu, J., Gupta, M., Mitsunaga, T., Nayar, S.K.: Video from a single coded exposure photograph using a learned over-complete dictionary. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV) (2011)

  13. Karr, A.F.: Probability. Springer Texts in Statistics. Springer, New York (1993)

    Google Scholar 

  14. Koppal, S.J., Yamazaki, S., Narasimhan, S.G.: Exploiting DLP illumination dithering for reconstruction and photography of high-speed scenes. Int. J. Comput. Vis. 96, 125–144 (2012)

    Article  MathSciNet  Google Scholar 

  15. Kozuka, H.: Image sensor, October 29, 2002. US Patent 6,473,538

  16. Krantz, S.G.: A Panorama of Harmonic Analysis. Carus Mathematical Monographs, vol. 27. Mathematical Association of America, New York (1999)

    Google Scholar 

  17. Levin, A., Sand, P., Cho, T.S., Durand, F., Freeman, W.T.: Motion-invariant photography. ACM Trans. Graphics (TOG) 27(3), 71:1–71:9 (2008)

    Article  Google Scholar 

  18. Levin, A., Sand, P., Cho, T.S., Durand, F., Freeman, W.T.: Method and apparatus for motion invariant imaging, October 1 2009. US Patent 20,090,244,300

  19. McCloskey, S.: Temporally coded flash illumination for motion deblurring. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 683–690 (2011)

  20. McCloskey, S., Au, W., Jelinek, J.: Iris capture from moving subjects using a fluttering shutter. In: Proceedings of the IEEE International Conference on Biometrics: Theory Applications and Systems (BTAS), pp. 1–6 (2010)

  21. McCloskey, S., Muldoon, K., Venkatesha, S.: Motion invariance and custom blur from lens motion. In: Proceedings of the IEEE International Conference on Computational Photography (ICCP), pp. 1–8 (2011)

  22. Mitra, K., Cossairt, O., Veeraraghavan, A.: A framework for the analysis of computational imaging systems with practical applications. arXiv:1308.1981, PAMI (submitted) (2013)

  23. Mitra, K., Cossairt, O., Veeraraghavan, A.: Performance limits for computational photography. In: International Workshop on Advanced Optical Imaging and Metrology (2013)

  24. Nakamura, K., Ohzu, H., Ueno, I.: Image sensor in which reading and resetting are simultaneously performed, November 16, 1993. US Patent 5,262,870

  25. Narasimhan, S.G., Koppal, S.J., Yamazaki, S.: Temporal dithering of illumination for fast active vision. In: Proceedings of the Springer-Verlag European Conference on Computer Vision (ECCV), pp. 830–844 (2008)

  26. Nayar, S.K., Ben-Ezra, M.: Motion-based motion deblurring. IEEE Trans. Pattern Anal. Mach. Intell. 26(6), 689–698 (2004)

    Article  Google Scholar 

  27. Raskar, R.: Method and apparatus for deblurring images, July 13, 2010. US Patent 7,756,407

  28. Raskar, R., Agrawal, A., Tumblin, J.: Coded exposure photography: motion deblurring using fluttered shutter. ACM Trans. Graphics (TOG) 25(3), 795–804 (2006)

    Article  Google Scholar 

  29. Raskar, R., Tumblin, J., Agrawal, A.: Method for deblurring images using optimized temporal coding patterns, August 25, 2009. US Patent 7,580,620

  30. Reddy, D., Veeraraghavan, A., Chellappa, R.: P2C2: programmable pixel compressive camera for high speed imaging. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 329–336 (2011)

  31. Reddy, D., Veeraraghavan, A., Raskar, R.: Coded strobing photography for high-speed periodic events. In: Imaging Systems. Optical Society of America (2010)

  32. Sarker, A., Hamey, L.G.C.: Improved reconstruction of flutter shutter images for motion blur reduction. In: Proceedings of the IEEE International Conference on Digital Image Computing: Techniques and Applications (DICTA), pp. 417–422 (2010)

  33. Shi, G., Gao, D., Liu, D., Wang, L.: High resolution image reconstruction: a new imager via movable random exposure. In: Proceedings of the IEEE International Conference on Image Processing (ICIP), pp. 1177–1180 (2009)

  34. Shiryaev, A.N.: Probability. Graduate Texts in Mathematics. Springer, New York (1984)

    Google Scholar 

  35. Tai, Y.W., Du, H., Brown, M.S., Lin, S.: Correction of spatially varying image and video motion blur using a hybrid camera. IEEE Trans. Pattern Anal. Mach. Intell. 32(6), 1012–1028 (2010)

    Article  Google Scholar 

  36. Tai, Y.W., Kong, N., Lin, S., Shin, S.Y.: Coded exposure imaging for projective motion deblurring. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2408–2415 (2010)

  37. Telleen, J., Sullivan, A., Yee, J., Wang, O., Gunawardane, P., Collins, I., Davis, J.: Synthetic shutter speed imaging. Comput. Graphics Forum 26(3), 591–598 (2007)

    Article  Google Scholar 

  38. Tendero, Y.: The flutter shutter camera simulator. Image Process. Line 2, 225–242 (2012)

    Article  Google Scholar 

  39. Tendero, Y.: Are random flutter shutter codes good enough? Submittted (UCLA CAM report 13-84), pp. 1–23 (2013)

  40. Tendero, Y., Morel, J.-M.: An optimal blind temporal motion blur deconvolution filter. IEEE Trans. Signal Process. Lett. 20(5), 523–526 (2013)

    Article  Google Scholar 

  41. Tendero, Y., Morel, J.-M., Rougé, B.: A formalization of the flutter shutter. J. Phys. 386(1), 012001 (2012)

    Google Scholar 

  42. Tendero, Y., Morel, J.-M., Rougé, B.: The flutter shutter paradox. SIAM J. Imaging Sci. 6(2), 813–847 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  43. Tendero, Y., Osher, S.: Blind uniform motion blur deconvolution for image bursts and video sequences based on sensor characteristics. Submittted (UCLA CAM report 14-17), pp. 1–4 (2014)

  44. Tsai, R.: Pulsed control of camera flash, June 14, 2011. US Patent 7,962,031

  45. Veeraraghavan, A., Reddy, D., Raskar, R.: Coded strobing photography: compressive sensing of high speed periodic events. IEEE Trans. Pattern Anal. Mach. Intell. 33(4), 671–686 (2011)

    Article  Google Scholar 

  46. Wilburn, B., Joshi, N., Vaish, V., Talvala, E.V., Antunez, E., Barth, A., Adams, A., Horowitz, M., Levoy, M.: High performance imaging using large camera arrays. ACM Trans. Graphics (TOG) 24(3), 765–776 (2005)

    Article  Google Scholar 

  47. Xu, S., Liang, H., Tu, D., Li, G.: A deblurring technique for large scale motion blur images using a hybrid camera. Proceedings of the IEEE International Congress on Image and Signal Processing (ICISP), vol. 2, pp. 806–810 (2010)

  48. Xu, W., McCloskey, S.: 2D Barcode localization and motion deblurring using a flutter shutter camera. In: Proceedings of the IEEE Workshop on Applications of Computer Vision (WACV), pp. 159–165 (2011)

Download references

Acknowledgments

Yohann Tendero is partially supported on the grants ONR No: N00014-14-1-0683 and BEA Appareo. Part of this work was done at the UCLA Math. dept. Jean-Michel Morel is partially founded by the European Research Council (advanced grant Twelve Labours No: 246961), the office of Naval research (ONR grant N00014-14-1-0023), and ANR-DGA project ANR-12-ASTR-0035.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Yohann Tendero.

Additional information

Part of this work was done at the UCLA Mathematics Department.

Appendices

Annex 1: Proof of Lemma 1

For any \(x \in \mathbb {R}\), we have

$$\begin{aligned} \nonumber&\sum _{n=-\infty }^\infty \left( \frac{1}{|v|}\alpha \left( \frac{\cdot }{v}\right) *u\right) (n)\gamma (x-n) \nonumber \\&\quad = \int _{-\infty }^{+\infty }\left( \frac{1}{|v|}\alpha \left( \frac{\cdot }{v}\right) *u\right) (y)\gamma (x-y)\mathrm{d}y\end{aligned}$$
(44)
$$\begin{aligned}&\quad =\left[ \left( \frac{1}{|v|}\alpha \left( \frac{\cdot }{v}\right) *u\right) *\gamma \right] (x)\end{aligned}$$
(45)
$$\begin{aligned} \nonumber&\quad =\mathcal {F}^{-1}\left[ \mathcal {F}\left( \frac{1}{|v|}\alpha \left( \frac{\cdot }{v}\right) *u\right) (\xi ) \mathcal {F}(\gamma )(\xi )\right] (x)\nonumber \\&\quad =\mathcal {F}^{-1}\left[ \hat{\alpha }(\xi v)\hat{u}(\xi ) \hat{\gamma }(\xi )\right] (x)\end{aligned}$$
(46)
$$\begin{aligned}&\quad =\frac{1}{2\pi }\int _{-\infty }^{+\infty }\hat{\alpha }(\xi v) \hat{u}(\xi )\hat{\gamma }(\xi )e^{ix\xi }\mathrm{d}\xi \end{aligned}$$
(47)
$$\begin{aligned}&\quad =\frac{1}{2\pi }\int _{-\pi }^{+\pi }\hat{\alpha }(\xi v) \hat{u}(\xi )\hat{\gamma }(\xi )e^{ix\xi }\mathrm{d}\xi \end{aligned}$$
(48)
$$\begin{aligned}&\quad =\frac{1}{2\pi }\int _{-\pi }^{\pi } \hat{u}(\xi )e^{ix\xi }\mathrm{d}\xi = u(x). \end{aligned}$$
(49)

Poisson’s formula (xxv) justifies (44). Indeed, consider

$$\begin{aligned} f_x(y)=\left[ \frac{1}{|v|}\alpha \left( \frac{\cdot }{v}\right) *u\right] (y)\gamma (x-y). \end{aligned}$$
(50)

From Cauchy–Schwartz’s inequality we have

$$\begin{aligned} \Vert f_x\Vert _{L^1(\mathbb {R})} \leqslant \left\| \frac{1}{|v|}\alpha \left( \frac{\cdot }{v}\right) *u \right\| _{L^2(\mathbb {R})} \Vert \gamma \Vert _{L^2(\mathbb {R})}. \end{aligned}$$

Indeed, since \(u \in L^1(\mathbb {R})\) by Young’s inequality we have

$$\begin{aligned} \left\| \frac{1}{|v|}\alpha \left( \frac{\cdot }{v}\right) *u \right\| _{L^2(\mathbb {R})}\leqslant & {} \left\| \frac{1}{|v|}\alpha \left( \frac{\cdot }{v}\right) \right\| _{L^2(\mathbb {R})} \Vert u \Vert _{L^1(\mathbb {R})}\\= & {} \Vert \alpha \Vert _{L^2(\mathbb {R})} \Vert u \Vert _{L^1(\mathbb {R})} < + \infty , \end{aligned}$$

because \( \alpha \in L^2(\mathbb {R})\) and \( u \in L^1(\mathbb {R})\). Since \(\gamma \in L^2(\mathbb {R})\), from (50), we deduce that \(f_x \in L^1(\mathbb {R})\). Moreover, we have

$$\begin{aligned} \hat{f}_x(\xi )=\left( \left( \hat{\alpha }\left( \cdot v\right) \hat{u}\left( \cdot \right) \right) *\hat{\gamma }\left( \cdot \right) e^{-ix\xi }\right) (\xi ). \end{aligned}$$
(51)

Since u is \([-\pi ,\pi ]\) band-limited and \(\gamma \) is \([-\pi ,\pi ]\) band-limited from its Definition (19) we deduce that \(\hat{f}_x(\xi )=0\) for all \(\xi \in \mathbb {R}\) such that \(|\xi |>\pi \). Yet, from Poisson’s formula (xxv) this only implies that

$$\begin{aligned} \sum _n f_x(n)=\hat{f}_x(0)+\hat{f}_x(-2\pi )+\hat{f}_x(2\pi ). \end{aligned}$$

From (51), \(f_x(y)\) is given by a convolution. Thus,

$$\begin{aligned} \hat{f}_x(2\pi )= & {} \int _\mathbb {R}\hat{\alpha }(\xi v) \hat{u}(\xi )\hat{\gamma }(\xi -2\pi )e^{-ix\xi }\mathrm{d}\xi \\= & {} \int _{-\pi }^\pi \hat{\alpha }(\xi v) \hat{u}(\xi )\hat{\gamma }(\xi -2\pi )e^{-ix\xi }\mathrm{d}\xi \end{aligned}$$

since \(\hat{u}\) is \([-\pi ,\pi ]\) band-limited. Since the integrand is non-zero on the set \(\{\pi \}\), which has Lebesgue measure zero, we deduce that \(\hat{f}_x(2\pi )=0\). Similarly we obtain \(\hat{f}_x( -2\pi )=0\). The validity of (45)–(46) is, from the continuity of the \(L^2(\mathbb {R})\) function \(\mathbb {R}\ni x \mapsto \left[ \left( \frac{1}{|v|}\alpha \left( \frac{\cdot }{v}\right) *u\right) *\gamma \right] (x)\), obvious. Not less obvious is (47) from the inverse Fourier transform definition (xxiii). The definition (19) of \(\hat{\gamma }\) justifies (48). The fact that u is \([-\pi ,\pi ]\) band-limited and again the definition of \(\hat{\gamma }\) (19) justifies (49) and completes our proof.

Annex 2: Proof of Lemma 2

We first prove (23) then prove that \(\mathrm {var}\left( {\widetilde{\mathbbm {u}}}_\mathrm{est,ana}(\cdot ) \right) \) can be written as the sum of an \(L^1(\mathbb {R})\) function and a constant. From Proposition 3, for any \(x\in \mathbb {R}\), we have

$$\begin{aligned} \mathrm {var}\left( {\widetilde{\mathbbm {u}}}_\mathrm{est,ana}(x) \right) =\sum _{n=-\infty }^\infty \mathrm {var}\left( obs(n)\right) (\gamma (x-n))^2. \end{aligned}$$
(52)

Therefore, combining Proposition 1 (Eq. (7)) and (52), for any \(x\in \mathbb {R}\), we have

$$\begin{aligned} \nonumber&\mathrm {var}\left( {\widetilde{\mathbbm {u}}}_\mathrm{est,ana}(x) \right) \\ \nonumber&\quad =\sum _{n=-\infty }^\infty \left[ \left( \frac{1}{|v|} \alpha \left( \frac{\cdot }{v}\right) *u\right) (n)\right. \nonumber \\&\qquad \left. + \left( \Vert \alpha \Vert _{L^1(\mathbb {R})} \left( \mu +\sigma _o^2\right) +\sigma ^2_r\right) \right] \left( \gamma (x-n)\right) ^2 \nonumber \\&\quad =\sum _{n=-\infty }^\infty \left( \frac{1}{|v|} \alpha \left( \frac{\cdot }{v}\right) *u\right) (n)(\gamma (x-n))^2 \nonumber \\&\qquad +\left( \Vert \alpha \Vert _{L^1(\mathbb {R})} \left( \mu +\sigma _o^2\right) +\sigma ^2_r\right) \sum _{n=-\infty }^\infty (\gamma (x-n))^2 \end{aligned}$$
(53)
$$\begin{aligned}&\quad =\sum _{n=-\infty }^\infty \left( \frac{1}{|v|} \alpha \left( \frac{\cdot }{v}\right) *u\right) (n)(\gamma (x-n))^2 \\ \nonumber&\qquad +\,\Vert \gamma \Vert _{L^2(\mathbb {R})}^2\left( \Vert \alpha \Vert _{L^1(\mathbb {R})} \left( \mu +\sigma _o^2\right) +\sigma ^2_r\right) , \end{aligned}$$
(54)

and (23) is proved. As a consequence, the function \(\mathbb {R}\ni x \mapsto \mathrm {var}\left( {\widetilde{\mathbbm {u}}}_\mathrm{est,ana}(x) \right) \) can be written as

$$\begin{aligned} \mathrm {var}\left( {\widetilde{\mathbbm {u}}}_\mathrm{est,ana}(x) \right)= & {} f_{\text {ana}}(x) +\Vert \gamma \Vert _{L^2(\mathbb {R})}^2\\&\times \left( \Vert \alpha \Vert _{L^1(\mathbb {R})} \left( \mu +\sigma _o^2\right) +\sigma ^2_r\right) , \end{aligned}$$

where \(f_{\text {ana}}(\cdot )\) is defined by (24). It remains to justify that \(f_{\text {ana}} \in L^1(\mathbb {R})\). We have

$$\begin{aligned}&\int _{-\infty }^{+\infty } \left| \sum _{n=-\infty }^\infty \left[ \left( \frac{1}{|v|} \alpha \left( \frac{\cdot }{v}\right) *u\right) (n)(\gamma (x-n))^2\right] \right| \mathrm{d}x\\&\quad \leqslant \int _{-\infty }^{+\infty }\sum _{n=-\infty }^\infty \left[ \left| \left( \frac{1}{|v|} \alpha \left( \frac{\cdot }{v}\right) *u\right) (n)\right| (\gamma (x-n))^2 \right] \mathrm{d}x\\&\quad =\sum _{n=-\infty }^\infty \left[ \left| \left( \frac{1}{|v|} \alpha \left( \frac{\cdot }{v}\right) *u\right) (n)\right| \int _{-\infty }^{+\infty }(\gamma (x-n))^2dx \right] \\&\quad =\sum _{n=-\infty }^\infty \left[ \left| \left( \frac{1}{|v|} \alpha \left( \frac{\cdot }{v}\right) *u\right) (n)\right| \Vert \gamma \Vert _{L^2(\mathbb {R})}^2 \right] \\&\quad =\Vert \gamma \Vert _{L^2(\mathbb {R})}^2 \sum _{n=-\infty }^\infty \left| \left( \frac{1}{|v|} \alpha \left( \frac{\cdot }{v}\right) *u\right) (n)\right| \\&\quad =\Vert \gamma \Vert _{L^2(\mathbb {R})}^2 \int _{-\infty }^{+\infty } \left| \left( \frac{1}{|v|} \alpha \left( \frac{\cdot }{v}\right) *u\right) (x)\right| \mathrm{d}x\\&\hbox { (by~(xxv) equation~(57))}\\&\quad = \Vert \gamma \Vert _{L^2(\mathbb {R})} \left\| \frac{1}{|v|} \alpha \left( \frac{\cdot }{v}\right) *u \right\| _{L^1(\mathbb {R})}<\\&\qquad +\infty (L^1 *L^1 \subset L^1 \hbox { convolution)}. \end{aligned}$$

This concludes our proof.

Annex 3: Proof of Proposition 4

We first prove (26) then (27). From the numerical flutter shutter samples definition (Definition 6), since, by assumption, \(\mathbb {E}(\eta (n))=0\) we have

$$\begin{aligned} \mathbb {E}\left( obs(n) \right)= & {} \sum _{k=0}^{L-1} \alpha _k \int _{k \Delta t}^{(k+1)\Delta t} u(n-vt)+\mu +\sigma _o^2 \mathrm{d}t\\= & {} \int _0^{L\Delta t} \alpha (t) \left( u(n-v t)+\mu +\sigma _o^2\right) \mathrm{d}t . \end{aligned}$$

Thus,

$$\begin{aligned} \nonumber \mathbb {E}\left( obs(n) \right)= & {} \int _0^{L|v|\Delta t} \frac{1}{|v|} \alpha \left( \frac{t}{v}\right) \left( u(n-t)+\mu +\sigma _o^2 \right) \mathrm{d}t\\= & {} \left( \frac{1}{|v|} \alpha \left( \frac{\cdot }{v}\right) *u \right) (n)+\left( \mu + \sigma _o^2\right) \left( \int _{-\infty }^\infty \alpha (t) \mathrm{d}t \right) \end{aligned}$$

and (26) is proved. We now prove (27). From Definition 6, since \(\mathrm {var}\left( \eta (n) \right) =\sigma ^2_r\) we have

$$\begin{aligned}&\mathrm {var}\left( obs(n) \right) \\&\quad = \sum _{k=0}^{L-1} \alpha _k^2 \left( \int _{k\Delta t}^{(k+1)\Delta t} u(n-vt)+\mu +\sigma _o^2 dt+\mathrm {var}\left( \eta (k) \right) \right) \\&\quad =\left( \frac{1}{|v|} \alpha ^2\left( \frac{\cdot }{v}\right) *\left( u +\mu +\sigma _o^2\right) \right) (n) + \sum _{k=0}^{L-1} \alpha _k^2 \mathrm {var}\left( \eta (k) \right) \\&\quad =\left( \frac{1}{|v|} \alpha ^2\left( \frac{\cdot }{v}\right) *u \right) (n)+ \Vert \alpha \Vert ^2_{L^2(\mathbb {R})} \left( \mu +\sigma _o^2\right) + \sum _{k=0}^{L-1} \alpha _k^2 \sigma _r^2 \\&\quad =\left( \frac{1}{|v|} \alpha ^2\left( \frac{\cdot }{v}\right) *u \right) (n) + \Vert \alpha \Vert _{L^2(\mathbb {R})}^2 \left( \mu +\sigma _o^2\right) + \frac{ \Vert \alpha \Vert _{L^2(\mathbb {R})}^2}{\Delta t}\sigma _r^2 \\&\quad = \left( \frac{1}{|v|} \alpha ^2\left( \frac{\cdot }{v}\right) *u \right) (n) + \Vert \alpha \Vert _{L^2(\mathbb {R})}^2 \left( \mu +\sigma ^2_o + \frac{\sigma _r^2}{\Delta t} \right) \end{aligned}$$

and (27) is proved. This concludes our proof.

Annex 4: Proof of Theorem 2

The proof follows the same path as the proof of Lemma 2 on Annex 1: we first justify that \(\mathrm {var}\left( {\widetilde{\mathbbm {u}}}_\mathrm{est,num}(x)\right) \) can be written as the sum of an \(L^1(\mathbb {R})\) function and a constant. We then prove the formula in Theorem 2.

From Proposition 5, we have that

$$\begin{aligned}&\mathrm {var}\left( {\widetilde{\mathbbm {u}}}_\mathrm{est,num}(x) \right) \\&\quad =\sum _{n=-\infty }^\infty \mathrm {var}\left( obs(n) \right) (\gamma (x-n))^2\\&\quad =\sum _{n=-\infty }^\infty \left[ \left( \frac{1}{|v|} \alpha ^2\left( \frac{\cdot }{v}\right) *u\right) (n)+ \Vert \alpha \Vert _{L^2(\mathbb {R})}^2\right. \\&\qquad \left. \left( \frac{\sigma _r^2}{\Delta t} + \mu +\sigma _o^2\right) \right] (\gamma (x-n))^2, \end{aligned}$$

where the last equality is justified by Proposition 4. The rest of the calculation is similar to (53)–(54) in Annex 1, and we obtain that

$$\begin{aligned} \mathrm {var}\left( {\widetilde{\mathbbm {u}}}_\mathrm{est,num}(x) \right)= & {} f_{\text {num}}(x)+\Vert \alpha \Vert _{L^2(\mathbb {R})}^2 \Vert \gamma \Vert _{L^2(\mathbb {R})}^2 \\&\times \left( \frac{\sigma _r^2}{\Delta t} + \mu +\sigma _o^2\right) , \end{aligned}$$

for any \(x \in \mathbb {R}\), where

$$\begin{aligned} f_{\text {num}}(\cdot )\!:=\! \sum _{n=-\infty }^\infty \left[ \left( \frac{1}{|v|} \alpha ^2\left( \frac{\cdot }{v}\right) *u\right) (n)(\gamma (\cdot -n))^2\right] \!\in \! L^1(\mathbb {R}). \end{aligned}$$

The proof that \(f_{\text {num}} \in L^1(\mathbb {R})\) is identical to the proof that \(f_{\text {ana}} \in L^1(\mathbb {R})\) given in Annex 1. Indeed, \(\alpha \in L^1(\mathbb {R}) \cap L^2(\mathbb {R})\) and therefore \(\alpha ^2 \in L^1(\mathbb {R})\). We now justify the formula in Theorem 2. From Proposition 5, for any \(x\in \mathbb {R}\) we have \(\mathbb {E}({\widetilde{\mathbbm {u}}}_\mathrm{est,num}(x))=\widetilde{u}(x)\) and therefore \(\mathbb {E}|{\widetilde{\mathbbm {u}}}_\mathrm{est,num}(x) - \widetilde{u}(x) |^2=\mathrm {var}\left( {\widetilde{\mathbbm {u}}}_\mathrm{est,num}(x) \right) .\) Thus, by similar calculations between Lemma 2 and Theorem 1 we obtain

$$\begin{aligned}&\lim _{T \rightarrow +\infty }\frac{1}{2T} \int _{-T}^T \mathbb {E}|{\widetilde{\mathbbm {u}}}_\mathrm{est,num}(x) - \widetilde{u}(x)|^2 \\&\quad dx= \Vert \alpha \Vert _{L^2(\mathbb {R})}^2 \Vert \gamma \Vert _{L^2(\mathbb {R})}^2 \left( \frac{\sigma _r^2}{\Delta t} + \mu +\sigma _o^2\right) \end{aligned}$$

and Theorem 2 is proved.

Annex 5: Main Notations and Formulae

  1. (i)

    t time variable

  2. (ii)

    \(\Delta t\) length of a time interval (exposure time)

  3. (iii)

    \(x \in \mathbb {R}\) spatial variable

  4. (iv)

    \(X \sim Y\) means that the random variables X and Y have the same law

  5. (v)

    \(\mathbb {P}(A)\) probability of an event A

  6. (vi)

    \(\mathbb {E}(X)\) expected value of a random variable X

  7. (vii)

    \(\mathrm {var}(X)\) variance of a random variable X

  8. (viii)

    \(f *g\) convolution of two \(L^2 (\mathbb {R})\) functions \((f *g)(x) = \int _{\mathbb {R}} f(y) g(x-y) dy\) (the validity is discussed as soon as both functions are not in \(L^2(\mathbb {R})\))

  9. (ix)

    \(l(t,x)>0 ~ \forall ~ x \in \mathbb {R}^+ \times \mathbb {R}\) continuous landscape before passing through the optical system

  10. (x)

    \(\mathcal P(\lambda )\) Poisson random variable with intensity \(\lambda >0\)

  11. (xi)

    \(g\) point-spread-function of the optical system. Assumption \(g\in S(\mathbb {R})\) (\(S(\mathbb {R})\) denotes the Schwartz class on \(\mathbb {R}\))

  12. (xii)

    \(\sigma _o^2<+\infty \) variance of the thermal noise

  13. (xiii)

    \(\sigma _r^2<+\infty \) variance of the additive (sensor readout, quantization) noise

  14. (xiv)

    \(\mu <+\infty \) average mean of \(\widetilde{u}\) (see (xv))

  15. (xv)

    \(\widetilde{u}=\mathbbm {1}_{[-\frac{1}{2},\frac{1}{2}]}*g*l\) ideal observable landscape just before sampling. Assumption: \(\widetilde{u}\) \([-\pi ,\pi ]\) band-limited, \(\mu :=\lim _{T\rightarrow +\infty } \frac{1}{2T} \int _{-T}^T \widetilde{u}(x) \mathrm{d}x\) and \(u:=(\widetilde{u}-\mu ) \in L^1(\mathbb {R})\cap L^2(\mathbb {R})\)

  16. (xvi)

    \(obs(n)\), \(n \in \mathbb {Z}\) observation of the landscape at pixel n

  17. (xvii)

    \({\widetilde{\mathbbm {u}}}_\mathrm{est,ana}\) estimated landscape for the analog flutter shutter from the observed samples \(obs(n)\) where \(n \in \mathbb {Z}\). It is defined to have \(\mathbb {E}({\widetilde{\mathbbm {u}}}_\mathrm{est,ana}(x))=\widetilde{u}(x)\) for any \(x \in \mathbb {R}\).

  18. (xviii)

    \({\widetilde{\mathbbm {u}}}_\mathrm{est,num}\) estimated landscape for the numerical flutter shutter from the observed samples \(obs(n)\) where \(n \in \mathbb {Z}\). It is defined to have \(\mathbb {E}({\widetilde{\mathbbm {u}}}_\mathrm{est,ana}(x))=\widetilde{u}(x)\) for any \(x \in \mathbb {R}\).

  19. (xix)

    v relative velocity between the landscape and the camera (unit: pixels per second)

  20. (xx)

    \(\alpha (t)\) flutter shutter gain function.

  21. (xxi)

    \(\mathbbm {1}_{[a,b]}\) indicator function of an interval [ab]

  22. (xxii)

    \( \Vert f \Vert _{L^1(\mathbb {R})}= \int _\mathbb {R}|f(x)| \mathrm{d}x\), \( \Vert f \Vert _{L^2(\mathbb {R})}= \sqrt{\int _\mathbb {R}|f(x)|^2 \mathrm{d}x}\)

  23. (xxiii)

    Let \(f,g \in L^1(\mathbb {R})\) or \(L^2(\mathbb {R})\), then

    Moreover \(\mathcal {F}(f*g)(\xi )=\mathcal {F}(f)(\xi )\mathcal {F}(g)(\xi )\) and (Plancherel)

    $$\begin{aligned} \int _{\mathbb {R}} |f(x)|^2dx= & {} \Vert f \Vert _{L^2(\mathbb {R})}^2=\frac{1}{2 \pi } \int _{\mathbb {R}} |\mathcal {F}(f)|^2(\xi ) \mathrm{d}\xi \nonumber \\&= \frac{1}{2\pi } \left\| \mathcal {F}(f) \right\| _{L^2(\mathbb {R})}^2. \end{aligned}$$
    (55)
  24. (xxiv)

    \(\mathrm {sinc}(x)=\frac{\sin (\pi x)}{\pi x}=\frac{1}{2 \pi } \mathcal {F}(\mathbbm {1}_{[-\pi ,\pi ]})(x)=\mathcal {F}^{-1}(\mathbbm {1}_{[-\pi ,\pi ]})(x)\)

  25. (xxv)

    (Poisson summation formula) Let \(\varphi \in L^1(\mathbb {R})\) be \([-\pi ,\pi ]\) band-limited.

    $$\begin{aligned} \sum _n \varphi (n)=\sum _k \hat{\varphi }(2k\pi ). \end{aligned}$$
    (56)

    Since \(\varphi \) is \([-\pi ,\pi ]\) band-limited \(\hat{\varphi }(\xi )=0\) \(\forall \xi \in \mathbb {R}\) such that \(|\xi |>\pi \), then from Poisson’s formula (xxv) Eq. (56) we have

    $$\begin{aligned} \sum _n\varphi (n)=\hat{\varphi }(0)=\int _{\mathbb {R}}\varphi (x)\mathrm{d}x. \end{aligned}$$
    (57)

    This applies to any shift of \(\varphi \), so we also have

    $$\begin{aligned} \sum _n\varphi (x+n)=\int _{\mathbb {R}}\varphi (x)\mathrm{d}x. \end{aligned}$$
    (58)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Tendero, Y., Morel, JM. On the Mathematical Foundations of Computational Photography. J Math Imaging Vis 54, 378–397 (2016). https://doi.org/10.1007/s10851-015-0609-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10851-015-0609-5

Keywords

Navigation