Skip to main content
SpringerLink
Log in

Wave-digital filter circuits for single-chip 4-D light field depth-based enhancement

  • Published:
Multidimensional Systems and Signal Processing Aims and scope Submit manuscript

Abstract

In four-dimensional (4-D) light field (LF) processing, 4-D linear shift-invariant filters having hyperplanar passbands are used for depth-based scene enhancement. In this paper, two low-sensitivity and low-complexity field programmable gate array (FPGA)-based digital hardware architectures for 4-D hyperplanar filters are proposed for on-chip real-time processing of LFs. Both 4-D filters are designed exploiting resonant properties of multi-dimensional passive prototype networks, and are realized as wave digital filters (WDFs). The two 4-D WDF realizations are implemented as raster-scanned processing architectures on a Xilinx Virtex 6 Sx35 FPGA with a real-time clock speed of up to 33 MHz. This corresponds to a real-time throughout of 16.8 LFs/s for an LF of size \(11 \times 11 \times 128 \times 128\).

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  • Agathoklis, P., & Bruton, L. T. (1983). Practical-BIBO stability of n-dimensional discrete systems. IEE Proceedings G (Electronic Circuits and Systems), 130(6), 236. https://doi.org/10.1049/ip-g-1.1983.0045.

    Article  Google Scholar 

  • Bajpayee, A., Techet, A. H., & Singh, H. (2018). Real-time light field processing for autonomous robotics. In 2018 IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 4218–4225).

  • Bruton, L. T. (1980). RC-active circuits: theory and design. Englewood Cliffs, NJ: Prentice-Hall Inc.

    Google Scholar 

  • Chai, J. X., Tong, X., Chan, S. C., & Shum, H. Y. (2000). Plenoptic sampling. In Proceedings annual conference on computer graphics (SIGGRAPH) (pp. 307–318).

  • Chang, C. W., Chen, M. R., Hsu, P. H., & Lu, Y. C. (2014). A pixel-based depth estimation algorithm and its hardware implementation for 4-D light field data. In IEEE international symposium on circuits and systems (pp. 786–789). IEEE.

  • Chen, L. D., Lu, Y. T., Hiao, Y. L., Yang, B. H., Chen, W. C., & Huang, C. T. (2018). A 95pJ/label wide-range depth-estimation processor for full-HD light-field applications on FPGA. In Proceedings of IEEE Asian solid-state circuits conference (pp. 261–262). IEEE.

  • Chen, X., Xie, W., Ma, H., Chu, J., Qi, B., Ren, G., et al. (2020). Wavefront measurement method based on improved light field camera. Results in Physics, 17, 1–7.

    Google Scholar 

  • Dansereau, D., & Bruton, L. T. (2003). A 4D frequency-planar IIR filter and its application to light field processing. In: Proceedings of IEEE international symposium on circuits and systems (Vol. 4, pp. IV–476–IV–479).

  • Dansereau, D., & Bruton, L. T. (2004). Gradient-based depth estimation from 4D light fields. In Proceedings IEEE IEEE international symposium on circuits and systems.

  • Dansereau, D., & Bruton, L. T. (2007). A 4-D dual-fan filter bank for depth filtering in light fields. IEEE Transactions on Signal Processing, 55(2), 542–549.

    Article  MathSciNet  Google Scholar 

  • Dansereau, D. G., Bongiorno, D. L., Pizarro, O., & Williams, S. B. (2013). Light field image denoising using a linear 4D frequency-hyperfan all-in-focus filter. Proc SPIE Comput Imag XI, 8657, 86,570P–1—86,570P–14. https://doi.org/10.1117/12.2002239.

  • Dansereau, D. G., Pizarro, O., & Williams, S. B. (2015). Linear volumetric focus for light field cameras. ACM Transactions on Graphics, 34(2), 1–20. https://doi.org/10.1145/2665074.

    Article  Google Scholar 

  • Domínguez Conde, C., Lüke, J. P., & Rosa González, F. (2019). Implementation of a depth from light field algorithm on FPGA. Sensors, 19(16), 3562.

    Article  Google Scholar 

  • Dong, F., Ieng, S. H., Savatier, X., Etienne-Cummings, R., & Benosman, R. (2013). Plenoptic cameras in real-time robotics. The International Journal of Robotics Research, 32(2), 206–217.

    Article  Google Scholar 

  • Edussooriya, C. U. S., Bruton, L. T., & Agathoklis, P. (2015a). A 5-D IIR depth-velocity filter for enhancing objects moving on linear-trajectories in light field videos. In Proceedings of IEEE international symposium on circuits and systems (pp. 2381–2384).

  • Edussooriya, C. U. S., Dansereau, D. G., Bruton, L. T., & Agathoklis, P. (2015b). Five-dimensional depth-velocity filtering for enhancing moving objects in light field videos. IEEE Transactions on Signal Processing, 63(8), 2151–2163.

    Article  MathSciNet  Google Scholar 

  • Fettweis, A. (1986). Wave digital filters: Theory and practice. Proceedings of the IEEE, 74(2), 270–327.

    Article  Google Scholar 

  • Fettweis, A., & Meerkotter, K. (1975). On adaptors for wave digital filters. IEEE Transactions on Acoustics, Speech, and Signal Processing, 23(6), 516–525.

    Article  Google Scholar 

  • Hahne, C., & Aggoun, A. (2014a). Embedded FIR filter design for real-time refocusing using a standard plenoptic video camera. In Proceedings of SPIE 9023. https://doi.org/10.1117/12.2042495.

  • Hahne, C., & Aggoun, A. (2014b). Embedded FIR filter design for real-time refocusing using a standard plenoptic video camera. In Proceedings SPIE electronic imaging (Vol. 9023, pp. 902,305–1–902,305–12).

  • Hahne, C., Lumsdaine, A., Aggoun, A., & Velisavljevic, V. (2018). Real-time refocusing using an FPGA-based standard plenoptic camera. IEEE Transactions on Industrial Electronics, 65(12), 9757–9766.

    Article  Google Scholar 

  • Isaksen, A., McMillan, L., & Gortler, S.J. (2000). Dynamically reparameterized light fields. In Proceedings annual conference on computer graphics (SIGGRAPH) (pp. 297–306).

  • Jayaweera, S. S., Edussooriya, C. U. S., Wijenayake, C., Agathoklis, P., & Bruton, L. (2020). Multi-volumetric refocusing of light fields. TechRxiv https://doi.org/10.36227/techrxiv.12349835.v1, https://www.techrxiv.org/articles/preprint/Multi-Volumetric_Refocusing_of_Light_Fields/12349835.

  • Levin, A., Hasinoff, S. W., Green, P., Durand, F., & Freeman, W. T. (2009). 4D frequency analysis of computational cameras for depth of field extension. ACM Transactions on Graphics, 28(3), 1. https://doi.org/10.1145/1531326.1531403.

    Article  Google Scholar 

  • Levoy, M., & Hanrahan, P. (1996). Light field rendering. In Proceedings annual conference computer graphics (SIGGRAPH) (pp 31–42).

  • Li, H., Kummert, A., Schauland, S., & Velten, J. (2009). 3D wave digital filter implementation on a virtex2 FPGA board with external SDRAM. In Proceedings international workshop on multidimensional (nD) systems.

  • Liyanage, N., Wijenayake, C., Edussooriya, C., Madanayake, A., Agathoklis, P., Ambikairajah, E., & Bruton, L. (2018). Low-complexity 4-D IIR filters for multi-depth filtering and occlusion suppression in light fields. In Proceedings IEEE international symposium circuits system (pp. 1–5).

  • Liyanage, N., Wijenayake, C., Edussooriya, C., Madanayake, A., Agathoklis, P., Bruton, L. T., et al. (2019). Multi-depth filtering and occlusion suppression in 4-D light fields: Algorithms and architectures. Signal Process, 167, 1–13.

    Google Scholar 

  • Liyanage, N., Wijenayake, C., Edussooriya, C. U. S., & Ambikairajah, E. (2019). Reduced-complexity depth filtering and occlusion suppression using modulated-sparse light fields. In Proceedings IEEE international symposium circuits systems (pp. 1–5).

  • Lumsdaine, A., & Georgiev, T. (2009). The focused plenoptic camera. In Proceedings of IEEE international conference computer photography (pp. 1–8).

  • Madanayake, A., Wimalagunaratne, R., Dansereau, D. G., & Bruton, L. T. (2011a). Design and FPGA-implementation of 1st-order 4D IIR frequency-hyperplanar digital filters. In Proceedings IEEE 54th midwest symposium on circuits and systems (pp. 1–4).

  • Madanayake, A., Wimalagunaratne, R., Dansereau, D. G., & Bruton, L. T. (2011b). Design and FPGA-implementation of 1st-order 4D IIR frequency-hyperplanar digital filters. In IEEE 54th international midwest symposium on circuits and systems (MWSCAS), pp 1–4.

  • Madanayake, A., Wijenayake, C., Dansereau, D. G., Gunaratne, T. K., Bruton, L. T., & Williams, S. B. (2013). Multidimensional (MD) circuits and systems for emerging applications including cognitive radio, radio astronomy, robot vision and imaging. IEEE Circuits and Systems Magazine, 13(1), 10–43.

    Article  Google Scholar 

  • Madanayake, A., Wimalagunarathne, R., Dansereau, D. G., Cintra, R. J., & Bruton, L. T. (2015). VLSI architecture for 4-D depth filtering. Signal, Image and Video Process, 9(4), 809–818. https://doi.org/10.1007/s11760-013-0511-9.

    Article  Google Scholar 

  • McCloskey, S. (2014). Masking light fields to remove partial occlusion. In 22nd international conferences on pattern recognition (pp. 2053–2058).

  • Meerkötter, K. (2018). On the passivity of wave digital networks. IEEE Circuits and Systems Magazine, 18(4), 40–57.

    Article  Google Scholar 

  • Ng, R. (2005). Fourier slice photography. In Proceedings annual conference on computer graphics (SIGGRAPH) (pp. 735–744).

  • Ng, R., Levoy, M., Brédif, M., Duval, G., Horowitz, M., & Hanrahan, P. (2005). Light field photography with a hand-held plenoptic camera. Technical report CTSR 2005-02, Stanford Univ, Stanford, CA.

  • Orchard, J. (1966). Inductorless filters. Electronic Letters, 2(2), 224–225.

    Article  Google Scholar 

  • Premaratne, S. U., Edussooriya, C. U. S., Wijenayake, C., Bruton, L. T., & Agathoklis, P. (2018). A 4-D sparse FIR hyperfan filter for volumetric refocusing of light fields by hard thresholding. In Proceedings IEEE international conference on digital signal processing (pp. 1–5).

  • Premaratne, S. U., Liyanage, N., Edussooriya, C. U. S., & Wijenayake, C. (2020). Real-time light field denoising using a novel linear 4-D hyperfan filter. IEEE Transactions on Circuits and Systems I, 67(8), 2693–2706.

    Article  Google Scholar 

  • Rajapaksha, N., Madanayake, A., & Bruton, L. T. (2015). Systolic array architecture for steerable multibeam VHF wave-digital RF apertures. IEEE Trans on Aerospace and Electronic Systems, 51(1), 669–687.

    Article  Google Scholar 

  • Rerabek, M., & Ebrahimi, T. (2016). New light field image dataset. In Proceedings of the 8th international conference quality multimedia experience (pp. 1–2). http://mmspg.epfl.ch/EPFL-light-field-image-dataset.

  • Rodríguez-Ramos, L., Marín, Y., Díaz, J., Piqueras, J., García-Jiménez, J., Rodríguez-Ramos, J. (2010). Fpga-based real time processing of the plenoptic wavefront sensor. In 1st AO4ELT conference-adaptive optics for extremely large telescopes (pp. 1–6).

  • Rodríguez-Ramos, L.F., Montilla, I., Fernández-Valdivia, J., Trujillo-Sevilla, J., & Rodríguez-Ramos, J. (2012). Concepts, laboratory, and telescope test results of the plenoptic camera as a wavefront sensor. In Adaptive optics systems III, international society for optics and photonics (Vol. 8447, p. 844745).

  • Schauland, S., Velten, J., & Kummert, A. (2008). Implementation of hardware-optimized 3-D wave digital filters for motion-based object detection in video scenes. In Proceedings 16th European signal processing conference.

  • Tao, M. W., Hadap, S., Malik, J., & Ramamoorthi, R. (2013). Depth from combining defocus and correspondence using light-field cameras. In Proceedings IEEE international conference on computer vision (pp. 673–680).

  • Temes, G. C., & Orchard, H. J. (1973). First-order sensitivity and worstcase analysis of doubly-terminated reactance two-ports. IEEE Transactions on Circuit Theory, 20(5), 567–571.

    Article  Google Scholar 

  • Tsai, D., Dansereau, D. G., Peynot, T., & Corke, P. (2017). Image-based visual servoing with light field cameras. IEEE Robotics and Automation Letters, 2(2), 912–919.

    Article  Google Scholar 

  • Veeraraghavan, A., Raskar, R., Agrawal, A., Mohan, A., & Tumblin, J. (2007). Dappled photography: Mask enhanced cameras for heterodyned light fields and coded aperture refocusing. ACM Transactions on Graphics, 26(3), 69-2–69-12.

    Article  Google Scholar 

  • Wang, T. C., Efros, A. A., & Ramamoorthi, R. (2015). Occlusion-aware depth estimation using light-field cameras. In IEEE international conferences on computer vision (ICCV). IEEE (pp. 3487–3495). https://doi.org/10.1109/ICCV.2015.398.

  • Wang, T. C., Efros, A. A., & Ramamoorthi, R. (2016). Depth estimation with occlusion modeling using light-field cameras. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(11), 2170–2181.

    Article  Google Scholar 

  • Wanner, S., & Goldluecke, B. (2012). Globally consistent depth labeling of 4D light fields. In 2012 IEEE conference on computer vision and pattern recognition. IEEE (pp. 41–48). https://doi.org/10.1109/CVPR.2012.6247656, http://ieeexplore.ieee.org/document/6247656/.

  • Wijenayake, C., Liyanage, N., Edussooriya, C. U. S., Seatang, H., Agathoklis, P., & Bruton, L. T. (2019). Design and implementation of 5-D IIR depth velocity filters for light field video processing. IEEE Transactions on Circuits and Systems II, 66(7), 1267–1271. https://doi.org/10.1109/TCSII.2018.2874940.

    Article  Google Scholar 

  • Wimalagunarathne, R., Madanayake, A., Dansereau, D. G., & Bruton, L. T. (2012). A systolic-array architecture for first-order 4-D IIR frequency-planar digital filters. In Proceedings IEEE international symposium on circuits and systems (pp 3069–3072).

  • Wimalagunarathne, R., Wijenayake, C., Madanayake, A., Dansereau, D. G., & Bruton, L. T. (2013). Integral form 4-D light field filters using Xilinx FPGAs and 45 nm CMOS technology. 288 Multidimensional Systems and Signal Processing, 26(1), 47–65. https://doi.org/10.1007/s11045-013-0235-6.

    Article  MathSciNet  Google Scholar 

  • Zhang, Y., & Bruton, L. T. (1994). Applications of 3-D LCR networks in the design of 3-D recursive filters for processing image sequences. IEEE Transactions on Circuits and Systems for Video Technology, 4(4), 369–382.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Electrical and Computer Engineering, University of Akron, Akron, OH, USA

    Sai K. Gullapalli

  2. Department of Electronic and Telecommunication Engineering, University of Moratuwa, Moratuwa, Sri Lanka

    Chamira U. S. Edussooriya

  3. School of Information Technology and Electrical Engineering, University of Queensland, Brisbane, Australia

    Chamith Wijenayake

  4. Sydney Institute for Robotics and Intelligent Systems, University of Sydney, Sydney, NSW, Australia

    Donald G. Dansereau

  5. Department of Electrical and Computer Engineering, University of Calgary, Calgary, AB, Canada

    Len T. Bruton

  6. Department of Electrical and Computer Engineering, Florida International University, Miami, FL, USA

    Chamira U. S. Edussooriya & Arjuna Madanayake

Authors
  1. Sai K. Gullapalli
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chamira U. S. Edussooriya
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chamith Wijenayake
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Donald G. Dansereau
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Len T. Bruton
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Arjuna Madanayake
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chamira U. S. Edussooriya.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gullapalli, S.K., Edussooriya, C.U.S., Wijenayake, C. et al. Wave-digital filter circuits for single-chip 4-D light field depth-based enhancement. Multidim Syst Sign Process 32, 607–631 (2021). https://doi.org/10.1007/s11045-020-00751-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11045-020-00751-y

Keywords

Search

Navigation