Abstract
CMOS sensors are now more and more frequently integrated into popular consumer products. Images from these sensors thus need to be digitally processed for display purposes. To do so, CMOS sensors are associated with dedicated components that keep power consumption low. However, use of dedicated components limits hardware flexibility and prevents updating of image processing algorithms. This paper describes the eISP, a programmable processing architecture that combines enough computational efficiency for 1080p HD video with silicon area and power characteristics suitable for the next generation of mobile phones (lower than 1 mm2 and 500 mW in TSMC 65 nm).
Similar content being viewed by others
Notes
Several processes can be inserted, and the order of processes may differ from the one presented here.
References
Alleysson, D., Süsstrunk, S., Hérault, J.: Linear demosaicing inspired by the human visual system. IEEE Trans. Image Process. 14(4), 439–449 (2005). doi:10.1109/TIP.2004.841200
Bigas, M., Cabruja, E., Forest, J., Salvi, J.: Review of CMOS image sensors. Microelectron. J. 37(5), 433–451 (2006)
Cat, H.H., Gentile, A., Eble, J.C., Lee, M., Vendier, O., Joo, Y.J., Wills, D.S., Brooke, M., Jokerst, N.M., Brown, A.S.: SIMPil: An OE integrated SIMD architecture for focal plane processing applications. In: Proceedings of Third International Conference on Massively Parallel Processing using Optical Interconnections, Maui, pp. 44–52 (1996)
Chen, J., Chien, S.Y.: Crisp: coarse-grained reconfigurable image stream processor for digital still cameras and camcorders. IEEE Trans. Circuits Syst. Video Technol. 18(9), 1223–1236 (2008). doi:10.1109/TCSVT.2008.928529
Chen, T., Catrysse, P., Gamal, A.E., W.B.: How small should pixel size be. In: Proceedings of SPIE, April 2000, pp. 451–459 (2000)
Cheung, N., Henkel, J., Parameswaran, S.: Rapid configuration and instruction selection for an ASIP: a case study. In: Design, Automation and Test in Europe Conference and Exhibition, 2003, pp. 802–807 (2003)
Chou, C., Liu, K., Lee, W.: Adaptive color filter array demosaicking based on constant hue and local properties of luminance. In: Advances in Image and Video Technology, pp. 357–370 (2007)
Ding, L., Goshtasby, A.: On the canny edge detector. Pattern Recognit. 34(3), 721–725 (2001)
Gentile, A., Vitabile, S., Verdoscia, L., Sorbello, F.: Image processing chain for digital still cameras based on the SIMPil architecture. In: 2005 ICPP 2005 Workshops International Conference Workshops on Parallel Processing, pp. 215–222 (2005). doi:10.1109/ICPPW.2005.41
Goodwin, D., Rowen, C., Martin, G.: Configurable multi-processor platforms for next generation embedded systems. In: 2007 ASP-DAC ’07 Asia and South Pacific Design Automation Conference, pp. 744–746 (2007). doi:10.1109/ASPDAC.2007.358076
Graham, S.L., Kessler, P.B., Mckusick, M.K.: Gprof: a call graph execution profiler. SIGPLAN Not 17(6), 120–126 (1982). doi:10.1145/872726.806987
Horiuchi, T., Watanabe, K., Tominaga, S.: Adaptive filtering for color image sharpening and denoising. In: 2007 ICIAPW 2007 14th International Conference on Image Analysis and Processing Workshops, pp. 196–201 (2007). doi:10.1109/ICIAPW.2007.11
Kehtarnavaz, N., Oh, H.J., Yoo, Y.: Development and real-time implementation of auto white balancing scoring algorithm. Real Time Imaging 8(5), 379–386 (2002). doi:10.1006/rtim.2001.0287
Khailany, B., Dally, W.J., Kapasi, U.J., Mattson, P., Namkoong, J., Owens, J.D., Towles, B., Chang, A., Rixner, S.: Imagine: Media processing with streams. IEEE Micro 21(2), 35–46 (2001). doi:10.1109/40.918001
Khailany, B., Williams, T., Lin, J., Long, E., Rygh, M., Tovey, D., Dally, W.: A programmable 512 GOPS stream processor for signal, image, and video processing. IEEE J. Solid State Circuits 43(1), 202–213 (2008). doi:10.1109/JSSC.2007.909331
Khawam, S., Nousias, I., Milward, M., Yi, Y., Muir, M., Arslan, T.: The reconfigurable instruction cell array. IEEE Trans. Very Large Scale Integr. Syst. 16(1), 75–85 (2008). doi:10.1109/TVLSI.2007.912133
Kyo, S., Okazaki, S., Arai, T.: An integrated memory array processor architecture for embedded image recognition systems. In: 2005 ISCA ’05 Proceedings 32nd International Symposium on Computer Architecture, pp. 134–145 (2005). doi:10.1109/ISCA.2005.11
Lee, S.Y., Kumar, Y., Cho, J.M., Kim, S.W.L.S.W.: Enhanced autofocus algorithm using robust focus measure and fuzzy reasoning. IEEE Trans. Circuits Syst. Video Technol. 18, 1237–1246 (2008). doi:10.1109/TCSVT.2008.924105
Meylan, L., Süsstrunk, S.: High dynamic range image rendering using a Retinex-based Adaptive filter. IEEE Trans. Image Process 15(9), 2820–2830 (2006). doi:http://rr.epfl.ch/12/
Mucci, C., Campi, F., Deledda, A., Fazzi, A., Ferri, M., Bocchi, M.: A cycle-accurate ISS for a dynamically reconfigurable processor architecture. Parallel and Distributed Processing Symposium, 2005 Proceedings 19th IEEE International 4(8), 8 (2005). doi:10.1109/IPDPS.2005.14
Nethercote, N.: Dynamic binary analysis and instrumentation. Ph.D. thesis, Trinity College (2004)
Pinto, C.A., Beric, A., Singh, S.P., Farfade, S.: Hiveflex-video vsp1: Video signal processing architecture for video coding and post-processing. In: Multimedia, 2006 ISM’06 Eighth IEEE International Symposium, pp. 493–500 (2006). doi:10.1109/ISM.2006.83
Ramanath, R., Snyder, W.E., Bilbro, G.L., Iii, W.A.S.: Demosaicking methods for bayer color arrays. J. Electron. Imaging 11, 306–315 (2002)
Russ, J.C.: The Image Processing Handbook, 5th edn. CRC Press, New York (1997)
Sakaue, S., Nakayama, M., Tamura, A., Maruno, S.: Adaptive gamma processing of the video cameras for the expansion of the dynamic range. IEEE Trans. Consum. Electron. 41(3), 555–562 (1995). doi:10.1109/30.468094
Shimizu, S., Kondo, T., Kohashi, T., Tsurata, M., Komuro, T.: A new algorithm for exposure control based on fuzzy logic for video cameras. IEEE Trans. Consum. Electron. 38(3), 617–623 (1992). doi:10.1109/30.156745
Tamburrino, D., Alleysson, D., Meylan, L., Süsstrunk, S.: Digital camera workflow for high dynamic range images using a model of retinal processing. In: IS& T/SPIE Electronic Imaging: Digital Photography IV, vol. 6817 (2008)
Thevenin, M., Paindavoine, M., Letellier, L., Heyrman, B.: Processor extensions for CMOS sensor imaging in camera phone. In: Photonics Europe 2008—Photonics in Multimedia, vol. 7001 (2008)
Tomasi, C., Manduchi, R.: Bilateral filtering for gray and color images. In: Proceedings of the 1998 IEEE International Conference on Computer Vision, IEEE, Bombay, India (1998)
van de Weijer, J., Gevers, T.: Color constancy based on the grey-edge hypothesis. In: ICIP (2), pp. 722–725 (2005)
Yoo, Y., Lee, S., Choe, W., Kim, C.Y.: CMOS image sensor noise reduction method for image signal processor in digital cameras and camera phones. In: Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, SPIE Conference, vol. 6502 (2007). doi:10.1117/12.702758
Zhiming, W., Jianhua, T.: A fast implementation of adaptive histogram equalization. In: 2006 8th International Conference on Signal Processing (2006). doi:10.1109/ICOSP.2006.345602
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Thevennin, M., Paindavoine, M., Letellier, L. et al. The eISP low-power and tiny silicon footprint programmable video architecture. J Real-Time Image Proc 6, 33–46 (2011). https://doi.org/10.1007/s11554-010-0163-8
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11554-010-0163-8