Skip to main content
SpringerLink
Log in

High-level dataflow programming for real-time image processing on smart cameras

  • Special Issue Paper
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

We describe the application of the caph programming language to the implementation of complex real-time image processing application on a FPGA embedded in a smart camera architecture. caph is based upon the dataflow model of computation. Applications are described as networks of purely dataflow actors exchanging tokens through unidirectional channels and the behavior of each actor is defined as a set of transition rules using pattern matching. We show that this model is naturally suited to the description of applications operating on the fly on digital video streams and supports a fully automated compilation path producing efficient VHDL code. This is demonstrated on an application performing the extraction of HOG (histogram of oriented gradient) feature vectors in real time on the dreamcam smart camera, an experimental platform developed at our institute.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Notes

  1. Field Programmable Gate Arrays.

  2. In Fig. 1, streams are denoted (ordered) from right to left; for example, the actor ADD first processes the token 1, then the token 2, etc. Since streams are potentially infinite, their end is denoted ”...”. However, when describing actors textually, streams will be denoted from left to right; for example: ADD:1,2,... = 2,3,....

  3. Pop the value from the connected FIFO.

  4. caph is a strongly typed language: every value has a type, which is computed statically. Static typing allows the detection of erroneous programs early in the development process, which is crucial when targeting FPGA-based systems because target-level debugging is notoriously difficult in this case. Moreover, types form the basis of many optimisation techniques at the implementation level. The type system of caph is directly inspired from that equipping modern functional programming languages such as Haskell or ML. It is an instance of the classical Hindley–Millner system, supporting automatic type inference, parametric polymorphism and higher-order functions. The ability to define polymorphic functions is crucial for building libraries of reusable actor components. Higher-order functions nicely match the concept of graph pattern, which is the other major source of abstraction and reusability.

  5. Technically, this type can be defined in caph as

    type $t img = SoF | EoF | SoL | EoL | Data of $t where tags correspond to value constructors and \(\$t\) denotes a type variable.

  6. The (sub)graphs resulting from the application of the neigh13 and neigh33 wiring functions are delineated using boxes drawn with short and long dash, respectively, on the figure.

  7. Ill-formed networks and inconsistent use of actors can be readily detected using a classical Hindley–Millner polymorphic type inference and checking phase.

  8. In other words, the underlying dataflow model of computation is not a static one.

  9. According to [6], using \(L_1\) norm instead of \(L_2\) only slightly decreases the performance of the classifier; on the other hand, this significantly reduces resource usage on the target FPGA.

  10. In Listing 3, we adopted an abbreviated syntax, allowed in caph, for describing rules. Instead of prefixing each component of the rule by the name of the involved input, output or variable, we start the rule set by giving a rule format, which applies to all the subsequent rules. This is equivalent to going from an explicit naming of function parameters to an implicit one, by position.

  11. The size of the sum array is given by the number of blocks per line: \(1280/8=160\), in our particular case.

  12. Introduced by the when keyword in Listing 4.

  13. They can be customized, however, if necessary.

  14. For the VHDL code, we aggregated the lines describing the related processes and those instantiating and wiring these processes.

  15. No external memory is used in the studied implementations.

References

  1. Ahmed, S.: Application of a Dataflow Programming Language to the High Level Synthesis of Real-Time Vision Systems on Reconfigurable Hardware. PhD thesis, Université Blaise Pascal-Clermont-Ferrand II, (2013)

  2. Bauer, S., Kohler, S., Doll, K., Brunsmann, U.: Fpga-gpu architecture for kernel svm pedestrian detection. In: 2010 IEEE Computer Society Conference on, Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 61–68 (2010)

  3. Birem, M., Berry, F.: DreamCam: a modular FPGA-based smart camera architecture. J. Syst. Archit. 60(6), 519–527 (2014). doi:10.1016/j.sysarc.2014.01.006

  4. Sérot, J.: The caph software and reference manual. http://caph.univ-bpclermont.fr

  5. Cao, P., Deng, G.: Real-time vision- based stop sign detection system on FPGA. In: International Conference on Digital Image Computing: Techniques and Applications (DICTA), IEEE Computer Society, pp. 465–471 (2008)

  6. Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: Schmid, C., Soatto, S., Tomasi, C. (eds) International Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 886–893. June (2005)

  7. Dahlin, A., Ersfolk, J., Yang, G., Habli, H., Lilius, J.: The Canals language and its compiler. In: Proceedings of th 12th International Workshop on Software and Compilers for Embedded Systems, SCOPES ’09, ACM, pp. 43–52. New York, NY, USA (2009)

  8. Dollr, P., Wojek, C., Schiele, B., Perona, P.: Pedestrian detection: A benchmark. In: IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), (2009)

  9. Enzweiler, M., Gavrila, D.M.: Monocular Pedestrian Detection: Survey and Experiments. In: IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), (2008)

  10. Frigo, J., Gokhale, M., Lavenier, D.: Evaluation of the streams-C C-to-FPGA compiler: an applications perspective. In: Proceedings of the 2001 ACM/SIGDA ninth international symposium on Field programmable gate arrays, FPGA ’01, pp. 134–140. ACM, New York, NY (2001). doi:10.1145/360276.360326

  11. Gupta, S., Dutt, N., Gupta, R., Nicolau, A.: Spark A high-level synthesis framework for applying parallelizing compiler transformations. In: International Conference on VLSI Design, pp. 461–466 (2003)

  12. Kadota, R., Sugano, H., Hiromoto, M., Ochi, H., Miyamoto, R., Nakamura, Y.: Hardware architecture for hog feature extraction. In: Proceedings of the 2009 Fifth International Conference on Intelligent Information Hiding and Multimedia Signal Processing, IEEE Computer Society, pp. 1330–1333. Washington, DC, USA (2009)

  13. Lucarz, C., Mattavelli, M., Wipliez, M., Roquier, G., Raulet, M., Janneck, J., Miller, I., Parlour, D.: Dataflow/Actor-Oriented language for the design of complex signal processing systems. In: Proceedings of the 2008 Conference on Design and Architectures for Signal and Image Processing, DASIP 2008, pp. 168–175 (2008)

  14. Najjar, W.A., Lee, E.A., Gao, G.R.: Advances in the dataflow computational model. Parallel Comput. 25(13–14), 1907–1929 (1999)

    Article  Google Scholar 

  15. Najjar, W.A., Boehm, W., Draper, B.A., Hammes, J., Rinker, R., Beveridge, J.R., Chawathe, M., Ross, C.: High-level language abstraction for reconfigurable computing. Computer 36, 63–69 (2003)

    Article  Google Scholar 

  16. Sérot, J.: The semantics of a purely functional graph notation system. In: Achten, P., Koopman, P.W.M., Morazán, M.T. (eds.) Draft Proceedings of the Ninth Symposium on Trends in Functional Programming, TFP 2008, May 26–28. Nijmegen, The Netherlands (2008)

  17. Sérot, J., Berry, F., Ahmed, S.: Implementing stream-processing applications on fpgas: a dsl-based approach. In: Proceedings of the 2011 21st International Conference on Field Programmable Logic and Applications, FPL ’11, IEEE Computer Society, pp. 130–137. Washington, DC, USA (2011)

  18. Sérot, J., Berry, F., Ahmed, S.: CAPH: a language for implementing stream-processing applications on FPGAs. In: Athanas, P. (ed.) Embedded Systems Design with FPGAs. Springer, New York (2013)

    Google Scholar 

  19. Zhu, Q., Yeh, M.-C., Cheng, K.-T., Avidan, S.: Fast human detection using a cascade of histograms of oriented gradients. In: Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, IEEE Computer Society, vol 2, pp. 1491–1498. Washington, DC, USA (2006)

  20. Handel-c language reference manual (2009). http://www.agilityds.com/literature/HandelC_Language_Reference_Manual

  21. Impulse accelerated technologies. http://www.impulsec.com

Download references

Author information

Authors and Affiliations

  1. Institut Pascal, Université Blaise Pascal/CNRS, Clermont-Ferrand, France

    Jocelyn Sérot, François Berry & Cédric Bourrasset

Authors
  1. Jocelyn Sérot
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. François Berry
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Cédric Bourrasset
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jocelyn Sérot.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sérot, J., Berry, F. & Bourrasset, C. High-level dataflow programming for real-time image processing on smart cameras. J Real-Time Image Proc 12, 635–647 (2016). https://doi.org/10.1007/s11554-014-0462-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11554-014-0462-6

Keywords

Search

Navigation