Skip to main content
SpringerLink
Log in

Exploiting On-Device Image Classification for Energy Efficiency in Ambient-Aware Systems

  • Chapter
  • First Online:
Mobile Cloud Visual Media Computing

Abstract

Ambient-aware applications need to know what objects are in the environment. Although video data contains this information, analyzing it is a challenge esp. on portable devices that are constrained in energy and storage. A naïve solution is to sample and stream video to the cloud, where advanced algorithms can be used for analysis. However, this increases communication energy costs, making this approach impractical. In this article, we show how to reduce energy in such systems by employing simple on-device computations. In particular, we use a low-complexity feature-based image classifier to filter out unnecessary frames from video. To lower the processing energy and sustain a high throughput, we propose a hierarchically pipelined hardware architecture for the image classifier. Based on synthesis results from an ASIC in a 45 nm SOI process, we demonstrate that the classifier can achieve minimum-energy operation at a frame rate of 12 fps, while consuming only 3 mJ of energy per frame. Using a prototype system, we estimate about 70 % reduction in communication energy when 5 % of frames are interesting in a video stream.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 54.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. D’Andrea, R.: Can drones deliver? IEEE Trans. Autom. Sci. Eng. 138–141 (2014)

    Google Scholar 

  2. Randell, C.: Wearable computing: a review. Technical Report Number CSTR-06-004. University of Bristol (2005)

    Google Scholar 

  3. Geronimo, D., Lopez, A.M., Sappa, A.D., Graf, T.: Survey of pedestrian detection for advanced driver assistance systems. IEEE Trans. Pattern Anal. Mach. Intell. 32(7), 1239–1258 (2009). doi:10.1109/TPAMI.2009.122

    Article  Google Scholar 

  4. Baber, C., Smith, P., Cross, J., Zasikowski, D., Hunter, J.: Wearable technology for crime scene investigation. In: Proceedings of the IEEE International Symposium on Wearable Computers, pp. 138–141 (2005)

    Google Scholar 

  5. Mann, S.: WearCam (the wearable camera): personal imaging systems for long-term use in wearable tetherless computer-mediated reality and personal photo/videographic memory prosthesis. In: Proceedings of the IEEE International Symposium on Wearable Computers, pp. 124–131 (1998)

    Google Scholar 

  6. Kelly, P., Marshall, S.J., Badland, H., Kerr, J., Oliver, M., Doherty, A.R., Foster, C.: An ethical framework for automated, wearable cameras in health behavior research. Am. J. Prev. Med. 44(3), 314–319 (2013). doi:10.1016/j.amepre.2012.11.006

    Article  Google Scholar 

  7. Navab, N.: Developing killer apps for industrial augmented reality. IEEE Comput. Graph. Appl. 24(3), 16–20 (2004)

    Article  Google Scholar 

  8. Aleksya, M., Rissanenb, M.J., Maczeya, S., Dixa, M.: Wearable computing in industrial service applications. Int. Conf. Ambient Syst. Netw. Technol. 5, 394–400 (2011). doi:10.1016/j.procs.2011.07.051

    Google Scholar 

  9. Weinland, D., Ronfard, R., Boyer, E.: A survey of vision-based methods for action representation, segmentation and recognition. Elsevier Comput. Vis. Image Underst. 115(2), 224–241 (2011). doi:10.1016/j.cviu.2010.10.002

    Article  Google Scholar 

  10. Poppe, R.: A survey on vision-based human action recognition. Elsevier Image Vis. Comput. 28(6), 976–990 (2010). doi:10.1016/j.imavis.2009.11.014

    Article  Google Scholar 

  11. Crevier, D., Lepage, R.: Knowledge-based image understanding systems: a survey. Elsevier Comput. Vis. Image Underst. 67(2), 161–185 (1997). doi:10.1006/cviu.1996.0520

    Article  Google Scholar 

  12. LiKamWa, R., Wang, Z., Carroll, A., Lin, X.F., Zong, L.: Draining our glass: an energy and heat characterization of Google Glass. In: Proceedings of Asia-Pacific Workshop on Systems, Article no. 10, (2014). doi:10.1145/2637166.2637230

  13. Ha, K., Chen, Z., Hu, W., Richter, W., Pillai, P., Satyanarayanan, M.: Towards wearable cognitive assistance. In: Proceedings of the International Conference on Mobile Systems, Applications, and Services, pp. 68–81 (2014). doi:10.1145/2594368.2594383

  14. Jia, Z., Balasuriya, A., Challa, S.: Vision based target tracking for autonomous land vehicle navigation: a brief survey. Recent Pat. Comput. Sci. 2(1), 32–42 (2009)

    Google Scholar 

  15. Soro, S., Heinzelman, W.: A survey of visual sensor networks. Hindawi Advances in Multimedia, Article no. 640386 (2009). doi:10.1155/2009/640386

  16. Kyono, Y., Yonezawa, T., Nozaki, H., Keio, M.O., Keio, T.I., Keio, J.N., Takashio, K., Tokuda, H.: EverCopter: continuous and adaptive over-the-air sensing with detachable wired flying objects. In: Proceedings of the ACM Conference on Pervasive and Ubiquitous Computing Adjunct Publication, pp. 299–302 (2013). doi:10.1145/2494091.2494183

  17. Kemp, R., Palmer, N., Kielmann, T., Bal, H.: Cuckoo: a computation offloading framework for smartphones. In: Mobile Computing, Applications, and Services. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecom. Engineering, vol. 76, pp. 59–79 (2012)

    Google Scholar 

  18. Ra, M-R., Sheth, A., Mummert, L., Pillai, P., Wetherall, D., Govindan, R.: Odessa: enabling interactive perception applications on mobile devices. In: Proceedings of the International Conference on Mobile Systems, Applications, and Services, pp. 43–56, (2011). doi:10.1145/1999995.2000000

  19. Nishikawa, T., Takahashi, M., Hamada, M., Takayanagi, T., Arakida, H., Machiada, N., Yamamoto, H., Fujiyoshi, T., Matsumoto, Y., Yamagishi, O., Samata, T., Asano, A., Terazawa, T., Ohmori, K., Shirakura, J., Watanabe, Y., Nakamura, H., Minami, S., Kuroda, T., Furuyama, T.: A 60MHz 240mW MPEG-4 video-phone LSI with 16Mb embedded DRAM. IEEE J. Solid-State Circuits 35, 1713–1721 (2000)

    Article  Google Scholar 

  20. Worlds most power-efficient 1080p/60 high definition imag sensor for front-facing camera applications. OV2740 1080p Product Brief. www.ovt.com (2014)

  21. Halperin, D., Greenstein, B., Sheth, A., Wetherall, D.: Demystifying 802.11n power consumption. In: Proceedings of the International Conference on Power Aware Computing and Systems, Article no. 1 (2010)

    Google Scholar 

  22. Brostow, G., Shotton, J., Fauquer, J., Cipolla, R.: Segmentation and recognition using structure from motion point clouds. In: Proceedings of the European Conference on Computer Vision, pp. 44–57 (2008). doi:10.1007/978-3-540-88682-2_5

  23. Venkataramani, S., Bahl, V., Hua, X.-S., Liu, J., Li, J., Phillipose, M., Priyantha, B., Shoaib, M.: SAPPHIRE: an always-on context-aware computer-vision system for portable devices. In: Proceedings of Conference on Design Automation and Test in Europe, to appear (2015)

    Google Scholar 

  24. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Proceedings of Neural Information Processing Systems, pp. 1106–1114 (2012)

    Google Scholar 

  25. Jin, J., Gokhale, V., Dundar, A., Krishnamurthy, B., Martini, B., Culurciello, E.: An efficient implementation of deep convolutional neural networks on a mobile coprocessor. IEEE Int. Midwest Symp. Circuits Syst. 133–136 (2014). doi:10.1109/MWSCAS.2014.6908370

  26. Gokhale, V., Jin, J., Dundar, A., Martini, B., Culurciello, E.: A 240 G-ops/s mobile coprocessor for deep neural networks. IEEE Conf. Comput. Vis. Pattern Recognit. Workshops 696–701 (2014). doi:10.1109/CVPRW.2014.106

  27. Chen, T., Du, Z., Sun, N., Wang, J., Wu, C., Chen, Y., Temam, O.: DianNao: a small-footprint high-throughput accelerator for ubiquitous machine-learning. In: Proceedings of the International Conference on Architectural Support for Programming Languages and Operating Systems, pp. 269–284 (2014). doi:10.1145/2541940.2541967

  28. Perronnin, F., Sanchez, J., Mensink, T.: Improving the fisher kernel for large-scale image classification. In: Proceedings of the European Conference on Computer Vision, pp. 143–156 (2010)

    Google Scholar 

  29. Harris, C., Stephens, M.: A combined corner and edge detector. In: Proceedings of the Fourth Alvey Vision Conference, pp. 147–151 (1988)

    Google Scholar 

  30. Winder, S., Hua, G., Brown, M.: Picking the best daisy. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition (2009)

    Google Scholar 

  31. Winder, S., Hua, G., Brown, M.: Discriminative learning of local image descriptors. IEEE Trans. Pattern Anal. Mach. Intell. 33(1), 43–57 (2009). doi:10.1109/TPAMI.2010.54

    Google Scholar 

  32. Shotton, J., Johnson, M., Cipolla, R.: Semantic texton forests for image categorization and segmentation. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition, pp. 1–8 (2008). doi:10.1109/CVPR.2008.4587503

  33. He, X., Yan, S., Hu, Y., Niyogi, P., Zhang, H.-J.: Face recognition using Laplacian faces. IEEE Trans. Pattern Anal. Mach. Intell. 27(3), 328–340 (2005)

    Article  Google Scholar 

  34. Chen, H.-T., Chang, H.-W., Liu, T.-L.: Local discriminant embedding and its variants. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition, pp. 846–853 (2005). doi:10.1109/CVPR.2005.216

  35. Winder, S.A.J., Brown, M.: Learning local image descriptors. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition, pp. 1–8 (2007)

    Google Scholar 

  36. Sanchez, J., Perronnin, F., Mensink, T., Jakob, V.: Image classification with the Fisher vector: theory and practice. Int. J. Comput. Vis. 105(3), 222–245 (2013). doi:10.1007/s11263-013-0636-x

    Article  MathSciNet  MATH  Google Scholar 

  37. Leijen, D., Schulte, W., Burchardt, S.: The design of a task parallel library. In: Proceedings of the Conference on Object Oriented Programming Systems Languages and Applications, pp. 227–242(2009). doi:10.1145/1640089.1640106

  38. Griffin, G., Holum, A., Perona, P.: Caltech-256 object category dataset. Caltech Technical Report Number: CNS-TR-2007-001. http://authors.library.caltech.edu/7694 (2011)

  39. LeCun, Y., Huang, F.J., Bottou, L.: Learning methods for generic object recognition with invariance to pose and lighting. Proc. Int. Conf. Comput. Vis. Pattern Recognit. (2004). doi:10.1109/CVPR.2004.1315150

  40. Everingham, M., Ali, E.S.M., Luc, V.G., Williams, C.K.I., Winn, J., Zisserman, A.: The pascal visual object classes challenge: a retrospective. Int. J. Comput. Vis. 1–39 (2014). doi:10.1007/s11263-014-0733-5

  41. OmniVision OV7735 Product Brief. www.ovt.com (2014)

  42. Chen, S., Bermak, A., Wang, Y.: A CMOS image sensor with on-chip image compression based on predictive boundary adaptation and memoryless QTD algorithm. IEEE Trans. VLSI Syst. 19(4), 538–547 (2011)

    Article  Google Scholar 

  43. Low Power Advantage of 802.11a/g vs. 802.11b. Whitepaper, Texas Instruments. www.ti.com (2003)

Download references

Author information

Authors and Affiliations

  1. Microsoft Research, Redmond, WA, 98052, USA

    Mohammed Shoaib, Xian-Sheng Hua, Jie Liu & Jin Li

  2. School of ECE, Purdue University, West Lafayette, IN, 47907, USA

    Swagath Venkataramani

Authors
  1. Mohammed Shoaib
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Swagath Venkataramani
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xian-Sheng Hua
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jie Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jin Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mohammed Shoaib .

Editor information

Editors and Affiliations

  1. Visual Computing Group, Microsoft Research Asia, Beijing, Beijing, China

    Gang Hua

  2. Alibaba Group, Hangzhou, Zhejiang, China

    Xian-Sheng Hua

Rights and permissions

Reprints and permissions

Copyright information

© 2015 Springer International Publishing Switzerland

About this chapter

Cite this chapter

Shoaib, M., Venkataramani, S., Hua, XS., Liu, J., Li, J. (2015). Exploiting On-Device Image Classification for Energy Efficiency in Ambient-Aware Systems. In: Hua, G., Hua, XS. (eds) Mobile Cloud Visual Media Computing. Springer, Cham. https://doi.org/10.1007/978-3-319-24702-1_7

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-24702-1_7

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-24700-7

  • Online ISBN: 978-3-319-24702-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation