skip to main content
10.1145/3307334.3326092acmconferencesArticle/Chapter ViewAbstractPublication PagesmobisysConference Proceedingsconference-collections
research-article
Public Access

Banner: An Image Sensor Reconfiguration Framework for Seamless Resolution-based Tradeoffs

Published: 12 June 2019 Publication History

Abstract

Mobile vision systems would benefit from the ability to situationally sacrifice image resolution to save system energy when imaging detail is unnecessary. Unfortunately, any change in sensor resolution leads to a substantial pause in frame delivery -- as much as 280 ms. Frame delivery is bottlenecked by a sequence of reconfiguration procedures and memory management in current operating systems before it resumes at the new resolution. This latency from reconfiguration impedes the adoption of otherwise beneficial resolution-energy tradeoff mechanisms. We propose Banner as a media framework that provides a rapid sensor resolution reconfiguration service as a modification to common media frameworks, e.g., V4L2. Banner completely eliminates the frame-to-frame reconfiguration latency (226 ms to 33 ms), i.e., removing the frame drop during sensor resolution reconfiguration. Banner also halves the end-to-end resolution reconfiguration latency (226 ms to 105 ms). This enables a more than 49% reduction of system power consumption by allowing continuous vision applications to reconfigure the sensor resolution to 480p compared with downsampling from 1080p to 480p, as measured in a cloud-based offloading workload running on a Jetson TX2 board. As a result, Banner unlocks unprecedented capabilities for mobile vision applications to dynamically reconfigure sensor resolutions to balance the energy efficiency and task accuracy tradeoff.

References

[1]
Andrew Adams, Eino-Ville Talvala, Sung Hee Park, David E. Jacobs, Boris Ajdin, Natasha Gelfand, Jennifer Dolson, Daniel Vaquero, Jongmin Baek, Marius Tico, Hendrik P. A. Lensch, Wojciech Matusik, Kari Pulli, Mark Horowitz, and Marc Levoy. 2010. The Frankencamera: An Experimental Platform for Computational Photography. In ACM SIGGRAPH 2010 Papers (SIGGRAPH '10). ACM.
[2]
Apple. 2018. Use HDR on your iPhone, iPad, and iPod touch. (2018). https://support.apple.com/en-us/HT207470
[3]
Apple. 2019. ARKit. (2019). https://developer.apple.com/arkit//
[4]
Saul Berenbaum. 2013. Google Glass Explorer Edition has a 30-minute battery life while shooting video. (2013). https://www.digitaltrends.com/mobile/google-glass-30-minute-videobattery/
[5]
Mark Buckler, Philip Bedoukian, Suren Jayasuriya, and Adrian Sampson. 2018. EVA2: Exploiting Temporal Redundancy in Live Computer Vision. In Proceedings of the 45th Annual International Symposium on Computer Architecture (ISCA '18). IEEE Press, Piscataway, NJ, USA, 533--546.
[6]
Mark Buckler, Suren Jayasuriya, and Adrian Sampson. 2017. Reconfiguring the Imaging Pipeline for Computer Vision. In The IEEE International Conference on Computer Vision (ICCV) .
[7]
Tiffany Chen, Hari Balakrishnan, Lenin Ravindranath, and Paramvir Bahl. 2016. Glimpse: Continuous, Real-Time Object Recognition on Mobile Devices. GetMobile: Mobile Computing and Communications, Vol. 20 (07 2016), 26--29.
[8]
David Chu, Nicholas D. Lane, Ted Tsung-Te Lai, Cong Pang, Xiangying Meng, Qing Guo, Fan Li, and Feng Zhao. 2011. Balancing Energy, Latency and Accuracy for Mobile Sensor Data Classification. In Proceedings of the 9th ACM Conference on Embedded Networked Sensor Systems (SenSys '11). ACM.
[9]
Google. 2019. ARcore. (2019). https://developers.google.com/ar//
[10]
KHRONOS Group. 2013. Camera BOF. (2013). https://www.khronos.org/assets/uploads/developers/library/2013-siggraph-camera-bof/Camera-BOF_SIGGRAPH-2013.pdf
[11]
Kiryong Ha, Zhuo Chen, Wenlu Hu, Wolfgang Richter, Padmanabhan Pillai, and Mahadev Satyanarayanan. 2014. Towards Wearable Cognitive Assistance. In Proceedings of the 12th Annual International Conference on Mobile Systems, Applications, and Services (MobiSys '14). ACM.
[12]
Muhammad Haris, Greg Shakhnarovich, and Norimichi Ukita. 2018. Task-Driven Super Resolution: Object Detection in Low-resolution Images. CoRR, Vol. abs/1803.11316 (2018). arxiv: 1803.11316 http://arxiv.org/abs/1803.11316
[13]
James Hegarty, John Brunhaver, Zachary DeVito, Jonathan Ragan-Kelley, Noy Cohen, Steven Bell, Artem Vasilyev, Mark Horowitz, and Pat Hanrahan. 2014. Darkroom: Compiling High-level Image Processing Code into Hardware Pipelines. ACM Trans. Graph., Vol. 33, 4, Article 144 (July 2014), bibinfonumpages11 pages.
[14]
James Hegarty, Ross Daly, Zachary DeVito, Jonathan Ragan-Kelley, Mark Horowitz, and Pat Hanrahan. 2016. Rigel: Flexible Multi-rate Image Processing Hardware. ACM Trans. Graph., Vol. 35, 4, Article 85 (July 2016), bibinfonumpages11 pages.
[15]
Jinhan Hu, Jianan Yang, Vraj Delhivala, and Robert LiKamWa. 2018. Characterizing the Reconfiguration Latency of Image Sensor Resolution on Android Devices. In Proceedings of the 19th International Workshop on Mobile Computing Systems & Applications (HotMobile '18). ACM.
[16]
Rizwan Mohamed Ibrahim. 2019. Camera. (2019). https://github.com/rizwankce/Camera/
[17]
Seungwoo Kang, Jinwon Lee, Hyukjae Jang, Hyonik Lee, Youngki Lee, Souneil Park, Taiwoo Park, and Junehwa Song. 2008. SeeMon: Scalable and Energy-efficient Context Monitoring Framework for Sensor-rich Mobile Environments. In Proceedings of the 6th International Conference on Mobile Systems, Applications, and Services (MobiSys '08). ACM.
[18]
The kernel development community. V4L2 video capture example.
[19]
The kernel development community. 2019. ioctl VIDIOC_REQBUFS. (2019). https://linuxtv.org/downloads/v4l-dvb-apis/uapi/v4l/vidioc-reqbufs.html
[20]
Venkatesh Kodukula, Sai Bharadwaj Medapuram, Britton Jones, and Robert LiKamWa. 2018. A Case for Temperature-Driven Task Migration to Balance Energy Efficiency and Image Quality of Vision Processing Workloads. Proceedings of the 19th International Workshop on Mobile Computing Systems & Applications (HotMobile '18). ACM.
[21]
Athindran Ramesh Kumar, Balaraman Ravindran, and Anand Raghunathan. 2018. Pack and Detect: Fast Object Detection in Videos Using Region-of- Interest Packing . ArXiv e-prints, Article arXiv:1809.01701 (Sept. 2018), bibinfonumpagesarXiv:1809.01701 pages.arxiv: cs.CV/1809.01701
[22]
Robert LiKamWa, Bodhi Priyantha, Matthai Philipose, Lin Zhong, and Paramvir Bahl. 2013. Energy Characterization and Optimization of Image Sensing Toward Continuous Mobile Vision. In Proceeding of the 11th Annual International Conference on Mobile Systems, Applications, and Services (MobiSys '13). ACM.
[23]
Robert LiKamWa and Lin Zhong. 2015. Starfish: Efficient Concurrency Support for Computer Vision Applications. In Proceedings of the 13th Annual International Conference on Mobile Systems, Applications, and Services, MobiSys '15. ACM.
[24]
Felix Xiaozhu Lin, Zhen Wang, Robert LiKamWa, and Lin Zhong. 2012. Reflex: Using Low-power Processors in Smartphones Without Knowing Them. SIGPLAN Not., Vol. 47, 4 (March 2012), 13--24.
[25]
Kaisen Lin, Aman Kansal, Dimitrios Lymberopoulos, and Feng Zhao. 2010. Energy-accuracy Trade-off for Continuous Mobile Device Location. In Proceedings of the 8th International Conference on Mobile Systems, Applications, and Services (MobiSys '10). ACM.
[26]
Tsung-Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollá r. 2017. Focal Loss for Dense Object Detection. CoRR, Vol. abs/1708.02002 (2017). arxiv: 1708.02002 http://arxiv.org/abs/1708.02002
[27]
Bodhi Priyantha, Dimitrios Lymberopoulos, and Jie Liu. EERS: Energy Efficient Responsive Sleeping on Mobile Phones.
[28]
B. Priyantha, D. Lymberopoulos, and J. Liu. 2011. LittleRock: Enabling Energy-Efficient Continuous Sensing on Mobile Phones. IEEE Pervasive Computing, Vol. 10, 2 (April 2011), 12--15.
[29]
Joseph Redmon and Ali Farhadi. 2018. YOLOv3: An Incremental Improvement. CoRR, Vol. abs/1804.02767 (2018). arxiv: 1804.02767 http://arxiv.org/abs/1804.02767
[30]
N. Roy, A. Misra, C. Julien, S. K. Das, and J. Biswas. 2011. An energy-efficient quality adaptive framework for multi-modal sensor context recognition. In 2011 IEEE International Conference on Pervasive Computing and Communications (PerCom) .
[31]
V'i t Ruzicka and Franz Franchetti. 2018. Fast and accurate object detection in high resolution 4K and 8K video using GPUs. CoRR, Vol. abs/1810.10551 (2018). arxiv: 1810.10551 http://arxiv.org/abs/1810.10551
[32]
Samsung. 2017. {In-Depth Look} Fast, Fun and In-Focus: The Galaxy S8 Camera. (2017). https://news.samsung.com/global/in-depth-look-fast-fun-and-in-focus-the-galaxy-s8-camera
[33]
Narayanan Sundaram. 2012. Making computer vision computationally efficient . Ph.D. Dissertation. EECS Department, University of California, Berkeley. http://www2.eecs.berkeley.edu/Pubs/TechRpts/2012/EECS-2012--106.html
[34]
Micron Technology. Calculating Memory System Power for DDR . https://www.micron.com/ /media/Documents/Products/Technical%20Note/DRAM/TN4603.pdf
[35]
Vuforia. 2019. Innovate With Industrial Augmented Reality. (2019). https://www.ptc.com/en/products/augmented-reality/
[36]
Wikipedia. 2018. High-dynamic-range imaging. (2018).
[37]
Boyu Zhang, Azadeh Davoodi, and Yu-Hen Hu. 2018. Exploring Energy and Accuracy Tradeoff in Structure Simplification of Trained Deep Neural Networks. In Proceedings of the 23rd Asia and South Pacific Design Automation Conference (ASPDAC '18). IEEE Press. http://dl.acm.org/citation.cfm?id=3201607.3201693

Cited By

View all
  • (2024)NIR-sighted: A Programmable Streaming Architecture for Low-Energy Human-Centric Vision ApplicationsACM Transactions on Embedded Computing Systems10.1145/367207623:6(1-26)Online publication date: 11-Sep-2024
  • (2023)Software-Defined Imaging: A SurveyProceedings of the IEEE10.1109/JPROC.2023.3266736111:5(445-464)Online publication date: May-2023
  • (2023)MRIM: Lightweight saliency-based mixed-resolution imaging for low-power pervasive visionPervasive and Mobile Computing10.1016/j.pmcj.2023.10185896(101858)Online publication date: Dec-2023
  • Show More Cited By

Index Terms

  1. Banner: An Image Sensor Reconfiguration Framework for Seamless Resolution-based Tradeoffs

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    MobiSys '19: Proceedings of the 17th Annual International Conference on Mobile Systems, Applications, and Services
    June 2019
    736 pages
    ISBN:9781450366618
    DOI:10.1145/3307334
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    In-Cooperation

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 12 June 2019

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. device drivers
    2. efficient visual computing
    3. energy efficiency
    4. operating systems
    5. reconfiguration
    6. resolution-based tradeoff

    Qualifiers

    • Research-article

    Funding Sources

    Conference

    MobiSys '19
    Sponsor:

    Acceptance Rates

    Overall Acceptance Rate 274 of 1,679 submissions, 16%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)154
    • Downloads (Last 6 weeks)28
    Reflects downloads up to 27 Jan 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)NIR-sighted: A Programmable Streaming Architecture for Low-Energy Human-Centric Vision ApplicationsACM Transactions on Embedded Computing Systems10.1145/367207623:6(1-26)Online publication date: 11-Sep-2024
    • (2023)Software-Defined Imaging: A SurveyProceedings of the IEEE10.1109/JPROC.2023.3266736111:5(445-464)Online publication date: May-2023
    • (2023)MRIM: Lightweight saliency-based mixed-resolution imaging for low-power pervasive visionPervasive and Mobile Computing10.1016/j.pmcj.2023.10185896(101858)Online publication date: Dec-2023
    • (2022)MRIM: Enabling Mixed-Resolution Imaging for Low-Power Pervasive Vision Tasks2022 IEEE International Conference on Pervasive Computing and Communications (PerCom)10.1109/PerCom53586.2022.9762398(44-53)Online publication date: 21-Mar-2022
    • (2021)Characterizing real-time dense point cloud capture and streaming on mobile devicesProceedings of the 3rd ACM Workshop on Hot Topics in Video Analytics and Intelligent Edges10.1145/3477083.3480155(1-6)Online publication date: 25-Oct-2021
    • (2021)LensCapProceedings of the 19th Annual International Conference on Mobile Systems, Applications, and Services10.1145/3458864.3467676(14-27)Online publication date: 24-Jun-2021
    • (2021)CODS: Cloud-assisted Object Detection for Streaming Videos on Edge Devices2021 IEEE International Performance, Computing, and Communications Conference (IPCCC)10.1109/IPCCC51483.2021.9679395(1-6)Online publication date: 29-Oct-2021

    View Options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Login options

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media