Mingyang Zhang
ABSTRACT
While Siamese object tracking has witnessed significant advancements, its hard real-time behaviour on embedded devices remains inadequately addressed. In many application cases, an embedded implementation should not only have a minimal execution latency, but this latency should ideally also be having zero variance, i.e. predictable. To bridge this gap, we firstly analyse the real-time predictability of components of a state-of-the-art deep-learning-based object video object tracking system. Our detailed experiments indicate the superiority of FPGA implementations in terms of hard real-time behaviour, but unveil important time-predictability bottlenecks. Then, we craft a dedicated hardware accelerator specifically for the bottleneck. Our method seamlessly integrates advanced tracker features and improves greatly the tracker’s speed and time-predictability on embedded systems. Implemented on a KV260 board, our quantized tracker demonstrates superior performance. These findings spotlight the immense promise of hardware acceleration in real-time object tracking and set a benchmark for forthcoming hardware-software co-design pursuits focused on achieving time-predictable object tracking.
- Luca Bertinetto, Jack Valmadre, João F. Henriques, Andrea Vedaldi, and Philip H. S. Torr. 2016. Fully-Convolutional Siamese Networks for Object Tracking. In Proceedings of the European Conference on Computer Vision (ECCV), Gang Hua and Hervé Jégou (Eds.). Springer, 850–865.Google Scholar
- Yi Cao, Hongbing Ji, Wenbo Zhang, and Shahram Shirani. 2019. Extremely Tiny Siamese Networks with Multi-level Fusions for Visual Object Tracking. In 2019 22th International Conference on Information Fusion (FUSION). 1–7. https://doi.org/10.23919/FUSION43075.2019.9011338Google ScholarCross Ref
- Zhoujuan Cui and Junshe An. 2020. Heterogeneous Siamese Tracking System Based on PYNQ Framework. In 2020 6th International Conference on Control, Automation and Robotics (ICCAR). 16–20. https://doi.org/10.1109/ICCAR49639.2020.9108096Google ScholarCross Ref
- Itay Hubara, Yury Nahshan, Yair Hanani, Ron Banner, and Daniel Soudry. 2020. Improving post training neural quantization: Layer-wise calibration and integer programming. arXiv preprint arXiv:2006.10518 (2020).Google Scholar
- Vinod Kathail. 2020. Xilinx vitis unified software platform. In Proceedings of the 2020 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays. 173–174.Google ScholarDigital Library
- Bo Li, Wei Wu, Qiang Wang, Fangyi Zhang, Junliang Xing, and Junjie Yan. 2019. SiamRPN++: Evolution of Siamese Visual Tracking With Very Deep Networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 4282–4291.Google ScholarCross Ref
- Bo Li, Junjie Yan, Wei Wu, Zheng Zhu, and Xiaolin Hu. 2018. High Performance Visual Tracking With Siamese Region Proposal Network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 8971–8980.Google ScholarCross Ref
- Dominika Przewlocka, Mateusz Wasala, Hubert Szolc, Krzysztof Blachut, and Tomasz Kryjak. 2020. Optimisation of a Siamese Neural Network for Real-Time Energy Efficient Object Tracking. In Computer Vision and Graphics, Leszek J. Chmielewski, Ryszard Kozera, and Arkadiusz Orłowski (Eds.). Springer International Publishing, Cham, 151–163.Google Scholar
- Dominika Przewlocka-Rus and Tomasz Kryjak. 2022. Towards Real-Time and Energy Efficient Siamese Tracking – A Hardware-Software Approach. In Design and Architecture for Signal and Image Processing, Karol Desnos and Sergio Pertuz (Eds.). Springer International Publishing, Cham, 162–173.Google Scholar
- Pauli Virtanen, Ralf Gommers, Travis E Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, 2020. SciPy 1.0: fundamental algorithms for scientific computing in Python. Nature methods 17, 3 (2020), 261–272.Google Scholar
- Xilinx. [n. d.]. PYNQ - Python Productivity for Zynq. https://pynq.io/Google Scholar
- Xilinx. 2022. DPUCZDX8G for zynq UltraScale+ MPSoCs product guide (PG338). https://docs.xilinx.com/r/en-US/pg338-dpu?tocId=3xsG16y_QFTWvAJKHbisEwGoogle Scholar
- Xilinx. 2022. Vitis HLS - Xilinx. https://www.xilinx.com/products/design-tools/vitis/vitis-hls.htmlGoogle Scholar
- Xilinx. 2023. Vitis AI User Guide (UG1414). https://docs.xilinx.com/r/en-US/ug1414-vitis-ai/Vitis-AI-RuntimeGoogle Scholar
- Bingyi Zhang, Xin Li, Jun Han, and Xiaoyang Zeng. 2018. MiniTracker: A Lightweight CNN-based System for Visual Object Tracking on Embedded Device. In 2018 IEEE 23rd International Conference on Digital Signal Processing (DSP). 1–5. https://doi.org/10.1109/ICDSP.2018.8631813Google ScholarCross Ref
Index Terms
- Real-Time Predictability Analysis and Enhancement of Deep-Learning-Based Object Tracking
Recommendations
Real-Time Object Tracking System on FPGAs
SAAHPC '11: Proceedings of the 2011 Symposium on Application Accelerators in High-Performance ComputingObject tracking is an important task in computer vision applications. One of the crucial challenges is the real-time speed requirement. In this paper we implement an object tracking system in reconfigurable hardware using an efficient parallel ...
Real-time multiple object centroid tracking for gesture recognition based on FPGA
ICUIMC '13: Proceedings of the 7th International Conference on Ubiquitous Information Management and CommunicationIn this paper, we present the design and implementation of real-time multiple object centroid tracking for gesture recognition. Our multiple object tracking design consists of four stages: preprocessing, local intensity accumulation, object observation, ...
Automatically controlled pan-tilt smart camera with FPGA based image analysis system dedicated to real-time tracking of a moving object
An intelligent, automatically controlled camera based on visual feedback.A system for acquisition, processing image analysis and a camera driver are implemented in the FPGA Xilinx Spartan-6 device.The FPGA device is connected directly to the eight ...
Comments