Skip to main content
SpringerLink
Log in

DSP-based parallel optimization for real-time video stitching

  • Original Research Paper
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

Video stitching is a technique that stitches multiple overlapped videos acquired from different cameras, which is widely applied in many applications, including video surveillance, autonomous driving, and virtual reality. Feature-based stitching methods are popular in this area because of their invariance property and efficiency. However, the video stitching pipeline is relatively complicated and the amount of data calculation is large, which impedes its real-time applications. In this paper, we propose a real-time video stitching framework based on vision Digital Signal Processing (DSP). Real-time processing is achieved by the algorithm-level and system-level optimizations. In the algorithm of ORB feature extraction, methods, including look-up table, linear approximation, and single instruction multiple data (SIMD), are adopted to optimize its computation on DSP. In the algorithm of feature matching, the Hamming distance calculation is eliminated for some feature point pairs when their coordinates and angles are not satisfied with certain conditions. Reverse feature matching is proposed to improve registration accuracy. In system-level optimization, the directed acyclic graph (DAG)-based scheduling is proposed to improve the calculation efficiency on dual DSPs, and a ping-pong buffer is utilized to speed up the data transmission between DSP and external memory. Experimental results show that the proposed method can achieve a ten times speedup than that of the CPU, and it can achieve 1536\(\times\)1024@37fps real-time processing on vision DSP.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18

Similar content being viewed by others

References

  1. Abbadi, N.K.E.L., Al Hassani, S.A., Abdulkhaleq, A.H.: A review over panoramic image stitching techniques. J. Phys.: Conf. Ser. 1999, 0121115 (2021). (IOP Publishing)

    Google Scholar 

  2. Li, C., Liu, J.: Parallax-tolerant image stitching for outdoor scenes. J. Phys.: Conf. Ser. 1651, 012186 (2020). (IOP Publishing)

    Google Scholar 

  3. Guo, C., Jia, F., Tang, W., Huang, P.: A fast method for image matching and registration based on sift algorithm and image pyramid. J. Phys.: Conf. Ser. 1449, 012119 (2020). (IOP Publishing)

    Google Scholar 

  4. Ali, I.H., Salman, S.: 360-degree panoramic image stitching for un-ordered images based on harris corner detection. Indian J. Sci. Technol. 12, 4 (2019)

    Google Scholar 

  5. Brown, M., Lowe, D.G.: Automatic panoramic image stitching using invariant features. Int. J. Comput. Vis. 74(1), 59–73 (2007)

    Article  Google Scholar 

  6. Shin, J., Rahim, M.A., Yun, K.S.: Panoramic image stitching with efficient brightness fusion using Ransac algorithm. Int. J. Eng. Technol. 7(3.34), 267–272 (2018)

    Article  Google Scholar 

  7. HajiRassouliha, A., Taberner, A.J., Nash, M.P., Nielsen, P.M.F.: Suitability of recent hardware accelerators (dsps, fpgas, and gpus) for computer vision and image processing algorithms. Signal Process.: Image Commun. 68, 101–119 (2018)

    Google Scholar 

  8. Cadence Design Systems, Inc. Vision P6 DSP User’s Guide, 03 2017

  9. Lee, J.-H.: Panoramic image stitching using feature extracting and matching on embedded system. Trans. Electr. Electron. Mater. 18(5), 273–278 (2017)

    Google Scholar 

  10. Jeon, H., Jeong, J., and Lee, K.: An implementation of the real-time panoramic image stitching using orb and prosac. In: 2015 International SoC Design Conference (ISOCC), pages 91–92. IEEE, (2015)

  11. Qingyi, G., Raut, S., Okumura, K., Aoyama, T., Takaki, T., Ishii, I.: Real-time image mosaicing system using a high-frame-rate video sequence. J. Robot. Mechatron. 27(1), 12–23 (2015)

    Article  Google Scholar 

  12. Jin, H.: Method and apparatus for estimating rotation, focal lengths and radial distortion in panoramic image stitching. US Patent 8, 131–113 (2012)

    Google Scholar 

  13. Hu, K.C., Lin, F.-Y., Chien, C.-C., Tsai, T.-S., Hsia, C.-H., and Chiang, J.-S.. Panoramic image stitching system for automotive applications. In: 2014 IEEE International Conference on Consumer Electronics-Taiwan, pages 203–204. IEEE (2014)

  14. Zhang, H., Zhao, M.: Panoramic image stitching using double encoder-decoders. SN Comput. Sci. 2(2), 1–12 (2021)

    Article  Google Scholar 

  15. Chen, L., Han, J., Zhang, Y., and Bai, L.: Real-time panoramic image mosaic via harris corner detection on fpga. In: Image and Graphics, pages 111–124. Springer (2015)

  16. Tong, L.I.N.G., Xin, Z.H.A.O., Zhe, H.O.U., Kai-wei, W.A.N.G., Jian, B.A.I.: Fast panoramic annular image stretching based on cuda. Comput. Technol. Develop. 27(4), 008–012 (2011)

    Google Scholar 

  17. Torii, A., Dong, Y., Okutomi, M., Sivic, J., Pajdla, T.: Efficient localization of panoramic images using tiled image descriptors. Inf. Media Technol. 9(3), 351–355 (2014)

    Google Scholar 

  18. Lowe, D.G.: Object recognition from local scale-invariant features. In: Proceeding of IEEE International Conference on Computer Vision (1999)

  19. Bay, H., Tuytelaars, T., and Van Gool, L.: Surf: speeded up robust features. In European Conference on Computer Vision, pages 404–417. Springer (2006)

  20. Rublee, E., Rabaud, V., Konolige, K., and Bradski, G.: Orb: an efficient alternative to sift or surf. In 2011 International Conference on Computer Vision, pages 2564–2571. IEEE (2011)

  21. Minchen, Z., Weizhi, W., Binghan, L., Jingshan, H., Derek, A.: Efficient video panoramic image stitching based on an improved selection of Harris corners and a multiple-constraint corner matching. Plos One 8(12), e81182 (2013)

    Article  Google Scholar 

  22. Wang, X., Cao, W., Yao, C., and Yin, H.: Feature matching algorithm based on surf and lowes algorithm. In: 2020 39th Chinese Control Conference (CCC) (2020)

  23. Jae Chang Kwak: An implementation of the real-time image stitching algorithm based on roi. J. IKEEE 19(4), 460–464 (2015)

    Article  Google Scholar 

  24. Bhat, A. S., Shivaprakash, A. V., Prasad, N. S., and Nagaraj, C.: Template matching technique for panoramic image stitching. In: Modelling Symposium (AMS), 2013 7th Asia (2013)

  25. Lo, I. C., Shih, K. T., Yu, P. C.,Hung, C. T. , and Chen, H. H. : Seamless stitching dual fisheye images for 360\(^\circ\) free view. In 2019 IEEE International Conference on Image Processing (ICIP) (2019)

  26. Lee, K. Y. and Sim, J. Y.: Stitching for multi-view videos with large parallax based on adaptive pixel warping. IEEE Access, pp. 1–1 (2018)

  27. Chengyao, D., Yuan, J., Dong, J., Li, L., Chen, M., Li, T.: Gpu based parallel optimization for real time panoramic video stitching. Pattern Recogn. Lett. 133, 62–69 (2020)

    Article  Google Scholar 

  28. Zhi, X., Yan, J., Hang, Y., Wang, S.: Realization of cuda-based real-time registration and target localization for high-resolution video images. J. Real-Time Image Proc. 16(4), 1025–1036 (2019)

    Article  Google Scholar 

  29. Liao, W.-S., Hsieh, T.-J., and Chang, Y.-L.: Gpu parallel computing of spherical panorama video stitching. In 2012 IEEE 18th International Conference on Parallel and Distributed Systems, pages 890–895. IEEE (2012)

  30. Gong, X., Le, Z.: Research and implementation of multi-object tracking based on vision dsp. J. Real-Time Image Proc. 17(6), 1801–1809 (2020)

    Article  Google Scholar 

  31. Turturici, M., Saponara, S., Fanucci, L., and Franchi, E.: Low-power embedded system for real-time correction of fish-eye automotive cameras. In 2012 Design, Automation & Test in Europe Conference & Exhibition (DATE) 2012

  32. Imsaengsuk, T., and Pumrin, S.: Feature detection and description based on orb algorithm for fpga-based image processing. In 2021 9th International Electrical Engineering Congress (iEECON) (2021)

  33. Rosten, E. and Drummond, T.: Machine learning for high-speed corner detection. In European Conference on Computer Vision, pages 430–443. Springer (2006)

  34. Bpl Rosin. Measuring corner properties. In: Computer Vision and Image Understanding. In Cviu (1999)

  35. Calonder, M., Lepetit, V., Strecha, C., and Fua, P.: Brief: binary robust independent elementary features. In European Conference on Computer Vision, pages 778–792. Springer (2010)

  36. Pappalardo, F., Calonaci, C., Pennisi, M., Mastriani, E., and Motta, S.: Hamfast: Fast hamming distance computation. In 2009 WRI World Congress on Computer Science and Information Engineering, volume 1, pages 569–572 (2009)

  37. Yan, C., Zhang, Y., Xu, J., Feng, D., Zhang, J., Dai, Q., Feng, W.: Efficient parallel framework for hevc motion estimation on many-core processors. IEEE Trans. Circ. Syst. Video Technol. 24(12), 2077–2089 (2014)

    Article  Google Scholar 

  38. Gao, W., Ma, S., Duan, L., Tian, Y., Xing, P., Wang, Y., Wang, S., Jia, H., Huang, T.: Digital retina: a way to make the city brain more efficient by visual coding. IEEE Trans. Circ. Syst. Video Technol. 31(11), 4147–4161 (2021)

    Article  Google Scholar 

  39. Balntas, V., Lenc, K., Vedaldi, A., and Mikolajczyk, K.: Hpatches: a benchmark and evaluation of handcrafted and learned local descriptors. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3852–3861 (2017)

  40. Chai, T., Draxler, R.R.: Root mean square error (rmse) or mean absolute error (mae). Geosci. Model Dev. Discuss. 7(1), 1525–1534 (2014)

    Google Scholar 

  41. Sara, U., Akter, M., Uddin, M.S.: Image quality assessment through fsim, ssim, mse and psnr-a comparative study. J. Comput. Commun. 7(3), 8–18 (2019)

    Article  Google Scholar 

  42. Sharma, S.K., Jain, K., and Suresh, M.: Quantitative evaluation of panorama softwares. In International Conference on Communications and Cyber Physical Engineering 2018, pages 543–561. Springer (2018)

  43. Pavan Chennagiri Madhusudana and Rajiv Soundararajan: Subjective and objective quality assessment of stitched images for virtual reality. IEEE Trans. Image Process. 28(11), 5620–5635 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  44. Jose, A., Pachath, A., Rajesh, A., Chandhan, P., and Shenil, P.S.: Fpga based novel architecture for real-time video stitching. In 2021 Innovations in Power and Advanced Computing Technologies (i-PACT), pages 1–7. IEEE (2021)

  45. Yeh, S.-H., and Lai, S.-H.: Real-time video stitching. In 2017 IEEE International Conference on Image Processing (ICIP), pages 1482–1486. IEEE (2017)

  46. Wang, G., Zhai, Z., Xu, B., and Cheng, Y.: A parallel method for aerial image stitching using orb feature points. In 2017 IEEE/ACIS 16th International Conference on Computer and Information Science (ICIS), pages 769–773. IEEE (2017)

  47. Choi, K., Jun, K.: Real-time panorama video system using networked multiple cameras. J. Syst. Architect. 64, 110–121 (2016)

    Article  Google Scholar 

  48. Mukherjee, S., Su, G.-M., and Cheng, I.: Adaptive dithering using curved Markov-gaussian noise in the quantized domain for mapping sdr to hdr image. In International Conference on Smart Multimedia, pages 193–203. Springer (2018)

Download references

Acknowledgements

This work was supported in part by the National Key R &D Program of China (2021ZD0109802), and by  the National Natural Science Foundation of China under Grant Nos. 61901150, 61931008, and 61972123.

Author information

Authors and Affiliations

  1. School of Communication Engineering, Hangzhou Dianzi University, Hangzhou, 310018, China

    Xiaofeng Huang, Ran Tang, Yang Zhou & Haibing Yin

  2. Advanced Institute of Information Technology, Peking University, Hangzhou, 311215, China

    Xiaofeng Huang

  3. School of Automation, Hangzhou Dianzi University, Hangzhou, 310018, China

    Chenggang Yan

Authors
  1. Xiaofeng Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ran Tang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yang Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Haibing Yin
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Chenggang Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yang Zhou.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Huang, X., Tang, R., Zhou, Y. et al. DSP-based parallel optimization for real-time video stitching. J Real-Time Image Proc 20, 28 (2023). https://doi.org/10.1007/s11554-023-01275-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11554-023-01275-x

Keywords

Search

Navigation