Skip to main content
SpringerLink
Log in

Adaptive video stabilization based on feature point detection and full-reference stability assessment

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Video stabilization is an important video enhancement technique that removes shaky motion and produces stable videos with good visual quality. Previous feature point matching methods are proposed to estimate motion information. However, video stabilization based on feature points matching will decrease or fail to match feature for video sequences lacking feature point, which causes poor results. Also, there is not a recognized method to measure the performance of video stabilization. To solve these problems, we propose an adaptive video stabilization based on feature point detection and full-reference stability assessment. In the proposed method, appropriate video stabilization algorithms are firstly selected by detecting the number of feature points and camera trajectory optimization is used to retain original motion information. Secondly, we propose a full-reference stability assessment to measure video stabilization performance. Furthermore, video stabilization is assessed from three aspects on the video dataset DeepStab. Finally, experimental results demonstrate the promising performance of our proposed algorithm in terms of subjective and objective evaluations.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Algorithm 1
Algorithm 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Data Availability

The authors declare that data supporting the findings of this study are available within the article

References

  1. Grundmann M, Kwatra V, Essa IA (2011) Auto-directed video stabilization with robust L1 optimal camera paths. In: The 24th IEEE conference on computer vision and pattern recognition, CVPR 2011, Colorado springs, CO, USA, 20-25 June 2011, pp 225–232. IEEE computer society, CVPR. https://doi.org/10.1109/CVPR.2011.5995525

  2. Liu F, Gleicher M, Jin H, Agarwala A (2009) Content-preserving warps for 3d video stabilization. ACM Trans Graph 28(3):44. https://doi.org/10.1145/1531326.1531350

  3. Liu F, Gleicher M, Wang J, Jin H, Agarwala A (2011) Subspace video stabilization. ACM Trans Graph 30(1):4–1410. https://doi.org/10.1145/1899404.1899408

  4. Lin S, Le TNH, Wu P, Lee T (2021) Content-and-disparity-aware stereoscopic video stabilization. Multim Tools Appl 80(1):1545–1564. https://doi.org/10.1007/s11042-020-09767-9

  5. Laboratories K Multimedia: Use Image Stabilization. http://www.websiteoptimization.com/speed/tweak/stabilizer/

  6. Rawat P, Singhai J (2011) Review of motion estimation and video stabilization techniques for hand held mobile video. Signal Image Process: An Int J (SIPIJ) Vol 2

  7. Sato K, Ishizuka S, Nikami A, Sato M (1993) Control techniques for optical image stabilizing system. IEEE Trans Consumer Electron 39(3):461–466. https://doi.org/10.1109/30.234621

  8. Guilluy W, Oudre L, Beghdadi A (2021) Video stabilization: overview, challenges and perspectives. Signal Process Image Commun 90:116015. https://doi.org/10.1016/j.image.2020.116015

  9. Morimoto C, Chellappa R (1998) Evaluation of image stabilization algorithms. In: Proceedings of the 1998 IEEE international conference on acoustics, speech and signal processing, ICASSP ’98, Seattle, Washington, USA, May 12-15, 1998, pp 2789–2792. IEEE, ICASSP. https://doi.org/10.1109/ICASSP.1998.678102

  10. Raj R, Rajiv P, Kumar P, Khari M, Verdú E, Crespo RG, Manogaran G (2020) Feature based video stabilization based on boosted HAAR cascade and representative point matching algorithm. Image Vis Comput 101:103957. https://doi.org/10.1016/j.imavis.2020.103957

  11. Prasertsakul P, Kondo T, Iida H, Phatrapornnant T (2020) Camera operation estimation from video shot using 2d motion vector histogram. Multim Tools Appl 79(25–26):17403–17426. https://doi.org/10.1007/s11042-019-08378-3

  12. Wang Y, Huang Q, Jiang C, Liu J, Shang M, Miao Z (2023) Video stabilization: a comprehensive survey. Neurocomputing 516:205–230. https://doi.org/10.1016/j.neucom.2022.10.008

  13. Matsushita Y, Ofek E, Ge W, Tang X, Shum H (2006) Full-frame video stabilization with motion inpainting. IEEE Trans Pattern Anal Mach Intell 28(7):1150–1163. https://doi.org/10.1109/TPAMI.2006.141

  14. Grundmann M, Kwatra V, Castro D, Essa IA (2012) Calibration-free rolling shutter removal. In: 2012 IEEE international conference on computational photography, ICCP 2012, Seattle, WA, USA, April 28-29, 2012, pp 1–8. IEEE computer society, ICCP. https://doi.org/10.1109/ICCPhot.2012.6215213

  15. Guo H, Liu S, He T, Zhu S, Zeng B, Gabbouj M (2016) Joint video stitching and stabilization from moving cameras. IEEE Trans Image Process 25(11):5491–5503. https://doi.org/10.1109/TIP.2016.2607419

  16. Liu S, Yuan L, Tan P, Sun J (2013) Bundled camera paths for video stabilization. ACM Trans Graph 32(4):78–17810. https://doi.org/10.1145/2461912.2461995

  17. Wang Y, Liu F, Hsu P, Lee T (2013) Spatially and temporally optimized video stabilization. IEEE Trans Vis Comput Graph 19(8):1354–1361. https://doi.org/10.1109/TVCG.2013.11

  18. Hu W, Chen C, Chen T, Peng M, Su Y (2018) Real-time video stabilization for fast-moving vehicle cameras. Multim Tools Appl 77(1):1237–1260. https://doi.org/10.1007/s11042-016-4291-4

  19. Wu R, Xu Z, Zhang J, Zhang L (2021) Robust global motion estimation for video stabilization based on improved k-means clustering and superpixel. Sensors 21(7):2505. https://doi.org/10.3390/s21072505

  20. Hu W, Chen C, Su Y, Chang T (2018) Feature-based real-time video stabilization for vehicle video recorder system. Multim Tools Appl 77(5):5107–5127. https://doi.org/10.1007/s11042-017-4369-7

  21. Krishnakumar K, Gandhi SI (2019) Video stitching using interacting multiple model based feature tracking. Multim Tools Appl 78(2):1375–1397. https://doi.org/10.1007/s11042-018-6116-0

  22. Liu F, Niu Y, Jin H (2013) Joint subspace stabilization for stereoscopic video. In: IEEE international conference on computer vision, ICCV 2013, Sydney, Australia, December 1-8, 2013, pp 73–80. IEEE computer society, ICCV. https://doi.org/10.1109/ICCV.2013.16

  23. Liu S, Yuan L, Tan P, Sun J (2014) Steadyflow: Spatially smooth optical flow for video stabilization. In: 2014 IEEE conference on computer vision and pattern recognition, CVPR 2014, Columbus, OH, USA, June 23-28, 2014, pp 4209–4216. IEEE computer society, CVPR. https://doi.org/10.1109/CVPR.2014.536

  24. Liu S, Tan P, Yuan L, Sun J, Zeng B (2016) Meshflow: minimum latency online video stabilization. In: Leibe B, Matas J, Sebe N, Welling M (eds.) Computer vision - ECCV 2016 - 14th European conference, Amsterdam, The Netherlands, October 11-14, 2016, proceedings, part VI. Lecture notes in computer science, 9910:800–815. Springer, ECCV. https://doi.org/10.1007/978-3-319-46466-4_48

  25. Liu S, Li M, Zhu S, Zeng B (2017) Codingflow: enable video coding for video stabilization. IEEE Trans Image Process 26(7):3291–3302. https://doi.org/10.1109/TIP.2017.2697759

  26. Huang H, Wei X, Zhang L (2019) Encoding shaky videos by integrating efficient video stabilization. IEEE Trans Circuits Syst Video Technol 29(5):1503–1514. https://doi.org/10.1109/TCSVT.2018.2833476

  27. Chi X, Zhang Y, Di Maio D, Lieven NA (2021) Viability of image compression in vibrothermography. Exp Tech 45:345–362. https://doi.org/10.1007/s40799-020-00395-4

  28. Wang Y, Liu F, Hsu P, Lee T (2013) Spatially and temporally optimized video stabilization. IEEE Trans Vis Comput Graph 19(8):1354–1361. https://doi.org/10.1109/TVCG.2013.11

  29. Bai J, Agarwala A, Agrawala M, Ramamoorthi R (2014) User-assisted video stabilization 33:61–70. https://doi.org/10.1111/cgf.12413

  30. Liu S, Wang Y, Yuan L, Bu J, Tan P, Sun J (2012) Video stabilization with a depth camera. In: 2012 IEEE conference on computer vision and pattern recognition, Providence, RI, USA, June 16-21, 2012, 89–95. IEEE computer society, CVPR. https://doi.org/10.1109/CVPR.2012.6247662

  31. Zhou Z, Jin H, Ma Y () Plane-based content preserving warps for video stabilization. In: 2013 IEEE conference on computer vision and pattern recognition, Portland, OR, USA, June 23-28, 2013, 2299–2306. IEEE computer society, CVPR. https://doi.org/10.1109/CVPR.2013.298

  32. Buehler C, Bosse M, McMillan L (2001) Non-metric image-based rendering for video stabilization. In: 2001 IEEE computer society conference on computer vision and pattern recognition (CVPR 2001), with CD-ROM, 8-14 December 2001, Kauai, HI, USA, pp 609–614. IEEE Computer Society, CVPR. https://doi.org/10.1109/CVPR.2001.991019

  33. Smith BM, Zhang L, Jin H, Agarwala A (2009) Light field video stabilization. In: IEEE 12th international conference on computer vision, ICCV 2009, Kyoto, Japan, September 27 - October 4, 2009, pp 341–348. IEEE computer society, ICCV. https://doi.org/10.1109/ICCV.2009.5459270

  34. Bell S, Troccoli AJ, Pulli K (2014) A non-linear filter for gyroscope-based video stabilization. In: Fleet DJ, Pajdla T, Schiele B, Tuytelaars T (eds.) Computer vision - ECCV 2014 - 13th European conference, Zurich, Switzerland, September 6-12, 2014, proceedings, Part IV. Lecture notes in computer science, 8692:294–308. Springer, ECCV. https://doi.org/10.1007/978-3-319-10593-2_20

  35. Karpenko A, Jacobs D, Baek J, Levoy M (2011) Digital video stabilization and rolling shutter correction using gyroscopes. CSTR 1(2):13

    Google Scholar 

  36. Ovren H, Forssén P (2015) Gyroscope-based video stabilisation with auto-calibration. In: IEEE international conference on robotics and automation, ICRA 2015, Seattle, WA, USA, 26-30 May, 2015, pp 2090–2097. IEEE, ICRA. https://doi.org/10.1109/ICRA.2015.7139474

  37. Goldstein A, Fattal R (2012) Video stabilization using epipolar geometry. ACM Trans Graph 31(5):126–112610. https://doi.org/10.1145/2231816.2231824

  38. Zhang L, Xu Q, Huang H (2017) A global approach to fast video stabilization. IEEE Trans Circuits Syst Video Technol 27(2):225–235. https://doi.org/10.1109/TCSVT.2015.2501941

  39. Li S, Handa A, Zhang Y, Calway A (2016) Hdrfusion: HDR SLAM using a low-cost auto-exposure RGB-D sensor. In: Fourth international conference on 3D vision, 3DV 2016, Stanford, CA, USA, October 25-28, 2016, pp 314–322. IEEE computer society, 3DV. https://doi.org/10.1109/3DV.2016.40

  40. Valero MM, Verstockt S, Mata C, Jimenez D, Queen L, Rios O, Pastor E, Planas E (2020) Image similarity metrics suitable for infrared video stabilization during active wildfire monitoring: a comparative analysis. Remote Sens 12(3):540. https://doi.org/10.3390/rs12030540

  41. Streijl RC, Winkler S, Hands DS (2016) Mean opinion score (MOS) revisited: methods and applications, limitations and alternatives. Multim Syst 22(2):213–227. https://doi.org/10.1007/s00530-014-0446-1

  42. Korhonen J, You J (2012) Peak signal-to-noise ratio revisited: is simple beautiful? In: Burnett IS (ed.) Fourth international workshop on quality of multimedia experience, QoMEX 2012, Melbourne, Australia, July 5-7, 2012, pp 37–38. IEEE, QoMEX. https://doi.org/10.1109/QoMEX.2012.6263880

  43. Wang Z, Bovik AC (2009) Mean squared error: love it or leave it? A new look at signal fidelity measures. IEEE Signal Process Mag 26(1):98–117

    Article  ADS  Google Scholar 

  44. Wang Z, Simoncelli EP, Bovik AC (2003) Multiscale structural similarity for image quality assessment. In: The thrity-seventh Asilomar conference on signals, systems & computers, 2003, 2:1398–14022. https://doi.org/10.1109/ACSSC.2003.1292216

  45. Niskanen M, Silvén O, Tico M (2006) Video stabilization performance assessment. In: Proceedings of the 2006 IEEE international conference on multimedia and expo, ICME 2006, July 9-12 2006, Toronto, Ontario, Canada, 405–408. IEEE computer society, ICME. https://doi.org/10.1109/ICME.2006.262522

  46. Jia S, Zhang Y (2018) Saliency-based deep convolutional neural network for no-reference image quality assessment. Multim Tools Appl 77(12):14859–14872. https://doi.org/10.1007/s11042-017-5070-6

  47. Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vis 60(2):91–110. https://doi.org/10.1023/B:VISI.0000029664.99615.94

  48. Subbarao R, Meer P (2006) Beyond RANSAC: user independent robust regression. In: IEEE conference on computer vision and pattern recognition, CVPR workshops 2006, New York, NY, USA, 17-22 June, 2006, p 101. IEEE computer society, CVPR. https://doi.org/10.1109/CVPRW.2006.43

  49. Qu H, Song L (2013) Video stabilization with L1-L2 optimization. In: IEEE international conference on image processing, ICIP 2013, Melbourne, Australia, September 15-18, 2013, pp 29–33. IEEE, ICIP. https://doi.org/10.1109/ICIP.2013.6738007

  50. Lee K, Chuang Y, Chen B, Ouhyoung M (2009) Video stabilization using robust feature trajectories. In: IEEE 12th international conference on computer vision, ICCV 2009, Kyoto, Japan, September 27 - October 4, 2009, pp 1397–1404. IEEE computer society, ICCV. https://doi.org/10.1109/ICCV.2009.5459297

  51. Qu H, Song L, Xue G (2013) Shaking video synthesis for video stabilization performance assessment. In: 2013 visual communications and image processing, VCIP 2013, Kuching, Malaysia, November 17-20, 2013, pp 1–6. IEEE, VCIP. https://doi.org/10.1109/VCIP.2013.6706422

  52. Wang M, Yang G, Lin J, Zhang S, Shamir A, Lu S, Hu S (2019) Deep online video stabilization with multi-grid warping transformation learning. IEEE Trans Image Process 28(5):2283–2292. https://doi.org/10.1109/TIP.2018.2884280

  53. Deshaker. http://www.guthspot.se/video/deshaker.htm

  54. Fang M, Li H, Si S (2018) A video stabilization algorithm based on affine sift. In: 2018 international conference on computing, mathematics and engineering technologies (iCoMET), pp 1–4. https://doi.org/10.1109/ICOMET.2018.8346332

  55. Wang M, Yang G, Lin J, Zhang S, Shamir A, Lu S, Hu S (2019) Deep online video stabilization with multi-grid warping transformation learning. IEEE Trans Image Process 28(5):2283–2292. https://doi.org/10.1109/TIP.2018.2884280

  56. Yu J, Ramamoorthi R, Cheng K, Sarkis M, Bi N (2021) Real-time selfie video stabilization. In: IEEE conference on computer vision and pattern recognition, CVPR 2021, virtual, June 19-25, 2021, pp 12036–12044. Computer vision foundation / IEEE, CVPR. https://doi.org/10.1109/CVPR46437.2021.01186. https://openaccess.thecvf.com/content/CVPR2021/html/Yu_Real-Time_Selfie_Video_Stabilization_CVPR_2021_paper.html

  57. Liu Y, Lai W, Yang M, Chuang Y, Huang J (2021) Hybrid neural fusion for full-frame video stabilization. In: 2021 IEEE/CVF international conference on computer vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pp 2279–2288. IEEE, ICCV. https://doi.org/10.1109/ICCV48922.2021.00230

  58. Xu Y, Zhang J, Maybank SJ, Tao D (2022) DUT: learning video stabilization by simply watching unstable videos. IEEE Trans Image Process 31:4306–4320. https://doi.org/10.1109/TIP.2022.3182887

Download references

Acknowledgements

This work was supported in part by the Fundamental Research Funds for the Central Universities under Grant No.B230205048, the Postgraduate Research & Practice Innovation Program of Jiangsu Province under Grant (No.KYCX23_0685, No.SJCX22_0161), the Key Research and Development Program of Yunnan Province under grant No. 202203AA080009, the 14th Five-Year Plan for Educational Science of Jiangsu Province under grant No. D/2021/01/39, the Jiangsu Higher Education Reform Research Project under grant No.2021JSJG143, and the 2022 Undergraduate Practice Teaching Reform Research Project of Hohai University

Author information

Authors and Affiliations

  1. School of Computer and Information, Hohai University, Nanjing, China

    Yiming Wang, Qian Huang, Jiwen Liu, Chuanxu Jiang & Mingzhou Shang

  2. Key Laboratory of Water Big Data Technology of Ministry of Water Resources, Hohai University, Nanjing, China

    Yiming Wang, Qian Huang, Jiwen Liu, Chuanxu Jiang & Mingzhou Shang

Authors
  1. Yiming Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qian Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jiwen Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Chuanxu Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Mingzhou Shang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Qian Huang.

Ethics declarations

Conflicts of interest

The authors declared that they have no conflicts of interest to this work

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, Y., Huang, Q., Liu, J. et al. Adaptive video stabilization based on feature point detection and full-reference stability assessment. Multimed Tools Appl 83, 32497–32524 (2024). https://doi.org/10.1007/s11042-023-16607-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-16607-z

Keywords

Search

Navigation