skip to main content
research-article

Image/Video Restoration via Multiplanar Autoregressive Model and Low-Rank Optimization

Published: 16 December 2019 Publication History

Abstract

In this article, we introduce an image/video restoration approach by utilizing the high-dimensional similarity in images/videos. After grouping similar patches from neighboring frames, we propose to build a multiplanar autoregressive (AR) model to exploit the correlation in cross-dimensional planes of the patch group, which has long been neglected by previous AR models. To further utilize the nonlocal self-similarity in images/videos, a joint multiplanar AR and low-rank based approach is proposed (MARLow) to reconstruct patch groups more effectively. Moreover, for video restoration, the temporal smoothness of the restored video is constrained by the Markov random field (MRF), where MRF encodes a priori knowledge about consistency of patches from neighboring frames. Specifically, we treat different restoration results (from different patch groups) of a certain patch as labels of an MRF, and temporal consistency among these restored patches is imposed. The proposed method is also suitable for other restoration applications such as interpolation and text removal. Extensive experimental results demonstrate that the proposed approach obtains encouraging performance comparing with state-of-the-art methods.

References

[1]
Adobe Research. n.d. Content-Aware Fill. Retrieved October 23, 2019 from https://research.adobe.com/project/content-aware-fill/.
[2]
Y. Boykov, O. Veksler, and R. Zabih. 2001. Fast approximate energy minimization via graph cuts. IEEE Transactions on Pattern Analysis and Machine Intelligence 23, 11 (Nov. 2001), 1222--1239.
[3]
Antoni Buades, Bartomeu Coll, and Jean-Michel Morel. 2005. A review of image denoising algorithms, with a new one. Multiscale Modeling 8 Simulation 4, 2 (2005), 490--530.
[4]
Jian-Feng Cai, Emmanuel J. Candès, and Zuowei Shen. 2010. A singular value thresholding algorithm for matrix completion. SIAM Journal on Optimization 20, 4 (2010), 1956--1982.
[5]
Emmanuel J. Candès and Benjamin Recht. 2009. Exact matrix completion via convex optimization. Foundations of Computational Mathematics 9, 6 (2009), 717--772.
[6]
Y.-L. Chen, C.-T. Hsu, and H.-Y. M. Liao. 2014. Simultaneous tensor decomposition and completion using factor priors. IEEE Transactions on Pattern Analysis and Machine Intelligence 36, 3 (March 2014), 577--591.
[7]
G. Chierchia, N. Pustelnik, B. Pesquet-Popescu, and J.-C. Pesquet. 2014. A nonlocal structure tensor-based approach for multicomponent image recovery problems. IEEE Transactions on Image Processing 23, 12 (Dec. 2014), 5531--5544.
[8]
K. Dabov, A. Foi, and K. Egiazarian. 2007. Video denoising by sparse 3D transform-domain collaborative filtering. In Proceedings of the European Signal Processing Conference. 145--149.
[9]
K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian. 2007. Image denoising by sparse 3-D transform-domain collaborative filtering. IEEE Transactions on Image Processing 16, 8 (Aug. 2007), 2080--2095.
[10]
Weisheng Dong, Guangming Shi, and Xin Li. 2013. Nonlocal image restoration with bilateral variance estimation: A low-rank approach. IEEE Transactions on Image Processing 22, 2 (Feb. 2013), 700--711.
[11]
W. Dong, L. Zhang, R. Lukac, and G. Shi. 2013. Sparse representation based image interpolation with nonlocal autoregressive modeling. IEEE Transactions on Image Processing 22, 4 (April 2013), 1382--1394.
[12]
J.-J. Fadili, J.-L. Starck, and F. Murtagh. 2009. Inpainting and zooming using sparse representations. Computer Journal 52, 1 (2009), 64--79.
[13]
W. B. Goh, M. N. Chong, S. Kalra, and D. Krishnan. 1996. Bi-directional 3D auto-regressive model approach to motion picture restoration. In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP ’96), Vol. 4. 2275--2278.
[14]
Mohammad Golbabaee and Pierre Vandergheynst. 2012. Hyperspectral image compressed sensing via low-rank and joint-sparse matrix recovery. In Proceedings of the 2012 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP’12). IEEE, Los Alamitos, CA, 2741--2744.
[15]
Liangtian He and Yilun Wang. 2014. Iterative support detection-based split Bregman method for wavelet frame-based image inpainting. IEEE Transactions on Image Processing 23, 12 (Dec. 2014), 5470--5485.
[16]
F. Heide, W. Heidrich, and G. Wetzstein. 2015. Fast and flexible convolutional sparse coding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’15). 5135--5143.
[17]
Wenrui Hu, Dacheng Tao, Wensheng Zhang, Yuan Xie, and Yehui Yang. 2015. A new low-rank tensor model for video completion. arXiv:1509.02027.
[18]
Yao Hu, Debing Zhang, Jieping Ye, Xuelong Li, and Xiaofei He. 2013. Fast and accurate matrix completion via truncated nuclear norm regularization. IEEE Transactions on Pattern Analysis and Machine Intelligence 35, 9 (2013), 2117--2130.
[19]
Yao Hu, Chen Zhao, Deng Cai, Xiaofei He, and Xuelong Li. 2016. Atom decomposition with adaptive basis selection strategy for matrix completion. ACM Transactions on Multimedia Computing, Communications, and Applications 12, 3 (June 2016), Article 43, 25 pages.
[20]
J. B. Huang, A. Singh, and N. Ahuja. 2015. Single image super-resolution from transformed self-exemplars. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’15). 5197--5206.
[21]
V. Jakhetiya, W. Lin, S. P. Jaiswal, S. C. Guntuku, and O. C. Au. 2017. Maximum a posterior and perceptually motivated reconstruction algorithm: A generic framework. IEEE Transactions on Multimedia 19, 1 (Jan. 2017), 93--106.
[22]
Hui Ji, Sibin Huang, Zuowei Shen, and Yuhong Xu. 2011. Robust video restoration by joint sparse and low rank matrix approximation. SIAM Journal on Imaging Sciences 4, 4 (2011), 1122--1142.
[23]
Hui Ji, Chaoqiang Liu, Zuowei Shen, and Yuhong Xu. 2010. Robust video denoising using low rank matrix completion. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’10). 1791--1798.
[24]
A. Kokaram and P. Rayner. 1994. Detection and interpolation of replacement noise in motion picture sequences using 3D autoregressive modelling. In Proceedings of the IEEE International Symposium on Circuits and Systems (ISCAS’94), Vol. 3. 21--24.
[25]
Chao Li, Lili Guo, and Andrzej Cichocki. 2014. Multi-tensor completion for estimating missing values in video data. In Proceedings of the 2014 Joint 7th International Conference on Soft Computing and Intelligent Systems (SCIS’14) and the 15th International Symposium on Advanced Intelligent Systems (ISIS’14). IEEE, Los Alamitos, CA, 1339--1342.
[26]
Chao Li, Qibin Zhao, Junhua Li, Andrzej Cichocki, and Lili Guo. 2015. Multi-Tensor Completion with Common Structures. Retrieved October 23, 2019 from http://www.aaai.org/ocs/index.php/AAAI/AAAI15/paper/view/9437.
[27]
Mading Li, Jiaying Liu, Zhiwei Xiong, Xiaoyan Sun, and Zongming Guo. 2016. MARLow: A joint multiplanar autoregressive and low-rank approach for image completion. In Proceedings of the European Conference on Computer Vision (ECCV’16). 819--834.
[28]
X. Li and M. T. Orchard. 2001. New edge-directed interpolation. IEEE Transactions on Image Processing 10, 10 (Oct. 2001), 1521--1527.
[29]
R. Liao, X. Tao, R. Li, Z. Ma, and J. Jia. 2015. Video super-resolution via deep draft-ensemble learning. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’15). 531--539.
[30]
Ji Liu, P. Musialski, P. Wonka, and J. Ye. 2009. Tensor completion for estimating missing values in visual data. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’09). 2114--2121.
[31]
J. Liu, P. Musialski, P. Wonka, and J. Ye. 2013. Tensor completion for estimating missing values in visual data. IEEE Transactions on Pattern Analysis and Machine Intelligence 35, 1 (2013), 208--220.
[32]
Luca Lorenzi, Farid Melgani, and Grégoire Mercier. 2013. Missing-area reconstruction in multispectral images under a compressive sensing perspective. IEEE Transactions on Geoscience and Remote Sensing 51, 7 (2013), 3998--4008.
[33]
M. Maggioni, G. Boracchi, A. Foi, and K. Egiazarian. 2012. Video denoising, deblocking, and enhancement through separable 4-D nonlocal spatiotemporal transforms. IEEE Transactions on Image Processing 21, 9 (Sept. 2012), 3952--3966.
[34]
M. Maggioni, V. Katkovnik, K. Egiazarian, and A. Foi. 2013. Nonlocal transform-domain filter for volumetric data denoising and reconstruction. IEEE Transactions on Image Processing 22, 1 (Jan. 2013), 119--133.
[35]
J. Mairal, F. Bach, J. Ponce, G. Sapiro, and A. Zisserman. 2009. Non-local sparse models for image restoration. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’09). 2272--2279.
[36]
J. Mairal, M. Elad, and G. Sapiro. 2008. Sparse representation for color image restoration. IEEE Transactions on Image Processing 17, 1 (Jan. 2008), 53--69.
[37]
D. Martin, C. Fowlkes, D. Tal, and J. Malik. 2001. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proceedings of the International Conference on Computer Vision (ICCV’01), Vol. 2. 416--423.
[38]
S. Ono, T. Miyata, and I. Yamada. 2014. Cartoon-texture image decomposition using blockwise low-rank texture characterization. IEEE Transactions on Image Processing 23, 3 (March 2014), 1128--1142.
[39]
Holger Rauhut, Karin Schnass, and Pierre Vandergheynst. 2008. Compressed sensing and redundant dictionaries. IEEE Transactions on Information Theory 54, 5 (2008), 2210--2219.
[40]
Justin Romberg. 2009. Compressive sensing by random convolution. SIAM Journal on Imaging Sciences 2, 4 (2009), 1098--1128.
[41]
Stefan Roth and Michael J. Black. 2009. Fields of experts. International Journal of Computer Vision 82, 2 (2009), 205--229.
[42]
Tijana Ružić, Bruno Cornelis, Ljiljana Platiša, Aleksandra Pižurica, Ann Dooms, Wilfried Philips, Maximiliaan Martens, Marc De Mey, and Ingrid Daubechies. 2011. Virtual restoration of the Ghent Altarpiece using crack detection and inpainting. In Proceedings of the International Conference on Advanced Concepts for Intelligent Vision Systems. 417--428.
[43]
Huanfeng Shen, Xinghua Li, Liangpei Zhang, Dacheng Tao, and Chao Zeng. 2014. Compressed sensing-based inpainting of aqua moderate resolution imaging spectroradiometer band 6 using adaptive spectrum-weighted sparse Bayesian dictionary learning. IEEE Transactions on Geoscience and Remote Sensing 52, 2 (2014), 894--906.
[44]
L. Sun and J. Hays. 2012. Super-resolution from Internet-scale scene matching. In Proceedings of the 2012 IEEE International Conference on Computational Photography (ICCP’12). 1--12.
[45]
Kenny Kal Vin Toh and Nor Ashidi Mat Isa. 2010. Noise adaptive fuzzy switching median filter for salt-and-pepper noise reduction. IEEE Signal Processing Letters 17, 3 (2010), 281--284.
[46]
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. 2018. Deep image prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’18). 9446--9454.
[47]
H. Wang, Y. Cen, Z. He, R. Zhao, Y. Cen, and F. Zhang. 2017. Robust generalized low-rank decomposition of multimatrices for image recovery. IEEE Transactions on Multimedia 19, 5 (May 2017), 969--983.
[48]
Hua Wang, Feiping Nie, and Heng Huang. 2014. Low-Rank Tensor Completion with Spatio-Temporal Consistency. Retrieved October 23, 2019 from http://www.aaai.org/ocs/index.php/AAAI/AAAI14/paper/view/8580.
[49]
Debing Zhang, Yao Hu, Jieping Ye, Xuelong Li, and Xiaofei He. 2012. Matrix completion by truncated nuclear norm regularization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’12). 2192--2199.
[50]
Jian Zhang, Debin Zhao, and Wen Gao. 2014. Group-based sparse representation for image restoration. IEEE Transactions on Image Processing 23, 8 (Aug. 2014), 3336--3351.
[51]
Jian Zhang, Debin Zhao, Ruiqin Xiong, Siwei Ma, and Wen Gao. 2014. Image restoration using joint statistical modeling in a space-transform domain. IEEE Transactions on Circuits and Systems for Video Technology 24, 6 (June 2014), 915--928.
[52]
Peixuan Zhang and Fang Li. 2014. A new adaptive weighted mean filter for removing salt-and-pepper noise. IEEE Signal Processing Letters 21, 10 (2014), 1280--1283.
[53]
Xiangjun Zhang and Xiaolin Wu. 2008. Image interpolation by adaptive 2-D autoregressive modeling and soft-decision estimation. IEEE Transactions on Image Processing 17, 6 (June 2008), 887--896.
[54]
Z. Zhang, G. Ely, S. Aeron, N, Hao, and M. Kilmer. 2014. Novel methods for multilinear data completion and de-noising based on tensor-SVD. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’14). 3842--3849.
[55]
Dizhi Zhou, Xinyue Shen, and Wenjie Dong. 2012. Image zooming using directional cubic convolution interpolation. IET Image Processing 6, 6 (2012), 627--634.
[56]
M. Zhou, H. Chen, J. Paisley, L. Ren, L. Li, Z. Xing, D. Dunson, G. Sapiro, and L. Carin. 2012. Nonparametric Bayesian dictionary learning for analysis of noisy and incomplete images. IEEE Transactions on Image Processing 21, 1 (Jan. 2012), 130--144.

Cited By

View all
  • (2025)Predictive modeling and insight into protein fouling in microfiltration and ultrafiltration through one-dimensional convolutional modelsSeparation and Purification Technology10.1016/j.seppur.2024.128237352(128237)Online publication date: Jan-2025
  • (2023)Classifying Urban Functional Zones Based on Modeling POIs by DeepwalkSustainability10.3390/su1510799515:10(7995)Online publication date: 13-May-2023
  • (2023)Sparsity-Guided Discriminative Feature Encoding for Robust Keypoint DetectionACM Transactions on Multimedia Computing, Communications, and Applications10.1145/3628432Online publication date: 17-Oct-2023
  • Show More Cited By

Index Terms

  1. Image/Video Restoration via Multiplanar Autoregressive Model and Low-Rank Optimization

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Transactions on Multimedia Computing, Communications, and Applications
    ACM Transactions on Multimedia Computing, Communications, and Applications  Volume 15, Issue 4
    November 2019
    322 pages
    ISSN:1551-6857
    EISSN:1551-6865
    DOI:10.1145/3376119
    Issue’s Table of Contents
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 16 December 2019
    Accepted: 01 June 2019
    Revised: 01 May 2019
    Received: 01 August 2018
    Published in TOMM Volume 15, Issue 4

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. Image/video restoration
    2. Markov random field
    3. low-rank optimization
    4. multiplanar autoregressive model

    Qualifiers

    • Research-article
    • Research
    • Refereed

    Funding Sources

    • National Natural Science Foundation of China
    • Beijing Natural Science Foundation

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)22
    • Downloads (Last 6 weeks)0
    Reflects downloads up to 10 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2025)Predictive modeling and insight into protein fouling in microfiltration and ultrafiltration through one-dimensional convolutional modelsSeparation and Purification Technology10.1016/j.seppur.2024.128237352(128237)Online publication date: Jan-2025
    • (2023)Classifying Urban Functional Zones Based on Modeling POIs by DeepwalkSustainability10.3390/su1510799515:10(7995)Online publication date: 13-May-2023
    • (2023)Sparsity-Guided Discriminative Feature Encoding for Robust Keypoint DetectionACM Transactions on Multimedia Computing, Communications, and Applications10.1145/3628432Online publication date: 17-Oct-2023
    • (2023)Low-light Image Enhancement via a Frequency-based Model with Structure and Texture DecompositionACM Transactions on Multimedia Computing, Communications, and Applications10.1145/359096519:6(1-23)Online publication date: 31-May-2023
    • (2023)Aligning Image Semantics and Label Concepts for Image Multi-Label ClassificationACM Transactions on Multimedia Computing, Communications, and Applications10.1145/355027819:2(1-23)Online publication date: 6-Feb-2023
    • (2023)Image Quality Assessment–driven Reinforcement Learning for Mixed Distorted Image RestorationACM Transactions on Multimedia Computing, Communications, and Applications10.1145/353262519:1s(1-23)Online publication date: 3-Feb-2023
    • (2023)Deep Inter Prediction with Error-Corrected Auto-Regressive Network for Video CodingACM Transactions on Multimedia Computing, Communications, and Applications10.1145/352817319:1s(1-22)Online publication date: 23-Jan-2023
    • (2023)Double Attention Based on Graph Attention Network for Image Multi-Label ClassificationACM Transactions on Multimedia Computing, Communications, and Applications10.1145/351903019:1(1-23)Online publication date: 5-Jan-2023
    • (2023)Novel heuristic-based hybrid ResNeXt with recurrent neural network to handle multi class classification of sentiment analysisMachine Learning: Science and Technology10.1088/2632-2153/acc0d54:1(015033)Online publication date: 21-Mar-2023
    • (2022)Densely Enhanced Semantic Network for Conversation System in Social MediaACM Transactions on Multimedia Computing, Communications, and Applications10.1145/350179918:4(1-24)Online publication date: 4-Mar-2022
    • Show More Cited By

    View Options

    Login options

    Full Access

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    HTML Format

    View this article in HTML Format.

    HTML Format

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media