Abstract
Learning-based image restoration approaches typically learn to map distorted images to clean images. To remove multiple combined distortions with unknown mixture ratios, most of the existing methods have focused on the development of different deep neural network architectures and novel loss functions. Although these methods have proved their effectiveness on image restoration tasks, they require expensive training data and produce results in a noninterpretable way. In this work, we present a deep reinforcement learning (DRL) based method to restore the distorted images, which casts an image restoration Problem as a Partially Observable Markov Decision Process (POMDP) where actions are defined as multiple pixel-wise image denoising operations. In our method, each agent possesses a pixel, the agent learns to adjust the corresponding pixel value by determining the proper combination of the actions. We also develop a novel exploration scheme such that similar actions have similar value, thereby avoiding overfitting in state-action value estimation. Through extensive experiments, we show that our method can restore images with multiple combined distortions and our DRL approach performs comparable or better performance against previous learning-based approaches. By visualizing the process of weighting multiple pixel-wise operations, we can identify what combination of operations is employed for each pixel at each stage. We believe our work takes a step toward the explainability and interpretability of learning-based image restoration methods.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Agustsson E, Timofte R (2017) Ntire 2017 challenge on single image super-resolution: dataset and study. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 126–135
Anaya J, Barbu A (2018) RENOIR-a dataset for real low-light image noise reduction. J Vis Commun Image Represent 51:144–154
Bertsekas DP, Tsitsiklis JN (1995) Neuro-dynamic programming: an overview. In: Proceedings of 1995 34th IEEE conference on decision and control. IEEE, pp 560–564
Bowling M, Veloso M (2001) Rational and convergent learning in stochastic games. In: International joint conference on artificial intelligence. Lawrence Erlbaum Associates Ltd, pp 1021–1026
Buades A, Coll B, Morel J-M (2005) A non-local algorithm for image denoising. In: 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR’05). IEEE, pp 60–65
Burger HC, Schuler CJ, Harmeling S (2012) Image denoising: Can plain neural networks compete with BM3D? In: 2012 IEEE conference on computer vision and pattern recognition. IEEE, pp 2392–2399
Buşoniu L, Babuška R, De Schutter B (2010) Multi-agent reinforcement learning: an overview. In: Innovations in multi-agent systems and applications-1. Springer, pp 183–221
Cao Q, Lin L, Shi Y (2017) Attention-aware face hallucination via deep reinforcement learning. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 690–698
Chen Y, Yu W, Pock T (2015) On learning optimized reaction diffusion processes for effective image restoration. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5261–5269
Chen W, Wilson J, Tyree S (2015) Compressing neural networks with the hashing trick. In: International conference on machine learning, pp 2285–2294
Dabov K, Foi A, Egiazarian K (2007) Image denoising by sparse 3-D transform-domain collaborative filtering. IEEE Trans Image Process 16:2080–2095
Furuta R, Inoue N, Yamasaki T (2019) Pixelrl: Fully convolutional network with reinforcement learning for image processing. IEEE Trans Multimed 22(7):1704–1719
Guo J, Chao H (2017) One-to-many network for visually pleasing compression artifacts reduction. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3038–3047
Guestrin C, Koller D, Parr R (2001) Solving factored POMDPs with linear value functions. In: 17th international joint conference on artificial intelligence (IJCAI-01) workshop on planning under uncertainty and incomplete information. Citeseer, pp 67–75
Han S, Mao H, Dally WJ (2015) Deep compression: compressing deep neural networks with pruning, trained quantization and huffman coding. ArXiv preprint arXiv:1510.00149
Kim J, Lee JK, Lee KM (2016) Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1646–1654
Kingma DP, Ba J (2014) Adam: a method for stochastic optimization. ArXiv preprint arXiv:1412.6980
Koller D, Parr R (2013) Policy iteration for factored MDPs. ArXiv preprint arXiv:1301.3869
Li D, Wu H, Zhang J, Huang K (2018) A2-RL: Aesthetics aware reinforcement learning for image cropping. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 8193–8201
Li W, Feng X, An H (2020) MRI reconstruction with interpretable pixel-wise operations using reinforcement learning. In: Proceedings of the AAAI conference on artificial intelligence, pp 792–799
Li Z, Zhang X (2019) Deep reinforcement learning for automatic thumbnail generation. In: International conference on multimedia modeling. Springer, pp 41–53
Liao X, Li W, Xu Q (2020) Iteratively-refined interactive 3D medical image segmentation with multi-agent reinforcement learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 9394–9402
Lin L-J, Mitchell TM (1992) Memory approaches to reinforcement learning in non-Markovian domains. Carnegie-Mellon University. Department of Computer Science
Littman ML (1994) Markov games as a framework for multi-agent reinforcement learning. In: Machine learning proceedings. Elsevier, pp 157–163
Littman ML (2001) Friend-or-foe Q-learning in general-sum games. In: ICML, pp 322–328
Lowe R, Wu YI, Tamar A (2017) Multi-agent actor-critic for mixed cooperative-competitive environments. In: Advances in neural information processing systems, pp 6379–6390
Mairal J, Elad M, Sapiro G (2007) Sparse representation for color image restoration. IEEE Trans Image Process 17:53–69
Park J, Lee J-Y, Yoo D, So Kweon I (2018) Distort-and-recover: color enhancement using deep reinforcement learning. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5928–5936
Polikar R (2012) Ensemble learning. In: Ensemble machine learning. Springer, pp 1–34
Rudin LI, Osher S, Fatemi E (1992) Nonlinear total variation based noise removal algorithms. Phys Nonlinear Phenom 60:259–268
Rashid T, Samvelyan M, De Witt CS (2018) QMIX: monotonic value function factorisation for deep multi-agent reinforcement learning. ArXiv preprint arXiv:1803.11485
Singsanga S, Hattagam W, Tat EH (2010) Packet forwarding in overlay wireless sensor networks using NashQ reinforcement learning. In: 2010 6th international conference on intelligent sensors, sensor networks and information processing. IEEE, pp 85–90
Suganuma M, Liu X, Okatani T (2019) Attention-based adaptive selection of operations for image restoration in the presence of unknown combined distortions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 9039–9048
Tan M (1993) Multi-agent reinforcement learning: independent vs. cooperative agents. In: Proceedings of the 10th international conference on machine learning, pp 330–337
White CC III, Scherer WT (1989) Solution procedures for partially observed Markov decision processes. Oper Res 37:791–797
Xie C, Wu Y, Maaten LV, Yuille AL, He K (2019) Feature denoising for improving adversarial robustness. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 501–509
Yang Y, Luo R, Li M (2018) Mean field multi-agent reinforcement learning. ArXiv preprint ArXiv:1802.05438
Yu K, Dong C, Lin L, Loy CC (2018) Crafting a toolchain for image restoration by deep reinforcement learning. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2443–2452
Zhang K, Zuo W, Chen Y (2017) Beyond a gaussian denoiser: residual learning of deep CNN for image denoising. IEEE Trans Image Process 26:3142–3155
Zhang K, Yang Z, Başar T (2019) Multi-agent reinforcement learning: a selective overview of theories and algorithms. ArXiv preprint arXiv:1911.10635
Acknowledgements
We are very grateful to the anonymous reviewers for their constructive comments on improving this paper.
Funding
This work was supported by the Key-Area Research and Development Program of Guangdong Pro-vince, Grant No. 2019B02 0223003.
Author information
Authors and Affiliations
Contributions
Jie Zhang was responsible for experimental design, implementation, and paper editing. Qiyuan Zhang participated in the realization of the experimental process. Xixuan Zhao participated in the editing of the article, and Jiangming Kan guided the experiment.
Corresponding author
Ethics declarations
Conflict of interest
All the authors declare no conflict of interests.
Ethical approval
This paper does not contain any studies with human participants or animals performed by any of the authors.
Informed consent
This paper does not contain any studies with human participants performed by any of the authors, so there is no informed consent involved.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Zhang, J., Zhang, Q., Zhao, X. et al. Boosting denoisers with reinforcement learning for image restoration. Soft Comput 26, 3261–3272 (2022). https://doi.org/10.1007/s00500-022-06840-3
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00500-022-06840-3