research-article

Toward High-quality Face-Mask Occluded Restoration

Authors:

Han HongAuthors Info & Claims

ACM Transactions on Multimedia Computing, Communications and Applications, Volume 19, Issue 1

Article No.: 24, Pages 1 - 23

https://doi.org/10.1145/3524137

Published: 06 January 2023 Publication History

Abstract

Face-mask occluded restoration aims at restoring the masked region of a human face, which has attracted increasing attention in the context of the COVID-19 pandemic. One major challenge of this task is the large visual variance of masks in the real world. To solve it we first construct a large-scale Face-mask Occluded Restoration (FMOR) dataset, which contains 5,500 unmasked images and 5,500 face-mask occluded images with various illuminations, and involves 1,100 subjects of different races, face orientations, and mask types. Moreover, we propose a Face-Mask Occluded Detection and Restoration (FMODR) framework, which can detect face-mask regions with large visual variations and restore them to realistic human faces. In particular, our FMODR contains a self-adaptive contextual attention module specifically designed for this task, which is able to exploit the contextual information and correlations of adjacent pixels for achieving high realism of the restored faces, which are however often neglected in existing contextual attention models. Our framework achieves state-of-the-art results of face restoration on three datasets, including CelebA, AR, and our FMOR datasets. Moreover, experimental results on AR and FMOR datasets demonstrate that our framework can significantly improve masked face recognition and verification performance.

References

[1]

Sunil Arya and D. M. Mount. 1998. ANN: Library for approximate nearest neighbor searching. In Proceedings of the IEEE CGC Workshop on Computational Geometry, Providence, RI.

[2]

Michael Ashikhmin. 2001. Synthesizing natural textures. In Proceedings of the 2001 Symposium on Interactive 3D Graphics. 217–226.

Digital Library

[3]

Samik Banerjee and Sukhendu Das. 2020. SD-GAN: Structural and denoising GAN reveals facial parts under occlusion[J]. arXiv preprint arXiv:2002.08448(2020).

[4]

Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B. Goldman. 2009. PatchMatch: A randomized correspondence algorithm for structural image editing. ACM Transactions on Graphics 28, 3 (2009), 24.

Digital Library

[5]

Raphaël Bornard, Emmanuelle Lecan, Louis Laborelli, and Jean-Hugues Chenot. 2002. Missing data correction in still images and image sequences. In Proceedings of the 10th ACM International Conference on Multimedia. 355–361.

Digital Library

[6]

Emmanuel J. Candès, Xiaodong Li, Yi Ma, and John Wright. 2011. Robust principal component analysis? The Journal of the ACM 58, 3 (2011), 1–37.

Digital Library

[7]

Chaofeng Chen, Xiaoming Li, Lingbo Yang, Xianhui Lin, Lei Zhang, and Kwan-Yee K. Wong. 2021. Progressive semantic-aware style transformation for blind face restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 11896–11905.

[8]

Xiang Chen, Linbo Qing, Xiaohai He, Jie Su, and Yonghong Peng. 2018. From eyes to face synthesis: A new approach for human-centered smart surveillance. IEEE Access 6 (2018), 14567–14575.

[9]

Antonio Criminisi, Patrick Pérez, and Kentaro Toyama. 2004. Region filling and object removal by exemplar-based image inpainting. IEEE Transactions on Image Processing 13, 9 (2004), 1200–1212.

Digital Library

[10]

Jiankang Deng, Jia Guo, Tongliang Liu, Mingming Gong, and Stefanos Zafeiriou. 2020. Sub-center arcface: Boosting face recognition by large-scale noisy web faces. In Proceedings of the European Conference on Computer Vision. Springer, 741–757.

Digital Library

[11]

Alexei A. Efros and William T. Freeman. 2001. Image quilting for texture synthesis and transfer. In Proceedings of the CVPR. ACM, 341–346.

Digital Library

[12]

Alexei A. Efros and Thomas K. Leung. 1999. Texture synthesis by non-parametric sampling. In Proceedings of the ICCV. IEEE, 1033–1038.

Digital Library

[13]

Wen Gao, Bo Cao, Shiguang Shan, Xilin Chen, Delong Zhou, Xiaohua Zhang, and Debin Zhao. 2007. The CAS-PEAL large-scale chinese face database and baseline evaluations. IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans 38, 1 (2007), 149–161.

[14]

Shiming Ge, Jia Li, Qiting Ye, and Zhao Luo. 2017. Detecting masked faces in the wild with lle-cnns. In Proceedings of the CVPR. 2682–2690.

[15]

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Proceedings of the NeurIPS. 2672–2680.

[16]

Ralph Gross. 2005. Face databases. In Proceedings of the Handbook of Face Recognition. Springer, 301–327.

[17]

Ralph Gross, Iain Matthews, Jeffrey Cohn, Takeo Kanade, and Simon Baker. 2010. Multi-pie. Image and Vision Computing 28, 5 (2010), 807–813.

Digital Library

[18]

Xiefan Guo, Hongyu Yang, and Di Huang. 2021. Image inpainting via conditional texture and structure dual generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 14134–14143.

[19]

Kaiming He and Jian Sun. 2012. Statistics of patch offsets for image completion. In Proceedings of the ECCV. Springer, 16–29.

[20]

Lingxiao He, Haiqing Li, Qi Zhang, Zhenan Sun, and Zhaofeng He. 2016. Multiscale representation for partial face recognition under near infrared illumination. In Proceedings of the BTAS. IEEE, 1–7.

Digital Library

[21]

Xiaofei He and Partha Niyogi. 2004. Locality preserving projections. In Proceedings of the Advances in Neural Information Processing Systems.153–160.

[22]

Alexander Hermans, Lucas Beyer, and Bastian Leibe. 2017. In defense of the triplet loss for person re-identification[J]. arXiv preprint arXiv:1703.07737(2017).

[23]

Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. 2017. Globally and locally consistent image completion. ACM Transactions on Graphics 36, 4 (2017), 1–14.

Digital Library

[24]

Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. 2017. Image-to-image translation with conditional adversarial networks. In Proceedings of the CVPR. 1125–1134.

[25]

Wen Jian-Ke. 2013. Ways to retouch photos. Laboratory Ence 16, 4 (2013), 14–17.

[26]

Jino Lee, Dong-Kyu Lee, and Rae-Hong Park. 2012. Robust exemplar-based inpainting algorithm using region segmentation. IEEE Transactions on Consumer Electronics 58, 2 (2012), 553–561.

[27]

Zhen Lei, Shengcai Liao, Ran He, Matti Pietikainen, and Stan Z. Li. 2008. Gabor volume based local binary pattern for face representation and recognition. In Proceedings of the International Conference on Automatic Face Gesture Recognition. IEEE, 1–6.

[28]

Ang Li, Jianzhong Qi, Rui Zhang, and Ramamohanarao Kotagiri. 2019. Boosted gan with semantically interpretable information for image inpainting. In Proceedings of the IJCNN. IEEE, 1–8.

[29]

Chenyu Li, Shiming Ge, Daichi Zhang, and Jia Li. 2020. Look through masks: Towards masked face recognition with de-occlusion distillation. In Proceedings of the 28th ACM International Conference on Multimedia. 3016–3024.

Digital Library

[30]

Yijun Li, Sifei Liu, Jimei Yang, and Ming-Hsuan Yang. 2017. Generative face completion. In Proceedings of the CVPR. 3911–3919.

[31]

Ce Liu, Heung-Yeung Shum, and Chang-Shui Zhang. 2001. A two-step approach to hallucinating faces: Global parametric model and local nonparametric model. In Proceedings of the CVPR. IEEE, I–I.

[32]

Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. 2015. Deep learning face attributes in the wild. In Proceedings of the ICCV. 3730–3738.

Digital Library

[33]

Omkar M. Parkhi, Andrea Vedaldi, and Andrew Zisserman. 2015. Deep Face Recognition. British Machine Vision Association.

[34]

Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A. Efros. 2016. Context encoders: Feature learning by inpainting. In Proceedings of the CVPR. 2536–2544.

[35]

Bryan Christopher Russell, Antonio J. Torralba, Kevin Patrick Murphy, and William T. Freeman. 2008. LabelMe. International Journal of Computer Vision 77, 1 (2008), 157–173.

Digital Library

[36]

Antonio Torralba, Rob Fergus, and William T. Freeman. 2008. 80 million tiny images: A large data set for nonparametric object and scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 30, 11 (2008), 1958–1970.

Digital Library

[37]

Luan Tran, Xi Yin, and Xiaoming Liu. 2017. Disentangled representation learning gan for pose-invariant face recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1415–1424.

[38]

Matthew A. Turk and Alex P. Pentland. 1991. Face recognition using eigenfaces. In Proceedings of the CVPR. IEEE Computer Society, 586–587.

[39]

Qiong Wang and Jingyu Yang. 2006. Eye detection in facial images with unconstrained background. JPRR 1, 1 (2006), 55–62.

[40]

Xintao Wang, Yu Li, Honglun Zhang, and Ying Shan. 2021. Towards real-world blind face restoration with generative facial prior. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 9168–9178.

[41]

Yi Wang, Xin Tao, Xiaojuan Qi, Xiaoyong Shen, and Jiaya Jia. 2018. Image inpainting via generative multi-column convolutional neural networks. In Proceedings of the NeurIPS. 331–340.

[42]

Zhongyuan Wang, Guangcheng Wang, Baojin Huang, Zhangyang Xiong, Qi Hong, Hao Wu, Peng Yi, Kui Jiang, Nanxi Wang, Yingjiao Pei, Heling Chen, Yu Miao, Zhibing Huang, and Jinbi Liang. 2020. Masked face recognition dataset and application. arXiv preprint arXiv:2003.09093 (2020).

[43]

John Wright, Allen Y. Yang, Arvind Ganesh, S. Shankar Sastry, and Yi Ma. 2008. Robust face recognition via sparse representation. IEEE Transactions on Pattern Analysis and Machine Intelligence 31, 2 (2008), 210–227.

Digital Library

[44]

Weihao Xia, Yujiu Yang, Jing-Hao Xue, and Baoyuan Wu. 2021. Towards open-world text-guided face image generation and manipulation[J]. arXiv preprint arXiv:2104.08910 (2021).

[45]

Chao Yang, Xin Lu, Zhe Lin, Eli Shechtman, Oliver Wang, and Hao Li. 2017. High-resolution image inpainting using multi-scale neural patch synthesis. In Proceedings of the CVPR. 6721–6729.

[46]

Bangjie Yin, Luan Tran, Haoxiang Li, Xiaohui Shen, and Xiaoming Liu. 2019. Towards interpretable face recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 9348–9357.

[47]

Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S. Huang. 2018. Generative image inpainting with contextual attention. In Proceedings of the CVPR. 5505–5514.

[48]

Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S. Huang. 2019. Free-form image inpainting with gated convolution. In Proceedings of the ICCV. 4471–4480.

[49]

Lingyun Yu, Hongtao Xie, and Yongdong Zhang. 2021. Multimodal learning for temporally coherent talking face generation with articulator synergy. IEEE Transactions on Multimedia 24 (2021), 2950–2962.

[50]

Tao Yu, Zongyu Guo, Xin Jin, Shilin Wu, Zhibo Chen, Weiping Li, Zhizheng Zhang, and Sen Liu. 2020. Region normalization for image inpainting. In Proceedings of the AAAI Conference on Artificial Intelligence. 12733–12740.

[51]

Xian Zhang, Canghong Shi, Xin Wang, Xi Wu, Xiaojie Li, Jiancheng Lv, and Imran Mumtaz. 2021. Face inpainting based on GAN by facial prediction and fusion as guidance information. Applied Soft Computing 111 (2021), 107626.

Digital Library

[52]

Fang Zhao, Jiashi Feng, Jian Zhao, Wenhan Yang, and Shuicheng Yan. 2017. Robust lstm-autoencoders for face de-occlusion in the wild. IEEE Transactions on Image Processing 27, 2 (2017), 778–790.

[53]

Jian Zhao, Yu Cheng, Yan Xu, Lin Xiong, Jianshu Li, Fang Zhao, Karlekar Jayashree, Sugiri Pranata, Shengmei Shen, Junliang Xing, et al. 2018. Towards pose invariant face recognition in the wild. In Proceedings of the CVPR. 2207–2216.

[54]

Jian Zhao, Jianshu Li, Xiaoguang Tu, Fang Zhao, Yuan Xin, Junliang Xing, Hengzhu Liu, Shuicheng Yan, and Jiashi Feng. 2019. Multi-prototype networks for unconstrained set-based face recognition. In Proceedings of the 28th International Joint Conference on Artificial Intelligence. 4397–4403.

[55]

Jian Zhao, Lin Xiong, Panasonic Karlekar Jayashree, Jianshu Li, Fang Zhao, Zhecan Wang, Panasonic Sugiri Pranata, Panasonic Shengmei Shen, Shuicheng Yan, and Jiashi Feng. 2017. Dual-agent gans for photorealistic and identity preserving profile face synthesis. In Proceedings of the NeurIPS. 66–76.

[56]

Tong Zhou, Changxing Ding, Shaowen Lin, Xinchao Wang, and Dacheng Tao. 2020. Learning oracle attention for high-fidelity face completion. In Proceedings of the CVPR. 7680–7689.

[57]

Xiuzhuang Zhou, Kai Jin, Yuanyuan Shang, and Guodong Guo. 2018. Visually interpretable representation learning for depression recognition from facial images. IEEE Transactions on Affective Computing 11, 3 (2018), 542–552.

[58]

Zheng Zhu, Guan Huang, Jiankang Deng, Yun Ye, Junjie Huang, Xinze Chen, Jiagang Zhu, Tian Yang, Jiwen Lu, Dalong Du, and Jie Zhou. 2021. WebFace260M: A benchmark unveiling the power of million-scale deep face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10492–10502.

Cited By

Yuan TZhang XLiu BLiu KJin JJiao Z(2025)Surveillance Video-and-Language Understanding: From Small to Large Multimodal ModelsIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2024.346243335:1(300-314)Online publication date: 1-Jan-2025
https://dl.acm.org/doi/10.1109/TCSVT.2024.3462433
Wu JFeng YXu HZhu CZheng JWooldridge MDy JNatarajan S(2024)SyFormerProceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence10.1609/aaai.v38i6.28417(6021-6029)Online publication date: 20-Feb-2024
https://dl.acm.org/doi/10.1609/aaai.v38i6.28417
Lin JWang Y(2024)TSFormer: Tracking Structure Transformer for Image InpaintingACM Transactions on Multimedia Computing, Communications, and Applications10.1145/369645220:12(1-23)Online publication date: 20-Sep-2024
https://dl.acm.org/doi/10.1145/3696452
Show More Cited By

Index Terms

Toward High-quality Face-Mask Occluded Restoration
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
        Reconstruction

Recommendations

Does face restoration improve face verification?
Abstract
Methods for face verification works reasonably well on face images with standardized (frontal) face positions and good spatial resolution. However such methods have significant challenges on poor resolution images, poor lighting conditions and not ...
Restoration of a Frontal Illuminated Face Image Based on KPCA
ICPR '10: Proceedings of the 2010 20th International Conference on Pattern Recognition

In this paper, we propose a novel illumination-normalization method. By using the combination of the Kernel Principal Component Analysis (KPCA) and Pre-image technology, this method can restore the frontal-illuminated face image from a single non-frontal-...
Three-Dimensional Occlusion Detection and Restoration of Partially Occluded Faces

This paper presents an innovative three dimensional occlusion detection and restoration strategy for the recognition of three dimensional faces partially occluded by unforeseen, extraneous objects. The detection method considers occlusions as local ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Multimedia Computing, Communications, and Applications

ACM Transactions on Multimedia Computing, Communications, and Applications Volume 19, Issue 1

January 2023

505 pages

ISSN:1551-6857

EISSN:1551-6865

DOI:10.1145/3572858

Editor:
Abdulmotaleb El Saddik
Mohamed Bin Zayed University of Artificial Intelligence, UAE and University of Ottawa, Canada

Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 06 January 2023

Online AM: 25 March 2022

Accepted: 06 March 2022

Revised: 19 November 2021

Received: 05 July 2021

Published in TOMM Volume 19, Issue 1

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
586
Total Downloads

Downloads (Last 12 months)108
Downloads (Last 6 weeks)7

Reflects downloads up to 02 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Yuan TZhang XLiu BLiu KJin JJiao Z(2025)Surveillance Video-and-Language Understanding: From Small to Large Multimodal ModelsIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2024.346243335:1(300-314)Online publication date: 1-Jan-2025
https://dl.acm.org/doi/10.1109/TCSVT.2024.3462433
Wu JFeng YXu HZhu CZheng JWooldridge MDy JNatarajan S(2024)SyFormerProceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence10.1609/aaai.v38i6.28417(6021-6029)Online publication date: 20-Feb-2024
https://dl.acm.org/doi/10.1609/aaai.v38i6.28417
Lin JWang Y(2024)TSFormer: Tracking Structure Transformer for Image InpaintingACM Transactions on Multimedia Computing, Communications, and Applications10.1145/369645220:12(1-23)Online publication date: 20-Sep-2024
https://dl.acm.org/doi/10.1145/3696452
Yue ZLoy C(2024)DifFace: Blind Face Restoration With Diffused Error ContractionIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2024.343265146:12(9991-10004)Online publication date: Dec-2024
https://doi.org/10.1109/TPAMI.2024.3432651
Chen XTan JWang TZhang KLuo WCao X(2024)Toward Real-World Blind Face Restoration With Generative Diffusion PriorIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2024.338365934:9(8494-8508)Online publication date: 1-Sep-2024
https://dl.acm.org/doi/10.1109/TCSVT.2024.3383659
Geetha RJebamalar GShiney SDao NMoon HCho S(2024)Enhancing Upscaled Image Resolution Using Hybrid Generative Adversarial Network-Enabled FrameworksIEEE Access10.1109/ACCESS.2024.336776312(27784-27793)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3367763
Jin YWu JWang WYan YJiang JZheng J(2023)Cascading Blend Network for Image InpaintingACM Transactions on Multimedia Computing, Communications, and Applications10.1145/360895220:1(1-21)Online publication date: 25-Aug-2023
https://dl.acm.org/doi/10.1145/3608952
Xiao YLi XZhang QLv RLi QWang R(2023)Spreading Mosaic: An Image Restoration-Inspired Social Rumor Propagation ModelIEEE Transactions on Multimedia10.1109/TMM.2023.330509526(2906-2917)Online publication date: 15-Aug-2023
https://dl.acm.org/doi/10.1109/TMM.2023.3305095

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View full text|Download PDF

View Issue’s Table of Contents