Skip to main content
SpringerLink
Log in

Selfie retoucher: subject-oriented self-portrait enhancement

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Sharing self-portraits starts trending nowadays with the boom of social networks and the rise of smartphones. However, limited by the hardware capabilities, self-portraits taken by the front cameras of portable media devices usually face quality problems such as an incomplete field of view and poor lighting style. In our paper, we introduce a selfie retoucher which enhances a self-portrait with the help of N supporting photos that share the same scene and similar shooting time. With the extra information brought by the supporting photos, a lager field of view and a better lighting style can be achieved. To accomplish this, we propose a novel subject-oriented self-portrait enhancement method with a cascaded illumination unification and photos registration framework. Based on the correspondences extracted from the input 1+N photos, our method estimates and updates the illumination and registration coefficients in a cascaded manner. Moreover, a subject-oriented enhancement algorithm is proposed to enhance the face of the photographer in the self-portrait. We adopt a face-specific illumination correction process over the self-portrait to further improve the visual quality of the subject. After the enhancement, we globally fuse the aligned photos by a Markov Random Field based optimization method. During the fusion, a body map is additionally derived from the subject for guidance. Experimental results demonstrate that the proposed method achieves high-quality results in this novel application scenario.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

References

  1. Adobe Photoshop PhotoMerge. http://helpx.adobe.com/en/photoshop/using/create-panoramic-images-photomerge.html http://helpx.adobe.com/en/photoshop/using/create-panoramic-images-photomerge.html

  2. Barnes C, Shechtman E, Finkelstein A, Goldman DB (2009) Patchmatch: a randomized correspondence algorithm for structural image editing. ACM Trans Graph 28(3):341–352

    Article  Google Scholar 

  3. Bernd H, Thomas S, Poggio T (2007) A component-based framework for face detection and identification. Int J Comput Vis 74(2):167–181

    Article  Google Scholar 

  4. Chen Y, Chuang Y (2016) Natural image stitching with the global similarity prior. In: Proc. European conf. computer vision

  5. Ebner M (2006) Evolving color constancy. Pattern Recogn Lett 27:1220–1229

    Article  Google Scholar 

  6. Finlayson G, Hordley S, HubeL P (2001) Color by correlation: a simple, unifying framework for color constancy. IEEE Trans Pattern Anal Mach Intell 23:1209–1221

    Article  Google Scholar 

  7. Finlayson GD, Hordley SD, Tastl I (2003) Gamut constrained illuminant estimation. In: Proc. Int’l Conf. Computer Vision

  8. Fischler MA, Bolles RC (1981) Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun ACM 24(6):381–395

    Article  MathSciNet  Google Scholar 

  9. Gijsenij A, Gevers T (2011) Color constancy using natural image statistics and scene semantics. IEEE Trans Pattern Anal Mach Intell 33(4):687–698

    Article  Google Scholar 

  10. Hacohen Y, Shechtman E, Goldman DB, Lischinski D (2013) Optimizing color consistency in photo collections. ACM Trans Graph 32(4):96–96

    Article  Google Scholar 

  11. Laffont PY, Bousseau A, Paris S, Durand F, Drettakis G (2012) Coherent intrinsic images from photo collections. ACM Transactions on Graphics 31

  12. Li M, Liu J, Ren J, Guo Z (2015) Adaptive general scale interpolation based on weighted autoregressive models. IEEE Trans Circuits Syst Video Technol 25 (2):200–211

    Article  Google Scholar 

  13. Li M, Liu J, Yang W, Sun X, Guo Z (2018) Structure-revealing low-light image enhancement via robust Retinex model. IEEE Trans Image Process 27 (6):2828–2841

    Article  MathSciNet  MATH  Google Scholar 

  14. Lin K, Jiang N, Cheong L, Do M, Lu J (2016) Seagull: Seam-guided local alignment for parallax-tolerant image stitching. In: Proc. European conf. computer vision

  15. Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vis 60(2):91–110

    Article  Google Scholar 

  16. Osadchy M, Le Cun Y, Miller ML (2007) Synergistic face detection and pose estimation with energy-based models. The Journal of Machine Learning Research

  17. Nie L, Wang M, Zha Z, Chua T (2012) Oracle in image search: A content-based approach to performance prediction. ACM Trans Inf Syst 30(2):1–23

    Article  Google Scholar 

  18. Nomura Y, Zhang L, Nayar SK (2007) Scene collages and flexible camera arrays. In: Proc. Eurographics conf. rendering techniques, pp 127–138

  19. Park J, Tai YW, Sinha NS, Kweon IS (2016) Efficient and robust color consistency for community photo collections. In: Proc. IEEE int’l conf. computer vision and pattern recognition

  20. Perez P, Gangnet M, Blake A (2003) Poisson image editing. ACM Trans Graph 22:313–318

    Article  Google Scholar 

  21. Kim SJ, Pollefeys M (2008) Robust radiometric calibration and vignetting correction. IEEE Trans Pattern Anal Mach Intell 30:562–576

    Article  Google Scholar 

  22. Shan Q, Curless B, Furukawa Y, Hernandez C, Seitz SM (2014) Photo uncrop. In: Proc. European conf. computer vision, pp 16–31

  23. Sivic J, Kaneva B, Torralba A, Avidan S, Freeman WT (2008) Creating and exploring a large photorealistic virtual space. In: Proc. IEEE int’l conf. computer vision and pattern recognition workshops, pp 1–8

  24. Song S, Zeng A, Chang A, Savva M, Savarese S, Funkhouser T (2018) Im2pano3d: Extrapolating 360 structure and semantics beyond the field of view. In: Proc. IEEE int’l conf. computer vision and pattern recognition

  25. Valle R, Buenaposada J, Valdes A, Baumela L (2018) A deeply-initialized coarse-to-fine ensemble of regression trees for face alignment. In: Proc. European conf. computer vision

  26. Viola P, Jones MJ (2004) Robust real-time face detection. Int’l Journal of Computer Vision

  27. Wang M, Lai Y, Liang Y, Martin RR, Hu S (2014) Biggerpicture: data-driven image extrapolation using graph matching. ACM Trans Graph 33(6):1–13

    Google Scholar 

  28. Wexler Y, Shechtman E, Irani M (2007) Space-time completion of video. IEEE Trans Pattern Anal Mach Intell 29(3):463–476

    Article  Google Scholar 

  29. Xiong X, la Torre FD (2013) Supervised descent method and its applications to face alignment. In: Proc. IEEE int’l conf. computer vision and pattern recognition

  30. Yang S, Liu J, Xia S, Guo Z (2017) 1+n fusion: Cascaded self-portrait enhancement. In: Proc. IEEE int’l conf. acoustics, speech, and signal processing

  31. Zhang Y, Xiao J, Hays J, Tan P (2013) Framebreak: Dramatic image extrapolation by guided shift-maps. In: Proc. IEEE int’l conf. computer vision and pattern recognition

  32. Zhang W, Li G, Ying Z (2017) A new underwater image enhancing method via color correction and illumination adjustment. In: Proc. IEEE visual communication and image processing

  33. Zhou Q, Park J, Koltun V (2016) Fast global registration. In: Proc. European conf. computer vision

  34. Zhu X, Ramanan D (2012) Face detection, pose estimation, and landmark localization in the wild. In: Proc. IEEE int’l conf. computer vision and pattern recognition

Download references

Acknowledgements

This work was supported in part by National Natural Science Foundation of China under contract No. 61772043 and in part by Beijing Natural Science Foundation under contract No. L182002 and No. 4192025.

Author information

Authors and Affiliations

  1. Institute of Computer Science and Technology, Peking University, Beijing, China

    Sifeng Xia, Shuai Yang & Jiaying Liu

Authors
  1. Sifeng Xia
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shuai Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jiaying Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jiaying Liu.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xia, S., Yang, S. & Liu, J. Selfie retoucher: subject-oriented self-portrait enhancement. Multimed Tools Appl 78, 27591–27609 (2019). https://doi.org/10.1007/s11042-019-07873-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-019-07873-x

Keywords

Search

Navigation