Abstract
Under the perspective projection assumption, non-Lambertian photometric stereo is a highly non-linear problem. In this study, we present an optimized framework for reconstructing the surface normal and depth with non-Lambertian reflection models under perspective projection. By decomposing the images into diffuse and specular components, we compute the surface normal and reflectance simultaneously. We also propose a variational formulation that is robust and useful for surface reconstruction. The experiments show that our method accurately reconstructs both the surface shape and reflectance of colorful objects with non-Lambertian surfaces.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Agrawal A, Raskar R, Nayar SK, et al., 2005. Removing photography artifacts using gradient projection and flash-exposure sampling. ACM Trans Graph, 24(3):828–835. https://doi.org/10.1145/1073204.1073269
Agrawal A, Raskar R, Chellappa R, 2006a. Edge suppression by gradient field transformation using cross-projection tensors. IEEE Computer Society Conf on Computer Vision and Pattern Recognition, p.2301–2308. https://doi.org/10.1109/CVPR.2006.106
Agrawal A, Raskar R, Chellappa R, 2006b. What is the range of surface reconstructions from a gradient field? 9th European Conf on Computer Vision, p.578–591. https://doi.org/10.1007/11744023_45
Barsky S, Petrou M, 2003. The 4-source photometric stereo technique for three-dimensional surfaces in the presence of highlights and shadows. IEEE Trans Patt Anal Mach Intell, 25(10):1239–1252. https://doi.org/10.1109/TPAMI.2003.1233898
Chandraker M, Agarwal S, Kriegman D, 2007. Shadowcuts: photometric stereo with shadows. IEEE Conf on Computer Vision and Pattern Recognition, p.1–8. https://doi.org/10.1109/CVPR.2007.383288
Chung HS, Jia JY, 2008. Efficient photometric stereo on glossy surfaces with wide specular lobes. IEEE Conf on Computer Vision and Pattern Recognition, p.1–8. https://doi.org/10.1109/CVPR.2008.4587771
Coleman ENJr, Jain R, 1982. Obtaining 3-dimensional shape of textured and specular surfaces using four-source photometry. Comput Graph Image Process, 18(4):309–328. https://doi.org/10.1016/0146-664X(82)90001-6
Donoho DL, 2006. Compressed sensing. IEEE Trans Inform Theory, 52(4):1289–1306. https://doi.org/10.1109/TIT.2006.871582
Esteban CH, Vogiatzis G, Cipolla R, 2008. Multiview photometric stereo. IEEE Trans Patt Anal Mach Intell, 30(3):548–554. https://doi.org/10.1109/TPAMI.2007.70820
Finlayson GD, Schaefer G, 2001a. Convex and non-convex illuminant constraints for dichromatic colour constancy. Proc IEEE Computer Society Conf on Computer Vision and Pattern Recognition, p.598–604 https://doi.org/10.1109/CVPR.2001.990528
Finlayson GD, Schaefer G, 2001b. Solving for colour constancy using a constrained dichromatic reflection model. Int J Comput Vis, 42(3):127–144. https://doi.org/10.1023/A:1011120214885
Gabay D, Mercier B, 1976. A dual algorithm for the solution of nonlinear variational problems via finite element approximation. Comput Math Appl, 2(1):17–40. https://doi.org/10.1016/0898-1221(76)90003-1
Galliani S, Lasinger K, Schindler K, 2015. Massively parallel multiview stereopsis by surface normal diffusion. Proc IEEE Int Conf on Computer Vision, p.873–881. https://doi.org/10.1109/ICCV.2015.106
Georghiades AS, 2003. Incorporating the Torrance and Sparrow model of reflectance in uncalibrated photometric stereo. Proc 9th IEEE Int Conf on Computer Vision, Article 816. https://doi.org/10.1109/ICCV.2003.1238432
Ghosh A, Chen TB, Peers P, et al., 2009. Estimating specular roughness and anisotropy from second order spherical gradient illumination. Comput Graph For, 28(4):1161–1170. https://doi.org/10.1111/j.1467-8659.2009.01493.x
Goldman DB, Curless B, Hertzmann A, et al., 2010. Shape and spatially-varying BRDFs from photometric stereo. IEEE Trans Patt Anal Mach Intell, 32(6):1060–1071. https://doi.org/10.1109/TPAMI.2009.102
Guo XJ, Cao XC, Ma Y, 2014. Robust separation of reflection from multiple images. Proc IEEE Conf on Computer Vision and Pattern Recognition, p.2187–2194. https://doi.org/10.1109/CVPR.2014.281
Han BJ, Sim JY, 2018. Glass reflection removal using co-saliency-based image alignment and low-rank matrix completion in gradient domain. IEEE Trans Image Process, 27(10):4873–4888. https://doi.org/10.1109/TIP.2018.2849880
Han TQ, Shen HL, 2015. Photometric stereo for general BRDFs via reflection sparsity modeling. IEEE Trans Image Process, 24(12):4888–4903. https://doi.org/10.1109/TIP.2015.2471081
He XT, Peng YX, 2019. Fine-grained visual-textual representation learning. IEEE Trans Circ Syst Video Technol, online. https://doi.org/10.1109/TCSVT.2019.2892802
He XT, Peng YX, Zhao JJ, 2019a. Fast fine-grained image classification via weakly supervised discriminative localization. IEEE Trans Circ Syst Video Technol, 29(5):1394–1407. https://doi.org/10.1109/TCSVT.2018.2834480
He XT, Peng YX, Zhao JJ, 2019b. Which and how many regions to gaze: focus discriminative regions for finegrained visual categorization. Int J Comput Vis, 127(9):1235–1255. https://doi.org/10.1007/s11263-019-01176-2
Huynh CP, Robles-Kelly A, 2010. A solution of the dichromatic model for multispectral photometric invariance. Int J Comput Vis, 90(1):1–27. https://doi.org/10.1007/s11263-010-0333-y
Ikehata S, 2018. CNN-PS: CNN-based photometric stereo for general non-convex surfaces. European Conf on Computer Vision, p.3–19. https://doi.org/10.1007/978-3-030-01267-0_1
Ikehata S, Aizawa K, 2014. Photometric stereo using constrained bivariate regression for general isotropic surfaces. Proc IEEE Conf on Computer Vision and Pattern Recognition, p.2179–2186. https://doi.org/10.1109/CVPR.2014.280
Ikehata S, Wipf D, Matsushita Y, et al., 2012. Robust photometric stereo using sparse regression. IEEE Conf on Computer Vision and Pattern Recognition, p.318–325. https://doi.org/10.1109/CVPR.2012.6247691
Kong N, Tai YW, Shin SY, 2012. A physically-based approach to reflection separation. IEEE Conf on Computer Vision and Pattern Recognition, p.9–16. https://doi.org/10.1109/CVPR.2012.6247652
Levin A, Weiss Y, 2007. User assisted separation of reflections from a single image using a sparsity prior. IEEE Trans Patt Anal Mach Intell, 29(9):1647–1654. https://doi.org/10.1109/TPAMI.2007.1106
Li Y, Brown MS, 2013. Exploiting reflection change for automatic reflection removal. Proc IEEE Int Conf on Computer Vision, p.2432–2439. https://doi.org/10.1109/ICCV.2013.302
Mallick SP, Zickler TE, Kriegman DJ, et al., 2005. Beyond Lambert: reconstructing specular surfaces using color. IEEE Computer Society Conf on Computer Vision and Pattern Recognition, p.619–626. https://doi.org/10.1109/CVPR.2005.88
Nayar SK, Fang XS, Boult T, 1997. Separation of reflection components using color and polarization. Int J Comput Vis, 21(3):163–186. https://doi.org/10.1023/A:1007937815113
Papadhimitri T, Favaro P, 2013. A new perspective on uncalibrated photometric stereo. Proc IEEE Conf on Computer Vision and Pattern Recognition, p.1474–1481. https://doi.org/10.1109/CVPR.2013.194
Peng YX, He XT, Zhao JJ, 2018. Object-part attention model for fine-grained image classification. IEEE Trans Image Process, 27(3):1487–1500. https://doi.org/10.1109/TIP.2017.2774041
Quéau Y, Wu T, Lauze F, et al., 2017. A non-convex variational approach to photometric stereo under inaccurate lighting. Proc IEEE Conf on Computer Vision and Pattern Recognition, p.350–359. https://doi.org/10.1109/CVPR.2017.45
Quéau Y, Durix B, Wu T, et al., 2018. LED-based photometric stereo: modeling, calibration and numerical solution. J Math Imag Vis, 60(3):313–340. https://doi.org/10.1007/s10851-017-0761-1
Santo H, Samejima M, Sugano Y, et al., 2017. Deep photometric stereo network. Proc IEEE Int Conf on Computer Vision, p.501–509. https://doi.org/10.1109/ICCVW.2017.66
Schechner YY, Kiryati N, Basri R, 2000. Separation of transparent layers using focus. Int J Comput Vis, 39(1):25–39. https://doi.org/10.1023/A:1008166017466
Shafer SA, 1985. Using color to separate reflection components. Color Res Appl, 10(4):210–218. https://doi.org/10.1002/col.5080100409
Shen HL, Han TQ, Li CG, 2017. Efficient photometric stereo using kernel regression. IEEE Trans Image Process, 26(1):439–451. https://doi.org/10.1109/TIP.2016.2627805
Shi BX, Tan P, Matsushita Y, et al., 2014. Bi-polynomial modeling of low-frequency reflectances. IEEE Trans Patt Anal Mach Intell, 36(6):1078–1091. https://doi.org/10.1109/TPAMI.2013.196
Shi BX, Mo ZP, Wu Z, et al., 2019. A benchmark dataset and evaluation for non-Lambertian and uncalibrated photometric stereo. IEEE Trans Patt Anal Mach Intell, 41(2):271–284. https://doi.org/10.1109/TPAMI.2018.2799222
Shih Y, Krishnan D, Durand F, et al., 2015. Reflection removal using ghosting cues. Proc IEEE Conf on Computer Vision and Pattern Recognition, p.3193–3201. https://doi.org/10.1109/CVPR.2015.7298939
Solomon F, Ikeuchi K, 1996. Extracting the shape and roughness of specular lobe objects using four light photometric stereo. IEEE Trans Patt Anal Mach Intell, 18(4):449–454. https://doi.org/10.1109/34.491627
Sun C, Liu SC, Yang TT, et al., 2016. Automatic reflection removal using gradient intensity and motion cues. Proc 24th ACM Int Conf on Multimedia, p.466–470. https://doi.org/10.1145/2964284.2967264
Sunkavalli K, Zickler T, Pfister H, 2010. Visibility subspaces: uncalibrated photometric stereo with shadows. 11th European Conf on Computer Vision, p.251–264. https://doi.org/10.1007/978-3-642-15552-9_19
Tan RT, Ikeuchi K, 2008. Separating reflection components of textured surfaces using a single image. IEEE Trans Patt Anal Mach Intell, 27(2):353–384. https://doi.org/10.1109/TPAMI.2005.36
Tankus A, Kiryati N, 2005. Photometric stereo under perspective projection. 10th IEEE Int Conf on Computer Vision, p.611–616. https://doi.org/10.1109/ICCV.2005.190
Wan RJ, Shi BX, Hwee TA, et al., 2016. Depth of field guided reflection removal. IEEE Int Conf on Image Processing, p.21–25. https://doi.org/10.1109/ICIP.2016.7532311
Wan RJ, Shi BX, Duan LY, et al., 2018. CRRN: multi-scale guided concurrent reflection removal network. Proc IEEE Conf on Computer Vision and Pattern Recognition, p.4777–4785.
Wang F, Ainouz S, Petitjean C, et al., 2017. Specularity removal: a global energy minimization approach based on polarization imaging. Comput Vis Image Underst, 158:31–39. https://doi.org/10.1016/j.cviu.2017.03.003
Woodham RJ, 1979. Photometric stereo: a reflectance map technique for determining surface orientation from image intensity. 22nd Annual Technical Symp on Image Understanding Systems and Industrial Applications I, p.136–143. https://doi.org/10.1117/12.956740
Wu L, Ganesh A, Shi B, et al., 2010. Robust photometric stereo via low-rank matrix completion and recovery. 10th Asian Conf on Computer Vision, p.703–717. https://doi.org/10.1007/978-3-642-19318-7_55
Yang J, Gong D, Liu LQ, et al., 2018. Seeing deeply and bidirectionally: a deep learning approach for single image reflection removal. Proc European Conf on Computer Vision, p.654–669. https://doi.org/10.1007/978-3-030-01219-9_40
Yang QX, Tang JH, Ahuja N, 2015. Efficient and robust specular highlight removal. IEEE Trans Patt Anal Mach Intell, 37(6):1304–1311. https://doi.org/10.1109/TPAMI.2014.2360402
Yeung SK, Wu TP, Tang CK, et al., 2015. Normal estimation of a transparent ob ject using a video. IEEE Trans Patt Anal Mach Intell, 37(4):890–897. https://doi.org/10.1109/TPAMI.2014.2346195
Yu C, Seo Y, Lee SW, 2010. Photometric stereo from maximum feasible Lambertian reflections. 11th European Conf on Computer Vision, p.115–126. https://doi.org/10.1007/978-3-642-15561-1_9
Zhang XE, Ng R, Chen QF, 2018. Single image reflection separation with perceptual losses. Proc IEEE Conf on Computer Vision and Pattern Recognition, p.4786–4794. https://doi.org/10.1109/CVPR.2018.00503
Zhou ZL, Wu Z, Tan P, 2013. Multi-view photometric stereo with spatially varying isotropic materials. IEEE Conf on Computer Vision and Pattern Recognition, p.1482–1489. https://doi.org/10.1109/CVPR.2013.195
Author information
Authors and Affiliations
Corresponding author
Additional information
Project supported by the Technological Program of Cultural Relics Preservation of Zhejiang Province, China, the Key Research and Development Program of Zhejiang Province, China (No. 2018C03051), and the National Standard Development Program of Cultural Relics Protection of China (No. 581250-T0170B)
Compliance with ethics guidelines
Min LI, Chang-yu DIAO, Duan-qing XU, Wei XING, and Dong-ming LU declare that they have no conflict of interest.
Rights and permissions
About this article
Cite this article
Li, M., Diao, Cy., Xu, Dq. et al. A non-Lambertian photometric stereo under perspective projection. Front Inform Technol Electron Eng 21, 1191–1205 (2020). https://doi.org/10.1631/FITEE.1900156
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1631/FITEE.1900156