Skip to main content
Log in

Variational formulation of a hybrid perspective shape from shading model

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Over a long period of time, it is challenging to solve the completed shape from shading (SFS) problem which has no limitations on light position and surface roughness. Many existing researches developed Hamilton–Jacobi equation (HJE)-based method of such SFS problem. However, HJE-based methods have one major weakness of boundary condition requirement. Finding a proper algorithm for calculating direct depth in a completed SFS model is desirable. In this paper, a completed hybrid perspective shape from shading model based on Cook–Torrance BRDF reflectance model is established. The depth is expressed and calculated directly on the Cartesian coordinates rather than spherical coordinates, which prevents exponential explosion risks caused by substitution of natural exponential. A direct variational formulation method is employed to solve the complex partial differential equation (PDE) of the image irradiance model. The major contribution of the method is that depth is considered as the only variable to deduce iterative equation. After discretizing the partial derivative terms of the iterative equation, the variational method could be programed conveniently and intuitively. Simulations and experiments demonstrate the accuracy and robustness of the method.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Malti, A., Bartoli, A.: Combining conformal deformation and Cook–Torrance shading for 3-d reconstruction in laparoscopy. IEEE Trans. Biomed. Eng 61(6), 1684–1692 (2014)

    Article  Google Scholar 

  2. Lei, L., Li, J., Liu, M., Hu, X., Zhou, Y., Yang, S.: Shape from shading and optical flow used for 3-dimensional reconstruction of endoscope image. Acta Oto Laryngol. 136(11), 1190–1192 (2016)

    Article  Google Scholar 

  3. Wang, C., Wang, C., Qin, H., Zhang, T.-Y.: Video-based fluid reconstruction and its coupling with SPH simulation. Vis. Comput. 33(9), 1211–1224 (2017)

    Article  Google Scholar 

  4. Nie, X., Hu, Y., Shen, X.: Physics-preserving fluid reconstruction from monocular video coupling with SFS and SPH. Vis. Comput. 36(6), 1247–1257 (2020)

    Article  Google Scholar 

  5. Maurer, D., Ju, Y.C., Breuß, M., Bruhn, A.: Combining shape from shading and stereo: a joint variational method for estimating depth, illumination and albedo. Int. J. Comput. Vis. 126(12), 1342–1366 (2018)

    Article  Google Scholar 

  6. Horn, B.K.: Shape from shading: A method for obtaining the shape of a smooth opaque object from one view. PhD thesis, Massachusetts Institute of Technology (1970)

  7. Ikeuchi, K., Horn, B.K.: Numerical shape from shading and occluding boundaries. Artif. intell. 17(1–3), 141–184 (1981)

    Article  Google Scholar 

  8. Horn, B.K., Brooks, M.J.: The variational approach to shape from shading. Comput. Vis. Graph. Image Process. 33(2), 174–208 (1986)

    Article  Google Scholar 

  9. Zhang, R., Tsai, P.-S., Cryer, J.E., Shah, M.: Shape-from-shading: a survey. IEEE Trans. Pattern Anal. Mach. Intell. 21(8), 690–706 (1999)

    Article  Google Scholar 

  10. Lee, K.M., Kuo, C.-C.J.: Shape from shading with a generalized reflectance map model. Comput. Vis. Image Underst. 67(2), 143–160 (1997)

    Article  Google Scholar 

  11. Wu, C., Narasimhan, S.G., Jaramaz, B.: A multi-image shape-from-shading framework for near-lighting perspective endoscopes. Int. J. Comput. Vis. 86(2–3), 211–228 (2010)

    Article  MathSciNet  Google Scholar 

  12. Wang, G., Cheng, J.: Three-dimensional reconstruction of hybrid surfaces using perspective shape from shading. Optik 127(19), 7740–7751 (2016)

    Article  Google Scholar 

  13. Ahmed, A.H., Farag, A.A.: Shape from shading under various imaging conditions. In: 2007 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8. IEEE (2007)

  14. Ahmed, A., Farag, A.: Shape from shading for hybrid surfaces. In: 2007 IEEE International Conference on Image Processing, vol. 2, pp. II–525, IEEE (2007)

  15. Galliani, S., Ju, Y.C., Breuß, M., Bruhn, A.: Generalised perspective shape from shading in spherical coordinates. In: International Conference on Scale Space and Variational Methods in Computer Vision, pp. 222–233, Springer (2013)

  16. Prados, E., Faugeras, O.: Shape from shading: a well-posed problem? In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), vol. 2, pp. 870–877. IEEE (2005)

  17. Ju, Y.C., Bruhn, A., Breuß, M.: Variational perspective shape from shading. In: International Conference on Scale Space and Variational Methods in Computer Vision, pp. 538–550. Springer (2015)

  18. Zhang, Z.: A flexible new technique for camera calibration. IEEE Trans. Pattern Anal. Machi. Intell. 22(11), 1330–1334 (2000)

    Article  Google Scholar 

  19. Schlick, C.: An inexpensive BRDF model for physically-based rendering. In: Computer Graphics Forum, vol. 13, pp. 233–246, Wiley Online Library (1994)

  20. Kelemen, C., Szirmay-Kalos, L.: A microfacet based coupled specular-matte BRDF model with importance sampling. In: Eurographics Short Presentations, vol. 2, p. 4 (2001)

  21. Horn, B.K.P., Brooks, M.J.: Shape from Shading. MIT Press, Cambridge (1989)

    MATH  Google Scholar 

  22. Durou, J.-D., Falcone, M., Sagona, M.: Numerical methods for shape-from-shading: a new survey with benchmarks. Comput. Vis. Image Underst. 109(1), 22–43 (2008)

    Article  Google Scholar 

  23. Wang, G., Han, J., Jia, H., Zhang, X.: Fast viscosity solutions for shape from shading under a more realistic imaging model. Opt. Eng. 48(11), 117201 (2009)

    Article  Google Scholar 

  24. Dacorogna, B.: Direct Methods in the Calculus of Variations, vol. 78. Springer, Berlin (2007)

    MATH  Google Scholar 

  25. Prasath, V.S., Figueiredo, I.N., Figueiredo, P.N., Palaniappan, K.: Mucosal region detection and 3d reconstruction in wireless capsule endoscopy videos using active contours. In: 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 4014–4017. IEEE (2012)

  26. Weyrich, T., Matusik, W., Pfister, H., Bickel, B., Donner, C., Tu, C., Mcandless, J., Lee, J., Ngan, A., Jensen, H.: Analysis of human faces using a measurement-based skin reflectance model. ACM Trans. Graph. 25(3), 1013–1024 (2006)

    Article  Google Scholar 

  27. Matusik, W.: A data-driven reflectance model. PhD thesis, Massachusetts Institute of Technology (2003)

  28. Qin, C., Shi, G., Tao, J., Yu, H., Jin, Y., Lei, J., Liu, C.: Precise cutterhead torque prediction for shield tunneling machines using a novel hybrid deep neural network. Mech. Syst. Signal Process. 151, 107386 (2021)

    Article  Google Scholar 

  29. Ron, G., Peleg, S.: Multiresolution shape from shading. In: 1989 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 350–351 (1989)

  30. Jin, Y., Qin, C., Huang, Y., Liu, C.: Actual bearing compound fault diagnosis based on active learning and decoupling attentional residual network. Measurement, p. 108500 (2020)

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Qinghua Liang.

Ethics declarations

Conflict of interest

The authors declare no conflicts of interest.

Funding

This study was funded by National Key R&D Program of China (No. 2017YFB1302901).

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendices

.

A.1 Convergence

Equation (22) is a forward iteration method. We discuss the convergence of this equation theoretically in the following part. Usually, iteration equation is written in the form of \(x = \varphi (x)\) with one fixed point \(x^ * \). Local convergence of iterative method in the neighborhood of the fixed point \(x^ * \) is equivalent to \(\left| {\varphi '({x^ * })} \right| < 1\). In our model, we first write iteration equation (22) in traditional form:

$$\begin{aligned} Z= & {} \varphi (Z) = {\tilde{Z}} \nonumber \\&+\frac{{{\Delta ^4}}}{{20\lambda }}\{ (I - R){R_Z} - \frac{\partial }{{\partial x}}[(I - R){R_p}] - \frac{\partial }{{\partial y}}[(I - R){R_q}]\}\nonumber \\ \end{aligned}$$
(A.1)

where \({\tilde{Z}}\) is the average value of Z in the 5*5 neighborhood domain according to equation (23). We take \({\tilde{Z}}\) to be approximately equal to Z here.

The fixed point of equation (A.1) is \(Z^ *\), which represents a certain true depth value \(Z=Z^*\). Fixed point \(Z^*\) has following equation when the illumination parameters are correct. The value of function R on point \(Z^*\) is denoted as \(R|_{Z^*}\).

$$\begin{aligned} I - R|_{Z^*} = 0. \end{aligned}$$
(A.2)

Based on this characteristic, we are able to make simplifications of \(\varphi '({Z^ * })\). Each terms in \(\varphi (Z)\) were taken derivatives one by one on fixed point \(Z^*\). We denote the value of a function on point \(Z^*\) same as the above equation, for example, the value of function \([(I-R)R_Z]'\) on point \(Z^*\) is denoted as \([(I-R)R_Z]'|_{Z^*}\).

$$\begin{aligned} \begin{array}{l} [(I - R){R_Z}]'|_{Z^*} \\ \qquad = [(I - R)'{R_Z} + (I - R){R_Z}^\prime ]|_{Z^*} = - {({R_Z})^2}|_{Z^*} \\ \{ \frac{\partial }{{\partial x}}[(I - R){R_p}]\} '|_{Z^*} \\ \qquad =[\frac{\partial }{{\partial x}}(I{R_{pZ}} - {R_Z}{R_p} - R{R_{pZ}})]|_{Z^*} = [\frac{\partial }{{\partial x}}( - {R_Z}{R_p})]|_{Z^*} \\ \{ \frac{\partial }{{\partial y}}[(I - R){R_q}]\} '|_{Z^*} =\\ \qquad =[\frac{\partial }{{\partial y}}(I{R_{qZ}} - {R_Z}{R_q} - R{R_{qZ}})]|_{Z^*} = [\frac{\partial }{{\partial y}}( - {R_Z}{R_q})]|_{Z^*} \end{array} \end{aligned}$$
(A.3)

The final \(\varphi '({Z^ * })\) is:

$$\begin{aligned} \begin{array}{l} \varphi '(Z^ *) = 1 + \frac{{{\Delta ^4}}}{{20\lambda }}D|_{Z^*} \\ D|_{Z^*}=[ - {({R_Z})^2} + \frac{\partial }{{\partial x}}({R_Z}{R_p}) + \frac{\partial }{{\partial y}}({R_Z}{R_q})]|_{Z^*}. \end{array} \end{aligned}$$
(A.4)

It can be observed from the equation that the local convergence relies on the sign of \(D|_{Z^*}\). If \(D|_{Z^*}<0\), then there exists \(\lambda >0\) who can make \(\left| {\varphi '(Z^ *)} \right| < 1\). Although the first term \(-{({R_Z})^2}|_{Z^*}\) is less than 0 without any doubt, the last two terms are uncertain. The analytical formula of \(D|_{Z^*}\) is too complicated to derive a universal condition equation of convergence. Besides, the value of \(Z^*\), \(\{a,b,c,x,y,p,q,k,m,F_0,I_0,\rho \}\) parameters may affect the value of \(D|_{Z^*}\). In view of this, we numerically evaluate the convergence. It is achieved by substituting parameters uniformly chosen from the parameter space \(\{a,b,c,x,y,Z^*,p,q,k,m,F_0,I_0,\rho \}\) into \(D|_{Z^*}\). The results of a large amount of data suggest that three parameters km\(\mathbf {l} \cdot \mathbf {n}\) are significant, where \(\mathbf {l} \cdot \mathbf {n}\) actually represents comprehensive combination of parameter \(\{a,b,c,x,y,Z^*,p,q\}\). Therefore, \(D|_{Z^*}\approx D(k,m,\mathbf {l} \cdot \mathbf {n})\).

Based on the sign of \(D(k,m,\mathbf {l} \cdot \mathbf {n})\), we mark convergence and non-convergence points on coordinate system \(O-k-m-\mathbf {l} \cdot \mathbf {n}\). Those points on the boundary between convergence and non-convergence are used to fit boundary surface. The surface fitted by polynomial is shown in Fig. 11. When parameters \(k,m,\mathbf {l} \cdot \mathbf {n}\) are chosen above the surface, iteration equation will converge. Boundary surface function is:

$$\begin{aligned} \begin{array}{l} \mathbf {l} \cdot \mathbf {n} = \varepsilon _{00} + \varepsilon _{10}k + \varepsilon _{01}m + \varepsilon _{20}k^2 + \varepsilon _{11}km + \varepsilon _{02}m^2 \\ \quad \quad + \varepsilon _{30}k^3 + \varepsilon _{21}k^2m + \varepsilon _{12}km^2 + \varepsilon _{03}m^3 \\ \quad \quad + \varepsilon _{40}k^4 + \varepsilon _{31}k^3m + \varepsilon _{22}k^2m^2 + \varepsilon _{13}km^3 \\ \quad \quad + \varepsilon _{50}k^5 + \varepsilon _{41}k^4m + \varepsilon _{32}k^3m^2 + \varepsilon _{23}k^2m^3\\ \varepsilon _{00}=-0.501,\varepsilon _{10}=2.865,\varepsilon _{01}=5.653, \\ \varepsilon _{20}=-5.952,\varepsilon _{11}=-22.83,\varepsilon _{02}=-14.52, \\ \varepsilon _{30}=8.520,\varepsilon _{21}=20.71,\varepsilon _{12}=61.63,\varepsilon _{03}=10.65, \\ \varepsilon _{40}=-8.698,\varepsilon _{31}=3.078,\varepsilon _{22}=-61.89,\varepsilon _{13}=-44.04,\\ \varepsilon _{50}=3.690,\varepsilon _{41}=-5.135,\varepsilon _{32}=10.33,\varepsilon _{23}=37.00.\\ \end{array} \end{aligned}$$
(A.5)
Fig. 11
figure 11

Boundary surface of convergence area and non-convergence area

Given k and m, if the value of \(\mathbf {l} \cdot \mathbf {n}\) is bigger than the value calculated in (A.5), equation (22) is local convergent. For example, \(m>0.7\) and \(\mathbf {l} \cdot \mathbf {n}>0\) can make the equation converge, \(k=0.5,m=0.5\) and \(\mathbf {l} \cdot \mathbf {n}>0.0961\) can make the equation converge, \(k=0.3,\) \(m=0.3\) and \(\mathbf {l} \cdot \mathbf {n}>\) 0.1402 can make the equation converge. It can be seen that our method converges on large areas. Non-convergent circumstances are mainly when \(\mathbf {l}\) and \(\mathbf {n}\) are almost perpendicular to each other.

A.2 Parameters of the real image experiments

Figure 9a. Cook–Torrance illumination parameters are that: \({I_0} = 3.45,\rho = 0.8,\) \(k=0.4,m=0.5,F_0=0.7\). Blinn–Phong illumination parameters are: \({I_0} = 3.6,kd=0.3,ks=0.7,\alpha =4\). Lambertian illumination parameters are \({I_0}\rho =3.08\). Figure 9b. Cook–Torrance illumination parameters are that: \({I_0} = 3.45,\rho = 0.8,\) \(k=0.7,m=0.7,F_0=0.9\). Blinn–Phong illumination parameters are: \({I_0} = 3.6,kd=0.7,ks=0.3,\alpha =4\). Lambertian illumination parameters are \({I_0}\rho =2.64\).

Figure 9c. Cook–Torrance illumination parameters are that: \({I_0} = 3.45,\rho = 0.8,k=0.2,m=0.6,F_0=0.8\). Blinn–Phong illumination parameters are: \({I_0} = 3.6,kd=0.2,ks=0.8,\alpha =3\). Lambertian illumination parameters are \({I_0}\rho =2.50\).

Figure 9d. Cook–Torrance illumination parameters are that: \({I_0} = 3.45,\rho = 0.8,k=0.7,m=0.5,F_0=0.8\). Blinn–Phong illumination parameters are: \({I_0} = 3.6,kd=0.6,ks=0.4,\alpha =2\). Lambertian illumination parameters are \({I_0}\rho =2.38\).

Light source position of Fig. (a), (b) is \(L=(19.093,-15.568, 23.885)\). Light source position of Fig.(c), (d) is \(L=(-1.151,10.431,21.885)\).

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Fan, J., Chen, M., Mo, J. et al. Variational formulation of a hybrid perspective shape from shading model. Vis Comput 38, 1469–1482 (2022). https://doi.org/10.1007/s00371-021-02081-x

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-021-02081-x

Keywords

Navigation