Abstract
Recent deep learning algorithms for single-view 3D reconstruction recover rough 3D layout but fail to capture the crisp linear structures that grace our urban landscape. Here we show that for the particular problem of 3D Manhattan building reconstruction, the explicit application of linear perspective and Manhattan constraints within a classical constructive perceptual organization framework allows accurate and meaningful reconstructions to be computed. The proposed Line-Segment-to-3D (LS3D) algorithm computes a hierarchical representation through repeated application of the Gestalt principle of proximity. Edges are first organized into line segments, and the subset that conforms to a Manhattan frame is extracted. Optimal bipartite grouping of orthogonal line segments by proximity minimizes the total gap and generates a set of Manhattan spanning trees, each of which is then lifted to 3D. For each 3D Manhattan tree we identify the complete set of 3D 3-junctions and 3-paths, and show that each defines a unique minimal spanning cuboid. The cuboids generated by each Manhattan tree together define a solid model and the visible surface for that tree. The relative depths of these solid models are determined by an L1 minimization that is again rooted in a principle of proximity in both depth and image dimensions. The method has relatively fewer parameters and requires no training. For quantitative evaluation, we introduce a new 3D Manhattan building dataset (3DBM). We find that the proposed LS3D method generates 3D reconstructions that are both qualitatively and quantitatively superior to reconstructions produced by state-of-the-art deep learning approaches.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Coughlan, J.M., Yuille, A.L.: Manhattan world: orientation and outlier detection by Bayesian inference. Neural Comput. 15, 1063–1088 (2003)
Kubovy, M., Wagemans, J.: Grouping by proximity and multistability in dot lattices: a quantitative Gestalt theory. Psychol. Sci. 6, 225–234 (1995)
Kubovy, M., Holcombe, A.O., Wagemans, J.: On the lawfulness of grouping by proximity. Cogn. Psychol. 35, 71–98 (1998)
Elder, J.H., Goldberg, R.M.: Ecological statistics of Gestalt laws for the perceptual organization of contours. J. Vis. 2, 324–353 (2002)
Wagemans, J., et al.: A century of Gestalt psychology in visual perception: I. Perceptual grouping and figure-ground organization. Psychol. Bull. 138, 1172 (2012)
Gupta, A., Efros, A.A., Hebert, M.: Blocks world revisited: image understanding using qualitative geometry and mechanics. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6314, pp. 482–496. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15561-1_35
Roberts, L.G.: Machine perception of three-dimensional solids. Ph.D. thesis, Massachusetts Institute of Technology (1963)
Guzman, A.: Computer recognition of three-dimensional objects in a visual scene. Ph.D. thesis, MIT (1968)
Waltz, D.L.: Generating semantic descriptions from drawings of scenes with shadows. Technical Report AITR-271, MIT (1972)
Kanade, T.: A theory of Origami world. Artif. Intell. 13, 279–311 (1980)
Sugihara, K.: Machine Interpretation of Line Drawings, vol. 1. MIT Press, Cambridge (1986)
Hoiem, D., Efros, A.A., Hebert, M.: Recovering surface layout from an image. Int. J. Comput. Vis. 75, 151–172 (2007)
Barinova, O., Konushin, V., Yakubenko, A., Lee, K.C., Lim, H., Konushin, A.: Fast automatic single-view 3-D reconstruction of urban scenes. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008. LNCS, vol. 5303, pp. 100–113. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-88688-4_8
Haines, O., Calway, A.: Recognising planes in a single image. IEEE TPAMI 37, 1849–1861 (2015)
Coughlan, J.M., Yuille, A.L.: Manhattan world: compass direction from a single image by Bayesian inference. In: CVPR, vol. 2, pp. 941–947 (1999)
Denis, P., Elder, J.H., Estrada, F.J.: Efficient edge-based methods for estimating Manhattan frames in urban imagery. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008. LNCS, vol. 5303, pp. 197–210. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-88688-4_15
Tal, R., Elder, J.H.: An accurate method for line detection and Manhattan frame estimation. In: Park, J.-I., Kim, J. (eds.) ACCV 2012. LNCS, vol. 7729, pp. 580–593. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-37484-5_47
Delage, E., Lee, H., Ng, A.Y.: Automatic single-image 3D reconstructions of indoor Manhattan world scenes. In: Thrun, S., Brooks, R., Durrant-Whyte, H. (eds.) Robotics Research. STAR, vol. 28, pp. 305–321. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-48113-3_28
Hedau, V., Hoiem, D., Forsyth, D.: Recovering the spatial layout of cluttered rooms. In: ICCV, pp. 1849–1856 (2009)
Gupta, A., Hebert, M., Kanade, T., Blei, D.M.: Estimating spatial layout of rooms using volumetric reasoning about objects and surfaces. In: Lafferty, J.D., Williams, C.K.I., Shawe-Taylor, J., Zemel, R.S., Culotta, A. (eds.) NIPS. Curran Associates, Inc. (2010)
Ramalingam, S., Pillai, J.K., Jain, A., Taguchi, Y.: Manhattan junction catalogue for spatial reasoning of indoor scenes. In: CVPR 2013, pp. 3065–3072 (2013)
Mallya, A., Lazebnik, S.: Learning informative edge maps for indoor scene layout prediction. In: ICCV, pp. 936–944 (2015)
Pero, L.D., Bowdish, J., Fried, D., Kermgard, B., Hartley, E., Barnard, K.: Bayesian geometric modeling of indoor scenes. In: CVPR, pp. 2719–2726 (2012)
Felzenszwalb, P.F., Veksler, O.: Tiered scene labeling with dynamic programming. In: CVPR, pp. 3097–3104 (2010)
Schwing, A.G., Urtasun, R.: Efficient exact inference for 3D indoor scene understanding. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7577, pp. 299–313. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33783-3_22
Yang, H., Zhang, H.: Efficient 3D room shape recovery from a single panorama. In: CVPR, pp. 5422–5430 (2016)
Dasgupta, S., Fang, K., Chen, K., Savarese, S.: Delay: robust spatial layout estimation for cluttered indoor scenes. In: CVPR, pp. 616–624 (2016)
Ramalingam, S., Brand, M.: Lifting 3D Manhattan lines from a single image. In: ICCV, pp. 497–504 (2013)
Kushal, A., Seitz, S.M.: Single view reconstruction of piecewise swept surfaces. In: 3DV, pp. 239–246 (2013)
Saxena, A., Sun, M., Ng, A.Y.: Make3D: learning 3D scene structure from a single still image. IEEE TPAMI 31, 824–840 (2009)
Eigen, D., Fergus, R.: Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture. In: CVPR, pp. 2650–2658 (2015)
Liu, F., Shen, C., Lin, G., Reid, I.: Learning depth from single monocular images using deep convolutional neural fields. IEEE TPAMI 38, 2024–2039 (2016)
Laina, I., Rupprecht, C., Belagiannis, V., Tombari, F., Navab, N.: Deeper depth prediction with fully convolutional residual networks. In: 3DV, pp. 239–248 (2016)
Liu, F., Shen, C., Lin, G.: Deep convolutional neural fields for depth estimation from a single image. In: CVPR, pp. 5162–5170 (2015)
Zhuo, W., Salzmann, M., He, X., Liu, M.: 3D box proposals from a single monocular image of an indoor scene. In: AAAI (2018)
Fu, H., Gong, M., Wang, C., Batmanghelich, K., Tao, D.: Deep ordinal regression network for monocular depth estimation. In: CVPR (2018)
Xu, D., Ouyang, W., Wang, X., Sebe, N.: PAD-Net: multi-tasks guided prediction-and-distillation network for simultaneous depth estimation and scene parsing. In: CVPR (2018)
Qi, X., Liao, R., Liu, Z., Urtasun, R., Jia, J.: GeoNet: geometric neural network for joint depth and surface normal estimation. In: CVPR (2018)
Li, Z., Snavely, N.: MegaDepth: learning single-view depth prediction from internet photos. In: CVPR (2018)
Geiger, A., Lenz, P., Stiller, C., Urtasun, R.: Vision meets robotics: the KITTI dataset. Int. J. Rob. Res. 32, 1231–1237 (2013)
Garg, R., B.G., V.K., Carneiro, G., Reid, I.: Unsupervised CNN for single view depth estimation: geometry to the rescue. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9912, pp. 740–756. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46484-8_45
Zhou, T., Brown, M., Snavely, N., Lowe, D.G.: Unsupervised learning of depth and ego-motion from video. In: CVPR (2017)
Izadinia, H., Shan, Q., Seitz, S.M.: IM2CAD. In: CVPR, pp. 2422–2431. IEEE (2017)
Almazan, E.J., Tal, R., Qian, Y., Elder, J.H.: MCMLSD: a dynamic programming approach to line segment detection. In: CVPR (2017)
Lee, D., Hebert, M., Kanade, T.: Geometric reasoning for single image structure recovery. In: CVPR, pp. 2136–2143. IEEE (2009)
Munkres, J.: Algorithms for the assignment and transportation problems. J. Soc. Ind. Appl. Math. 5, 32–38 (1957)
Cignoni, P., Callieri, M., Corsini, M., Dellepiane, M., Ganovelli, F., Ranzuglia, G.: MeshLab: an open-source mesh processing tool. In: Eurographics Italian Chapter Conference (2008)
Eigen, D., Puhrsch, C., Fergus, R.: Depth map prediction from a single image using a multi-scale deep network. In: NIPS, pp. 2366–2374 (2014)
Liu, C., Yang, J., Ceylan, D., Yumer, E., Furukawa, Y.: PlaneNet: piece-wise planar reconstruction from a single RGB image. In: CVPR, pp. 2579–2588 (2018)
Acknowledgements
This research was supported by the NSERC Discovery program and the NSERC CREATE Training Program in Data Analytics & Visualization, the Ontario Research Fund, and the York University VISTA and Research Chair programs.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Qian, Y., Ramalingam, S., Elder, J.H. (2019). LS3D: Single-View Gestalt 3D Surface Reconstruction from Manhattan Line Segments. In: Jawahar, C., Li, H., Mori, G., Schindler, K. (eds) Computer Vision – ACCV 2018. ACCV 2018. Lecture Notes in Computer Science(), vol 11364. Springer, Cham. https://doi.org/10.1007/978-3-030-20870-7_25
Download citation
DOI: https://doi.org/10.1007/978-3-030-20870-7_25
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-20869-1
Online ISBN: 978-3-030-20870-7
eBook Packages: Computer ScienceComputer Science (R0)