Skip to main content
SpringerLink
Log in

Detecting and inferring repetitive elements with accurate locations and shapes from façades

  • Original Article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

The use of repetition detection is an effective approach for increasing the efficiency of urban modeling. In practice, repetition detection can benefit from the apparent regularities and strong contextual relationships in façades. In view of this, we propose a novel algorithm for automatically detecting and inferring repetitive elements with accurate locations and shapes from façades. More specifically, firstly, starting from a rectification of the input façade, we employ the color clustering method to automatically derive candidate templates. Secondly, to detect the non- and partially occluded repetitive elements matching with the derived templates, we construct an adaptive region descriptor and a repetitive characteristic curve. Finally, the fully occluded elements are inferred by utilizing the Bayesian probability network, which can be learned from a database of the selected façades. The accuracy of our detection and inference is tested through a variety of experiments, and all of them justify the robustness of our algorithm to outliers such as appearance variations and occlusions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Müller, P., Zeng, G., Wonka, P., Van Gool, L.: Image-based procedural modeling of façades. ACM Trans. Gr. 26(3), 85 (2007)

    Article  Google Scholar 

  2. Wu, C., Frahm, J., Pollefeys, M.: Repetition-based dense single-view reconstruction. Rodng 42(7), 3113–3120 (2011)

    Google Scholar 

  3. Hou, F., Qin, H., Qi, Y.: Procedure-based component and architecture modeling from a single image. Vis. Comput. 32(2), 151–166 (2016)

    Article  Google Scholar 

  4. Ji, L., Przemyslaw, M., Peter, W., Jieping, Y.: Tensor completion for estimating missing values in visual data. IEEE Trans. Pattern Anal. Mach. Intell. 35(1), 208–220 (2013)

    Article  Google Scholar 

  5. Musialski, P., Wonka, P., Recheis, M., Maierhofer, S., Purgathofer, W.: Symmetry-based façade repair. In: Vision Modeling and Visualization, Workshop, pp. 3–10 (2009)

  6. Fuzhang, W., Yan, D.M., Dong, W., Zhang, X., Wonka, P.: Inverse procedural modeling of façade layouts. ACM Trans. Gr. 33(4), 70–79 (2014)

    Google Scholar 

  7. Zhang, H., Kai, X., Jiang, W., Lin, J., Cohen-Or, D., Chen, B.: Layered analysis of irregular façades via symmetry maximization. ACM Trans. Gr. 32(4), 96–96 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  8. Yunliang, C., George, B.: Detecting, grouping, and structure inference for invariant repetitive patterns in images. IEEE Trans. Image Process. 22(6), 2343–2355 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  9. Minwoo, P., Kyle, B., Collins, R.T., Yanxi, L.: Deformed lattice detection in real-world images using mean-shift belief propagation. IEEE Trans. Pattern Anal. Mach. Intell. 31(10), 1804–1816 (2009)

    Article  Google Scholar 

  10. Shen, C., Huang, S., Hu, S.: Adaptive partitioning of urban façades. ACM Trans. Gr. 30(6), 61–64 (2012)

    Google Scholar 

  11. Teboul, O., Kokkinos, I., Simon, L., Koutsourakis, P.,  Paragios, N.: Shape grammar parsing via reinforcement learning. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2273–2280 (2011)

  12. Changchang, W., Frahm, J.M., Pollefeys, M.: Detecting large repetitive structures with salient boundaries. Lect. Notes Comput. Sci. 6312, 142–155 (2010)

    Article  Google Scholar 

  13. Zhang, H., Yang, L., Zhao, P., Quan, L.: Per-pixel translational symmetry detection, optimization, and segmentation. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 526–533 (2012)

  14. Zhao, P., Quan, L.: Translation symmetry detection in a fronto-parallel view. In: Proceedings/CVPR, IEEE Computer Society Conference on Computer Vision and Pattern Recognition. IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 1009–1016 (2011)

  15. Furukawa, Y., Curless, B., Seitz, S.M., Szeliski, R.: Manhattan-world stereo. In: IEEE Conference on Computer Vision and Pattern Recognition, 2009. CVPR 2009, pp. 1422–1429. IEEE (2009)

  16. Lian, Y., Shen, X.: Probabilistic reasoning for repeatability detection from urban facade image. In: International Conference on Optical and Photonic Engineering (icOPEN2015), pp. 952429–952429. International Society for Optics and Photonics (2015)

  17. Lian, Y., Shen, X.: Detecting repetitive elements with accurate locations and shapes from urban façade. In: 2015 IEEE International Conference on Image Processing (ICIP), pp. 1920–1924. IEEE (2015)

  18. Martinovi, A., Mathias, M., Weissenberg, J., Van Gool, L.: A three-layered approach to facade parsing. Int. J. Comput. Vis. 7578(1), 416–429 (2012)

    MathSciNet  Google Scholar 

  19. Zhao, P., Quan, L.: Translation symmetry detection in a fronto-parallel view. In: IEEE Conference on Computer Vision and Pattern Recognition 2011, pp. 1009–1016 (2011)

  20. Peng, Z., Lei, Y., Honghui, Z., Long, Q.: Per-pixel translational symmetry detection, optimization, and segmentation. In: Computer Vision and Pattern Recognition, pp. 526–533 (2012)

  21. Lee, S.C., Nevatia, R.: Extraction and integration of window in a 3d building model from ground view images. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 2, pp. II-113–II-120 (2004)

  22. Zitnick, C.L., Dollár, P.: Edge boxes: locating object proposals from edges. In: European Conference on Computer Vision, pp. 391–405. Springer (2014)

  23. Loy, G., Eklundh, J.O.: Detecting Symmetry and Symmetric Constellations of Features. Springer, Berlin (2006)

    Book  Google Scholar 

  24. Jian, L., Ye, Z., Zou, Y.: Automatic generation of colorful patterns with wallpaper symmetries from dynamics. Vis. Comput. 23(6), 445–449 (2007)

    Article  Google Scholar 

  25. Olivier, T., Iasonas, K., Loic, S., Panagiotis, K., Nikos, P.: Parsing façades with shape grammars and reinforcement learning. IEEE Trans. Pattern Anal. Mach. Intell. 35(7), 1744–1756 (2013)

    Article  Google Scholar 

  26. Fan, L., Musialski, P., Liu, L., Wonka, P.: Structure completion for façade layouts. ACM Trans. Gr. 33(6), 210:1–210:11 (2014)

    Article  Google Scholar 

  27. Nan, L., Sharf, A., Zhang, H., Cohen-Or, D., Chen, B.: Smartboxes for interactive urban reconstruction. ACM Trans. Gr. 29(4), 157–166 (2010)

    Article  Google Scholar 

  28. Nan, L., Jiang, C., Bernard, G., Peter, W.: Template assembly for detailed urban reconstruction. Comput. Gr. Forum 34(2), 217–228 (2015)

    Article  Google Scholar 

  29. Rasmussen, C., Korah, T.: Spatiotemporal inpainting for recovering texture maps of partially occluded building facades. In: IEEE International Conference on Image Processing 2005, vol. 3, p. III-125. IEEE (2005)

  30. Korah, T., Rasmussen, C.: Spatiotemporal inpainting for recovering texture maps of occluded building facades. IEEE Trans. Image Process. 16(9), 2262–2271 (2007)

    Article  MathSciNet  Google Scholar 

  31. Böhm, J.: Multi-image fusion for occlusion-free façade texturing. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. 35(5), 867–872 (2004)

    Google Scholar 

  32. Ortin, D., Remondino, F.: Occlusion-free image generation for realistic texture mapping. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. 36(5/W17), 7 (2005)

    Google Scholar 

  33. Wang, X., Totaro, S., Taillandier, F., Hanson, A.R, Teller, S.: Recovering façade texture and microstructure from real-world images. In: Proceedings of International Workshop on Texture Analysis and Synthesis at ECCV, pp. 381–386 (2002)

  34. Mu, T.-J., Wang, J.-H., Du, S.-P., Hu, S.-M.: Stereoscopic image completion and depth recovery. Vis Comput 30(6), 833–843 (2014)

    Article  Google Scholar 

  35. Yang, C., Han, T., Quan, L., Tai, C.-L.: Parsing façade with rank-one approximation. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1720–1727. IEEE (2012)

  36. Liang, X., Ren, X., Zhang, Z., Ma, Y.: Repairing sparse low-rank texture. In: European Conference on Computer Vision, pp. 482–495. Springer (2012)

  37. Liang, X., Ren, X., Zhang, Z., Ma, Y.: Texture repairing by unified low rank optimization. J. Comput. Sci. Technol. 31(3), 525–546 (2016)

    Article  MathSciNet  Google Scholar 

  38. He, K., Sun, J.: Statistics of patch offsets for image completion. In: Computer Vision-ECCV 2012, pp. 16–29. Springer (2012)

  39. Yeh, Y.-T., Breeden, K., Yang, L., Fisher, M., Hanrahan, P.: Synthesis of tiled patterns using factor graphs. ACM Trans. Gr. (TOG) 32(1), 3 (2013)

    MATH  Google Scholar 

  40. Komodakis, N.: Image completion using global optimization. In: 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), vol. 1, pp. 442–452. IEEE (2006)

  41. Pavić, D., Schönefeld, V., Kobbelt, L.: Interactive image completion with perspective correction. Vis. Comput. 22(9–11), 671–681 (2006)

    Google Scholar 

  42. Liu, J., Musialski, P., Wonka, P., Ye, J.: Tensor completion for estimating missing values in visual data. IEEE Trans. Pattern Anal. Mach. Intell. 35(1), 208–220 (2013)

    Article  Google Scholar 

  43. Comaniciu, D., Meer, P.: Mean shift: a robust approach toward feature space analysis. IEEE Trans. Pattern Anal. Mach. Intell. 24(5), 603–619 (2002)

    Article  Google Scholar 

  44. Spath, H.: The cluster dissection and analysis theory FORTRAN programs examples. Prentice-Hall, Inc, Upper Saddle River, NJ, USA (1985)

  45. Serra, J., Soille, P.: Mathematical Morphology and Its Applications to Image Processing. Kluwer, Dordrecht (1994)

    Book  MATH  Google Scholar 

  46. Adams, R., Bischof, L.: Seeded region growing. IEEE Trans. Pattern Anal. Mach. Intell. 16(6), 641–647 (1994)

    Article  Google Scholar 

  47. Kumar, S., Chauhan, A.: A survey on image feature selection techniques. Int. J. Comput. Sci. Inf. Technol. 5(5), 6449–6452 (2014)

    Google Scholar 

  48. Kazmi, I.K., You, L., Zhang, J.J.: A survey of 2d and 3d shape descriptors. In: Computer Graphics, Imaging and Visualization (CGIV), 2013 10th International Conference, pp. 1–10. IEEE (2013)

  49. Heider, P., Pierre-Pierre, A., Li, R., Mueller, R., Grimm, C.: Comparing local shape descriptors. Vis. Comput. 28(9), 919–929 (2012)

    Article  Google Scholar 

  50. Theodoridis, S., Koutroumbas, K.: Pattern Recognition, 4th edn. Academic Press, New York (2008)

    MATH  Google Scholar 

  51. Szegedy, C., Liu, W., Jia, Y., Sermanet, P.: Going deeper with convolutions. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1–9 (2014)

  52. LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015)

    Article  Google Scholar 

  53. Bengio, Y.: Learning deep architectures for AI. Found. Trends\({\textregistered }\) Mach. Learn. 2(1), 1–127 (2009)

  54. Huang, R., Ye, M., Pei, X., Li, T., Dou, Y.: Learning to pool high-level features for face representation. Vis. Comput. 31(12), 1683–1695 (2015)

    Article  Google Scholar 

  55. Hinton, G.E., Salakhutdinov, R.R.: Reducing the dimensionality of data with neural networks. Science 313(5786), 504–507 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  56. Hinton, G.E., Osindero, S., Teh, Y.-W.: A fast learning algorithm for deep belief nets. Neural Comput. 18(7), 1527–1554 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  57. Tylecek, R., Šára, R.: Spatial pattern templates for recognition of objects with regular structure. In: German Conference on Pattern Recognition, pp. 364–374. Springer (2013)

  58. Shao, H., Svoboda, T., Van Gool, L.: Zubud–zurich buildings database for image based recognition. Comput. Vis. Lab. Swiss Federal Inst. Technol. Switz. Tech. Rep. 260, 20 (2003)

    Google Scholar 

  59. Jensen, F.V.: An Introduction to Bayesian Networks, vol. 210. UCL Press, London (1996)

    Google Scholar 

  60. Koller, D., Friedman, N.: Probabilistic Graphical Models: Principles and Techniques. MIT Press, Cambridge (2009)

    MATH  Google Scholar 

  61. Pearl, J.: Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Morgan Kaufmann, Burlington (2014)

    MATH  Google Scholar 

  62. Rumí, R., Salmerón, A., Moral, S.: Estimating mixtures of truncated exponentials in hybrid Bayesian networks. Test 15(2), 397–421 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  63. Cobb, B.R., Rumí, R., Salmerón, A.: Bayesian network models with discrete and continuous variables. In: Advances in Probabilistic Graphical Models, pp. 81–102. Springer Berlin Heidelberg (2007)

  64. Romero, V., Rum, R., Salmern, A.: Learning hybrid Bayesian networks using mixtures of truncated exponentials. Int. J. Approx. Reason. 42(1), 54–68 (2006)

    Article  MathSciNet  Google Scholar 

  65. Cobb, B.R., Shenoy, P.P.: Inference in hybrid Bayesian networks with mixtures of truncated exponentials. Int. J. Approx. Reason. 41(3), 257–286 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  66. Goutte, C., Gaussier, E.: A probabilistic interpretation of precision, recall and F-score, with implication for evaluation. In: European Conference on Advances in Information Retrieval, pp. 345–359. Springer Berlin Heidelberg (2005)

  67. Geng, Y., Hai-Miao, H., Zeng, G., Zheng, J.: A person re-identification algorithm by exploiting region-based feature salience. J. Vis. Commun. Image Rep. 29, 89–102 (2015)

    Article  Google Scholar 

  68. Torralba, A., Murphy, K.P., Freeman, W.T.: Sharing features: efficient boosting procedures for multiclass object detection. In: Computer Vision and Pattern Recognition, 2004. CVPR 2004. Proceedings of the 2004 IEEE Computer Society Conference on, vol. 2, pp. II-762. IEEE (2004)

  69. Krizhevsky, A., Sutskever, I., Hinton, G.E: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105. Curran Associates, Inc (2012)

  70. Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 580–587 (2014)

Download references

Acknowledgements

This work was partially supported by the National High-Tech Research and Development Program of China (Grant No. 2013AA013803-1) and the National Natural Science Foundation of China (Grant No. 61202235).

Author information

Authors and Affiliations

  1. State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing, 100191, China

    Yongjian Lian, Xukun Shen & Yong Hu

  2. Modern Education Technology and Information Center, North University of China, Taiyuan, 030051, China

    Yongjian Lian

Authors
  1. Yongjian Lian
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xukun Shen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yong Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yong Hu.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lian, Y., Shen, X. & Hu, Y. Detecting and inferring repetitive elements with accurate locations and shapes from façades. Vis Comput 34, 491–506 (2018). https://doi.org/10.1007/s00371-017-1355-z

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-017-1355-z

Keywords

Search

Navigation