Skip to main content
SpringerLink
Log in

Automatic façade recovery from single nighttime image

  • Research Article
  • Published:
Frontiers of Computer Science Aims and scope Submit manuscript

Abstract

Nighttime images are difficult to process due to insufficient brightness, lots of noise, and lack of details. Therefore, they are always removed from time-lapsed image analysis. It is interesting that nighttime images have a unique and wonderful building features that have robust and salient lighting cues from human activities. Lighting variation depicts both the statistical and individual habitation, and it has an inherent man-made repetitive structure from architectural theory. Inspired by this, we propose an automatic nighttime façade recovery method that exploits the lattice structures of window lighting. First, a simple but efficient classification method is employed to determine the salient bright regions, which may be lit windows. Then we group windows into multiple lattice proposals with respect to façades by patch matching, followed by greedily removing overlapping lattices. Using the horizon constraint, we solve the ambiguous proposals problem and obtain the correct orientation. Finally, we complete the generated façades by filling in the missing windows. This method is well suited for use in urban environments, and the results can be used as a good single-view compensation method for daytime images. The method also acts as a semantic input to other learning-based 3D image reconstruction techniques. The experiment demonstrates that our method works well in nighttime image datasets, and we obtain a high lattice detection rate of 82.1% of 82 challenging images with a low mean orientation error of 12.1 ± 4.5 degrees.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Pan Z, Zhang Y, Kwong S. Efficient motion and disparity estimation optimization for low complexity multiview video coding. IEEE Transactions on Broadcasting, 2015, 61(2): 166–176

    Article  Google Scholar 

  2. Gonzalez R. C, Woods R. E. Digital Image Processing. New Jersey: Prentice Hall, 2008

    Google Scholar 

  3. Durand F, Dorsey J. Fast bilateral filtering for the display of high dynamic range images. In: Proceedings of the 29th International Conference and Exhibition on Computer Graphics and Interactive Techniques. 2002, 257–266

    Google Scholar 

  4. Rao Y, Chen L. A survey of video enhancement techniques. Journal of Information Hiding and Multimedia Signal Processing, 2012, 3(1): 71–99

    Google Scholar 

  5. Raskar R, llie A, Yu J. Image fusion for context enhancement and video surrealism. In: Proceedings of the 3rd international symposium on Non-photorealistic animation and rendering. 2004, 85–152

    Google Scholar 

  6. Cai Y, Huang K, Tan T, Wang Y. In: Proceedings of the 13rd International Conference on Pattern Recognition. 2006, 980–983

    Google Scholar 

  7. Dong X, Wang G, Pang Y, Li W, Wen J, Meng W, Lu Y. Fast efficient algorithm for enhancement of low lighting video. In: Proceedings of International Conference on Multimedia and Expo. 2011, 1–6

    Google Scholar 

  8. Micusik B, Wildenauer H, Kosecka J. Detection and matching of rectilinear structures. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2008, 1–7

    Google Scholar 

  9. Kosecka J, Zhang W. Extraction, matching, and pose recovery based on dominant rectangular structures. In: Proceedings of International Workshop on Higher-Level Knowledge in 3D Modeling and Motion Analysis. 2003

    Google Scholar 

  10. David L, Martial H, Takeo K. Geometric reasoning for single image structure recovery. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2009, 2136–2143

    Google Scholar 

  11. Wu C, Frahm J, Pollefeys M. Detecting large repetitive structures with salient boundaries. In: Proceedings of European Conference on Computer Vision, 2010, 142–155

    Google Scholar 

  12. Schindler G, Krishnamurthy P, Lublinerman R, Liu Y, Dellaert F. Detecting and matching repeated patterns for automatic geo-tagging in urban environments. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2008, 1–7

    Google Scholar 

  13. Hays J, Leordeanu M, Efros A. A, Liu Y. Discovering texture regularity as a higher-order correspondence problem. In: Proceedings of European Conference on Computer Vision. 2006, 522–535

    Google Scholar 

  14. Park M, Brocklehurst K, Collins R T, Liu Y. Deformed Lattice Detection in Real-World Images Using Mean-Shift Belief Propagation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2009, 31(10): 1804–1816

    Article  Google Scholar 

  15. Park M, Brocklehurst K, Collins R T, Liu Y. Translation-Symmetry-Based Perceptual Grouping with Applications to Urban Scenes. In: Proceedings of Asian Conference on Computer Vision. 2010, 329–342

    Google Scholar 

  16. Liu S, Ng T T, Sunkavalli K, Do M N, Shechtman E, Carr N. PatchMatch-based automatic lattice detection for near-regular textures. In: Proceedings of IEEE International Conference on Computer Vision. 2015, 181–189

    Google Scholar 

  17. Mobahi H, Zhou Z, Yang A Y, Ma Y. Holistic 3d reconstruction of urban structures from low-rank textures. In: Proceedings of IEEE International Conference on Computer Vision Workshops. 2011, 593–600

    Google Scholar 

  18. Zhang Z, Ganesh A, Liang X, Ma Y. TILT: transform invariant lowrank textures. International Journal of Computer Vision, 2012, 99(1): 1–24

    Article  MathSciNet  Google Scholar 

  19. Liu J, Liu Y. Local regularity-driven city-scale facade detection from aerial images. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2014, 3778–3785

    Google Scholar 

  20. Matas J, Chum O, Urban M, Pajdla T. Robust wide baseline stereo from maximally stable extremal regions. Image and Vision Computing, 2004, 22(10): 761–767

    Article  Google Scholar 

  21. Forssen P E, Lowe D G. Shape descriptors for maximally stable extremal regions. In: Proceedings of IEEE International Conference on Computer Vision. 2007, 1–8

    Google Scholar 

  22. Guillou E, Meneveaux D, Maisel E, Bouatouch K. Using vanishing points for camera calibration and coarse 3D reconstruction from a single image. The Visual Computer, 2000, 16(7): 396–410

    Article  Google Scholar 

  23. Hoiem D, Efros A A, Hebert M. Geometric context from a single image. In: Proceedings of IEEE International Conference on Computer Vision. 2005, 654–661

    Google Scholar 

  24. Yang G, Stewart C V, Sofka M, Tsai C. Registration of challenging image pairs: initialization, estimation, and decision. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2007, 29(11): 1973–1989

    Article  Google Scholar 

Download references

Acknowledgements

We gratefully acknowledge the proofreading of Doctor Yuehua Wang. This work was supported by the National High-tech R&D Program (2015AA016403), the National Natural Science Foundation of China (Grant Nos. 61572061, 61472020, 61502020), and the China Postdoctoral Science Foundation (2013M540039).

Author information

Authors and Affiliations

  1. State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing, 100083, China

    Yi Zhou, Qichuan Geng, Zhong Zhou & Wei Wu

Authors
  1. Yi Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qichuan Geng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhong Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Wei Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Qichuan Geng or Zhong Zhou.

Additional information

Yi Zhou received the bachelor’s degree in computer science and technology from Beihang University, China in 2010. Currently, he is a PhD candidate at the State Key Lab of Virtual Reality Technology and Systems, Beihang University. His research interests include augmented virtual reality and time-lapsed video understanding. He is a student member of the CCF.

Qichuan Geng received the bachelor’s degree from the School of Automation Science and Electrical Engineering, Beihang University, China in 2012. He is currently a PhD candidate at the State Key Lab of Virtual Reality Technology and Systems, Beihang University. His research interests include augmented virtual reality and semantic segmentation understanding. He is a student member of the CCF.

Zhong Zhou is a professor at the State Key Lab of Virtual Reality Technology and Systems, Beihang University, China. He received his BS degree from Nanjing University, China and PhD degree from Beihang University in 1999 and 2005, respectively. His main research interests include natural phenomena simulation, augmented virtual reality, and Internet-based virtual reality technologies. He is a member of the China Computer Federation and the Institute of Electrical and Electronics Engineers.

Wei Wu is a professor in the School of Computer Science and Engineering at Beihang University, China and is currently the chair of the Technical Committee on Virtual Reality and Visualization (TCVRV) of the China Computer Federation (CCF). He received the PhD degree from Harbin Institute of Technology, China in 1995. He has published more than 90 papers, 33 issued patents, and one book. His current research interests involve real-time 3D reconstruction, remote immersive systems, and augmented reality.

Electronic supplementary material

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhou, Y., Geng, Q., Zhou, Z. et al. Automatic façade recovery from single nighttime image. Front. Comput. Sci. 14, 95–104 (2020). https://doi.org/10.1007/s11704-017-6457-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11704-017-6457-2

Keywords

Search

Navigation