Abstract
Nighttime images are difficult to process due to insufficient brightness, lots of noise, and lack of details. Therefore, they are always removed from time-lapsed image analysis. It is interesting that nighttime images have a unique and wonderful building features that have robust and salient lighting cues from human activities. Lighting variation depicts both the statistical and individual habitation, and it has an inherent man-made repetitive structure from architectural theory. Inspired by this, we propose an automatic nighttime façade recovery method that exploits the lattice structures of window lighting. First, a simple but efficient classification method is employed to determine the salient bright regions, which may be lit windows. Then we group windows into multiple lattice proposals with respect to façades by patch matching, followed by greedily removing overlapping lattices. Using the horizon constraint, we solve the ambiguous proposals problem and obtain the correct orientation. Finally, we complete the generated façades by filling in the missing windows. This method is well suited for use in urban environments, and the results can be used as a good single-view compensation method for daytime images. The method also acts as a semantic input to other learning-based 3D image reconstruction techniques. The experiment demonstrates that our method works well in nighttime image datasets, and we obtain a high lattice detection rate of 82.1% of 82 challenging images with a low mean orientation error of 12.1 ± 4.5 degrees.
Similar content being viewed by others
References
Pan Z, Zhang Y, Kwong S. Efficient motion and disparity estimation optimization for low complexity multiview video coding. IEEE Transactions on Broadcasting, 2015, 61(2): 166–176
Gonzalez R. C, Woods R. E. Digital Image Processing. New Jersey: Prentice Hall, 2008
Durand F, Dorsey J. Fast bilateral filtering for the display of high dynamic range images. In: Proceedings of the 29th International Conference and Exhibition on Computer Graphics and Interactive Techniques. 2002, 257–266
Rao Y, Chen L. A survey of video enhancement techniques. Journal of Information Hiding and Multimedia Signal Processing, 2012, 3(1): 71–99
Raskar R, llie A, Yu J. Image fusion for context enhancement and video surrealism. In: Proceedings of the 3rd international symposium on Non-photorealistic animation and rendering. 2004, 85–152
Cai Y, Huang K, Tan T, Wang Y. In: Proceedings of the 13rd International Conference on Pattern Recognition. 2006, 980–983
Dong X, Wang G, Pang Y, Li W, Wen J, Meng W, Lu Y. Fast efficient algorithm for enhancement of low lighting video. In: Proceedings of International Conference on Multimedia and Expo. 2011, 1–6
Micusik B, Wildenauer H, Kosecka J. Detection and matching of rectilinear structures. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2008, 1–7
Kosecka J, Zhang W. Extraction, matching, and pose recovery based on dominant rectangular structures. In: Proceedings of International Workshop on Higher-Level Knowledge in 3D Modeling and Motion Analysis. 2003
David L, Martial H, Takeo K. Geometric reasoning for single image structure recovery. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2009, 2136–2143
Wu C, Frahm J, Pollefeys M. Detecting large repetitive structures with salient boundaries. In: Proceedings of European Conference on Computer Vision, 2010, 142–155
Schindler G, Krishnamurthy P, Lublinerman R, Liu Y, Dellaert F. Detecting and matching repeated patterns for automatic geo-tagging in urban environments. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2008, 1–7
Hays J, Leordeanu M, Efros A. A, Liu Y. Discovering texture regularity as a higher-order correspondence problem. In: Proceedings of European Conference on Computer Vision. 2006, 522–535
Park M, Brocklehurst K, Collins R T, Liu Y. Deformed Lattice Detection in Real-World Images Using Mean-Shift Belief Propagation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2009, 31(10): 1804–1816
Park M, Brocklehurst K, Collins R T, Liu Y. Translation-Symmetry-Based Perceptual Grouping with Applications to Urban Scenes. In: Proceedings of Asian Conference on Computer Vision. 2010, 329–342
Liu S, Ng T T, Sunkavalli K, Do M N, Shechtman E, Carr N. PatchMatch-based automatic lattice detection for near-regular textures. In: Proceedings of IEEE International Conference on Computer Vision. 2015, 181–189
Mobahi H, Zhou Z, Yang A Y, Ma Y. Holistic 3d reconstruction of urban structures from low-rank textures. In: Proceedings of IEEE International Conference on Computer Vision Workshops. 2011, 593–600
Zhang Z, Ganesh A, Liang X, Ma Y. TILT: transform invariant lowrank textures. International Journal of Computer Vision, 2012, 99(1): 1–24
Liu J, Liu Y. Local regularity-driven city-scale facade detection from aerial images. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2014, 3778–3785
Matas J, Chum O, Urban M, Pajdla T. Robust wide baseline stereo from maximally stable extremal regions. Image and Vision Computing, 2004, 22(10): 761–767
Forssen P E, Lowe D G. Shape descriptors for maximally stable extremal regions. In: Proceedings of IEEE International Conference on Computer Vision. 2007, 1–8
Guillou E, Meneveaux D, Maisel E, Bouatouch K. Using vanishing points for camera calibration and coarse 3D reconstruction from a single image. The Visual Computer, 2000, 16(7): 396–410
Hoiem D, Efros A A, Hebert M. Geometric context from a single image. In: Proceedings of IEEE International Conference on Computer Vision. 2005, 654–661
Yang G, Stewart C V, Sofka M, Tsai C. Registration of challenging image pairs: initialization, estimation, and decision. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2007, 29(11): 1973–1989
Acknowledgements
We gratefully acknowledge the proofreading of Doctor Yuehua Wang. This work was supported by the National High-tech R&D Program (2015AA016403), the National Natural Science Foundation of China (Grant Nos. 61572061, 61472020, 61502020), and the China Postdoctoral Science Foundation (2013M540039).
Author information
Authors and Affiliations
Corresponding authors
Additional information
Yi Zhou received the bachelor’s degree in computer science and technology from Beihang University, China in 2010. Currently, he is a PhD candidate at the State Key Lab of Virtual Reality Technology and Systems, Beihang University. His research interests include augmented virtual reality and time-lapsed video understanding. He is a student member of the CCF.
Qichuan Geng received the bachelor’s degree from the School of Automation Science and Electrical Engineering, Beihang University, China in 2012. He is currently a PhD candidate at the State Key Lab of Virtual Reality Technology and Systems, Beihang University. His research interests include augmented virtual reality and semantic segmentation understanding. He is a student member of the CCF.
Zhong Zhou is a professor at the State Key Lab of Virtual Reality Technology and Systems, Beihang University, China. He received his BS degree from Nanjing University, China and PhD degree from Beihang University in 1999 and 2005, respectively. His main research interests include natural phenomena simulation, augmented virtual reality, and Internet-based virtual reality technologies. He is a member of the China Computer Federation and the Institute of Electrical and Electronics Engineers.
Wei Wu is a professor in the School of Computer Science and Engineering at Beihang University, China and is currently the chair of the Technical Committee on Virtual Reality and Visualization (TCVRV) of the China Computer Federation (CCF). He received the PhD degree from Harbin Institute of Technology, China in 1995. He has published more than 90 papers, 33 issued patents, and one book. His current research interests involve real-time 3D reconstruction, remote immersive systems, and augmented reality.
Electronic supplementary material
Rights and permissions
About this article
Cite this article
Zhou, Y., Geng, Q., Zhou, Z. et al. Automatic façade recovery from single nighttime image. Front. Comput. Sci. 14, 95–104 (2020). https://doi.org/10.1007/s11704-017-6457-2
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11704-017-6457-2