Skip to main content
SpringerLink
Log in

Enhancing performance-based generative architectural design with sketch-based image retrieval: a pilot study on designing building facade fenestrations

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

By coupling parametric modelling, building performance (like energy efficiency) simulation, and algorithmic optimization, performance-based generative architectural design (PGAD) can automatically generate lots of high-performance architectural design solutions. Although it is ‘performance-based’, the final selection of a real design project still needs to consider the aesthetics of design choices. However, due to the overwhelming number of design choices generated by PGAD, it is difficult for designers to choose the most favourable one from them. Therefore, the current study tries to integrate the technology of sketch-based image retrieval (SBIR) into the selecting stage of PGAD. Rather than navigating alternatives one from another and getting lost, designers can directly find the most aesthetically preferred one by inputting his/her hand-drawn design. A design project of fenestrating a multiple-floor office building is used to demonstrate this method and test three SBIR algorithms: Angular radial partitioning (ARP), Angular radial orientation partitioning (AROP), and Sketch-A-Net model (SAN). Test results show that AROP performs the best among these three algorithms. Its retrievals are most similar to inquiry images drawn by architects. Meanwhile, performances of AROP with different template combinations are also rated. After that, AROP with the best template is also tested with incompletely drawn inquiry images. In the end, investigation results are validated by another building façade design case. The current study automates the PGAD process stepwise, making it more applicable to real design projects.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Zhao, S., Angelis, D.E.: Performance-based generative architecture design: a review on design problem formulation and software utilization. J. Integr. Des. Process Sci. 22(3), 55–76 (2019)

    Article  Google Scholar 

  2. Gerber, D.J., Pantazis, E., Wang, A.: A multi-agent approach for performance based architecture: design exploring geometry, user, and environmental agencies in façades. Autom. Constr. 76(April), 45–58 (2017)

    Article  Google Scholar 

  3. Yang, D., Ren, S., Turrin, M., Sariyildiz, S., Sun, Y.: Multi-disciplinary and multi-objective optimization problem re-formulation in computational design exploration: a case of conceptual sports building design. Autom. Constr. 92, 242–269 (2018)

    Article  Google Scholar 

  4. Thornton, T.: Design Explorer 2. Github (2017). http://tt-acm.github.io/DesignExplorer/

  5. Autodesk: Project Refinery Beta (2019)

  6. Sileryte, R., Aquilio, A. D., Di Stefano, D., Yang, D., Turrin, M.: Supporting exploration of design alternatives using multivariate analysis algorithms. In: Proceedings of the Symposium on Simulation for Architecture and Urban Design (simAUD 2016), pp. 215–222 (2016)

  7. Nagy, D., et al.: Project discover: an application of generative design for architectural space planning. SimAUD 2017, 59–66 (2017)

    Google Scholar 

  8. Rodrigues, E., Sousa-Rodrigues, D., Teixeira de Sampayo, M., Gaspar, A.R., Gomes, Á., Henggeler Antunes, C.: Clustering of architectural floor plans: A comparison of shape representations. Autom. Constr. 80, 48–65 (2017)

    Article  Google Scholar 

  9. Liang, L.B., Jakubiec, J.A.: A three-part visualisation framework to navigate complex multi-objective (>3) building performance optimisation design space. In: BSO2018, pp. 11–12 (2018)

  10. Suga, K., Kato, S., Hiyama, K.: Structural analysis of pareto-optimal solution sets for multi-objective optimization: an application to outer window design problems using multiple objective genetic algorithms. Build. Environ. 45(5), 1144–1152 (2010)

    Article  Google Scholar 

  11. Yousif, S., Yan, W.: Shape clustering using K-medoids in architectural form finding. In: Computer-Aided Architectural Design. ``Hello, Culture’’, pp. 459–473 (2019)

  12. Xiao, R.: Comparing and clustering residential layouts using a novel measure of grating difference. Nexus Netw. J. 0123456789 (2020)

  13. Wallacei: Wallacei: an evolutionary and analytic engine for grasshopper 3D. https://www.wallacei.com/ (2019). Accessed 4 Apr, 2021

  14. Yang, J., et al.: k-Shape clustering algorithm for building energy usage patterns analysis and forecasting model accuracy improvement. Energy Build. 146, 27–37 (2017)

    Article  Google Scholar 

  15. Qian, X., Lu, D., Wang, Y., Zhu, L., Tang, Y.Y., Wang, M.: Image re-ranking based on topic diversity. IEEE Trans. Image Process. 26(8), 3734–3747 (2017)

    Article  MathSciNet  Google Scholar 

  16. Wang, Y., Zhu, L., Qian, X., Han, J.: Joint hypergraph learning for tag-based image retrieval. IEEE Trans. Image Process. 27(9), 4437–4451 (2018)

    Article  MathSciNet  Google Scholar 

  17. Wang, Y., et al.: Position focused attention network for image-text matching. In: IJCAI 2019 (2019)

  18. Zhao, W., Xie, F., Zhao, W., Wang, X., Chen, L., Peng, J.: Tag-based weakly-supervised hashing for image retrieval. In: IJCAI Int. Jt. Conf. Artif. Intell., 2018-July, pp. 3776–3782 (2018)

  19. Liu, P., Guo, J.M., Wu, C.Y., Cai, D.: Fusion of deep learning and compressed domain features for content-based image retrieval. IEEE Trans. Image Process. 26(12), 5706–5717 (2017)

    Article  MathSciNet  Google Scholar 

  20. Zhu, L., Shen, J., Xie, L., Cheng, Z.: Unsupervised visual hashing with semantic assistant for content-based image retrieval. IEEE Trans. Knowl. Data Eng. 29(2), 472–486 (2017)

    Article  Google Scholar 

  21. Xia, Z., Wang, X., Zhang, L., Qin, Z., Sun, X., Ren, K.: A privacy-preserving and copy-deterrence content-based image retrieval scheme in cloud computing. IEEE Trans. Inf. Forensics Secur. 11(11), 2594–2608 (2016)

    Article  Google Scholar 

  22. Hong, R., Li, L., Cai, J., Tao, D., Wang, M., Tian, Q.: Coherent semantic-visual indexing for large-scale image retrieval in the cloud. IEEE Trans. Image Process. 26(9), 4128–4138 (2017)

    Article  MathSciNet  Google Scholar 

  23. Qian, X., Tan, X., Zhang, Y., Hong, R., Wang, M.: Enhancing sketch-based image retrieval by re-ranking and relevance feedback. IEEE Trans. Image Process. 25(1), 195–208 (2016)

    Article  MathSciNet  Google Scholar 

  24. Zhang, Y., Qian, X., Tan, X., Han, J., Tang, Y.: Sketch-based image retrieval by salient contour reinforcement. IEEE Trans. Multimed. 18(8), 1604–1615 (2016)

    Article  Google Scholar 

  25. Eitz, M., Hildebrand, K., Boubekeur, T., Alexa, M.: Sketch-based image retrieval: benchmark and bag-of-features descriptors. IEEE Trans. Vis. Comput. Graph. 17(11), 1624–1636 (2011)

    Article  Google Scholar 

  26. Bui, T., Collomosse, J.: Scalable Sketch-based image retrieval using color gradient features. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1012–1019 (2015)

  27. Sun, X., Wang, C., Xu, C., Zhang, L.: Indexing billions of images for sketch-based retrieval. In: Proceedings of the 21st ACM international conference on Multimedia, pp. 233–242 (2013)

  28. Parui, S., Mittal, A.: Similarity-invariant sketch-based image retrieval in large databases. In: Computer Vision—ECCV 2014, pp. 398–414 (2014)

  29. Wang, L., Qian, X., Zhang, Y., Shen, J., Cao, X.: Enhancing sketch-based image retrieval by CNN semantic re-ranking. IEEE Trans. Cybern. 1–13 (2019)

  30. Sangkloy, P., Burnell, N., Ham, C., Hays, J.: The sketchy database: learning to retrieve badly drawn bunnies. ACM Trans. Graph. 35(4), 1–12 (2016)

    Article  Google Scholar 

  31. Liu, L., Shen, F., Shen, Y., Liu, X., Shao, L.: Deep sketch hashing: fast free-hand sketch-based image retrieval. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2862–2871 (2017)

  32. Seddati, O., Dupont, S., Mahmoudi, S.: Quadruplet networks for sketch-based image retrieval. In: ICMR’17, pp. 184–191 (2017)

  33. Qi, Y., Song, Y.Z., Zhang, H., Liu, J.: Sketch-based image retrieval via Siamese convolutional neural network. In: Proceedings—International Conference on Image Processing, ICIP, pp. 2460–2464 (2016)

  34. Bui, T., Ribeiro, L., Ponti, M., Collomosse, J.: Compact descriptors for sketch-based image retrieval using a triplet loss convolutional neural network. Comput. Vis. Image Underst. 164, 27–37 (2017)

    Article  Google Scholar 

  35. Li, Y., Li, W.: A survey of sketch-based image retrieval. Mach. Vis. Appl. 29(7), 1083–1100 (2018)

    Article  Google Scholar 

  36. Xu, J., Xue, X., Wu, Y., Mao, X.: Matching a composite sketch to a photographed face using fused HOG and deep feature models. Vis. Comput. (2020)

  37. Yu, Q., Yang, Y., Liu, F., Song, Y.Z., Xiang, T., Hospedales, T.M.: Sketch-a-Net: a deep neural network that beats humans. Int. J. Comput. Vis. 122(3), 411–425 (2017)

    Article  MathSciNet  Google Scholar 

  38. Takagi, H.: Interactive evolutionary computation: fusion of the capabilities of EC optimization and human evaluation. Proc. IEEE 89(9), 1275–1296 (2001)

    Article  Google Scholar 

  39. Brintrup, A.M., Ramsden, J., Takagi, H., Tiwari, A.: Ergonomic chair design by fusing qualitative and quantitative criteria using interactive genetic algorithms. IEEE Trans. Evol. Comput. 12(3), 343–354 (2008)

    Article  Google Scholar 

  40. Muehlbauer, M., Burry, J., Song, A.: An aesthetic-based fitness measure and a framework for guidance of evolutionary design in architecture. In: International Conference on Computational Intelligence in Music, Sound, Art and Design (Part of EvoStar), vol. 12103, pp. 134–149. LNCS (2020)

  41. Kowaliw, T., Dorin, A., McCormack, J.: Promoting creative design in interactive evolutionary computation. IEEE Trans. Evol. Comput. 16(4), 523–536 (2012)

    Article  Google Scholar 

  42. Weber, M., Langenhan, C., Roth-Berghofer, T., Liwicki, M., Dengel, A., Petzold, F.: a.SCatch: Semantic structure for architectural floor plan retrieval. In: ICCBR 2010: Case-Based Reasoning. Research and Development, pp. 510–524 (2010)

  43. Langenhan, C., Weber, M., Liwicki, M., Petzold, F., Dengel, A.: Sketch-based methods for researching building layouts through the semantic fingerprint of architecture. In: Designing Together: CAADFutures 2011—Proceedings of the 14th International conference on Computer Aided Architectural Design, pp. 85–102 (2011)

  44. Langenhan, C., Weber, M., Liwicki, M., Petzold, F., Dengel, A.: Graph-based retrieval of building information models for supporting the early design stages. Adv. Eng. Inform. 27(4), 413–426 (2013)

    Article  Google Scholar 

  45. Ahmed, S., Weber, M., Liwicki, M., Langenhan, C., Dengel, A., Petzold, F.: Automatic analysis and sketch-based retrieval of architectural floor plans. Pattern Recognit. Lett. 35(1), 91–100 (2014)

    Article  Google Scholar 

  46. Roith, J., Langenhan, C., Petzold, F.: Supporting the building design process with graph-based methods using centrally coordinated federated databases. Vis. Eng. 5(1) (2017).

  47. Kai, Q.: Drawing semantic retrieval algorithms based on deep multilayer convolutional network. In: Proceedings—2019 International Conference on Smart Grid and Electrical Automation, ICSGEA 2019, pp. 255–258 (2019)

  48. Wessel, R., Ochmann, S., Vock, R., Blümel, I., Klein, R.: Efficient retrieval of 3D building models using embeddings of attributed subgraphs. In: International Conference on Information and Knowledge Management, Proceedings, pp. 2097–2100 (2011)

  49. Wessel, R.: Shape Retrieval Methods for Architectural 3D Models. Universität Bonn (2013)

  50. Yasseen, Z., Verroust-Blondet, A., Nasri, A.: View selection for sketch-based 3D model retrieval using visual part shape description. Vis. Comput. 33(5), 565–583 (2017)

    Article  Google Scholar 

  51. Kazi, R.H., Grossman, T., Cheong, H., Hashemi, A., Fitzmaurice, G.: DreamSketch: early stage 3D design explorations with sketching and generative design. In: UIST 2017—Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology, pp. 401–414 (2017)

  52. Zhao, S.: Searching for office buildings’ fenestration geometries with a bi-phase optimization framework. Sci. Technol. Built Environ. 20(9), 1–22 (2020)

    Google Scholar 

  53. Roudsari M.S.: Honeybee. Github (2019)

  54. Jonatan: TT TOOLBOX. food4rhino (2017). https://www.food4rhino.com/app/tt-toolbox

  55. Canny, J.: A computational approach to edge detection. IEEE Trans. Pattern Anal. Mach. Intell. PAMI-8(6), 679–698 (1986)

  56. Martin, D.R., Fowlkes, C.C.: Learning to detect natural image boundaries using local brightness, color, and texture cues. In: IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 26, no. 5, pp. 530–549 (2004)

  57. Chalechale, A., Mertins, A., Naghdy, G.: Edge image description using angular radial partitioning. IEE Proc. Vis. Image Signal Process. 151(2), 93–101 (2004)

    Article  Google Scholar 

  58. Yu, Q., Yang, Y., Song, Y.-Z., Xiang, T., Hospedales, T.: Sketch-a-Net that beats humans (2015)

  59. Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255 (2009)

Download references

Acknowledgements

Ms. Jiewen WU contributes to plotting Fig. 9c.

Funding

Funding was provided by Key Technologies Research and Development Program (Grant No. 2020YFC2006602), National Natural Science Foundation of China (Grant No. 62072324) and Jiangsu Provincial Key Research and Development Program (Grant No. BE2020026).

Author information

Author notes

  1. Shenghuan Zhao and Luo Wang equally contributed to the paper and considered as co-first authors.

Authors and Affiliations

  1. School of Architecture and Urban Planning, Suzhou University of Science and Technology, Suzhou, China

    Shenghuan Zhao & Jianping Chen

  2. Jiangsu Key Laboratory of Intelligent Building Energy Efficiency, Suzhou University of Science and Technology, Suzhou, China

    Shenghuan Zhao & Jianping Chen

  3. School of Information and Communications Engineering, Xi’an Jiaotong University, Xi’an, China

    Luo Wang & Xueming Qian

  4. School of Electronic and Information Engineering, Suzhou University of Science and Technology, Suzhou, China

    Jianping Chen

Authors
  1. Shenghuan Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Luo Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xueming Qian
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jianping Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jianping Chen.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhao, S., Wang, L., Qian, X. et al. Enhancing performance-based generative architectural design with sketch-based image retrieval: a pilot study on designing building facade fenestrations. Vis Comput 38, 2981–2997 (2022). https://doi.org/10.1007/s00371-021-02170-x

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-021-02170-x

Keywords

Search

Navigation