Hostname: page-component-848d4c4894-wzw2p Total loading time: 0 Render date: 2024-05-07T23:48:05.643Z Has data issue: false hasContentIssue false
  • English
  • Français

Data-enabled sketch search and retrieval for visual design stimuli generation

Published online by Cambridge University Press:  02 August 2022

Zijian Zhang  and
Yan Jin Open the ORCID record for Yan Jin [Opens in a new window]
Zijian Zhang
Affiliation:
Department of Aerospace & Mechanical Engineering, University of Southern California, Los Angeles, CA 90089, USA
Yan Jin*
Affiliation:
Department of Aerospace & Mechanical Engineering, University of Southern California, Los Angeles, CA 90089, USA
*
Author for correspondence: Yan Jin, E-mail: yjin@usc.edu
Get access Rights & Permissions [Opens in a new window]

Abstract

Access to vast datasets of visual and textual materials has become significantly easier. How to take advantage of the conveniently available data to support creative design activities remains a challenge. In the phase of idea generation, the visual analogy is considered an effective strategy to stimulate designers to create innovative ideas. Designers can read useful information off vague and incomplete conceptual visual representations, or stimuli, to reach potential visual analogies. In this paper, a computational framework is proposed to search and retrieve visual stimulation cues, which is expected to have the potential to help designers generate more creative ideas by avoiding visual fixation. The research problems include identifying and detecting visual similarities between visual representations from various categories and quantitatifying the visual similarity measures serving as a distance metric for visual stimuli search and retrieval. A deep neural network model is developed to learn a latent space that can discover visual relationships between multiple categories of sketches. In addition, a top cluster detection-based method is proposed to quantify visual similarity based on the overlapped magnitude in the latent space and then effectively rank categories. The QuickDraw sketch dataset is applied as a backend for evaluating the functionality of our proposed framework. Beyond visual stimuli retrieval, this research opens up new opportunities for utilizing extensively available visual data as creative materials to benefit design-by-analogy.

Keywords

Data-enabled designdesign-by-analogysketchingunsupervised deep learningvisual fixation
Type
Research Article
Information
AI EDAM , Volume 36 , 2022 , e25
Check for updates
Copyright
Copyright © The Author(s), 2022. Published by Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Atilola, O, Tomko, M and Linsey, JS (2016) The effects of representation on idea generation and design fixation: a study comparing sketches and function trees. Design Studies 42, 110–136.CrossRefGoogle Scholar
Bogers, S, Frens, J, van Kollenburg, J, Deckers, E and Hummels, C (2016) Connected baby bottle: a design case study towards a framework for data-enabled design. Proceedings of the 2016 ACM Conference on Designing Interactive Systems, pp. 301–311.CrossRefGoogle Scholar
Cao, X, Zhang, H, Liu, S, Guo, X and Lin, L (2013) Sym-fish: a symmetry-aware flip invariant sketch histogram shape descriptor. Proceedings of the IEEE International Conference on Computer Vision, pp. 313–320.CrossRefGoogle Scholar
Casakin, H (2010) Visual analogy, visual displays, and the nature of design problems: the effect of expertise. Environment and Planning B: Planning and Design 37, 170–188.CrossRefGoogle Scholar
Chakrabarti, A, Sarkar, P, Leelavathamma, B and Nataraju, B (2005) A functional representation for aiding biomimetic and artificial inspiration of new ideas. AI EDAM 19, 113–132.Google Scholar
Chakrabarti, A, Siddharth, L, Dinakar, M, Panda, M, Palegar, N and Keshwani, S (2017) Idea Inspire 3.0—a tool for analogical design. International Conference on Research into Design. Springer, pp. 475–485.Google Scholar
Chen, T, Cheng, M-M, Tan, P, Shamir, A and Hu, S-M (2009) Sketch2photo: internet image montage. ACM Transactions on Graphics (TOG) 28, 124.CrossRefGoogle Scholar
Chen, Y, Tu, S, Yi, Y and Xu, L (2017) Sketch-pix2seq: a model to generate sketches of multiple categories. arXiv preprint arXiv:1709.04121.Google Scholar
Cheong, H, Chiu, I, Shu, L, Stone, RB and McAdams, DA (2011) Biologically meaningful keywords for functional terms of the functional basis. Journal of Mechanical Design 133, 021007.CrossRefGoogle Scholar
Chiu, I and Shu, L (2007) Biomimetic design through natural language analysis to facilitate cross-domain information retrieval. AI EDAM 21, 45–59.Google Scholar
Christensen, BT and Schunn, CD (2007) The relationship of analogical distance to analogical function and preinventive structure: the case of engineering design. Memory & Cognition 35, 29–38.CrossRefGoogle ScholarPubMed
Du, P and MacDonald, EF (2015) Products’ shared visual features do not cancel in consumer decisions. Journal of Mechanical Design 137, 071409.CrossRefGoogle Scholar
Eitz, M, Hildebrand, K, Boubekeur, T and Alexa, M (2010) An evaluation of descriptors for large-scale image retrieval from sketched feature lines. Computers & Graphics 34, 482–498.CrossRefGoogle Scholar
Fu, K, Cagan, J, Kotovsky, K and Wood, K (2013 a) Discovering structure in design databases through functional and surface based mapping. Journal of Mechanical Design 135, 031006.CrossRefGoogle Scholar
Fu, K, Chan, J, Cagan, J, Kotovsky, K, Schunn, C and Wood, K (2013 b) The meaning of “near” and “far”: the impact of structuring design databases and the effect of distance of analogy on design output. Journal of Mechanical Design 135, 021007.CrossRefGoogle Scholar
Goel, V (1995) Sketches of Thought. Cambridge, MA: MIT Press.CrossRefGoogle Scholar
Goel, AK, Vattam, S, Wiltgen, B and Helms, M (2012) Cognitive, collaborative, conceptual and creative—four characteristics of the next generation of knowledge-based CAD systems: a study in biologically inspired design. Computer-Aided Design 44, 879–900.CrossRefGoogle Scholar
Goldschmidt, G (1994) On visual design thinking: the vis kids of architecture. Design Studies 15, 158–174.CrossRefGoogle Scholar
Goldschmidt, G (2001) Visual analogy: A strategy for design reasoning and learning. In Eastman, CM, McCracken, WM and Newstetter, WC (eds), Design Knowing and Learning: Cognition in Design Education. Amsterdan: Elsevier, pp. 199–220.CrossRefGoogle Scholar
Goldschmidt, G and Smolkov, M (2006) Variances in the impact of visual stimuli on design problem solving performance. Design Studies 27, 549–569.CrossRefGoogle Scholar
Gonçalves, M, Cardoso, C and Badke-Schaub, P (2014) What inspires designers? Preferences on inspirational approaches during idea generation. Design Studies 35, 29–53.CrossRefGoogle Scholar
Goucher-Lambert, K, Moss, J and Cagan, J (2019) A neuroimaging investigation of design ideation with and without inspirational stimuli—understanding the meaning of near and far stimuli. Design Studies 60, 1–38.CrossRefGoogle Scholar
Goucher-Lambert, K, Gyory, JT, Kotovsky, K and Cagan, J (2020) Adaptive inspirational design stimuli: using design output to computationally search for stimuli that impact concept generation. ASME Journal of Mechanical Design 142, 091401.CrossRefGoogle Scholar
Ha, D and Eck, D (2017) A neural representation of sketch drawings. arXiv preprint arXiv:1704.03477.Google Scholar
Han, J, Shi, F, Chen, L and Childs, PR (2018) A computational tool for creative idea generation based on analogical reasoning and ontology. AI EDAM 32, 462–477.Google Scholar
Herring, SR, Chang, C-C, Krantzler, J and Bailey, BP (2009) Getting inspired!: understanding how and why examples are used in creative design practice. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, pp. 87–96.CrossRefGoogle Scholar
Hu, R, Barnard, M and Collomosse, J (2010) Gradient field descriptor for sketch based retrieval and localization. 2010 IEEE International Conference on Image Processing, IEEE, pp. 1025–1028.CrossRefGoogle Scholar
Jin, Y and Benami, O (2010) Creative patterns and stimulation in conceptual design. AI EDAM 24, 191–209.Google Scholar
Jongejan, J, Rowley, H, Kawashima, T, Kim, J and Fox-Gieg, N (2016) The Quick, Draw! - A.I. Experiment.Google Scholar
Karimi, P, Maher, ML, Davis, N and Grace, K (2019) Deep learning in a computational model for conceptual shifts in a co-creative design system. arXiv preprint arXiv:1906.10188.Google Scholar
Kingma, DP and Welling, M (2013) Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114.Google Scholar
Kuhn, HW (1955) The Hungarian method for the assignment problem. Naval Research Logistics Quarterly 2, 83–97.CrossRefGoogle Scholar
Kwon, E, Pehlken, A, Thoben, K-D, Bazylak, A and Shu, LH (2019) Visual similarity to aid alternative-use concept generation for retired wind-turbine blades. Journal of Mechanical Design 141, 031106.CrossRefGoogle Scholar
Linsey, JS, Clauss, E, Kurtoglu, T, Murphy, J, Wood, K and Markman, A (2011) An experimental study of group idea generation techniques: understanding the roles of idea representation and viewing methods. Journal of Mechanical Design 133, 031008.CrossRefGoogle Scholar
Luo, J, Yan, B and Wood, K (2017) InnoGPS for data-driven exploration of design opportunities and directions: the case of Google driverless car project. Journal of Mechanical Design 139, 111416.CrossRefGoogle Scholar
Maaten, LVD and Hinton, G (2008) Visualizing data using t-SNE. Journal of Machine Learning Research 9, 2579–2605.Google Scholar
Macomber, B and Yang, M (2011) The role of sketch finish and style in user responses to early stage design concepts. International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, vol. 54860, pp. 567–576.Google Scholar
McKoy, FL, Vargas-Hernández, N, Summers, JD and Shah, JJ (2001) Influence of design representation on effectiveness of idea generation. Proceedings of ASME DETC, Pittsburgh, PA, September 9–12.CrossRefGoogle Scholar
Sarkar, P and Chakrabarti, A (2008) The effect of representation of triggers on design outcomes. AI EDAM 22, 101–116.Google Scholar
Setchi, R and Bouchard, C (2010) In search of design inspiration: a semantic-based approach. Journal of Computing and Information Science in Engineering 10, 031006.CrossRefGoogle Scholar
Shah, JJ, Vargas-Hernandez, N, Summers, JD and Kulkarni, S (2001) Collaborative sketching (C-sketch)—an idea generation technique for engineering design. The Journal of Creative Behavior 35, 168–198.CrossRefGoogle Scholar
Shu, L (2010) A natural-language approach to biomimetic design. AI EDAM: Artificial Intelligence for Engineering Design, Analysis, and Manufacturing 24, 507–519.Google Scholar
Vattam, S, Wiltgen, B, Helms, M, Goel, AK and Yen, J (2011) DANE: fostering creativity in and through biologically inspired design. Design Creativity 2010. Springer, pp. 115–122.CrossRefGoogle Scholar
Walther, DB, Chai, B, Caddigan, E, Beck, DM and Fei-Fei, L (2011) Simple line drawings suffice for functional MRI decoding of natural scene categories. Proceedings of the National Academy of Sciences 108, 9661–9666.CrossRefGoogle ScholarPubMed
Yang, MC (2009) Observations on concept generation and sketching in engineering design. Research in Engineering Design 20, 1–11.CrossRefGoogle Scholar
Yu, Q, Liu, F, Song, Y-Z, Xiang, T, Hospedales, TM and Loy, C-C (2016) Sketch me that shoe. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 799–807.CrossRefGoogle Scholar
Zhang, Z and Jin, Y (2020) An unsupervised deep learning model to discover visual similarity between sketches for visual analogy support. International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, Vol. 83976. American Society of Mechanical Engineers, p. V008T008A003.Google Scholar
Zhang, Z and Jin, Y (2021) Toward computer aided visual analogy support (CAVAS): augment designers through deep learning. International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, Vol. 85420. American Society of Mechanical Engineers, p. V006T006A057.Google Scholar