research-article

Predicting Rating Distributions of Website Aesthetics with Deep Learning for AI-Based Research

Authors:
Simon Eisbach

Organizational and Business Psychology, University of Münster

Organizational and Business Psychology, University of Münster

0000-0001-8870-6646
View Profile

,
Fabian Daugs

Cologne Institute for Digital Ecosystems, Technische Hochschule Köln

Cologne Institute for Digital Ecosystems, Technische Hochschule Köln

0000-0002-8156-024X
View Profile

,
Meinald T. Thielsch

Organizational and Business Psychology, University of Münster

Organizational and Business Psychology, University of Münster

0000-0001-8493-9071
View Profile

,
Matthias Böhmer

Cologne Institute for Digital Ecosystems, Technische Hochschule Köln

Cologne Institute for Digital Ecosystems, Technische Hochschule Köln

0000-0001-8402-4859
View Profile

,
Guido Hertel

Organizational and Business Psychology, University of Münster

Organizational and Business Psychology, University of Münster

0000-0002-7754-2786
View Profile

Authors Info & Claims

ACM Transactions on Computer-Human Interaction Volume 30 Issue 3Article No.: 37pp 1–28https://doi.org/10.1145/3569889

Published:10 June 2023Publication History

ACM Transactions on Computer-Human Interaction

Abstract

The aesthetic appeal of a website has strong effects on users’ reactions, appraisals, and even behaviors. However, evaluating website aesthetics through user ratings is resource intensive, and extant models to predict website aesthetics are limited in performance and ability. We contribute a novel and more precise approach to predict website aesthetics that considers rating distributions. Moreover, we use this approach as a baseline model to illustrate how future research might be conducted using predictions instead of participants. Our approach is based on a deep convolutional neural network model and uses innovations in the field of image aesthetic prediction. It was trained with the dataset from Reinecke and Gajos [2014] and was validated using two independent large datasets. The final model reached an unprecedented cross-validated correlation between the ground truth and predicted rating of LCC = 0.752. We then used the model to successfully replicate prior findings and conduct original research as an illustration for AI-based research.

REFERENCES

[1] Altaboli Ahamed and Lin Yingzi. 2011. Objective and subjective measures of visual aesthetics of website interface design: The two sides of the coin. In Human-Computer Interaction, Julie A. Jacko (Ed.). Springer, 35–44.Google Scholar
[2] Arnheim Rudolf. 1974. Art and Visual Perception: A Psychology of the Creative Eye (2nd rev. ed.). University of California Press.Google ScholarCross Ref
[3] Barredo Arrieta Alejandro, Díaz-Rodríguez Natalia, Del Ser Javier, Bennetot Adrien, Tabik Siham, Barbado Alberto, Garcia Salvador, Gil-Lopez Sergio, Molina Daniel, Benjamins Richard, Chatila Raja, and Herrera Francisco. 2020. Explainable artificial intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Information Fusion 58, C (2020), 82–115. DOI:Google ScholarDigital Library
[4] Bao Xiaojie, Shi Ping, Pan Da, and You Jian. 2019. Image aesthetic prediction model based on deep convolutional neural network. In Proceedings of the 2019 2nd International Conference on Information Systems and Computer Aided Education. IEEE, 122–125. DOI:Google ScholarCross Ref
[5] Berlyne Daniel. 1971. Aesthetics and Psychobiology. Appleton-Century-Crofts.Google Scholar
[6] Birkhoff George David. 1933. Aesthetic Measure. Havard University Press.Google ScholarCross Ref
[7] Bölte Jens, Hösker Thomas, Hirschfeld Gerrit, and Thielsch Meinald. 2017. Electrophysiological correlates of aesthetic processing of webpages: A comparison of experts and laypersons. PeerJ 5, e3440 (June 2017). DOI:Google ScholarCross Ref
[8] Cyr Dianne, Head Milena, Larios Hector, and Pan Bing. 2009. Exploring human images in website design: A multi-method approach. MIS Quarterly 33, 3 (September 2009), 539–566. DOI:Google ScholarCross Ref
[9] Dhar Sagnik, Ordonez Vicente, and Berg Tamara L.. 2011. High level describable attributes for predicting aesthetics and interestingness. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 1657–1664. DOI:Google ScholarDigital Library
[10] Djamasbi Soussan, Siegel Marisa, and Tullis Thomas. 2012. Faces and viewing behavior: An exploratory investigation. AIS Transactions on Human-Computer Interaction 4, 3 (September 2012), 190–211. DOI:Google ScholarCross Ref
[11] Djamasbi Soussan, Siegel Marisa, and Tullis Tom. 2010. Generation Y, web design, and eye tracking. International Journal of Human-Computer Studies 68, 5 (May 2010), 307–323. DOI:Google ScholarDigital Library
[12] Djamasbi Soussan, Siegel Marisa, and Tullis Tom. 2014. Can fixation on main images predict visual appeal of homepages? In Proceedings of the 2014 47th Hawaii International Conference on System Sciences. 371–375. DOI:Google ScholarDigital Library
[13] Dodge Yadolah. 1987. An introduction to L1-norm based statistical data analysis. Computational Statistics & Data Analysis 5, 4 (1987), 239–253. DOI:Google ScholarDigital Library
[14] Dou Qi, Sam Zheng Xianjun, Sun Tongfang, and Heng Pheng-Ann. 2019. Webthetics: Quantifying webpage aesthetics with deep learning. International Journal of Human-Computer Studies 124, (2019), 56–66. DOI:Google ScholarCross Ref
[15] Douneva Maria, Jaron Rafael, and Thielsch Meinald T.. 2016. Effects of different website designs on first impressions, aesthetic judgements and memory performance after short presentation. Interacting with Computers 28, 4 (2016), 552–567. DOI:Google ScholarCross Ref
[16] Engholm Ida. 2002. Digital style history: The development of graphic design on the internet. Digital Creativity 13, 4 (December 2002), 193–211. DOI:Google ScholarCross Ref
[17] Fechner Gustav Theodor. 1876. Vorschule der Ästhetik. Breitkopf und Härtel.Google Scholar
[18] Folstein Marshal F., Robins Lee N., and Helzer John E.. 1983. The mini-mental state examination. Archives of General Psychiatry 40, 7 (1983), 812. DOI:Google ScholarCross Ref
[19] Frenzel Anne and Nett Ulrike. 2008. Berliner Intelligenzstrukturtest für Jugendliche : Begabungs- und Hochbegabungsdiagnostik (BIS-HB) von A. O. Jäger et al. ; [Rezension]. Zuerst ersch. in: Diagnostica 54, 4 (January 2008), 221–225.Google Scholar
[20] He Kaiming, Zhang Xiangyu, Ren Shaoqing, and Sun Jian. 2015. Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 37, 9 (2015), 1904–1916. DOI:Google ScholarDigital Library
[21] Hosu Vlad, Goldlucke Bastian, and Saupe Dietmar. 2019. Effective aesthetics prediction with multi-level spatially pooled features. In Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 9367–9375. DOI:Google ScholarCross Ref
[22] Hou Le, Yu Chen-Ping, and Samaras Dimitris. 2017. Squared earth mover's distance-based loss for training deep neural networks. arXiv:1611.05916. Retrieved from https://arxiv.org/abs/1611.05916.Google Scholar
[23] Howard Andrew, Zhu Menglong, Chen Bo, Kalenichenko D., Wang W., Weyand Tobias, Andreetto M., and Adam H.. 2017. MobileNets: Efficient convolutional neural networks for mobile vision applications. arXiv:1704.04861. Retrieved from https://arxiv.org/abs/1704.04861.Google Scholar
[24] Iten Glena H., Troendle Antonin, and Opwis Klaus. 2018. Aesthetics in context: The role of aesthetics and usage mode for a website's success. Interacting with Computers 30, 2 (2018), 133–149. DOI:Google ScholarCross Ref
[25] Ivory Melody Y. and Megraw Rodrick. 2005. Evolution of web site design patterns. ACM Transactions on Information Systems 23, 4 (October 2005), 463–497. DOI:Google ScholarDigital Library
[26] Jacobsen Thomas, Schubotz Ricarda I., Höfel Lea, and Yves von Cramon D.. 2006. Brain correlates of aesthetic judgment of beauty. Neuroimage 29, 1 (2006), 276–285. DOI:Google ScholarCross Ref
[27] Jia Yangqing, Shelhamer Evan, Donahue Jeff, Karayev Sergey, Long Jonathan, Girshick Ross, Guadarrama Sergio, and Darrell Trevor. 2014. Caffe: Convolutional architecture for fast feature embedding. In Proceedings of the ACM Conference on Multimedia. 675–678. Retrieved from https://arxiv.org/abs/1408.5093.Google Scholar
[28] Jin Bin, Segovia Maria V. Ortiz, and Susstrunk Sabine. 2016. Image aesthetic predictors based on weighted CNNs. In Proceedings of the 2016 IEEE International Conference on Image Processing. IEEE, Piscataway, NJ, 2291–2295. DOI: Google ScholarCross Ref
[29] Ke Yan, Tang Xiaoou, and Jing Feng. 2006. The design of high-level features for photo quality assessment. In Proceedings of the Conference on Computer Vision and Pattern Recognition. IEEE, Piscataway, NJ, 419–426. DOI:Google ScholarDigital Library
[30] Khani Masoud Ganj, Mazinani Mohammad Reza, Fayyaz Mohsen, and Hoseini Mojtaba. 2016. A novel approach for website aesthetic evaluation based on convolutional neural networks. In Proceedings of the 2016 2nd International Conference on Web Research. IEEE, Piscataway, NJ, 48–53. DOI:Google ScholarCross Ref
[31] Krizhevsky Alex, Sutskever Ilya, and Hinton Geoffrey E.. 2012. ImageNet classification with deep convolutional neural networks. In Proceedings of the 25th International Conference on Neural Information Processing Systems. Curran Associates Inc, Red Hook, NY, 1097–1105.Google Scholar
[32] Lavie Talia and Tractinsky Noam. 2004. Assessing dimensions of perceived visual aesthetics of web sites. International Journal of Human-Computer Studies 60, 3 (2004), 269–298. DOI:Google ScholarDigital Library
[33] Leder Helmut, Belke Benno, Oeberst Andries, and Augustin Dorothee. 2004. A model of aesthetic appreciation and aesthetic judgments. British Journal of Psychology 95, 4 (2004), 489–508. DOI:Google ScholarCross Ref
[34] Levina Elizabeth and Bickel Peter. 2001. The earth mover's distance is the Mallows distance: Some insights from statistics. In Proceedings of the 8th IEEE International Conference on Computer Vision. IEEE Computer Society, Los Alamitos, Calif, 251–256. DOI:Google ScholarCross Ref
[35] Lindgaard Gitte, Dudek Cathy, Sen Devjani, Sumegi Livia, and Noonan Patrick. 2011. An exploration of relations between visual appeal, trustworthiness and perceived usability of homepages. ACM Transactions on Computer-Human Interaction 18, 1 (2011), 1–30. DOI:Google ScholarDigital Library
[36] Lindgaard Gitte, Fernandes Gary, Dudek Cathy, and Brown J.. 2006. Attention web designers: You have 50 milliseconds to make a good first impression! Behaviour & Information Technology 25, 2 (2006), 115–126. DOI:Google ScholarCross Ref
[37] Liu Dong, Puri Rohit, Kamath Nagendra, and Bhattacharya Subhabrata. 2020. Composition-aware image aesthetics assessment. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 3558–3567. DOI:Google ScholarCross Ref
[38] Lu Xin, Lin Zhe, Jin Hailin, Yang Jianchao, and Wang James Z.. 2014. RAPID. In Proceedings of the 2014 ACM Conference on Multimedia. ACM, New York, NY, 457–466. DOI:Google ScholarDigital Library
[39] Mai Long, Jin Hailin, and Liu Feng. 2016. Composition-preserving deep photo aesthetics assessment. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Google Scholar
[40] Miniukovich Aliaksei and de Angeli Antonella. 2015. Computation of interface aesthetics. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 1163–1172. DOI:Google ScholarDigital Library
[41] Moshagen Morten and Thielsch Meinald T.. 2010. Facets of visual aesthetics. International Journal of Human-Computer Studies 68, 10 (2010), 689–709.Google ScholarDigital Library
[42] Moshagen Morten and Thielsch Meinald T.. 2013. A short version of the visual aesthetics of websites inventory. Behaviour & Information Technology 32, 12 (2013), 1305–1311.Google ScholarDigital Library
[43] Murray Naila and Gordo Albert. 2017. A deep architecture for unified aesthetic prediction. arXiv:1708.04890. Retrieved from https://arxiv.org/abs/1708.04890.Google Scholar
[44] Ngo David Chek Ling, Teo Lian Seng, and Byrne John G.. 2003. Modelling interface aesthetics. Information Sciences 152, 1 (June 2003), 25–46. DOI:Google ScholarDigital Library
[45] Nishiyama Masashi, Okabe Takahiro, Sato Imari, and Sato Yoichi. 2011. Aesthetic quality classification of photographs based on color harmony. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, Piscataway, NJ, 33–40. DOI:Google ScholarDigital Library
[46] PsyWeb. 2020. PsyWeb. Retrieved May 18, 2020 from https://psyweb.uni-muenster.de/.Google Scholar
[47] Reinecke Katharina and Gajos Krzysztof Z.. 2014. Quantifying visual preferences around the world. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, New York, NY, 11–20. DOI:Google ScholarDigital Library
[48] Reinecke Katharina, Yeh Tom, Miratrix Luke, Mardiko Rahmatri, Zhao Yuechen, Liu Jenny, and Gajos Krzysztof Z.. 2013. Predicting users’ first impressions of website aesthetics with a quantification of perceived visual complexity and colorfulness. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 2049–2058.Google ScholarDigital Library
[49] Ribeiro Marco Tulio, Singh Sameer, and Guestrin Carlos. 2016. Model-agnostic interpretability of machine learning. In Proceedings of the 2016 ICML Workshop on Human Interpretability in Machine Learning. DOI:Google ScholarCross Ref
[50] Robins David and Holmes Jason. 2008. Aesthetics and credibility in web site design. Information Processing & Management 44, 1 (2008), 386–399. DOI:Google ScholarDigital Library
[51] Roth Sandra P., Schmutz Peter, Pauwels Stefan L., Bargas-Avila Javier A., and Opwis Klaus. 2010. Mental models for web objects: Where do users expect to find the most frequent objects in online shops, news portals, and company web pages? Interacting with Computers 22, 2 (March 2010), 140–152. DOI:Google ScholarDigital Library
[52] Seckler Mirjam, Opwis Klaus, and Tuch Alexandre N.. 2015. Linking objective design factors with subjective aesthetics: An experimental study on how structure and color of websites affect the facets of users’ visual aesthetic perception. Computers in Human Behavior 49, C (2015), 375–389. DOI:Google ScholarDigital Library
[53] Simonyan Karen and Zisserman Andrew. 2015. Very deep convolutional networks for large-scale image recognition. In Proceedings of the 3rd International Conference on Learning Representations. 1–14. Retrieved from http://arxiv.org/pdf/1409.1556v6.Google Scholar
[54] Szegedy Christian, Vanhoucke V., Ioffe S., Shlens Jon, and Wojna Z.. 2016. Rethinking the inception architecture for computer vision. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. DOI:Google ScholarCross Ref
[55] Talebi Hossein and Milanfar Peyman. 2018. NIMA: Neural Image Assessment. IEEE Transactions on Image Processing : A Publication of the IEEE Signal Processing Society 27, 8 (2018), 3998–4011. DOI:Google ScholarCross Ref
[56] Theeuwes Jan and Van der Stigchel Stefan. 2006. Faces capture attention: Evidence from inhibition of return. Visual Cognition 13, 6 (April 2006), 657–665. DOI:Google ScholarCross Ref
[57] Thielsch Meinald T., Blotenberg I., and Jaron R.. 2014. User evaluation of websites: From first impression to recommendation. Interacting with Computers 26, 1 (2014), 89–102. DOI:Google ScholarCross Ref
[58] Thielsch Meinald T., Haines Russell, and Flacke Leonie. 2019. Experimental investigation on the effects of interface aesthetics on user performance in different virtual tasks. PeerJ 7 (2019), e6516. DOI:Google ScholarCross Ref
[59] Thielsch Meinald T. and Hirschfeld Gerrit. 2019. Facets of website content. Human–Computer Interaction 34, 4 (2019), 279–327. DOI:Google ScholarCross Ref
[60] Thielsch Meinald T. and Hirschfeld Gerrit. 2021. Quick assessment of web content perceptions. International Journal of Human-Computer Interaction 37, 1 (2021), 68–80. DOI:Google ScholarCross Ref
[61] Thielsch Meinald T., Scharfen Jana, Masoudi Ehsan, and Reuter Meike. 2019. Visual aesthetics and performance. In Proceedings of the Mensch und Computer 2019. ACM, New York, NY, 199–210. DOI:Google ScholarDigital Library
[62] Tractinsky Noam. 2013. Visual aesthetics. In The Encyclopedia of Human-Computer Interaction (2nd ed.). Soegaard M. and Dam R. F. (Eds.). The Interaction Design Foundation.Google Scholar
[63] Tractinsky Noam, Cokhavi Avivit, Kirschenbaum Moti, and Sharfi Tal. 2006. Evaluating the consistency of immediate aesthetic perceptions of web pages. International Journal of Human-Computer Studies 64, 11 (2006), 1071–1083. DOI:Google ScholarDigital Library
[64] Tuch Alexandre N., Presslaber Eva E., Stöcklin Markus, Opwis Klaus, and Bargas-Avila Javier A.. 2012. The role of visual complexity and prototypicality regarding first impression of websites: Working towards understanding aesthetic judgments. International Journal of Human-Computer Studies 70, 11 (November 2012), 794–811. DOI:Google ScholarDigital Library
[65] Webwiki. 2020. Webwiki - Das Bewertungsportal für Websites. Retrieved May 18, 2020 from https://www.webwiki.de/.Google Scholar
[66] Wu Ou, Hu Weiming, and Gao Jun. 2011. Learning to predict the perceived visual quality of photos. In Proceedings of the IEEE International Conference on Computer Vision. IEEE, Piscataway, NJ, 225–232. DOI:Google ScholarDigital Library
[67] Yarkoni Tal and Westfall Jacob. 2017. Choosing prediction over explanation in psychology: Lessons from machine learning. Perspectives on Psychological Science 12, 6 (2017), 1100–1122. DOI:Google ScholarCross Ref
[68] Sam Zheng Xianjun, Chakraborty Ishani, Jeng-Weei Lin James, and Rauschenberger Robert. 2009. Correlating low-level image statistics with users - rapid aesthetic and affective judgments of web pages. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, New York, NY, 1. DOI:Google ScholarDigital Library

Index Terms

Predicting Rating Distributions of Website Aesthetics with Deep Learning for AI-Based Research
1. Computing methodologies
  1. Machine learning
    1. Machine learning approaches
      1. Neural networks
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI design and evaluation methods
  2. Interaction design
    1. Interaction design process and methods
      1. User interface design

Recommendations

Predicting users' first impressions of website aesthetics with a quantification of perceived visual complexity and colorfulness
CHI '13: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

Users make lasting judgments about a website's appeal within a split second of seeing it for the first time. This first impression is influential enough to later affect their opinions of a site's usability and trustworthiness. In this paper, we ...
Read More
Rating prediction of recommended item based on review deep learning and rating probability matrix factorization
Highlights
- Combines the Deep Learning for review texts and the Probability Matrix Factorization method for rating data to predict the rating of the recommended items ...
Abstract
With a sharp improvement in E-commerce and data, the precise rating prediction of recommended items under user preferences has been a hot research topic in the EC intelligence domain. The rating data Matrix Factorization based methods ...
Read More
The effects of web site aesthetics and shopping task on consumer online purchasing behavior
CHI EA '08: CHI '08 Extended Abstracts on Human Factors in Computing Systems

This research-in-progress investigates the effects of website aesthetics and shopping task type on consumer attitude towards electronic shopping sites. The conceptualization of website aesthetics is based on the two-dimensional structure, namely, ...
Read More

Comments

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Published in
ACM Transactions on Computer-Human Interaction Volume 30, Issue 3
June 2023
544 pages
ISSN:1073-0516
EISSN:1557-7325
DOI:10.1145/3604411
Editors:
Kristina Höök
KTH Royal Institute of Technology, Sweden
,
Kasper Hornbæk
University of Copenhagen, Denmark
Issue’s Table of Contents
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s).
Sponsors
In-Cooperation
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
- Published: 10 June 2023
- Online AM: 29 October 2022
- Accepted: 31 August 2022
- Revised: 17 August 2022
- Received: 30 June 2021
Published in tochi Volume 30, Issue 3

Permissions
Request permissions about this article.
Request Permissions

Check for updates
Author Tags
Website aesthetics
aesthetics prediction
deep learning
Qualifiers
- research-article
Conference
Funding Sources
Other Metrics
View Article Metrics

Article Metrics
- 0
  Total Citations
  View Citations
- 610
  Total Downloads
- Downloads (Last 12 months)447
- Downloads (Last 6 weeks)50
Other Metrics
View Author Metrics
Cited By
This publication has not been cited yet

PDF Format

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Full Text

View this article in Full Text.

View Full Text

Predicting Rating Distributions of Website Aesthetics with Deep Learning for AI-Based Research

ACM Transactions on Computer-Human Interaction

Abstract

REFERENCES

Cited By

Index Terms

Recommendations

Predicting users' first impressions of website aesthetics with a quantification of perceived visual complexity and colorfulness

Rating prediction of recommended item based on review deep learning and rating probability matrix factorization

The effects of web site aesthetics and shopping task on consumer online purchasing behavior