User perception of sentiment-integrated critiquing in recommender systems
Introduction
In recommender systems, critiquing has been recognized as a distinct feedback mechanism to solve the popular cold-start problem in high-investment product domains (e.g., digital cameras, laptops, cars, apartments) (Chen and Pu, 2012). As users in those domains are usually new and do not have well-defined, fixed preferences initially, the critiquing system has been targeted to elicit users’ preferences for product attributes on site and allow them to incrementally refine preferences through posting critiques on the recommended product (such as “I would like something cheaper” or “with higher optical zoom” if the product is a digital camera). In this way, the system is able to improve recommendations in the next interaction cycle. Thus, in such a system, the initial user preference model does not influence the accuracy of their decision. Rather, it is the subsequent process of incremental critiquing that assists users in making more informed and confident decisions. According to prior experiments (Chen, Pu, 2006, Chen, Pu, 2007b), for a user to finally reach her/his ideal product, a number of critiquing cycles are often required. The studies from the areas of decision theory and consumer behavior also show that users are likely to construct their preferences in a context-dependent and adaptive manner during the decision process (Payne, Bettman, Johnson, 1993, Payne, Bettman, Schkade, 1999, Tversky, Simonson, 1993), and a typical buyer has some latent constraints and preferences that s/he may only become aware as s/he sees more options (Pu, Faltings, 2000, Pu, Faltings, 2002).
However, though critiquing has been popularly adopted in preference-based recommender systems (Chen, Pu, 2007c, Chen, Pu, 2010), knowledge-based recommender systems (Burke, 2000, Burke, Hammond, Young, 1997), and conversational recommender systems (McCarthy, Reilly, McGinty, Smyth, 2005, Shimazu, 2002, Smyth, McGinty, Reilly, McCarthy, 2004), the current methods are mainly based on products’ static attribute values (such as a digital camera’s screen size, effectiveness pixels, optical zoom) to elicit users’ critiques. Little work has studied whether and how other customers’ reviews could be leveraged into the critiquing interface for aiding the current user to construct her/his preferences. For example, suppose a user initially does not know the meaning of “optical zoom” when she searches for a digital camera, but after seeing the review “Nice 38X optical zoom lens for capturing beautiful close-ups of faraway action”, she may be able to specify preference for not only the optical zoom’s static value (e.g., “”) but also its associated sentiment (e.g., “ > 3” if the sentiment is in the range [1,5] from “least negative” to “very positive”). It hence implies that product reviews could be potentially useful for users to learn from others customers’ experiences (Aciar, Zhang, Simoff, Debenham, 2007, Kim, Srivastava, 2007, Wu, Wu, Sun, Yang, 2013), and hence possibly increase their own product knowledge and preference certainty.
Therefore, in this article, we propose a novel critiquing method that particularly extracts feature sentiments (i.e., opinions the other customers have expressed on some specific features in their reviews) and integrates them with products’ static attribute values for users to perform critiques. The user’s preference model is hence built on both static values and sentiments, for the system to compute product utility and return that with the highest utility as the recommendation in each interaction cycle. In the experiment, we report results of two user studies: before-after and within-subjects, which compared our method with the traditional critiquing system (without considering feature sentiments) in two different experimental settings. Both studies validate the superior performance of our method in terms of improving user perceptions, such as their product knowledge, preference certainty, decision confidence, perceived information usefulness, and purchase intention.
The remainder content is organized as follows. We first introduce related work in two branches, critiquing-based recommender systems and review-based recommender systems (Section 2). We then describe our method, i.e., the sentiment-integrated critiquing, in Section 3, followed by two experiments’ setup, materials, participants, and results analysis in Sections 4 and 5. Finally, we summarize our major findings and discuss their practical implications to the research field (Sections 6 and 7).
Section snippets
Critiquing-based recommender systems
Earlier critiquing-based recommender system mainly focused on pro-actively producing a set of critiques for users to pick (called system-suggested critiques), as improvement on the current recommendation. For example, one typical system is FindMe (Burke, Hammond, Young, 1996, Burke, Hammond, Young, 1997), which allows users to critique the currently recommended apartment by selecting one of the system’s pre-designed tweaks like “cheaper”, “bigger”, “nicer”, “safer”. However, because its
Sentiment-integrated critiquing
In this section, we introduce a sentiment-integrated critiquing approach to addressing the above-mentioned related work’s limitations. First of all, let’s see how users usually interact with a critiquing-based recommender system (Fig. 3) (Chen and Pu, 2012):
- •
Step 1: the user is asked to first specify a reference product as the starting query, or give some specific value preferences for the product’s attributes (e.g., searching criteria for the digital camera’s price, screen size, optical zoom,
User studies
We performed two experiments to empirically measure the performance of our sentiment-integrated critiquing interface (Senti-CBRS). In this section, we describe study materials, two experiments’ setups, participants, evaluation criteria, and our hypotheses.
Results
The software IBM SPSS Statistics V22.0 was used for data analysis. To identify whether the observed differences between the two systems (Senti-CBRS and CBRS) are statistically significant or not, we mainly ran one-way repeated measures ANOVA on before-after experimental results, and two-way mixed ANOVA (Field, 2013) on within-subjects results. In more details, one-way repeated measures ANOVA enables us to take the system as the independent factor for comparing the same participants’ differences
Critiquing interface design
Given that traditional critiquing-based recommender systems mainly exploit products’ static attribute values to elicit users’ critiquing feedback, in this paper, we propose a sentiment-integrated critiquing method, which particularly incorporates feature sentiments as extracted from product reviews into the process of assisting users in formulating and refining their preferences. We have actually extended our previous work on hybrid critiquing system (Chen and Pu, 2007a), to integrate feature
Conclusions
In conclusion, we have developed a sentiment-integrated critiquing interface for recommender systems (Senti-CBRS), which particularly utilizes feature sentiments of product reviews to support users to make critiques. By means of before-after and within-subjects experiments, we compared Senti-CBRS with the traditional CBRS, which demonstrate the impact brought by feature sentiments on improving users’ decision quality and perceptions of the system’s competence.
As mentioned at the beginning, our
Acknowledgements
This research work was supported by Hong Kong Research Grants Council (RGC) under projects ECS/HKBU211912 and RGC/HKBU12200415, and the Fundamental Research Funds of Shandong University, China. We also thank all participants who took part in our experiments.
References (53)
- et al.
Experimental methods: between-subject and within-subject design
J. Econ. Behav. Organ.
(2012) - et al.
Preference-based clustering reviews for augmenting e-commerce recommendation
Knowl.-Based Syst.
(2013) - et al.
Experiments in dynamic critiquing
Proceedings of the 10th International Conference on Intelligent User Interfaces
(2005) - et al.
Informed recommender: basing recommendations on consumer product reviews
IEEE Intell. Syst.
(2007) - et al.
Mining association rules between sets of items in large databases
Proceedings of the 1993 ACM SIGMOD International Conference on Management of Data
(1993) Knowledge-based recommender systems
Encycloped. Lib. Inf. Sci.
(2000)- et al.
Knowledge-based navigation of complex information spaces
Proceedings of the National Conference on Artificial Intelligence (AAAI’96)
(1996) - et al.
The findme approach to assisted browsing
IEEE Exp.
(1997) - et al.
Recommender systems based on user reviews: the state of the art
User Model. User-Adapt. Interact.
(2015) - et al.
Evaluating critiquing-based recommender agents
Proceedings of the Twenty-first National Conference on Artificial Intelligence (AAAI’06)
(2006)
The evaluation of a hybrid critiquing system with preference-based recommendations organization
Proceedings of the 2007 ACM Conference on Recommender Systems
Hybrid critiquing-based recommender systems
Proceedings of the 12th International Conference on Intelligent User Interfaces (IUI’07)
Preference-based organization interfaces: aiding user critiques in recommender systems
Proceedings of International Conference on User Modeling (UM’07)
Experiments on the preference-based organization interface in recommender systems
ACM Trans. Comput.-Human Interact. (TOCHI)
Critiquing-based recommenders: survey and emerging trends
User Model. User-Adapt. Interact.
An eye-tracking study: implication to implicit critiquing feedback elicitation in recommender systems
Proceedings of the 2016 Conference on User Modeling Adaptation and Personalization (UMAP’16)
Sentimental product recommendation
Proceedings of the 7th ACM Conference on Recommender Systems
Opinionated product recommendation
Proceedings of the 21st International Conference on Case-Based Reasoning
Sentiwordnet: a publicly available lexical resource for opinion mining
Proceedings of the 5th Conference on Language Resources and Evaluation
WordNet: An Electronic Lexical Database
Discovering Statistics Using IBM SPSS Statistics
Recomment: towards critiquing-based recommendation with speech interaction
Proceedings of the 7th ACM Conference on Recommender Systems
Mining and summarizing customer reviews
Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining
Mining opinion features in customer reviews
Proceedings of the 19th National Conference on Artifical Intelligence
Opinionminer: a novel machine learning system for web opinion mining and extraction
Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining
Mining comparative sentences and relations
Proceedings of the 21st National Conference on Artificial Intelligence - Volume 2
Cited by (16)
Analysis of sentiment expressions for user-centered design
2021, Expert Systems with ApplicationsCitation Excerpt :Therefore, a key requirement for these methods is generating labeled training data (Liu, 2010a); once a labeled dataset is generated, any supervised learning method can be applied to solve the problem. In previous studies on sentiment analysis of user reviews, researchers mostly focused on the polarity of users’ opinions (i.e., whether or not the user liked the product) (Chen, Yan, & Wang, 2017; Dhaoui, Webster, & Tan, 2017; Fernández-Gavilanes, Álvarez-López, Juncal-Martínez, Costa-Montenegro, & Javier González-Castaño, 2016; Yi, Nasukawa, Bunescu, & Niblack, 1889; Ma & Chen, 2017; Pham & Le, 2018; Wu, Wu, Chang, Wu, & Huang, 2019; Zhai, Liu, Xu, & Jia, 2011; Zheng, Lin, Wang, Lin, & Song, 2014). These methods generate predictive models given labeled datasets, based on a variety of machine learning and deep learning techniques, with the objective of classifying sentiment (Alqaryouti, Siyam, Monem, & Shaalan, 2019; Karimi, Rossi, Prati, & Full, 2020; Li, Bing, Li, Lam, & Yang, 2018; Soh, Yu, Narayanan, Duraisamy, & Chen, 2019; Suryadi & Kim, 2019; Toh & Su, 2015; Xu, Zhang, Xin, & Yang, 2019; Xu, Liu, Shu, & Yu, 2018; Xu, Lin, Wang, Yin, & Wang, 2017; Yang, Zhang, Jiang, & Li, 2019; Zhang et al., 2018a).
A conversational recommender system for diagnosis using fuzzy rules
2020, Expert Systems with ApplicationsCitation Excerpt :Interestingly, the proposed system, in that situation, would be extended to be able to rise alerts about possible unnoticed signs. We also have in mind to incorporate techniques based on sentiment analysis for critiquing-based recommender systems in the line of Chen, Yan, and Wang (2019). Regarding how to take advantage of the hidden knowledge in the dataset in order to guide the conversational process, we consider that FCA can bring some interesting ideas, such as the use of the implicit knowledge contained in the concept lattice, the use of minimal generators, attribute exploration techniques, and other implication bases which may reduce the computational complexity of the task.
An analytic approach to separate users by introducing new combinations of initial centers of clustering
2020, Physica A: Statistical Mechanics and its ApplicationsCitation Excerpt :Using the critiquing system, the authors extracted users’ preferences regarding the qualities of the site’s products and indicated that a critiquing-based approach can heighten users’ confidence and knowledge about products and considerably increase the site users’ urge to buy products. These approaches [50,53] did not separate gray sheep users and treat all users equally. Where users’ explicit ranking information is available, the clustering approaches [32] functions better in separating gray sheep users.
New perspectives on gray sheep behavior in E-commerce recommendations
2020, Journal of Retailing and Consumer ServicesCitation Excerpt :However, as both the approaches, i.e, clustering and social networking approach for GS users’ identification uses only the ratings (which are usually sparse) to measure the similarity between the users, they are unable to capture the entire difference in taste and preferences of users. Relying solely on ratings also ignores the implicit tastes of each user that gets reflected in the reviews and opinion, and ignoring this rich source of information is a major shortcoming of existing work on RS (Bandari et al., 2012; Boratto et al., 2016b; Chen et al., 2019; McAuley and Leskovec, 2013). As users’ taste and preferences are manifestation of their personality (Hu and Pu, 2011), in order to more effectively identify the different kind of users based on their taste and preferences, we propose the usage of the psychographic constructs based user models to specifically identify these GS users.
Towards a big data framework for analyzing social media content
2019, International Journal of Information ManagementCitation Excerpt :There are many applications for ML and NLP data analysis. Some of the most relevant ones are: sentiment analysis, recommender systems and user reviews analysis (Alahmadi & Zeng, 2015; Araque, Corcuera-Platas, Sánchez-Rada, & Iglesias, 2017; Chen, Yan, & Wang, 2017). The output of this phase is a method that integrates ML algorithms and NLP packages to perform data analysis.
Knowledge-based recommender systems: overview and research directions
2024, Frontiers in Big Data