Skip to main content

Advertisement

Springer Nature Link
Log in

A cross-site comparison of online review manipulation using Benford’s law

  • Published:
Electronic Commerce Research Aims and scope Submit manuscript

Abstract

There is a growing concern that online reviews are targets of systematic manipulation, and manipulated reviews serve purposes other than informing consumers. In this article, we report a cross-site comparison of the aggregate-level manipulation using Benford’s law to detect anomalies. Benford’s law states that digits in naturally occurring data follow a logarithmic distribution. Deviation from such distribution is considered as a sign of systematic manipulation. We empirically examine word-count distributions of reviews on a Chinese food delivery service platform (FDS), Dianping, Yelp, and Amazon. Our empirical analysis suggests, in general, word counts of online review contents do not obey Benford’s law, although Benford’s law holds among high-quality reviews. Deviation from Benford’s law is larger in emerging markets compared with mature online marketplaces. Further analyses reveal that positive reviews, especially positive and extreme reviews, exhibit more deviation from Benford’s law.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Notes

  1. For example, Amazon invests heavily in manual systems and automatic algorithms to identify fraudulent reviews. https://www.buzzfeednews.com/article/nicolenguyen/amazon-fake-review-problem.

  2. For example, Amazon prohibits third-party sellers from offering free discounts to consumers for exchanging reviews. https://www.digitalcommerce360.com/2016/10/07/amazon-tells-sellers-stop-offering-free-products-reviews/.

  3. Platforms set limits to the number of words contained in a review. For example, the FDS platform that we study has a limit of 500 characters, and Amazon sets 5000 as the ceiling of review word counts. Only 0.02% of FDS reviews in our dataset have more than 350 word counts, and 0.05% of Amazon reviews have more than 1000 word counts. The regulation aims to assure the function of the review systems rather than constrain review writing.

  4. To form a comparable sample with the FDS review data that covers reviews posted between July 2016 and September 2016, we randomly chose the latest reviews (submitted in August 2015) in the Amazon dataset.

  5. Consumers can click and access the detailed information of these suspected reviews from a separate page.

  6. No ground truth or algorithm can assure a review is truly manipulated or not in observational data. Reviews with helpful votes and invited reviews have been repeatedly used as high-quality reviews. Hence, we believe that these reviews are not subject to systematic manipulation. We further admit that it is better to conduct experiments. However, since the main purpose of the paper is to conduct the cross-sample analysis of the aggregate-level systematic manipulation, we leave this as a future research direction.

  7. These non-compliant promotional reviews are five-star reviews that violate Amazon’s guidelines. The guideline can be found on Amazon’s website.

  8. Due to the space limit, we present full tables in the appendix.

  9. Considering the smallest sample size is 6,930 for negative reviews on Dianping, we chose 6,500 as the sample size for all categories on the four sites.

  10. The number of review observations is greater than 900 on both sites in and after 2010, while less than 300 before 2010.

  11. For those years in which we have less than 3,000 observations, we randomly drew samples of 3,000 reviews with replacement. For the other years, we drew random samples without replacement.

  12. We thank an anonymous reviewer for pointing out that cultural differences could influence the effectiveness of manipulation detection methods. Benford’s law has been tested in different cultures, which strengthens its credibility in cross-cultural studies. However, it is an interesting extension to conduct more comprehensive empirical tests of Benford’s law applicability in different cultures. We do not include such tests in the paper due to data availability limitations.

References

  1. Kwark, Y., Chen, J., & Raghunathan, S. (2014). Online product reviews: Implications for retailers and competing manufacturers. Information Systems Research, 25(1), 93–110.

    Article  Google Scholar 

  2. Duan, W., Gu, B., & Whinston, A. B. (2008). Do online reviews matter? An empirical investigation of panel data. Decision Support Systems, 45(4), 1007–1016.

    Article  Google Scholar 

  3. Chevalier, J. A., & Mayzlin, D. (2006). The effect of word of mouth on sales: Online book reviews. Journal of Marketing Research, 43(3), 345–354.

    Article  Google Scholar 

  4. Forman, C., Ghose, A., & Wiesenfeld, B. (2008). Examining the relationship between reviews and sales: The role of reviewer identity disclosure in electronic markets. Information Systems Research, 19(3), 291–313.

    Article  Google Scholar 

  5. Wang, C., Zhang, X., & Hann, I.-H. (2018). Socially nudged: A quasi-experimental study of friends’ social influence in online product ratings. Information Systems Research, 29(3), 641–655.

    Article  Google Scholar 

  6. Luca, M., & Zervas, G. (2016). Fake it till you make it: Reputation, competition, and Yelp review fraud. Management Science, 62(12), 3412–3427.

    Article  Google Scholar 

  7. Mayzlin, D., Dover, Y., & Chevalier, J. (2014). Promotional reviews: An empirical investigation of online review manipulation. American Economic Review, 104(8), 2421–2455.

    Article  Google Scholar 

  8. Ghose, A., & Ipeirotis, P. G. (2011). Estimating the helpfulness and economic impact of product reviews: Mining text and reviewer characteristics. IEEE Transactions on Knowledge and Data Engineering, 23(10), 1498–1512.

    Article  Google Scholar 

  9. Susan, M. M., & David, S. (2010). What makes a helpful online review? A study of customer reviews on Amazon.com. MIS Quarterly, 34(1), 185–200.

    Article  Google Scholar 

  10. Ott, M., Choi, Y., Cardie, C., & Hancock, J. T. (2011). Finding deceptive opinion spam by any stretch of the imagination. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, 1, 309–319.

  11. Kumar, N., Venugopal, D., Qiu, L., & Kumar, S. (2019). Detecting anomalous online reviewers: An unsupervised approach using mixture models. Journal of Management Information Systems, 36(4), 1313–1346.

    Article  Google Scholar 

  12. Mukherjee, A., Kumar, A., Liu, B., Wang, J., Hsu, M., Castellanos, M., & Ghosh, R. (2013). Spotting opinion spammers using behavioral footprints. In Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 19, 632–640.

  13. Liang, N., Biros, D. P., & Luse, A. (2016). An empirical validation of malicious insider characteristics. Journal of Management Information Systems, 33(2), 361–392.

    Article  Google Scholar 

  14. Jindal, N., & Liu, B. Opinion spam and analysis. (2008). In Proceedings of the 2008 International Conference on Web Search and Data Mining, 219–230: ACM.

  15. Akoglu, L., Chandy, R., & Faloutsos, C. (2013). Opinion fraud detection in online reviews by network effects. In Proceedings of the 7th international AAAI Conference on Weblogs and Social Media, 7, 2–11.

  16. Rayana, S., & Akoglu, L. (2015). Collective opinion spam detection: Bridging review networks and metadata. In Proceedings of the 21st ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 21, 985–994.

  17. Wang, G., Xie, S., Liu, B., & Philip, S. Y. (2011). Review graph based online store review spammer detection. In Proceedings of the 11th IEEE International Conference on Data Mining, 11, 1242–1247.

  18. Judge, G., & Schechter, L. (2009). Detecting problems in survey data using Benford’s law. Journal of Human Resources, 44(1), 1–24.

    Article  Google Scholar 

  19. Skousen, C. J., Guan, L., & Wetzel, T. S. (2004). Anomalies and unusual patterns in reported earnings: Japanese managers round earnings. Journal of International Financial Management & Accounting, 15(3), 212–234.

    Article  Google Scholar 

  20. Rodriguez, R. J. (2004). First significant digit patterns from mixtures of uniform distributions. The American Statistician, 58(1), 64–71.

    Article  Google Scholar 

  21. Durtschi, C., Hillison, W., & Pacini, C. (2004). The effective use of Benford’s law to assist in detecting fraud in accounting data. Journal of Forensic Accounting, 5(1), 17–34.

    Google Scholar 

  22. Guan, L. M., Lin, F. Y., & Fang, W. C. (2008). Goal-oriented earnings management: Evidence from Taiwanese firms. Emerging Markets Finance and Trade, 44(4), 19–32.

    Article  Google Scholar 

  23. Lin, F., Zhao, L., & Guan, L. (2014). Window dressing in reported earnings: A comparison of high-tech and low-tech companies. Emerging Markets Finance and Trade, 50(sup1), 254–264.

    Article  Google Scholar 

  24. Lin, F., Lin, L.-J., Yeh, C.-C., & Wang, T.-S. (2018). Does the board of directors as Fat Cats exert more earnings management? Evidence from Benford’s law. The Quarterly Review of Economics and Finance, 68, 158–170.

    Article  Google Scholar 

  25. Chen, Z., & Lurie, N. H. (2013). Temporal contiguity and negativity bias in the impact of online word of mouth. Journal of Marketing Research, 50(4), 463–476.

    Article  Google Scholar 

  26. Yin, D., Mitra, S., & Zhang, H. (2016). When do consumers value positive vs. negative reviews? An empirical investigation of confirmation bias in online word of mouth. Information Systems Research, 27(1), 131–144.

    Article  Google Scholar 

  27. Shihab, M. R., & Putri, A. P. (2019). Negative online reviews of popular products: Understanding the effects of review proportion and quality on consumers’ attitude and intention to buy. Electronic Commerce Research, 19(1), 159–187.

    Article  Google Scholar 

  28. Godes, D., & Mayzlin, D. (2004). Using online conversations to study word-of-mouth communication. Marketing Science, 23(4), 545–560.

    Article  Google Scholar 

  29. Chintagunta, P. K., Gopinath, S., & Venkataraman, S. (2010). The effects of online user reviews on movie box office performance: Accounting for sequential rollout and aggregation across local markets. Marketing Science, 29(5), 944–957.

    Article  Google Scholar 

  30. Dellarocas, C., Zhang, X. M., & Awad, N. F. (2007). Exploring the value of online product reviews in forecasting sales: The case of motion pictures. Journal of Interactive Marketing, 21(4), 23–45.

    Article  Google Scholar 

  31. Duan, W., Gu, B., & Whinston, A. B. (2008). The dynamics of online word-of-mouth and product sales—An empirical investigation of the movie industry. Journal of Retailing, 84(2), 233–242.

    Article  Google Scholar 

  32. Gu, B., Park, J., & Konana, P. (2012). The impact of external word-of-mouth sources on retailer sales of high-involvement products. Information Systems Research, 23(1), 182–196.

    Article  Google Scholar 

  33. Zimmermann, S., Herrmann, P., Kundisch, D., & Nault, B. R. (2018). Decomposing the variance of consumer ratings and the impact on price and demand. Information Systems Research, 29(4), 984–1002.

    Article  Google Scholar 

  34. Cheng, L.-C., & Huang, C.-L. (2019). Exploring contextual factors from consumer reviews affecting movie sales: An opinion mining approach. Electronic Commerce Research, 1–26.

  35. Chen, Y., & Xie, J. (2008). Online consumer review: Word-of-mouth as a new element of marketing communication mix. Management Science, 54(3), 477–491.

    Article  Google Scholar 

  36. Hollenbeck, B., Moorthy, S., & Proserpio, D. (2019). Advertising strategy in the presence of reviews: An empirical analysis. Marketing Science, 38(5), 793–811.

    Article  Google Scholar 

  37. Lovett, M. J., Peres, R., & Shachar, R. (2013). On brands and word of mouth. Journal of Marketing Research, 50(4), 427–444.

    Article  Google Scholar 

  38. Li, X., & Hitt, L. M. (2008). Self-selection and information role of online product reviews. Information Systems Research, 19(4), 456–474.

    Article  Google Scholar 

  39. Dellarocas, C., & Wood, C. A. (2008). The sound of silence in online feedback: Estimating trading risks in the presence of reporting bias. Management Science, 54(3), 460–476.

    Article  Google Scholar 

  40. Hu, N., Pavlou, P. A., & Zhang, J. (2009). Overcoming the J-shaped distribution of product reviews. Communications of the ACM, 52(10), 144–147.

    Article  Google Scholar 

  41. Godes, D., & Silva, J. C. (2012). Sequential and temporal dynamics of online opinion. Marketing Science, 31(3), 448–473.

    Article  Google Scholar 

  42. Moe, W. W., & Trusov, M. (2011). The value of social dynamics in online product ratings forums. Journal of Marketing Research, 48(3), 444–456.

    Article  Google Scholar 

  43. Goes, P. B., Lin, M., & Au Yeung, C. (2014). “Popularity effect” in user-generated content: Evidence from online product reviews. Information Systems Research, 25(2), 222–238.

    Article  Google Scholar 

  44. Lee, Y.-J., Hosanagar, K., & Tan, Y. (2015). Do I follow my friends or the crowd? Information cascades in online movie ratings. Management Science, 61(9), 2241–2258.

    Article  Google Scholar 

  45. Burtch, G., Hong, Y., Bapna, R., & Griskevicius, V. (2017). Stimulating online reviews by combining financial incentives and social norms. Management Science, 64(5), 2065–2082.

    Article  Google Scholar 

  46. Huang, N., Hong, Y., & Burtch, G. (2016). Social network integration and user content generation: Evidence from natural experiments. MIS Quarterly, 17–001.

  47. Ho, Y.-C., Wu, J., & Tan, Y. (2017). Disconfirmation effect on online rating behavior: A structural model. Information Systems Research, 28(3), 626–642.

    Article  Google Scholar 

  48. Chen, P.-Y., Hong, Y., & Liu, Y. (2017). The value of multidimensional rating systems: Evidence from a natural experiment and randomized experiments. Management Science, 64(10), 4629–4647.

    Article  Google Scholar 

  49. Wang, H., Du, R., Li, J., & Fan, W. (2018). Subdivided or aggregated online review systems: Which is better for online takeaway vendors? Electronic Commerce Research, 1–30.

  50. Kumar, N., Qiu, L., & Kumar, S. (2018). Exit, voice, and response on digital platforms: An empirical investigation of online management response strategies. Information Systems Research, 29(4), 849–870.

    Article  Google Scholar 

  51. Lappas, T., Sabnis, G., & Valkanas, G. (2016). The impact of fake reviews on online visibility: A vulnerability assessment of the hotel industry. Information Systems Research, 27(4), 940–961.

    Article  Google Scholar 

  52. Zhao, Y., Yang, S., Narayan, V., & Zhao, Y. (2013). Modeling consumer learning from online product reviews. Marketing Science, 32(1), 153–169.

    Article  Google Scholar 

  53. Khern-am-nuai, W., Kannan, K., & Ghasemkhani, H. (2018). Extrinsic versus intrinsic rewards for contributing reviews in an online platform. Information Systems Research, 29(4), 871–892.

    Article  Google Scholar 

  54. Sun, Y., Dong, X., & McIntyre, S. (2017). Motivation of user-generated content: Social connectedness moderates the effects of monetary rewards. Marketing Science, 36(3), 329–337.

    Article  Google Scholar 

  55. Lin, Z., Zhang, Y., & Tan, Y. (2019). An empirical study of free product sampling and rating bias. Information Systems Research, 30(1), 260–275.

    Article  Google Scholar 

  56. Nunamaker, J. F., Jr., Burgoon, J. K., & Giboney, J. S. (2016). Information systems for deception detection. Journal of Management Information Systems, 33(2), 327–331.

    Article  Google Scholar 

  57. Zhang, D., Zhou, L., Kehoe, J. L., & Kilic, I. Y. (2016). What online reviewer behaviors really matter? Effects of verbal and nonverbal behaviors on detection of fake online reviews. Journal of Management Information Systems, 33(2), 456–481.

    Article  Google Scholar 

  58. Kumar, N., Venugopal, D., Qiu, L., & Kumar, S. (2018). Detecting review manipulation on online platforms with hierarchical supervised learning. Journal of Management Information Systems, 35(1), 350–380.

    Article  Google Scholar 

  59. Hu, N., Bose, I., Koh, N. S., & Liu, L. (2012). Manipulation of online reviews: An analysis of ratings, readability, and sentiments. Decision Support Systems, 52(3), 674–684.

    Article  Google Scholar 

  60. Saumya, S., & Singh, J. P. (2020). Spam review detection using LSTM autoencoder: An unsupervised approach. Electronic Commerce Research, 1–21.

  61. Feng, S., Xing, L., Gogar, A., & Choi, Y. (2012). Distributional footprints of deceptive product reviews. In Proceedings of the 6th International AAAI Conference on Weblogs and Social Media, 12(98), 105.

  62. Newcomb, S. (1881). Note on the frequency of use of the different digits in natural numbers. American Journal of Mathematics, 4(1), 39–40.

    Article  Google Scholar 

  63. Benford, F. (1938). The law of anomalous numbers. Proceedings of the American Philosophical Society, 551–572.

  64. Boyle, J. (1994). An application of Fourier series to the most significant digit problem. The American Mathematical Monthly, 101(9), 879–886.

    Article  Google Scholar 

  65. Hill, T. P. (1995). A statistical derivation of the significant-digit law. Statistical Science, 10(4), 354–363.

    Article  Google Scholar 

  66. Hill, T. P. (1995). The significant-digit phenomenon. The American Mathematical Monthly, 102(4), 322–327.

    Article  Google Scholar 

  67. Nigrini, M. J. (1996). A taxpayer compliance application of Benford’s law. The Journal of the American Taxation Association, 18(1), 72.

    Google Scholar 

  68. Nigrini, M. J., & Mittermaier, L. J. (1997). The use of Benford’s law as an aid in analytical procedures. Auditing-a Journal of Practice & Theory, 16(2), 52–67.

    Google Scholar 

  69. Jiang, L. C., Bazarova, N. N., & Hancock, J. T. (2013). From perception to behavior: Disclosure reciprocity and the intensification of intimacy in computer-mediated communication. Communication Research, 40(1), 125–143.

    Article  Google Scholar 

  70. Cao, Z., Hui, K.-L., & Xu, H. (2018). When discounts hurt sales: The case of daily-deal markets. Information Systems Research, 29(3), 567–591.

    Article  Google Scholar 

  71. Xiao, S., Tan, X., Dong, M., & Qi, J. (2014). How to design your project in the online crowdfunding market? Evidence from Kickstarter. In Proceedings of the 35th International Conference on Information Systems.

  72. Herbig, P. A., & Kramer, H. (1994). The effect of information overload on the innovation choice process. Journal of Consumer Marketing., 11(2), 45–54.

    Article  Google Scholar 

  73. Nigrini, M. J. (2012). Benford’s law: Applications for forensic accounting, auditing and fraud detection (Vol. 586). Hoboken: Wiley.

    Book  Google Scholar 

  74. Giles, D. E. (2007). Benford’s law and naturally occurring prices in certain eBay auctions. Applied Economics Letters, 14(3), 157–161.

    Article  Google Scholar 

  75. Nelson, P. (1970). Information and consumer behavior. Journal of Political Economy, 78(2), 311–329.

    Article  Google Scholar 

  76. Nelson, P. (1974). Advertising as information. Journal of Political Economy, 82(4), 729–754.

    Article  Google Scholar 

  77. Pu, J., Han, S., Kwark, Y., Gu, B., & Ye, Q. (2017). The double-edged sword of expert reviewer programs: The effects of offering expert reviewer status on review generation. In Proceedings of the 38th International Conference on Information Systems.

  78. Hong, Y., Huang, N., Burtch, G., & Li, C. (2016). Culture, conformity, and emotional suppression in online reviews. Journal of the Association for Information Systems, 17(11), 737–758.

    Article  Google Scholar 

  79. Koh, N. S., Hu, N., & Clemons, E. K. (2010). Do online reviews reflect a product’s true perceived quality? An investigation of online movie reviews across cultures. Electronic Commerce Research and Applications, 9(5), 374–385.

    Article  Google Scholar 

  80. Wang, Y., Wang, Z., Zhang, D., & Zhang, R. (2019). Discovering cultural differences in online consumer product reviews. Journal of Electronic Commerce Research, 20(3), 169–183.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Guanghua School of Management, Peking University, Yiheyuan Road, 5, Haidian District, Beijing, 100871, People’s Republic of China

    Cheng Zhao & Chong Alex Wang

Authors
  1. Cheng Zhao
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Chong Alex Wang
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Chong Alex Wang.

Ethics declarations

Conflict of interest

On behalf of all authors, the corresponding author states that there is no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendices

Appendix 1

The following tables show the types, impacts, and detection methods of online review manipulation.

See Table

Table 9 Different types of review manipulation
Full size table

9. Table 9 summarizes different types of review manipulation attempts and discusses their impacts.

See Table

Table 10 Different methods to detect review manipulation
Full size table

10. Table 10 presents a summary of the existing methods for review manipulation detection.

Appendix 2

The following figures show the distributions of review ratings and word counts in the four datasets.

See Fig.

Fig. 5
figure 5

The distributions of review ratings of reviews on the four sites

Full size image

5.

See Fig.

Fig. 6
figure 6

The distributions of word counts of reviews on the four sites

Full size image

6.

Appendix 3

Tables report test results using character counts to measure review length.

See Table 12. Table 12 reports Benford’s law tests for high-quality and low-quality reviews on Amazon, corresponding to Table 6.

See Table 13. Table 13 reports review manipulation in different categories, corresponding to Table 7.

See Table 14. Table 14 reports the interaction between extremity and valence results, corresponding to Table 8.

See Table 11. Table 11 reports review manipulation on different sites, corresponding to Table 5.

Table 11 Review manipulation on different sites using character countsa
Full size table
Table 12 Benford’s law tests: high-quality versus low-quality reviews on Amazon using character countsa
Full size table
Table 13 Review manipulation in different categories using character counts
Full size table
Table 14 Interaction of extremity and valence using character counts
Full size table

Appendix 4

Tables report all four statistical tests for Benford’s law violation.

See Table 15. Table 15 reports review manipulation in different categories, corresponding to Table 7.

Table 15 Review manipulation in different categories of all statistical testsa
Full size table

See Table 16. Table 16 reports the interaction between extremity and valence results, corresponding to Table 8.

Table 16 Interaction between extremity and valence of all statistical testsa
Full size table

See Table 17. Table 17 reports review manipulation in different categories using character counts as the review length measure, corresponding to Table 13.

Table 17 Review manipulation in different categories using character counts of all statistical testsa
Full size table

See Table 18. Table 18 reports the interaction between extremity and valence results using character counts as the review length measure, corresponding to Table 14.

Table 18 Interaction between extremity and valence using character counts of all statistical testsa
Full size table

Appendix 5

The table reports the result using an alternative measure of review negativity.

See Table 19. Table 1919 reports review manipulation in different categories using an alternative measure of review negativity, corresponding to Table 7.

Table 19 Review manipulation in different categories using an alternative measure of review negativitya
Full size table

Appendix 6

The table reports the result of controlling for the same sample size among different sub-categories.

See Table 20. Table 20 reports review manipulation in different categories using the same sample size, corresponding to Table 7.

Table 20 Review manipulation in different categories using the sample sizea
Full size table

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhao, C., Wang, C.A. A cross-site comparison of online review manipulation using Benford’s law. Electron Commer Res 23, 365–406 (2023). https://doi.org/10.1007/s10660-020-09455-8

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10660-020-09455-8

Keywords