Christopher G. Harris
ABSTRACT
When asked to rank or rate a list of items, humans are often affected by cognitive biases, which may lead to inconsistent decisions over time. These inconsistencies become part of machine learning prediction algorithms trained on human judgments, leading to misalignment and consequently affecting the metrics used to evaluate their correctness. In this paper, we propose new accuracy metrics, built upon commonly used statistics- and decision support-based metrics. Each of these metrics is designed to address the varying nature of human judgment and to evaluate the importance of decisions that change over time due to cognitive biases.
- Huber, J., Payne, J. W., & Puto, C. (1982). Adding asymmetrically dominated alternatives: Violations of regularity and the similarity hypothesis. Journal of consumer research, 9(1), 90-98.Google Scholar
Cross Ref
- Kahneman, D., Slovic, S. P., Slovic, P., & Tversky, A. (Eds.). (1982). Judgment under uncertainty: Heuristics and biases. Cambridge university press.Google ScholarCross Ref
- Kahneman, D. 2011. Thinking, fast and slow. New York, NY: Farrar, Straus, and Giroux.Google Scholar
- Pohl, R., & Pohl, R. F. (Eds.). (2004). Cognitive illusions: A handbook on fallacies and biases in thinking, judgment and memory. Psychology Press.Google Scholar
- Wilson, T. D., Houston, C. E., Etling, K. M., & Brekke, N. (1996). A new look at anchoring effects: basic anchoring and its antecedents. Journal of Experimental Psychology: General, 125(4), 387.Google ScholarCross Ref
- Lathia, N., Hailes, S., Capra, L., & Amatriain, X. (2010, July). Temporal diversity in recommender systems. In Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval (pp. 210-217).Google ScholarDigital Library
- Liu, F., Lee, J., & Shroff, N. (2018, April). A change-detection based framework for piecewise-stationary multi-armed bandit problem. In Thirty-Second AAAI Conference on Artificial Intelligence.Google Scholar
- Wu, Q., Iyer, N., & Wang, H. (2018, June). Learning contextual bandits in a non-stationary environment. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval (pp. 495-504).Google ScholarDigital Library
- Lefortier, D., Serdyukov, P., & De Rijke, M. (2014, November). Online exploration for detecting shifts in fresh intent. In Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management (pp. 589-598).Google ScholarDigital Library
- Whittle, P. (1988). Restless bandits: Activity allocation in a changing world. Journal of applied probability, 25(A), 287-298.Google ScholarCross Ref
- Tourangeau, R., Schwarz, N., & Sudman, S. (1992). Context effects on attitude responses: The role of retrieval and memory structures. Context effects in social and psychological research, 35-47.Google Scholar
- Kahneman, D., Slovic, S. P., Slovic, P., & Tversky, A. (Eds.). (1982). Judgment under uncertainty: Heuristics and biases. Cambridge university press.Google ScholarCross Ref
- Jagerman, R., Markov, I., & de Rijke, M. (2019). When people change their mind: Off-policy evaluation in non-stationary recommendation environments. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining (pp. 447-455).Google ScholarDigital Library
- Thomas, P. S., Theocharous, G., Ghavamzadeh, M., Durugkar, I., & Brunskill, E. (2017). Predictive off-policy policy evaluation for nonstationary decision problems, with applications to digital marketing. In Twenty-Ninth IAAI Conference.Google Scholar
- Adomavicius, G., & Tuzhilin, A. (2005). Toward the next generation of recommender systems: A survey of the state-of-the-art and possible extensions. IEEE transactions on knowledge and data engineering, 17(6), 734-749.Google ScholarDigital Library
Index Terms
- Methods to Evaluate Temporal Cognitive Biases in Machine Learning Prediction Models
Recommendations
Mitigating Cognitive Biases in Machine Learning Algorithms for Decision Making
WWW '20: Companion Proceedings of the Web Conference 2020Cognitive biases are an ingrained part of the human decision-making process. Nearly all machine learning algorithms that mimic human decision-making use human judgments as training data, which propagates these biases. In this paper, we conduct an ...
Personalizing with Human Cognitive Biases
UMAP'19 Adjunct: Adjunct Publication of the 27th Conference on User Modeling, Adaptation and PersonalizationHuman cognitive biases are numerous and well established. Due to inherent limitations in our knowledge of the world, and computational constraints, our judgments and decisions do not rigidly adhere to the principle of maximizing expected utility. We ...
Detecting Byzantine attack in cognitive radio networks using machine learning
AbstractOne primary function in a cognitive radio network (CRN) is spectrum sensing. In an infrastructure-based CRN, instead of individual nodes independently sensing the presence of the incumbent signal and taking decisions thereon, a fusion center (FC) ...
Comments