research-article

Methods to Evaluate Temporal Cognitive Biases in Machine Learning Prediction Models

Author:

Christopher G. HarrisAuthors Info & Claims

WWW '20: Companion Proceedings of the Web Conference 2020

Pages 572 - 575

https://doi.org/10.1145/3366424.3383418

Published: 20 April 2020 Publication History

Abstract

When asked to rank or rate a list of items, humans are often affected by cognitive biases, which may lead to inconsistent decisions over time. These inconsistencies become part of machine learning prediction algorithms trained on human judgments, leading to misalignment and consequently affecting the metrics used to evaluate their correctness. In this paper, we propose new accuracy metrics, built upon commonly used statistics- and decision support-based metrics. Each of these metrics is designed to address the varying nature of human judgment and to evaluate the importance of decisions that change over time due to cognitive biases.

References

[1]

Huber, J., Payne, J. W., & Puto, C. (1982). Adding asymmetrically dominated alternatives: Violations of regularity and the similarity hypothesis. Journal of consumer research, 9(1), 90-98.

[2]

Kahneman, D., Slovic, S. P., Slovic, P., & Tversky, A. (Eds.). (1982). Judgment under uncertainty: Heuristics and biases. Cambridge university press.

[3]

Kahneman, D. 2011. Thinking, fast and slow. New York, NY: Farrar, Straus, and Giroux.

[4]

Pohl, R., & Pohl, R. F. (Eds.). (2004). Cognitive illusions: A handbook on fallacies and biases in thinking, judgment and memory. Psychology Press.

[5]

Wilson, T. D., Houston, C. E., Etling, K. M., & Brekke, N. (1996). A new look at anchoring effects: basic anchoring and its antecedents. Journal of Experimental Psychology: General, 125(4), 387.

[6]

Lathia, N., Hailes, S., Capra, L., & Amatriain, X. (2010, July). Temporal diversity in recommender systems. In Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval (pp. 210-217).

Digital Library

[7]

Liu, F., Lee, J., & Shroff, N. (2018, April). A change-detection based framework for piecewise-stationary multi-armed bandit problem. In Thirty-Second AAAI Conference on Artificial Intelligence.

[8]

Wu, Q., Iyer, N., & Wang, H. (2018, June). Learning contextual bandits in a non-stationary environment. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval (pp. 495-504).

Digital Library

[9]

Lefortier, D., Serdyukov, P., & De Rijke, M. (2014, November). Online exploration for detecting shifts in fresh intent. In Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management (pp. 589-598).

Digital Library

[10]

Whittle, P. (1988). Restless bandits: Activity allocation in a changing world. Journal of applied probability, 25(A), 287-298.

[11]

Tourangeau, R., Schwarz, N., & Sudman, S. (1992). Context effects on attitude responses: The role of retrieval and memory structures. Context effects in social and psychological research, 35-47.

[12]

Kahneman, D., Slovic, S. P., Slovic, P., & Tversky, A. (Eds.). (1982). Judgment under uncertainty: Heuristics and biases. Cambridge university press.

[13]

Jagerman, R., Markov, I., & de Rijke, M. (2019). When people change their mind: Off-policy evaluation in non-stationary recommendation environments. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining (pp. 447-455).

Digital Library

[14]

Thomas, P. S., Theocharous, G., Ghavamzadeh, M., Durugkar, I., & Brunskill, E. (2017). Predictive off-policy policy evaluation for nonstationary decision problems, with applications to digital marketing. In Twenty-Ninth IAAI Conference.

[15]

Adomavicius, G., & Tuzhilin, A. (2005). Toward the next generation of recommender systems: A survey of the state-of-the-art and possible extensions. IEEE transactions on knowledge and data engineering, 17(6), 734-749.

Digital Library

Cited By

Almuzaini ABhatt CPennock DSingh V(2022)ABCinML: Anticipatory Bias Correction in Machine Learning ApplicationsProceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency10.1145/3531146.3533211(1552-1560)Online publication date: 21-Jun-2022
https://dl.acm.org/doi/10.1145/3531146.3533211

Index Terms

Methods to Evaluate Temporal Cognitive Biases in Machine Learning Prediction Models
1. Computing methodologies
  1. Machine learning
    1. Learning paradigms
      1. Supervised learning
    2. Machine learning approaches
2. Information systems

Index terms have been assigned to the content through auto-classification.

Recommendations

Mitigating Cognitive Biases in Machine Learning Algorithms for Decision Making
WWW '20: Companion Proceedings of the Web Conference 2020

Cognitive biases are an ingrained part of the human decision-making process. Nearly all machine learning algorithms that mimic human decision-making use human judgments as training data, which propagates these biases. In this paper, we conduct an ...
Detecting Byzantine attack in cognitive radio networks using machine learning
Abstract
One primary function in a cognitive radio network (CRN) is spectrum sensing. In an infrastructure-based CRN, instead of individual nodes independently sensing the presence of the incumbent signal and taking decisions thereon, a fusion center (FC) ...
Cognitive Biases in Visualizations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

WWW '20: Companion Proceedings of the Web Conference 2020

April 2020

854 pages

ISBN:9781450370240

DOI:10.1145/3366424

Editors:
Amal El Fallah Seghrouchni
Sorbonne University, France
,
Gita Sukthankar
University of Central Florida, United States
,
Tie-Yan Liu
Microsoft Research Asia, China
,
Maarten van Steen
University of Twente, Netherlands

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGWEB: ACM Special Interest Group on Hypertext, Hypermedia, and Web

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 April 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

WWW '20

Sponsor:

SIGWEB

WWW '20: The Web Conference 2020

April 20 - 24, 2020

Taipei, Taiwan

Acceptance Rates

Overall Acceptance Rate 1,899 of 8,196 submissions, 23%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
204
Total Downloads

Downloads (Last 12 months)16
Downloads (Last 6 weeks)1

Reflects downloads up to 02 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Almuzaini ABhatt CPennock DSingh V(2022)ABCinML: Anticipatory Bias Correction in Machine Learning ApplicationsProceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency10.1145/3531146.3533211(1552-1560)Online publication date: 21-Jun-2022
https://dl.acm.org/doi/10.1145/3531146.3533211

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Table of Conten