Abstract
Many data mining approaches aim at modelling and predicting human behaviour. An important quantity of interest is the quality of model-based predictions, e.g. for comparative analysis and finding a competition winner with best prediction performance. In real life, human beings meet their decisions with considerable uncertainty. Its assessment and resulting implications for the statistically evident evaluation of predictive models are in the main focus of this contribution. We identify relevant sources of uncertainty as well as the limited ability of its accurate measurement, propose an uncertainty-aware methodology for more evident evaluations of data mining approaches, and discuss its implications for existing quality assessment strategies. Specifically, our approach switches from common point-paradigm to more appropriate distribution-paradigm. The proposed methodology is exemplified in the context of recommender systems and their established metrics of prediction quality. The discussion is substantiated by comprehensive experiments with real users and large-scale simulations.
Similar content being viewed by others
Abbreviations
- Track: :
-
Empirical evaluation, Exploratory
References
ACM: Workshop on Recommendation Utitlity Evaluation: Beyond RMSE, vol. 9, Dublin, Ireland, September 2012
Amatriain, X., Pujol, J.M., Oliver, N.: I like It.. I like it not: evaluating user ratings noise in recommender systems. In: Houben, G.-J., McCalla, G., Pianesi, F., Zancanaro, M. (eds.) UMAP 2009. LNCS, vol. 5535, pp. 247–258. Springer, Heidelberg (2009). doi:10.1007/978-3-642-02247-0_24
Amatriain, X., Pujol, J.: Rate it again: increasing recommendation accuracy by user re-rating. In: Proceedings of the Third ACM Conference on Recommender Systems, pp. 173–180. ACM (2009)
Bobadilla, J., Ortega, F., Hernando, A., Gutiérrez, A.: Recommender systems survey. Knowl.-Based Syst. 46, 109–132 (2013)
Chan, F.K.: Miss distance-generalized variance non-central chi distribution. In: AAS/AIAA Space Flight Mechanics Meeting, pp. 11–175 (2011)
Döring, N., Bortz, J.: Forschungsmethoden und Evaluation in den Sozial- und Humanwissenschaften. Springer-Lehrbuch, 5th edn. Springer, Heidelberg (2016). doi:10.1007/978-3-642-41089-5
Grabe, M.: Grundriss der Generalisierten Gauß’schen Fehlerrechnung. Springer, Heidelberg (2011). doi:10.1007/978-3-642-17822-1
Henze, N.: Stochastik für Einsteiger. Springer Spektrum, Heidelberg (2013)
Herlocker, J.L.: Evaluating collaborative filtering recommender systems. ACM Trans. Inf. Syst. 22(1), 5–53 (2004)
Iannario, M.: Modelling uncertainty and overdispersion in ordinal data. Commun. Stat. - Theory Methods 43, 771–786 (2014)
Intraub, H.: Presentation rate and the representation of briefly glimpsed pictures in memory. J. Exp. Psychol. 6, 1–11 (1990)
JCGM: Guide to the expression of uncertainty in measurement. Technical report, BIPM (2008)
JCGM: Supplement 1 to the GUM - propagation of distributions using a Monte Carlo method. Technical report, BIPM (2008)
Ricci, F., Rokach, L., Shapira, B. (eds.): Recommender Systems Handbook. Springer, Boston (2015). doi:10.1007/978-1-4899-7637-6
Said, A., Jain, B.J., Narr, S., Plumbaum, T.: Users and noise: the magic barrier of recommender systems. In: Masthoff, J., Mobasher, B., Desmarais, M.C., Nkambou, R. (eds.) UMAP 2012. LNCS, vol. 7379, pp. 237–248. Springer, Heidelberg (2012). doi:10.1007/978-3-642-31454-4_20
Sauerbier, T.: Statistik für Wirtschaftswissenschaftler. Oldenbourg (2003)
Su, X., Khoshgoftaar, T.M.: A survey of collaborative filtering techniques. Adv. Artif. Intell., 01 (2009)
Vandekerckhove, J.: A cognitive latent variable model for the simultaneous analysis of behavioral and personality data. J. Math. Psychol. 60, 58–71 (2014)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Jasberg, K., Sizov, S. (2017). Assessment of Prediction Techniques: The Impact of Human Uncertainty. In: Bouguettaya, A., et al. Web Information Systems Engineering – WISE 2017. WISE 2017. Lecture Notes in Computer Science(), vol 10569. Springer, Cham. https://doi.org/10.1007/978-3-319-68783-4_8
Download citation
DOI: https://doi.org/10.1007/978-3-319-68783-4_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-68782-7
Online ISBN: 978-3-319-68783-4
eBook Packages: Computer ScienceComputer Science (R0)