Abstract:
Flexible cultural consensus theory (FlexCCT) provides an integrated framework and tool set for analyzing and aggregating ratings. It utilizes a likelihood-based statistic...Show MoreMetadata
Abstract:
Flexible cultural consensus theory (FlexCCT) provides an integrated framework and tool set for analyzing and aggregating ratings. It utilizes a likelihood-based statistical model to create aggregate ratings weighted for rater competencies and rater biases. It has features for the analysis of multiple rating cultures and for consensus adjusted reliability. Multiple optimization algorithms are implemented for FlexCCT, along with a range of model identifiability options to restrict certain subsets of the model parameters. Bootstrapping- and jackknifing-based methods are implemented to give confidence intervals for parameters. A holdout validation method that generates a measure of “prediction log-likelihood” allows for the testing of solution reliability and stability. Empirical work demonstrates the utility of FlexCCT on unsupervised wisdom of the crowd problems, and an example shows how FlexCCT can be used to analyze educational grading data.
Published in: IEEE Transactions on Computational Social Systems ( Volume: 5, Issue: 2, June 2018)