Yuqing Kong
Skip Abstract Section
Abstract
1 In the setting where participants are asked multiple similar possibly subjective multi-choice questions (e.g., Do you like Panda Express? Y/N; Do you like Chick-fil-A? Y/N), a series of peer prediction mechanisms have been designed to incentivize honest reports and some of them achieve dominantly truthfulness: Truth-telling is a dominant strategy and strictly dominates other “non-permutation strategy” with some mild conditions. However, those mechanisms require the participants to perform an infinite number of tasks. When the participants perform a finite number of tasks, these mechanisms only achieve approximated dominant truthfulness. The existence of a dominantly truthful multi-task peer prediction mechanism that only requires a finite number of tasks remains to be an open question that may have a negative result, even with full prior knowledge.
This article answers this open question by proposing a family of mechanisms, VMI-Mechanisms, that are dominantly truthful with a finite number of tasks. A special case of this family, DMI-Mechanism, only requires ≥ 2C tasks where C is the number of choices for each question (C=2 for binary-choice questions). The implementation of these mechanisms does not require any prior knowledge (detail-free) and only requires ≥ 2 participants. To the best of our knowledge, any mechanism of the family is the first dominantly truthful peer prediction mechanism that works for a finite number of tasks.
The core of these new mechanisms is a new family of information-monotone information measures: volume mutual information (VMI). VMI is based on a simple geometric information measure design method, the volume method. The volume method measures the informativeness of an object by “counting” the number of objects that are less informative than it. In other words, the more objects that the object of interest dominates, the more informative it is considered to be.
Finally, in the setting where agents need to invest efforts to obtain their private signals, we show how to select the mechanism to optimally incentivize efforts among a proper set of VMI-Mechanisms.
- . 2019. Preferences for truth-telling. Econometrica 87, 4 (2019), 1115–1153. Retrieved from http://www.jstor.org/stable/45172344Google Scholar
Cross Ref
- . 2020. Peer prediction with heterogeneous users. ACM Transactions on Economics and Computation 8, 1, Article
2 (Mar 2020), 34 pages.DOI: Google ScholarDigital Library
- . 1966. A general class of coefficients of divergence of one distribution from another. Journal of the Royal Statistical Society. Series B (Methodological) 28, 1 (1966), 131–142. Retrieved from http://www.jstor.org/stable/2984279Google ScholarCross Ref
- . 1967. The relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programming. USSR Computational Mathematics and Mathematical Physics 7, 3 (1967), 200–217.
DOI: Google ScholarCross Ref - . 2023. Measurement integrity in peer prediction: A peer assessment case study. In Proceedings of the 24th ACM Conference on Economics and Computation . ACM, New York, NY, 369–389.
DOI: Google ScholarDigital Library - . 2015. Optimum statistical estimation with strategic data sources. In Proceedings of the 28th Conference on Learning Theory . , , and (Eds.), PMLR,
Vol. 40, Paris, 280–296. Retrieved from https://proceedings.mlr.press/v40/Cai15.htmlGoogle Scholar - . 2020. Computing a pessimistic stackelberg equilibrium with multiple followers: The mixed-pure case. Algorithmica 82, 5 (2020), 1189–1238.
DOI: Google ScholarCross Ref - . 2006. Computing the optimal strategy to commit to. In Proceedings of the 7th ACM Conference on Electronic Commerce . ACM, New York, NY, 82–90.
DOI: Google ScholarDigital Library - . 2004. Information theory and statistics: A tutorial. Foundations and Trends® in Communications and Information Theory 1, 4 (2004), 417–528.
DOI: Google ScholarDigital Library - . 2013. Crowdsourced judgement elicitation with endogenous proficiency. In Proceedings of the 22nd International Conference on World Wide Web . ACM, New York, NY, 319–330.
DOI: Google ScholarDigital Library - . 2023. Game-theoretic mechanisms for eliciting accurate information. In Proceedings of the 32nd International Joint Conference on Artificial Intelligence. (Ed.), International Joint Conferences on Artificial Intelligence Organization, 6601–6609.
DOI: Survey Track .Google ScholarDigital Library - . 2014. Incentives to counter bias in human computation. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing. Vol. 2, 59–66.
DOI: Google ScholarCross Ref - . 2017. A geometric perspective on minimal peer prediction. ACM Transactions on Economics and Computation 5, 3, Article
17 (Jul 2017), 27 pages.DOI: Google ScholarDigital Library - . 2014. Trick or treat: Putting peer prediction to the test. In Proceedings of the 15th ACM Conference on Economics and Computation . ACM, New York, NY, 507–524.
DOI: Google ScholarDigital Library - . 2020. Incentivizing evaluation with peer prediction and limited access to ground truth (extended abstract). In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence (IJCAI’20). 5140–5144.
DOI: Google ScholarCross Ref - . 2007. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association 102, 477 (2007), 359–378.
DOI: Google ScholarCross Ref - . 2019. Deep bayesian trust: A dominant and fair incentive mechanism for crowd. In Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33, 1996–2003.
DOI: Google ScholarDigital Library - . 2023. Optimal scoring rules for multi-dimensional effort. In Proceedings of the 36th Conference on Learning Theory . and (Eds.), PMLR, Vol. 195, 2624–2650. Retrieved from https://proceedings.mlr.press/v195/hartline23a.htmlGoogle Scholar
- . 1981. The vec-permutation matrix, the vec operator and kronecker products: A review. Linear and Multilinear Algebra 9, 4 (1981), 271–288.
DOI: Google ScholarCross Ref - . 2007. Collusion-resistant, incentive-compatible feedback payments. In Proceedings of the 8th ACM Conference on Electronic Commerce . ACM, New York, NY, 200–209.
DOI: Google ScholarDigital Library - . 2009. Mechanisms for making crowds truthful. Journal of Artificial Intelligence Research 34, 1 (
Mar 2009), 209–253.Google ScholarDigital Library - . 2023. The square root agreement rule for incentivizing truthful feedback on online platforms. Management Science 69, 1 (
Jan 2023), 377–403.DOI: Google ScholarDigital Library - . 2020. Dominantly truthful multi-task peer prediction with a constant number of tasks. In Proceedings of the 31st Annual ACM-SIAM Symposium on Discrete Algorithms . Society for Industrial and Applied Mathematics, 2398–2411.Google ScholarCross Ref
- . 2022. More dominantly truthful multi-task peer prediction with a finite number of tasks. In Proceedings of the 13th Innovations in Theoretical Computer Science Conference . (Ed.),
Leibniz International Proceedings in Informatics (LIPIcs) , Vol. 215, Schloss Dagstuhl – Leibniz-Zentrum für Informatik, Dagstuhl, 95:1–95:20.DOI: Google ScholarCross Ref - . 2016. Putting peer prediction under the microeconomicscope and making truth-telling focal. In Proceedings of the 12th International Conference on Web and Internet Economics - Volume 10123 . Springer-Verlag, Berlin, 251–264.
DOI: Google ScholarDigital Library - . 2018c. Eliciting expertise without verification. In Proceedings of the 2018 ACM Conference on Economics and Computation . ACM, New York, NY, 195–212.
DOI: Google ScholarDigital Library - . 2018a. Equilibrium selection in information elicitation without verification via information monotonicity. In Proceedings of the 9th Innovations in Theoretical Computer Science Conference . (Ed.), Leibniz International Proceedings in Informatics (LIPIcs), Vol. 94, Schloss Dagstuhl–Leibniz-Zentrum fuer Informatik, Dagstuhl, 13:1–13:20.
DOI: Google ScholarCross Ref - . 2018b. Water from two rocks: Maximizing the mutual information. In Proceedings of the 2018 ACM Conference on Economics and Computation . ACM, New York, NY, 177–194.
DOI: Google ScholarDigital Library - . 2019. An information theoretic framework for designing information elicitation mechanisms that reward truth-telling. ACM Transactions on Economics and Computation 7, 1, Article
2 (Jan 2019), 33 pages.DOI: Google ScholarDigital Library - . 2020. Information elicitation mechanisms for statistical estimation. In Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34, 2095–2102.
DOI: Google ScholarCross Ref - . 2022. Optimization of scoring rules. In Proceedings of the 23rd ACM Conference on Economics and Computation . ACM, New York, NY, 988–989.
DOI: Google ScholarDigital Library - . 2017a. Machine-learning aided peer prediction. In Proceedings of the 2017 ACM Conference on Economics and Computation . ACM, New York, NY, 63–80.
DOI: Google ScholarDigital Library - . 2017b. Sequential peer prediction: Learning to elicit effort using posted prices. In Proceedings of the 31st AAAI Conference on Artificial Intelligence . AAAI Press, 607–613.Google ScholarCross Ref
- . 2023. Surrogate scoring rules. ACM Transactions on Economics and Computation 10, 3, Article
12 (Feb 2023), 36 pages.DOI: Google ScholarDigital Library - . 2016. Peer prediction with heterogeneous tasks. In Proceedings of CrowdML Workshop at (NIPS’6), Barcelona, Spain.Google Scholar
- . 2020. The effectiveness of peer prediction in long-term forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34, 2160–2167.
DOI: Google ScholarCross Ref - . 2013. Choosing a strictly proper scoring rule. Decision Analysis 10, 4 (2013), 292–304.
DOI: Google ScholarDigital Library - . 2005. Eliciting informative feedback: The peer-prediction method. Management Science 51, 9 (2005), 1359–1373.
DOI: Google ScholarDigital Library - . 2021. Binary scoring rules that incentivize precision. In Proceedings of the 22nd ACM Conference on Economics and Computation . ACM, New York, NY, 718–733.
DOI: Google ScholarDigital Library - . 1989. Optimal forecasting incentives. Journal of Political Economy 97, 5 (1989), 1091–1112. Retrieved from http://www.jstor.org/stable/1831887Google ScholarCross Ref
- . 2004. A Bayesian truth serum for subjective data. Science 306, 5695 (
Oct 2004), 462–466.Google ScholarCross Ref - . 2017. A solution to the single-question crowd wisdom problem. Nature 541, 7638 (
Jan 2017), 532–535.Google ScholarCross Ref - . 2013. A robust Bayesian truth serum for non-binary signals. In Proceedings of the 27th AAAI Conference on Artificial Intelligence . AAAI Press, 833–839.Google ScholarCross Ref
- . 2014. Incentives for truthful information elicitation of continuous signals. In Proceedings of the 28th AAAI Conference on Artificial Intelligence. AAAI Press, 770–776.Google ScholarDigital Library
- . 2015a. Incentive schemes for participatory sensing. In Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems . International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, 1081–1089.Google ScholarDigital Library
- . 2015b. Incentives for subjective evaluations with private beliefs. In Proceedings of the 29th AAAI Conference on Artificial Intelligence . AAAI Press, 1014–1020.Google ScholarCross Ref
- . 2016. Incentives for effort in crowdsourcing using the peer truth serum. ACM Transactions on Intelligent Systems and Technology 7, 4, Article
48 (Mar 2016), 28 pages.DOI: Google ScholarDigital Library - . 2016. Paying for the Truth: The Efficacy of a Peer Prediction Mechanism in the Field.
Working Paper .Google Scholar - . 2014. Minimum truth serums with optional predictions. In Proceedings of the 4th Workshop on Social Computing and User Generated Content.Google Scholar
- . 2021. Learning and strongly truthful multi-task peer prediction: A variational approach. In Proceedings of the 12th Innovations in Theoretical Computer Science Conference . (Ed.),
Leibniz International Proceedings in Informatics (LIPIcs) , Vol. 185, Schloss Dagstuhl–Leibniz-Zentrum für Informatik, Dagstuhl, 78:1–78:20.DOI: Google ScholarCross Ref - . 2023. Two strongly truthful mechanisms for three heterogeneous agents answering one question. ACM Transactions on Economics and Computation 10, 4, Article
14 (Feb 2023), 26 pages. Google ScholarDigital Library - . 2021. Information elicitation from rowdy crowds. In Proceedings of the Web Conference 2021 . ACM, New York, NY, 3974–3986.
DOI: Google ScholarDigital Library - . 2006. Non-Negative Matrices and Markov Chains. Springer Science & Business Media.Google Scholar
- . 2001. A mathematical theory of communication. SIGMOBILE Mobile Computing and Communications Review 5, 1 (
Jan 2001), 3–55.DOI: Google ScholarDigital Library - . 2016a. Informed truthfulness in multi-task peer prediction. In Proceedings of the 2016 ACM Conference on Economics and Computation . ACM, New York, NY, 179–196.
DOI: Google ScholarDigital Library - . 2016b. Measuring performance of peer prediction mechanisms using replicator dynamics. In Proceedings of the 25th International Joint Conference on Artificial Intelligence . AAAI Press, 2611–2617.Google Scholar
- . 2016. Practical peer prediction for peer assessment. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing. Vol. 4, 199–208.
DOI: Google ScholarCross Ref - . 1984. Lectures on Geometric Measure Theory. Vol. 3. The Australian National University, Mathematical Sciences Institute.Google Scholar
- . 2004. Labeling images with a computer game. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems . ACM, New York, NY, 319–326.
DOI: Google ScholarDigital Library - . 2010. Leadership games with convex strategy sets. Games and Economic Behavior 69, 2 (2010), 446–457.
DOI: Google ScholarCross Ref - . 2021. Forecast aggregation via peer prediction. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol. 9. 131–142.
DOI: Google ScholarCross Ref - . 1969. Scoring rules and the evaluation of probability assessors. Journal of the American Statistical Association 64, 327 (1969), 1073–1078. Retrieved from http://www.jstor.org/stable/2283486Google ScholarCross Ref
- . 2013. Dwelling on the negative: Incentivizing effort in peer prediction. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol. 1. 190–197.
DOI: Google ScholarCross Ref - . 2012b. Peer prediction without a common prior. In Proceedings of the 13th ACM Conference on Electronic Commerce . ACM, New York, NY, 964–981.
DOI: Google ScholarDigital Library - . 2012a. A robust Bayesian truth serum for small populations. In Proceedings of the 26th AAAI Conference on Artificial Intelligence . AAAI Press, 1492–1498.Google Scholar
- . 2013. Learning the prior in minimal peer prediction. In Proceedings of the 3rd Workshop on Social Computing and User Generated Content.Google Scholar
- . 2019. L_DMI: A novel information-theoretic loss function for training deep nets robust to label noise. In Advances in Neural Information Processing Systems. , , , , , and (Eds.), Vol. 32, Curran Associates, Inc. Retrieved from https://proceedings.neurips.cc/paper_files/paper/2019/file/8a1ee9f2b7abe6e88d1a479ab6a42c5e-Paper.pdfGoogle Scholar
- . 2014. Elicitability and knowledge-free elicitation with peer prediction. In Proceedings of the 2014 International Conference on Autonomous Agents and Multi-Agent Systems . International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, 245–252.Google ScholarDigital Library
- . 2021. The limits of multi-task peer prediction. In Proceedings of the 22nd ACM Conference on Economics and Computation . ACM, New York, NY, 907–926.
DOI: Google ScholarDigital Library
Index Terms
- Dominantly Truthful Peer Prediction Mechanisms with a Finite Number of Tasks
Recommendations
Putting Peer Prediction Under the Microeconomicscope and Making Truth-Telling Focal
WINE 2016: Proceedings of the 12th International Conference on Web and Internet Economics - Volume 10123Peer-predictionï ź[19] is a meta-mechanism which, given any proper scoring rule, produces a mechanism to elicit prie information from self-interested agents. Formally, truth-telling is a strict Nash equilibrium of the mechanism. Unfortunately, there may ...
The Limits of Multi-task Peer Prediction
EC '21: Proceedings of the 22nd ACM Conference on Economics and ComputationRecent advances in multi-task peer prediction have greatly expanded our knowledge about the power of multi-task peer prediction mechanisms. Various mechanisms have been proposed in different settings to elicit different types of information. But we ...
Informed Truthfulness in Multi-Task Peer Prediction
EC '16: Proceedings of the 2016 ACM Conference on Economics and ComputationThe problem of peer prediction is to elicit information from agents in settings without any objective ground truth against which to score reports. Peer prediction mechanisms seek to exploit correlations between signals to align incentives with truthful ...
Comments