Skip to main content
Springer Nature Link
Log in

Team selection for prediction tasks

  • Published:
Journal of Combinatorial Optimization Aims and scope Submit manuscript

Abstract

Given a random variable \(O \in \mathbb {R}\) and a set of experts \(E\), we describe a method for finding a subset of experts \(S \subseteq E\) whose aggregated opinion best predicts the outcome of \(O\). Therefore, the problem can be regarded as a team formation for performing a prediction task. We show that in case of aggregating experts’ opinions by simple averaging, finding the best team (the team with the lowest total error during past \(k\) rounds) can be modeled with an integer quadratic programming and we prove its NP-hardness whereas its relaxation is solvable in polynomial time. At the end, we do an experimental comparison between different rounding and greedy heuristics on artificial datasets which are generated based on calibration and informativeness of exprets’ information and show that our suggested tabu search works effectively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+
from $39.99 /Month
  • Starting from 10 chapters or articles per month
  • Access and download chapters and articles from more than 300k books and 2,500 journals
  • Cancel anytime
View plans

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Explore related subjects

Discover the latest articles and news from researchers in related subjects, suggested using machine learning.

References

  • Attarzadeh I, Ow SH (2011) Software development cost and time forecasting using a high performance artificial neural network model. In: Intelligent Computing and Information Science, pp 18–26. Springer, Berlin

  • Boutilier C (2012) Eliciting forecasts from self-interested experts: scoring rules for decision makers. In: Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems, Vol 2, pp 737–744. International Foundation for Autonomous Agents and Multiagent Systems

  • Chen Y, Kash I (2011) Information elicitation for decision making. In The 10th International Conference on Autonomous Agents and Multiagent Systems, Vol 1, pp 175–182. International Foundation for Autonomous Agents and Multiagent Systems

  • Chen Y, Chu C-H, Mullen T, Pennock DM (2005) Information markets vs. opinion pools: an empirical comparison. In: Proceedings of the 6th ACM Conference on Electronic Commerce, pp 58–67. ACM,Vancouver, Vancouver

  • Chen Y, Chu C-H, Mullen T (2006) Predicting uncertain outcomes using information markets: trader behavior and information aggregation. New Math Nat Comput 2(03):281–297

    Article  MathSciNet  MATH  Google Scholar 

  • Chhabra M, Das S, Szymanski B (2013) Team formation in social networks. In: Computer and Information Sciences III, pp 291–299. Springer, Berlin

  • Clemen RT, Winkler RL (1999) Combining probability distributions from experts in risk analysis. Risk Anal 19(2):187–203

    Google Scholar 

  • Clemen RT, Winkler RL (2007) Aggregating probability distributions. Advances in Decision Analysis: From Foundations to Applications. Cambridge University Press, Cambridge, pp 154–176

    Chapter  Google Scholar 

  • Cooke RM (1991) Experts in uncertainty: opinion and subjective probability in science. Oxford University Press, New York, NY

    Google Scholar 

  • Dani V, Madani O, Pennock DM, Sanghai S, Galebach B (2012) An empirical comparison of algorithms for aggregating expert predictions. arXiv preprint arXiv:1206.6814

  • French S (2011) Aggregating expert judgement. Rev R Acad A 105(1):181–206

    MathSciNet  MATH  Google Scholar 

  • Genest C, Zidek JV (1986) Combining probability distributions: a critique and an annotated bibliography. Stat Sci 1(1):114–135

    Article  MathSciNet  Google Scholar 

  • Graefe A, Scott AJ, Jones RJ Jr, Cuzán AG (2014) Combining forecasts: an application to elections. Int J Forecast 30(1):43–54

    Article  Google Scholar 

  • Hammitt JK, Yifan Z (2013) Combining experts’ judgments: comparison of algorithmic methods using synthetic data. Risk Anal 33(1):109–120

    Article  Google Scholar 

  • Hora SC (2007) Expert judgment. Encyclopedia of Quantitative Risk Analysis and Assessment

  • Jacobs RA (1995) Methods for combining experts’ probability assessments. Neural Comput 7(5):867–888

    Article  Google Scholar 

  • Jørgensen M (2007) Forecasting of software development work effort: evidence on expert judgement and formal models. Int J Forecast 23(3):449–462

    Article  Google Scholar 

  • Kallen MJ, Cooke RM (2002) Expert aggregation with dependence. In: Probabilistic Safety Assessment and Management, pp 1287–94. Elsevier, Oxford

  • Kargar M, Zihayat M, An A (2013) Affordable and collaborative team formation in an expert network. Department of Computer Science and Engineering, York University, Technical Report CSE-2013, 1

  • Lappas T, Liu K, Terzi E (2009) Finding a team of experts in social networks. In: Proceedings of the 15th ACM SIGKDD international conference on knowledge discovery and data mining, pp 467–476. ACM, Vancouver

  • Malhotra R, Jain A (2011) Software effort prediction using statistical and machine learning methods. Int J Adv Comput Sci Appl 2(1):1451–1521

    Google Scholar 

  • Michaeli R, Simon L (2008) An illustration of Bayes’ theorem and its use as a decision-making aid for competitive intelligence and marketing analysts. Eur J Market 42(7–8):804–813

    Google Scholar 

  • Morris PA (1974) Decision analysis expert use. Manag Sci 20(9):1233–1241

    MATH  Google Scholar 

  • Mostaghimi M (1996) Combining ranked mean value forecasts. Eur J Oper Res 94(3):505–516

    Article  MATH  Google Scholar 

  • Mostaghimi M (1997) Bayesian estimation of a decision using information theory. IEEE Trans Syst Man Cybern 27(4):506–517

    Article  Google Scholar 

  • Othman A, Sandholm T (2010) Decision rules and decision markets. In: Proceedings of the 9th international conference on autonomous agents and multiagent systems, Vol 1, pp 625–632. International Foundation for Autonomous Agents and Multiagent Systems

  • Reggiani P, Weerts AH (2008) A Bayesian approach to decision-making under uncertainty: an application to real-time forecasting in the river rhine. J Hydrol 356(1):56–69

    Article  Google Scholar 

  • Sprenger T, Bolster P, Venkateswaran A (2012) Conditional prediction markets as corporate decision support systems–an experimental comparison with group deliberations. J Predict Market 1(3):189–208

    Google Scholar 

  • Wu Q, Hao J-K (2012a) Coloring large graphs based on independent set extraction. Comput Oper Res 39(2):283–290

  • Wu Q, Hao J-K (2012b) An effective heuristic algorithm for sum coloring of graphs. Comput Oper Res 39(7):1593–1600

  • Wu Q, Hao J-K (2013) An adaptive multistart tabu search approach to solve the maximum clique problem. J Comb Optim 26(1):86–108

  • Zhang H, Horvitz E, Chen Y, Parkes DC (2012) Task routing for prediction tasks. In Proceedings of the 11th international conference on autonomous agents and multiagent systems, Vol 2, pp 889–896. International Foundation for Autonomous Agents and Multiagent Systems

Download references

Author information

Authors and Affiliations

  1. Department of Computer Engineering, Sharif University of Technology, Tehran, Iran

    MohammadAmin Fazli, Azin Ghazimatin, Jafar Habibi & Hamid Haghshenas

Authors
  1. MohammadAmin Fazli
    View author publications

    Search author on:PubMed Google Scholar

  2. Azin Ghazimatin
    View author publications

    Search author on:PubMed Google Scholar

  3. Jafar Habibi
    View author publications

    Search author on:PubMed Google Scholar

  4. Hamid Haghshenas
    View author publications

    Search author on:PubMed Google Scholar

Corresponding author

Correspondence to Azin Ghazimatin.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Fazli, M., Ghazimatin, A., Habibi, J. et al. Team selection for prediction tasks. J Comb Optim 31, 743–757 (2016). https://doi.org/10.1007/s10878-014-9784-3

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10878-014-9784-3

Keywords