Skip to main content
SpringerLink
Log in
Book cover

Pacific-Asia Conference on Knowledge Discovery and Data Mining

PAKDD 2015: Advances in Knowledge Discovery and Data Mining pp 538–549Cite as

Learning of Performance Measures from Crowd-Sourced Data with Application to Ranking of Investments

  • Conference paper
  • First Online:

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 9077))

Abstract

Interestingness measures stand as proxy for “real human interest,” but their effectiveness is rarely studied empirically due to the difficulty of obtaining ground-truth data. We propose a method based on learning-to-rank algorithms that enables pairwise rankings collected from domain community members to be used to learn a domain-specific measure. We apply this method to study the interestingness measures in finance, specifically, investment performance evaluation measures. More than 100 such measures have been proposed with no way of knowing which most closely matches the preferences of domain users. We use crowd-sourcing to collect gold-standard truth from traders and quantitative analysts in the form of pairwise rankings of equity graphs. With these rankings, we evaluate the accuracy with which each measure predicts the user-preferred equity graph. We then learn a new investment performance measure which has higher test accuracy than the currently proposed measures, in particular the commonly used Sharpe ratio.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Alexander, G.J., Baptista, A.M.: Portfolio performance evaluation using value at risk. The Journal of Portfolio Management 29(4), 93–102 (2003)

    Article  Google Scholar 

  2. Bacon, C.R.: Practical Risk-adjusted Performance Measurement. John Wiley & Sons (2012)

    Google Scholar 

  3. Blanchard, J., Guillet, F., Gras, R., Briand, H.: Using information-theoretic measures to assess association rule interestingness. In: Fifth IEEE International Conference on Data Mining, p. 8. IEEE (2005)

    Google Scholar 

  4. Breiman, L.: Random forests. Machine Learning 45(1), 5–32 (2001)

    Article  MATH  Google Scholar 

  5. Carvalho, D.R., Freitas, A.A., Ebecken, N.F.F.: Evaluating the Correlation Between Objective Rule Interestingness Measures and Real Human Interest. In: Jorge, A.M., Torgo, L., Brazdil, P.B., Camacho, R., Gama, J. (eds.) PKDD 2005. LNCS (LNAI), vol. 3721, pp. 453–461. Springer, Heidelberg (2005)

    Chapter  Google Scholar 

  6. Chang, C.-C., Lin, C.-J.: LIBSVM: a library for support vector machines. ACM Transactions on Intelligent Systems and Technology (TIST) 2(3), 27 (2011)

    Google Scholar 

  7. Cogneau, P., Hübner, G.: The (more than) 100 ways to measure portfolio performance. part 1: standardized risk-adjusted measures. Journal of Performance Measurement 13 (Summer 2009)

    Google Scholar 

  8. Dowd, K.: Beyond Value at Risk: The New Science of Risk Management, vol. 3. Wiley, Chichester (1998)

    Google Scholar 

  9. Eling, M.: Does the measure matter in the mutual fund industry? Financial Analysts Journal, 54–66 (2008)

    Google Scholar 

  10. Eling, M., Schuhmacher, F.: Does the choice of performance measure influence the evaluation of hedge funds? Journal of Banking & Finance 31(9), 2632–2647 (2007)

    Article  Google Scholar 

  11. Farinelli, S., Ferreira, M., Rossello, D., Thoeny, M., Tibiletti, L.: Beyond Sharpe ratio: Optimal asset allocation using different performance ratios. Journal of Banking & Finance 32(10), 2057–2063 (2008)

    Article  Google Scholar 

  12. Friedman, J., Hastie, T., Tibshirani, R.: Regularization paths for generalized linear models via coordinate descent. Journal of Statistical Software 33(1), 1 (2010)

    Google Scholar 

  13. Geng, L., Hamilton, H.J.: Interestingness measures for data mining: A survey. ACM Computing Surveys (CSUR) 38(3), 9 (2006)

    Google Scholar 

  14. Hahn, C., Peter Wagner, F., Pfingsten, A.: An empirical investigation of the rank correlation between different risk measures. In: EFA 2002 Berlin Meetings Presented Paper, pp. 02–01 (2002)

    Google Scholar 

  15. Herbrich, R., Graepel, T., Obermayer, K.: Large margin rank boundaries for ordinal regression. Advances in Neural Information Processing Systems, pp. 115–132 (1999)

    Google Scholar 

  16. Hilderman, R.J., Hamilton, H.J.: Evaluation of Interestingness Measures for Ranking Discovered Knowledge. In: Cheung, D., Williams, G.J., Li, Q. (eds.) PAKDD 2001. LNCS (LNAI), vol. 2035, p. 247. Springer, Heidelberg (2001)

    Chapter  Google Scholar 

  17. Keating, C., Shadwick, W.F.: A universal performance measure. Journal of Performance Measurement 6(3), 59–84 (2002)

    Google Scholar 

  18. Lease, M.: On quality control and machine learning in crowdsourcing. In: Human Computation (2011)

    Google Scholar 

  19. Lenca, P., Meyer, P., Vaillant, B., Lallich, S.: On selecting interestingness measures for association rules: User oriented description and multiple criteria decision aid. European Journal of Operational Research 184(2), 610–626 (2008)

    Article  MATH  Google Scholar 

  20. McGarry, K.: A survey of interestingness measures for knowledge discovery. Knowledge Eng. Review 20(1), 39–61 (2005)

    Article  Google Scholar 

  21. Mistry, J., Shah, J.: Dealing with the limitations of the Sharpe ratio for portfolio evaluation. Journal of Commerce and Accounting Research 2(3), 10–18 (2013)

    Google Scholar 

  22. Ohsaki, M., Kitaguchi, S., Okamoto, K., Yokoi, H., Yamaguchi, T.: Evaluation of Rule Interestingness Measures with a Clinical Dataset on Hepatitis. In: Boulicaut, J.-F., Esposito, F., Giannotti, F., Pedreschi, D. (eds.) PKDD 2004. LNCS (LNAI), vol. 3202, pp. 362–373. Springer, Heidelberg (2004)

    Chapter  Google Scholar 

  23. Sharpe, W.F.: Mutual fund performance. Journal of Business, 119–138 (1966)

    Google Scholar 

  24. Sharpe, W.F.: The Sharpe ratio. Journal of Portfolio Management 21, 49–58 (1994)

    Article  Google Scholar 

  25. Sortino, F.A., Van Der Meer, R.: Downside risk. The. Journal of Portfolio Management 17(4), 27–31 (1991)

    Google Scholar 

  26. Tan, P.-N., Kumar, V., Srivastava, J.: Selecting the right interestingness measure for association patterns. In: Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 32–41. ACM (2002)

    Google Scholar 

  27. Vaillant, B., Lenca, P., Lallich, S.: A Clustering of Interestingness Measures. In: Suzuki, E., Arikawa, S. (eds.) DS 2004. LNCS (LNAI), vol. 3245, pp. 290–297. Springer, Heidelberg (2004)

    Chapter  Google Scholar 

  28. Zakamouline, V.: The choice of performance measure does influence the evaluation of hedge funds. Available at SSRN 1403246 (2010)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, University of Southern California, Los Angeles, CA, USA

    Greg Harris

  2. Ming-Hsieh Department of Electrical Engineering, University of Southern California, Los Angeles, CA, USA

    Anand Panangadan & Viktor K. Prasanna

Authors
  1. Greg Harris
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Anand Panangadan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Viktor K. Prasanna
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Greg Harris .

Editor information

Editors and Affiliations

  1. Ho Chi Minh City University of Technology, Ho Chi Minh City, Vietnam

    Tru Cao

  2. Singapore Management University, Singapore, Singapore

    Ee-Peng Lim

  3. Nanjing University, Nanjing, China

    Zhi-Hua Zhou

  4. Japan Advanced Institute of Science and Technology, Nomi City, Japan

    Tu-Bao Ho

  5. University of Hong Kong, Hong Kong, Hong Kong SAR

    David Cheung

  6. Osaka University, Osaka, Japan

    Hiroshi Motoda

Rights and permissions

Reprints and permissions

Copyright information

© 2015 Springer International Publishing Switzerland

About this paper

Cite this paper

Harris, G., Panangadan, A., Prasanna, V.K. (2015). Learning of Performance Measures from Crowd-Sourced Data with Application to Ranking of Investments. In: Cao, T., Lim, EP., Zhou, ZH., Ho, TB., Cheung, D., Motoda, H. (eds) Advances in Knowledge Discovery and Data Mining. PAKDD 2015. Lecture Notes in Computer Science(), vol 9077. Springer, Cham. https://doi.org/10.1007/978-3-319-18038-0_42

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-18038-0_42

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-18037-3

  • Online ISBN: 978-3-319-18038-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation