Skip to main content
SpringerLink
Log in

Conformal predictions for information fusion

A comparative study of p-value combination methods

  • Published:
Annals of Mathematics and Artificial Intelligence Aims and scope Submit manuscript

Abstract

The increased availability of a wide range of sensing technologies over the last few decades has resulted in an equivalent increased need for reliable information fusion methods in machine learning applications. While existing theories such as the Dempster-Shafer theory and the possibility theory have been used for several years now, they do not provide guarantees of error calibration in information fusion settings. The Conformal Predictions (CP) framework is a new game-theoretic approach to reliable machine learning, which provides a methodology to obtain error calibration under classification and regression settings. In this work, we present a methodology to extend the Conformal Predictions framework to both classification and regression-based information fusion settings. This methodology is based on applying the CP framework to each data source as an independent hypothesis test, and subsequently using p-value combination methods as a test statistic for the combined hypothesis after fusion. The proposed methodology was studied in classification and regression settings within two real-world application contexts: person recognition using multiple modalities (classification), and head pose estimation using multiple image features (regression). Our experimental results showed that quantile methods of combining p-values (such as the Standard Normal Function and the Non-conformity Aggregation methods) provided the most statistically valid calibration results, and can be considered to extend the CP framework for information fusion settings.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Ali, H., Antenreiter, M., Auer, P., Csurka, G., de Campos, T., Hussain, Z., Laaksonen, J., Ortner, R., Pasupa, K., Perronnin, F., Saunders, C., Shawe-Taylor, J., Viitaniemi, V.: Description, analysis and evaluation of confidence estimation procedures for sub-categorization. Tech. Rep. D6.2.1, Xerox Research Center Europe (2009)

  2. Bailly, K., Milgram, M.: 2009 special issue: boosting feature selection for neural network based regression. Neural Netw. 22(5–6), 748–756 (2009)

    Article  Google Scholar 

  3. Balasubramanian, V., Chakraborty, S., Panchanathan, S.: Generalized query by transduction for online active learning. In: Proceedings of the International Conference on Computer Vision (ICCV 2009) Workshop on Online Learning for Computer Vision (2009)

  4. Balasubramanian, V., Gouripeddi, R., Panchanathan, S., Vermillion, J., Bhaskaran, A., Siegel, R.: Support vector machine based conformal predictors for risk of complications following a coronary drug eluting stent procedure. In: Computers in Cardiology, pp. 5–8 (2009)

  5. Balasubramanian, V., Panchanathan, S., Chakraborty, S.: Multiple cue integration in transductive confidence machines for head pose classification. In: Computer Vision and Pattern Recognition Workshops, CVPRW ’08. IEEE Computer Society Conference, pp. 1–8 (2008)

  6. Balasubramanian, V., Panchanathan, S., Chakraborty, S.: Multiple cue integration in transductive confidence machines for head pose classification. In: Computer Vision and Pattern Recognition Workshops, CVPRW ’08. IEEE Computer Society Conference, pp. 1–8 (2008). doi:10.1109/CVPRW.2008.4563070

  7. Balasubramanian, V., Ye, J., Panchanathan, S.: Biased manifold embedding: a framework for person-independent head pose estimation. In: Computer Vision and Pattern Recognition, CVPR ’07. IEEE Conference, pp. 1–7 (2007). doi:10.1109/CVPR.2007.383280

  8. Beiraghi, S., Ahmadi, M., Ahmed, M.S., Shridhar, M.: Application of fuzzy integrals in fusion of classifiers for low error rate handwritten numerals recognition. Int. Conf. Pattern Recog. (ICPR) 2, 2487 (2000)

    Google Scholar 

  9. Ben-Yacoub, S., Abdeljaoued, Y., Mayoraz, E.: Fusion of face and speech data for person identity verification. IEEE Trans. Neural Netw. 10(5), 1065–1074 (1999). doi:10.1109/72.788647

    Google Scholar 

  10. Bolme, D., Beveridge, J., Howe, A.: Person identification using text and image data. In: Biometrics: Theory, Applications, and Systems, BTAS 2007. First IEEE International Conference, pp. 1–6 (2007)

  11. Brown, L.M., Tian, Y.L.: Comparative study of coarse head pose estimation. In: IEEE Workshop on Motion and Video Computing, Orlando, Florida, pp. 125–130 (2002)

  12. Brunelli, R., Falavigna, D.: Person identification using multiple cues. IEEE Trans. Pattern. Anal. Mach. Intell. 17, 955–966 (1995)

    Article  Google Scholar 

  13. Buchanan, B.G., Shortliffe, E.H.: Rule Based Expert Systems: The Mycin Experiments of the Stanford Heuristic Programming Project (The Addison-Wesley series in artificial intelligence). Addison-Wesley Longman Publishing Co., Inc. (1984)

  14. Carrasco, M., Pizarro, L., Mery, D.: Bimodal biometric person identification system under perturbations. In: Advances in Image and Video Technology, pp. 114–127 (2007)

  15. Dasarathy, B.V.: Decision Fusion. IEEE Computer Society Press, Los Alamitos CA (1994)

    Google Scholar 

  16. Dashevskiy, M., Luo, Z.: Network traffic demand prediction with confidence. In: IEEE Global Telecommunications Conference IEEE GLOBECOM 2008, pp. 1–5 (2008). http://dx.doi.org/10.1109/GLOCOM.2008.ECP.284

  17. Dempster, A.: A generalization of bayesian inference. In: Classic Works of the Dempster-Shafer Theory of Belief Functions, pp. 73–104 (2008)

  18. Dubois, D., Prade, H.: Possibility theory and its applications: a retrospective and prospective view. In: Fuzzy Systems, FUZZ ’03. The 12th IEEE International Conference, vol. 1, pp. 5–11 (2003)

  19. Duc, B., Bign, E.S., Bign, J., Matre, G., Fischer, S.: Fusion of audio and video information for multi modal person authentication. Pattern Recognit. Lett. 18, 835–843 (1997)

    Article  Google Scholar 

  20. Edgington, E.S.: An AdditiveMethod for Combining Probability Values from Independent Experiments. J. Psychol. 80(2), 351–363 (1972). doi:10.1080/00223980.1972.9924813

    Google Scholar 

  21. Ekenel, H.K., Fischer, M., Jin, Q., Stiefelhagen, R.: Multi-modal person identification in a smart environment. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 1–8 (2007)

  22. Ekenel, H.K., Jin, Q., Fischer, M., Stiefelhagen, R.: ISL person identification systems in the CLEAR 2007 evaluations. In: Stiefelhagen, R., Bowers, R., Fiscus, J. (Eds.) Multimodal Technologies for Perception of Humans, vol. 4625, pp. 256–265. Springer, Berlin/Heidelberg (2008). doi:10.1007/978-3-540-68585-2_24

  23. Eugene, T.H., Weinstein, E., Kabir, R., Park, A.: Multi-modal face and speaker identification on a handheld device. In: Proceedings Workshop Multimodal User Authentication, pp. 120–132 (2003)

  24. Fisher, S.R.A.: Statistical methods for research workers, vol. 14, pp. 140–142. Edinburgh, Oliver and Boyd (1970)

  25. Smarandache, F., Dezert, J. (eds.): Advances and Applications of DSmT for Information Fusion (Collected works), 2nd volume, Am. Res Press (2006)

  26. Fox, N., Gross, R., Cohn, J., Reilly, R.: Robust biometric person identification using automatic classifier fusion of speech, mouth, and face experts. IEEE Trans. Multimed. 9(4), 701–714 (2007)

    Article  Google Scholar 

  27. Shafer, G.: Perspectives on the theory and practice of belief functions. Int. J. Approx. Reason. 4(5–6), 323–362 (1990)

    Article  MATH  MathSciNet  Google Scholar 

  28. Good, I.J.: On the weighted combination of significance tests. J. R. Stat. Soc. Ser. B Methodol. 17(2), 264–265 (1955)

    MATH  MathSciNet  Google Scholar 

  29. Hu, R., Damper, R.: Fusion of two classifiers for speaker identification: removing and not removing silence. In: Information Fusion, 2005 8th International Conference, vol. 1, p. 8 (2005)

  30. Jain, A., Nandakumar, K., Ross, A.: Score normalization in multimodal biometric systems. Pattern Recog 38(12), 2285, 2270 (2005)

    Article  Google Scholar 

  31. Jost, L.: Combining significance levels from multiple experiments or analyses. http://www.loujost.com/StatisticsandPhysics/StatsArticlesIndex.htm (2009). Accessed 18 June 2009

  32. Kostas, H.P., Proedrou, K., Vovk, V., Gammerman, A., Ex, S.T.: Inductive confidence machines for regression. In: Elomaa T., Mannila, H., Toivonen, H. (eds.) Proceedings of the 13th European Conference on Machine Learning, vol. 2430, pp. 345–356 (2002)

  33. Lambrou, A., Papadopoulos, H., Gammerman, A.: Reliable confidence measures for medical diagnosis with evolutionary algorithms. IEEE transactions on information technology in biomedicine: a publication of the IEEE Engineering. Med Biol Soc 15(1), 93–99 (2011). (PMID: 21062682) doi:10.1109/TITB.2010.2091144

    Article  Google Scholar 

  34. Lancaster, H.O.: The combination of probabilities: an application of orthonormal functions. Aust. N. Z. J. Stat. 3(1), 20–33 (1961)

    Article  MATH  MathSciNet  Google Scholar 

  35. Li, M., Vitányi, P.: An introduction to Kolmogorov complexity and its applications. 2nd edn. Springer, Secaucus (1997)

  36. Liptak, T.: On the combination of independent tests. Magyar Tud Akad Mat Kutato Int Kozl 3, 171–197 (1958)

    Google Scholar 

  37. Little, G., Krishna, S., Black, J., Panchanathan, S.: A methodology for evaluating robustness of face recognition algorithms with respect to variations in pose and illumination angle. In: Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 89–92. Philadelphia. (2005)

  38. Loughin, T.M.: A systematic comparison of methods for combining p-values from independent tests. Comput. Stat. Data Anal. 47(3), 467–485 (2004)

    Article  MATH  MathSciNet  Google Scholar 

  39. Luque, J., Morros, R., Garde, A., Anguita, J., Farrus, M., Macho, D., Marqus, F., Martnez, C., Vilaplana, V., Hernando, J.: Audio, video and multimodal person identification in a smart room. In: Multimodal Technologies for Perception of Humans, pp. 258–269 (2007)

  40. Marcel, S., McCool, C., Chakraborty, S., Balasubramanian, V., Panchanathan, S., Nolazco, J., Garcia, L., Aceves, R., et al.: Mobile biometry (mobio) face and speaker verification evaluation. In: Proceedings of the 20th International Conference on Pattern Recognition (ICPR2010) (2010)

  41. Michaelsen, E., Jaeger, K.: Evidence fusion using the GESTALT-system. In: Information Fusion, 2008 11th International Conference on, pp. 1–7. IEEE (2008)

  42. Mosteller, F., Bush, R.: Selected quantitative techniques. Handbook of Social Psychology (1954)

  43. Mudholkar, G.S., George, E.O.: The logit method for combining probabilities. In: Symposium on Optimizing Methods in Statistics, pp. 345–366. Academic Press, New York (1979)

  44. Murphy-Chutorian, E., Trivedi, M.: Head pose estimation in computer vision: a survey. IEEE Trans. Pattern Anal. Mach. Intell. 31(4), 607–626 (2009)

    Article  Google Scholar 

  45. Nolazco-Flores, J., Garcia-Perera, P.: Enhancing acoustic models for robust speaker verification. In: Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (2008)

  46. Nouretdinov, I., Melluish, T., Vovk, V.: Ridge regression confidence machine. In: Proceedings of the 18th International Conference on Machine Learning, pp. 385—392 (2001)

  47. Palanivel, S., Yegnanarayana, B.: Multimodal person authentication using speech, face and visual speech. Comput. Vis. Image Underst. 109(1), 44–55 (2008)

    Article  Google Scholar 

  48. Papadopoulos, H.: Inductive conformal prediction: theory and application to neural networks. In: Tools in Artificial Intelligence, pp. 315–329 (2008)

  49. Papadopoulos, H., Vovk, V., Gammerman, A.: Regression conformal prediction with nearest neighbours. J. Artif. Int. Res 40(1), 815–840 (2011). http://dl.acm.org/citation.cfm?id=20169452016967

    MATH  MathSciNet  Google Scholar 

  50. Pesarin, F.: Multivariate permutation tests: with applications in biostatistics, vol. 240. Wiley, Chichester (2001)

    Google Scholar 

  51. Poh, N., Korczak, J.: Hybrid biometric person authentication using face and voice features. In: Proceedings AVBPA, pp. 348–353 (2001)

  52. Proedrou, K.: Rigorous measures of confidence for pattern recognition and regression. PhD thesis, Royal Holloway College, University of London, Advisor-Alex Gammerman (2003)

  53. Rogova, G.L., Nimier, V.: Reliability in information fusion: literature survey. In: Svensson, P.,Schubert, J. (eds.) Proceedings of the 7th International Conference on Information Fusion. International Society of Information Fusion, vol. II, pp. 1158–1165. Mountain View, CA (2004)

  54. Sanderson, C.: Biometric person recognition: Face, speech and fusion. VDM Publishing (2008)

  55. Shafer, G.: A mathematical theory of evidence. Princeton University Press, Princeton (1976)

    MATH  Google Scholar 

  56. Shafer, G., Vovk, V.: A tutorial on conformal prediction. J. Mach. Learn. Res 9, 371–421 (2008)

    MATH  MathSciNet  Google Scholar 

  57. Shaffer, J.P.: Multiple hypothesis testing. Annual Review of Psychology 46(1), 561–584 (1995). doi:10.1146/annurev.ps.46.020195.003021

    Article  Google Scholar 

  58. Tippett, L.H.C.: The methods of statistics. 4th edn. Dover, New York (1963)

  59. Vovk, V.: On-line confidence machines are well-calibrated. In: FOCS ’02: 43rd Symposium on Foundations of Computer Science. IEEE Computer Society, pp. 187–196, Washington, DC. (2002)

  60. Vovk, V., Gammerman, A., Shafer, G.: Algorithmic Learning in a Random World. Springer, Secaucus (2005)

    MATH  Google Scholar 

  61. Wilkinson, B.: A statistical consideration in psychological research. Psychol. Bull. 48(3), 156–8 (1951)

    Article  MathSciNet  Google Scholar 

  62. Yang, F., Wang, H.-Z., Mi, H., de Lin, C., wen Cai, W.: Using random forest for reliable classification and cost-sensitive learning for medical diagnosis. BMC Bioinforma. 10(1), S22 (2009)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Indian Institute of Technology, Hyderabad, India

    Vineeth N. Balasubramanian, Shayok Chakraborty & Sethuraman Panchanathan

Authors
  1. Vineeth N. Balasubramanian
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shayok Chakraborty
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sethuraman Panchanathan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Vineeth N. Balasubramanian.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Balasubramanian, V.N., Chakraborty, S. & Panchanathan, S. Conformal predictions for information fusion. Ann Math Artif Intell 74, 45–65 (2015). https://doi.org/10.1007/s10472-013-9392-4

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10472-013-9392-4

Keywords

Mathematics Subject Classification

Search

Navigation