Skip to main content

Logistic Regression Bias Correction for Large Scale Data with Rare Events

  • Conference paper
Advanced Data Mining and Applications (ADMA 2013)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 8347))

Included in the following conference series:

Abstract

Logistic regression is a classical classification method, it has been used widely in many applications which have binary dependent variable. However, when the data sets are imbalanced, the probability of rare event is underestimated in the use of traditional logistic regression. With data explosion in recent years, some researchers propose large scale logistic regression which still fails to consider the rare event, therefore, there exists bias when applying their models for large scale data sets with rare events. To address the problems, this paper proposes LRBC method to correct bias of logistic regression for large scale data sets with rare events. Empirical studies compare LRBC with several state-of-the-art algorithms on an actual ad clicking data set. It demonstrates that LRBC method is able to exhibit much better classification performance, and the distributed process for bias correction also scales well.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Similar content being viewed by others

References

  1. Chan, P.K., Stolfo, S.J.: Toward scalable learning with non-uniform class and cost distributions: A case study in credit card fraud detection. In: KDD, vol. 1998, pp. 164–168 (1998)

    Google Scholar 

  2. Weiss, G.M., Hirsh, H.: Learning to predict extremely rare events. In: AAAI Workshop on Learning from Imbalanced Data Sets, pp. 64–68 (2000)

    Google Scholar 

  3. Kubat, M., Holte, R.C., Matwin, S.: Machine learning for the detection of oil spills in satellite radar images. Machine Learning 30(2-3), 195–215 (1998)

    Article  Google Scholar 

  4. King, G., Zeng, L.: Explaining rare events in international relations. International Organization 55(3), 693–715 (2001)

    Article  Google Scholar 

  5. King, G., Zeng, L.: Improving forecasts of state failure. World Politics 53(4), 623–658 (2001)

    Article  Google Scholar 

  6. Quigley, J., Bedford, T., Walls, L.: Estimating rate of occurrence of rare events with empirical bayes: A railway application. Reliability Engineering & System Safety 92(5), 619–627 (2007)

    Article  Google Scholar 

  7. Tsoucas, P.: Rare events in series of queues. Journal of Applied Probability, 168–175 (1992)

    Google Scholar 

  8. Lee, S.I., Lee, H., Abbeel, P., Ng, A.Y.: Efficient l1 regularized logistic regression. In: Proceedings of the National Conference on Artificial Intelligence, vol. 21, p. 401. AAAI Press, MIT Press, Menlo Park, Cambridge (2006)

    Google Scholar 

  9. Minka, T.P.: A comparison of numerical optimizers for logistic regression. Unpublished draft (2003)

    Google Scholar 

  10. Weiss, G.M.: Mining with rarity: a unifying framework. ACM SIGKDD Explorations Newsletter 6(1), 7–19 (2004)

    Article  Google Scholar 

  11. McCullagh, P., Nelder, J.A.: Generalized linear models. Monographs on statistics and applied probability, vol. 37. Chapman Hall, London (1989)

    Book  MATH  Google Scholar 

  12. Maalouf, M., Trafalis, T.B.: Robust weighted kernel logistic regression in imbalanced and rare events data. Computational Statistics & Data Analysis 55(1), 168–183 (2011)

    Article  MATH  MathSciNet  Google Scholar 

  13. Zhang, L., Jin, R., Chen, C., Bu, J., He, X.: Efficient online learning for large-scale sparse kernel logistic regression. In: AAAI (2012)

    Google Scholar 

  14. Liu, J., Chen, J., Ye, J.: Large-scale sparse logistic regression. In: Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 547–556. ACM (2009)

    Google Scholar 

  15. Genkin, A., Lewis, D.D., Madigan, D.: Large-scale bayesian logistic regression for text categorization. Technometrics 49(3), 291–304 (2007)

    Article  MathSciNet  Google Scholar 

  16. Lin, C.J., Weng, R.C., Keerthi, S.S.: Trust region newton method for large-scale logistic regression. The Journal of Machine Learning Research 9, 627–650 (2008)

    MATH  MathSciNet  Google Scholar 

  17. Yang, X., Sun, J.: An analytical performance model of mapreduce. In: 2011 IEEE International Conference on Cloud Computing and Intelligence Systems (CCIS), pp. 306–310. IEEE (2011)

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2013 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Qiu, Z., Li, H., Su, H., Ou, G., Wang, T. (2013). Logistic Regression Bias Correction for Large Scale Data with Rare Events. In: Motoda, H., Wu, Z., Cao, L., Zaiane, O., Yao, M., Wang, W. (eds) Advanced Data Mining and Applications. ADMA 2013. Lecture Notes in Computer Science(), vol 8347. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-53917-6_12

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-53917-6_12

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-53916-9

  • Online ISBN: 978-3-642-53917-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics