Skip to main content
SpringerLink
Log in

Gradient Ascent for Best Response Regression

  • Conference paper
  • First Online:
Advances in Intelligent Data Analysis XIX (IDA 2021)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 12695))

Included in the following conference series:

  • 820 Accesses

Abstract

Although regression is among the oldest areas of statistics, new approaches may still be found. One recent suggestion is Best Response Regression, where one tries to find a regression function that provides, for as many instances as possible, a better prediction than some reference regression function. In this paper we propose a new method for Best Response Regression that is based on gradient ascent rather than mixed integer programming. We evaluate our approach for a variety of noise (or error) distributions, showing that especially for heavy-tailed distributions best response regression outperforms, on unseen data, ordinary least squares regression, both w.r.t. the sum of squared errors as well as the number of instances for which better predictions are provided.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    These implementations are publicly available at www.borgelt.net/brreg.html.

  2. 2.

    All result diagrams are available at www.borgelt.net/docs/brreg.pdf.

References

  1. Beaton, A.E., Tukey, J.W.: The fitting of power series, meaning polynomials, illustrated on band-spectroscopic data. Technometrics 16, 147–185. American Society for Quality and the American Statistical Association, Milwaukee, WI and Boston, MA, USA (1974)

    Google Scholar 

  2. Ben-Porat, O., Tennenholtz, M.: Best response regression. In: Advances in Neural Information Processing Systems 30 (NIPS 2017, Long Beach, CA), Curran Associates, Red Hook, NY, USA pp. 1499–1508 (2017)

    Google Scholar 

  3. Embrechts, P., Klüppelberg, C., Mikosch, T.: Modelling Extremal Events: For Insurance and Finance. Springer Science & Business Media, Berlin, Germany (2013)

    MATH  Google Scholar 

  4. Dozat, T.: Incorporating nesterov momentum into adam. In: Proceedings International Conference on Learning Representations (ICLR Workshop 2016, San Juan, Puerto Rico). openreview.net (2016)

    Google Scholar 

  5. Duchi, J., Hazan, E., Singer, Y.: Adaptive subgradient methods for online learning and stochastic optimization. J. Mach. Learn. Res. 12, 2121–2159. Microtome Publishing, Brookline, MA, USA (2011)

    Google Scholar 

  6. Fahlman, S.E.: An Empirical Study of Learning Speed in Backpropagation Networks. Proceedings of the Connectionist Models Summer School (Carnegie Mellon University). Morgan Kaufman, San Mateo, CA, USA (1988)

    Google Scholar 

  7. Huber, P.J.: Robust estimation of a location parameter. In: Kotz, S., Johnson, N.L. (eds.) Breakthroughs in Statistics. Springer Series in Statistics (Perspectives in Statistics). Springer, New York, NY (1992). https://doi.org/10.1007/978-1-4612-4380-9_35

  8. Huber, P.J.: Robust Statistics. J. Wiley & Sons, New York, NY, USA (1981)

    Book  Google Scholar 

  9. D.P. Kingma and J. Ba. Adam: A method for stochastic optimization. In: Proceedings International Conference on Learning Representations (ICLR 2015, San Diego, CA). openreview.net (2015)

    Google Scholar 

  10. Nesterov, Y.E.: A method of solving a convex programming problem with convergence rate \(O(1/k^2)\). Soviet Math. Doklady 27(2), 372–376 (1983)

    MATH  Google Scholar 

  11. Polyak, B.T.: Some methods of speeding up the convergence of iteration methods. USSR Comput. Math. Math. Phys. 4(5), 1–17 (1964)

    Google Scholar 

  12. Riedmiller, M., Braun, H.: Rprop-a fast adaptive learning algorithm. Technical Report, University of Karlsruhe, Karlsruhe, Germany (1992)

    Google Scholar 

  13. Riedmiller, M., Braun, H.: A direct adaptive method for faster backpropagation learning: the RPROP algorithm. In: International Conference on Neural Networks (ICNN-93, San Francisco, CA), pp. 586–591. IEEE (1993)

    Google Scholar 

  14. Tieleman, T., Hinton, G.: Lecture 6.5-RMSPROP: divide the gradient by a running average of its recent magnitude. COURSERA: Neural Netw. Mach. Learn. 4(2), 26–31 (2012)

    Google Scholar 

  15. Tollenaere, T.: SuperSAB: Fast adaptive back propagation with good scaling properties. Neural Netw. 3(5), 561–573 (1990)

    Article  Google Scholar 

  16. Zeiler, M.D.: AdaDelta: an adaptive learning rate method. arXiv:1212.5701 (2012)

Download references

Acknowledgments

The second author gratefully acknowledges the financial support from Land Salzburg within the WISS 2025 project IDA-Lab (20102-F1901166-KZP and 20204-WISS/225/197-2019).

Author information

Authors and Affiliations

  1. Department of Mathematics, University of Salzburg, Salzburg, Austria

    Victoria Racher & Christian Borgelt

  2. Department of Computer Sciences, University of Salzburg, Salzburg, Austria

    Christian Borgelt

  3. Paracelsus Medical University, Salzburg, Austria

    Victoria Racher

Authors
  1. Victoria Racher
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Christian Borgelt
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Victoria Racher .

Editor information

Editors and Affiliations

  1. University of Coimbra, Coimbra, Portugal

    Pedro Henriques Abreu

  2. University of Porto, Porto, Portugal

    Pedro Pereira Rodrigues

  3. University of Granada, Granada, Spain

    Alberto Fernández

  4. University of Porto, Porto, Portugal

    João Gama

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Racher, V., Borgelt, C. (2021). Gradient Ascent for Best Response Regression. In: Abreu, P.H., Rodrigues, P.P., Fernández, A., Gama, J. (eds) Advances in Intelligent Data Analysis XIX. IDA 2021. Lecture Notes in Computer Science(), vol 12695. Springer, Cham. https://doi.org/10.1007/978-3-030-74251-5_12

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-74251-5_12

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-74250-8

  • Online ISBN: 978-3-030-74251-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation