Skip to main content

Cross Entropy

  • Reference work entry
  • First Online:
Computer Vision

Definition

Cross entropy is a concept in information theory to measure the independence of two probability distributions.

Theory

For two distributions p(x) and q(x) defined on the same space, the cross entropy is defined as

$$\displaystyle\begin{array}{rcl} H(p,q)& =& \mathrm{E}_{p}[-\log q(X)] \\ & =& \mathrm{E}_{p}[-\log p(X)] + \mathrm{E}_{p}[\log (p(X)/q(X))] \\ & =& H(p) + KL(p,q), \\ \end{array}$$

where \(H(p) = \mathrm{E}_{p}[-\log p(X)]\) is the entropy of p and \(KL(p,q) = \mathrm{E}_{p}[\log (p(X)/q(X))]\) is the Kullback-Leibler divergence from p to q. The Kullback-Leibler divergence is also called relative entropy.

The cross entropy method is a Monte Carlo method for rare event simulation and stochastic optimization [2].

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Cover TM, Thomas JA (1991) Elements of information theory. Wiley, New York

    Book  MATH  Google Scholar 

  2. Rubinstein RY (1997) Optimization of computer simulation models with rare events. Eur J Oper Res 99:89–112

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2014 Springer Science+Business Media New York

About this entry

Cite this entry

Wu, Y.N. (2014). Cross Entropy. In: Ikeuchi, K. (eds) Computer Vision. Springer, Boston, MA. https://doi.org/10.1007/978-0-387-31439-6_743

Download citation

Publish with us

Policies and ethics