Out-of-Sample Evaluation

Definition

Out-of-sample evaluation refers to algorithm evaluation whereby the learned model is evaluated on out-of-sample data, which are data that were not used in the process of learning the model. Out-of-sample evaluation provides a less biased estimate of learning performance than in-sample evaluation. Cross validation, holdout evaluation and prospective evaluation are three main approaches to out-of-sample evaluation. Cross validation and holdout evaluation run risks of overestimating performance relative to what should be expected on future data, especially if the data set used is not a true random sample of the distribution on which the learned models are to be applied in the future.

Cross-References

• Algorithm Evaluation