A fast algorithm for non-negativity model selection

Abstract

An efficient optimization algorithm for identifying the best least squares regression model under the condition of non-negative coefficients is proposed. The algorithm exposits an innovative solution via the unrestricted least squares and is based on the regression tree and branch-and-bound techniques for computing the best subset regression. The aim is to filling a gap in computationally tractable solutions to the non-negative least squares problem and model selection. The proposed method is illustrated with a real dataset. Experimental results on real and artificial random datasets confirm the computational efficacy of the new strategy and demonstrates its ability to solve large model selection problems that are subject to non-negativity constrains.

Appendices

Appendix A: Selection criteria equivalence

Lemma 2

Let V={1,2,…,n} denote a set of variables. Finding the best-subset regression with respect to some selection criterion f∈{RSS,AIC,BIC,…} reduces to finding the n-list of the best subset models corresponding to each model size p=1,…,n.

Proof

Without loss of generality it is assumed that the selection criterion f is to be minimized. The problem of finding the best subset-model with respect to f can be written as

(6)

Note that

Now, consider that the computed the n-list of the best submodels of each size p is given by

The problem (6) becomes

This completes the proof. □

There are $\binom{n}{p}$ candidate submodels that selects p variables out of n. The total number of submodels of V when excluding the empty model is $\sum_{p=1}^{n} \binom{n}{p} = 2^{n} - 1$. Lemma 2 states that once the n-list of the best submodels corresponding to each model size have been computed, the best submodel of V is obtained as the optimum of this n-best list.

Property 1

Let V={1,2,…,n} denote a set of variables. A selection criterion is said to be monotonous in the RSS for equal-size subsets if

Standard selection criteria such as AIC and BIC satisfies the above property, since for equal-size subsets the penalty term is constant.

Lemma 3

Let V={1,2,…,n} denote a set of variables and p such that 1≤p≤n. If f is a selection criterion that satisfies the monotonicity property 1, then

Proof

Let

It follows that

Using the monotonicity property 1 the latter becomes

which implies $f(W_{p}^{*}) =f(\widehat{W}_{p})$. This completes the proof. □

Lemma 3 states that if the model size p is fixed, then the same best submodel optimizes the RSS as well as any selection criterion that satisfies monotonicity in RSS Property 1.

Appendix B: The Survival data

Details and a description of the data can be found in Armstrong and Frome (1976) and Cutler (1993). The use of the original data derives the best-subset solution in the root node of the regression tree. Thus, for illustration purposes, in the case study the first and last independent variables (a ₂ and a ₄) have been interchanged to give the dataset below. The constant variable a ₁=1 has been added. The variable labels as they appear in the original data are indicated in the last column.