Extensions to the repetitive branch and bound algorithm for globally optimal clusterwise regression

Abstract

A branch and bound strategy is proposed for solving the clusterwise regression problem, extending Brusco's repetitive branch and bound algorithm (RBBA). The resulting strategy relies upon iterative heuristic optimization, new ways of observation sequencing, and branch and bound optimization of a limited number of ending subsets. These three key features lead to significantly faster optimization of the complete set and the strategy has more general applications than only for clusterwise regression. Additionally, an efficient implementation of incremental calculations within the branch and bound search algorithm eliminates most of the redundant ones. Experiments using both real and synthetic data compare the various features of the proposed optimization algorithm and contrasts them against a benchmark mixed logical-quadratic programming formulation optimized by CPLEX. The results indicate that all components of the proposed algorithm provide significant improvements in processing times, and, when combined, generally provide the best performance, significantly outperforming CPLEX.

Introduction

Clusterwise regression is a clustering technique where multiple lines or hyperplanes are fit to mutually exclusive subsets of a dataset such that the sum of squared errors (SSE) from each observation to its cluster's line is minimized [1], [2], [3]. The term line will be used in this paper for both line and hyperplane. Clusterwise regression has relevance to such areas as spline estimation, utility function clustering and response based segmentations of customers, markets, regions, subjects, strategies or investors [1], [2], [3], [4], [5]. Optimization for clusterwise regression is considered “a tough combinatorial optimization problem” [4], and it appears that the only currently known feasible method for global optimization is by mixed logical-quadratic programming (MLQP) [6].

A new paradigm, the repetitive branch and bound algorithm (RBBA) has recently been proposed by Brusco and Stahl for clustering, seriation and variable selection [7], [8]. It works by sequencing the data and optimizing by branch and bound (BB) a series of problems corresponding to ending subsets with one more observation at a time [8]. An ending subset is a sequential subset of observations whose last observation is also the last observation of the complete set. Values of the solved problems are used to strengthen the lower bounds of the search at the current iteration. Building on this previous work, the present paper proposes an extended branch and bound strategy which combines iterative heuristic optimization, new ways of sequencing the observations, and branch and bound optimization of a limited number of ending subsets. These three key features lead to significantly faster optimization of the complete set and the strategy has more general applications than only for clusterwise regression.

Clusterwise regression is a cubic optimization problem defined by the number of clusters (K), the number of independent dimensions (D), and the number of observations (O). The iterators are for a cluster (k∈{1,…,K}), an independent dimension (d∈{1,…,D}), and an observation (o∈{1,…,O}). The model parameters are the independent variable for an observation and dimension (x_od) and the dependent variable for an observation (y_o). The model variables are the cluster assignment of an observation to a cluster (z_ok), the regression coefficient (aka β) for a dimension of a cluster (b_dk) and the error for an observation of a cluster (e_ok). The cubic model is as follows:

$$\mathrm{SSE}=\min \sum _{k=1}^K\sum _{o=1}^O(z_{ok}{e}_{ok}^2)$$

$$\mathrm{s}.\mathrm{t}.\sum _{d=1}^D(b_{dk}{x}_{od})+e_{ok}=y_{o}o=1,\dots ,O,k=1,\dots ,K$$

$$\sum _{k=1}^K{z}_{ok}=1o=1,\dots ,O$$

$$z_{ok}\in \{0,1\}o=1,\dots ,O,k=1,\dots ,K$$

The objective (1) is the minimization of the sum over all clusters of the sum of squared errors (SSE) for their observations relative to their regression line. The constraint (2) fits the regression lines to the data by adjusting the coefficient and error terms. An observation can only be assigned to one cluster at a time (3) and the cluster assignment is binary (4). This formulation does not explicitly require an intercept but it can be included in the model simply by adding a variable to the data with a constant of one. All models in this paper include an intercept, thus D is always one more than the number of independent variables in the original dataset. Since there are K^O possible clustering configurations and since there is a minimum of D²O regression computations [9] to perform per clustering configuration, the enumeration of the complete problem search space requires at least K^OD²O operations.

Although identifying the globally optimal solution to a clusterwise regression problem by no means guarantees identifying the true model, on average, these globally optimal solutions will lead to better models than random local optima identified by heuristics. However, as stressed by Brusco et al., clusterwise regression makes no effort to distinguish between error explained by clustering and error explained by regression [10]. Also, since clusterwise regression fits multiple lines to the data, the overfitting potential is much greater than that of a single regression line. Consequently an evaluation procedure has been proposed to test if there is overfitting or not [10]. Nevertheless, evaluating and addressing this overfitting problem is not in the scope of the current research and neither is the statistical validity of identified optimal clusterwise regression models. This research considers only the feasibility and processing time for finding the optimal solution to a clusterwise multiple linear regression problem (OCMLR).

This paper is structured as follows: Section 2 provides an overview of some previous heuristic and exact optimization approaches, Section 3 details the proposed exact global optimization strategy, Section 4 overviews the experimental protocol and datasets, Section 5 presents the results and related discussion. Conclusions are presented in Section 6.