Unified generalized iterative scaling and its applications

Abstract

Generalized iterative scaling (GIS) has become a popular method for getting the maximum likelihood estimates for log-linear models. It is basically a sequence of successive $I$-projections onto sets of probability vectors with some given linear combinations of probability vectors. However, when a sequence of successive $I$-projections are applied onto some closed and convex sets (e.g., marginal stochastic order), they may not lead to the actual solution. In this manuscript, we present a unified generalized iterative scaling (UGIS) and the convergence of this algorithm to the optimal solution is shown. The relationship between the UGIS and the constrained maximum likelihood estimation for log-linear models is established. Applications to constrained Poisson regression modeling and marginal stochastic order are used to demonstrate the proposed UGIS.

Introduction

Let $p$ and $\pi$ be two probability vectors in $R^k$. The $I$-divergence of $p$ with respect to $\pi$ (also known as the Kullback–Liebler information number, cross entropy and information for discrimination) is defined by

$$I\left(p\right|\pi )=\sum _{i=1}^k{p}_{i}log(p_i/{\pi }_i)\text{.}$$

For any given $\pi$, it is often of interest to find $p^{\ast }$ in some set $E$ such that

$$I\left(p^{\ast }\right|\pi )=\sum _{i=1}^k{p}_i^{\ast }log(p_i^{\ast }/{\pi }_i)=\underset{p\in E}{min}\sum _{i=1}^k{p}_{i}log(p_i/{\pi }_i)\text{.}$$

The above minimization problem is usually called the $I$-projection problem, for $\pi$ onto $E$, and the associated solution $p^{\ast }$ is called the $I$-projection of $\pi$ onto $E$. It has long been known that $I$-projection plays a key role in the information theoretic approach to statistics (Kullback, 1959, Good, 1963, Bishop et al., 1975).

In some applications, $E$ is usually assumed to be of the form $\{p:h-A^{\prime }p\in K\}$ for some given vector $h$, matrix $A$ and convex cone $K$. This form often occurs in statistics. For $K=\left\{{(0,\dots ,0)}^{\prime }\right\}$, Deming and Stephan (1940) formally introduced the iterative proportional fitting procedure (IPFP) to adjust cell frequencies of contingency tables when all elements of $A$ are equal to 0 or 1. Ireland and Kullback (1968) showed the convergence of IPFP to the $I$-projection when marginals of contingency tables are given. Darroch and Ratcliff (1972) proposed the generalized iterative scaling (GIS) to obtain the $I$-projection for general $A$ and established the relations between maximum likelihood estimation for log-linear models and $I$-projections. Dykstra and Lemke (1988) demonstrated that the maximum likelihood estimation for discrete distributions has close relationships with $I$-projections onto $E=\{p:h-A^{\prime }p\in K\}$ for some $h$ and $A$. Dykstra (1985) proposed an iterative procedure for obtaining $I$-projections onto the intersection of convex sets, and Dykstra and Wollan (1987) devised a computer program based on the iterative procedure. Winkler (1990), Bhattacharya and Dykstra, 1995, Bhattacharya and Dykstra, 1997 and Kuroda and Geng (1999) considered similar problems.

Kullback (1968), Csiszar, 1975, Csiszar, 1989, Haberman (1984) and Ruschendorf and Thomsen (1993) considered $I$-projection problems for probability measures. Ruschendorf (1995) demonstrated that the IPFP for probability also converges to the $I$-projection with given marginals. Bhattacharya (2006) considered an iterative procedure for probability measures to obtain $I$-projections onto the intersection of convex sets.

Gao and Shi (2003) considered the $I$-projection problem when $K$ consists of some inequality constraints. They proposed an iterative algorithm for finding the solutions and proved that the proposed algorithm converges to an $I$-projection. The algorithm partly generalizes GIS. They also established the relationship between $I$-projections and the maximum likelihood estimation for log-linear models with ordered parameters.

Analysis of ordinal data is a challenging problem. One of the popular models for ordinal data is the log-linear model, whose parameters are often restricted by some ordering such as odds ratios increasing with ordinal categories (Agresti and Coull, 2002). The aim of this paper is to provide new algorithms for analyzing ordinal data. An iterative algorithm is proposed for computing $I$-projections for when $K=C^{\ast }$ and $C^{\ast }$ is the Fenchel dual cone of an isotonic cone $C$. The algorithm reduces to the famous GIS when $C^{\ast }=\left\{{(0,\dots ,0)}^{\prime }\right\}$ is chosen. The new method is called the unified generalized iterative scaling (UGIS). Relationships between $I$-projections and maximum likelihood estimations of restricted parameters for log-linear models are demonstrated. This paper is organized as follows. In Section 2, an $I$-projection problem is considered and relations between $I$-projections and log-linear models are given. The UGIS is introduced and the related algorithms are proposed in Section 3. The relationships between UGIS and maximum likelihood estimation of constrained parameters for log-linear models are established. Poisson regression modeling and marginal stochastic order are used to demonstrate the proposed algorithms in Section 4.