Simulation-based approach to estimation of latent variable models

https://doi.org/10.1016/j.csda.2006.02.016Get rights and content

Abstract

We propose a simulation-based method for calculating maximum likelihood estimators in latent variable models. The proposed method integrates a recently developed sampling strategy, the so-called Sample Average Approximation method, to efficiently compute high quality solutions of the estimation problem. Theoretical and algorithmic properties of the method are discussed. A computational study, involving two numerical examples, is presented to highlight a significant improvement of the proposed approach over existing methods.

Introduction

In recent years a considerable attention was attracted to parametric statistical models in which the probability distributions are defined in terms of multivariate integrals. Since, typically, the involved integrals cannot be written in a closed form, a popular approach to solving the obtained estimation problems is by using Monte Carlo (MC) simulation techniques (e.g., Bhat, 2001, Geyer and Thompson, 1992, Gouriéroux and Monfort, 1996, Lee, 1997). At the same time it was shown theoretically, and verified in numerical experiments, that MC simulation techniques can be surprisingly efficient in solving large scale stochastic programming problems (see Shapiro, 2003, Shapiro and Nemirovski, 2005, and references therein).

In this paper we discuss applications of the methodology and some theoretical results borrowed from stochastic programming to a class of latent variable models. Recent years have witnessed an increasing interest in using latent variable models in a large spectrum of applications, ranging from business/social science to medical science to engineering (cf., Bartholomew and Knott, 1999, Joreskog and Moustaki, 2001, Medel and Kamakura, 2001, Moustaki and Knott, 2000). As a defining feature of such models, latent variables are incorporated for modeling variance, correlation, dependence or other unobserved quantities. Maximum likelihood (ML) method and variants are standard techniques for parameter estimation in these models. The likelihood of such latent variable models are in forms of multivariate integrals, which makes calculating the ML estimators a challenging task.

The EM algorithm has been widely used as a solution to mitigate these computational difficulties (cf., Bartholomew and Knott, 1999, Moustaki and Knott, 2000). It starts with approximating the involved integrals by some deterministic methods, like Gaussian quadratures, and then iterates between E and M steps to calculate the estimators based on the approximated integrals. There are two major difficulties associated with an implementation of the EM algorithm for the considered class of problems. First, it is known that numerical approximations for multi-dimensional integrals are quite inaccurate and deterministic methods do not work in high dimensions (see, e.g., Evans and Swartz, 2000). Actually this would be a problem with any numerical procedure for solving such type of problems. Second, the iterative E and M schemes in the EM method do not make much sense from the optimization point of view, and it is known that convergence rates of the EM algorithm are rather slow. When dealing with small optimization problems, the drawback of slow convergence of the EM algorithm could be counterbalanced by simplicity of its implementation. However, for larger problems this slow convergence could be prohibitively expensive.

In this paper we study the ML estimation problem from a stochastic programming perspective. Stochastic programming is an emerging and important area in modern optimization that studies mathematical programming problems involving uncertainty. We show that the estimation problem can be framed as a generalized stochastic program. By integrating a sampling methodology, which became known in the area of stochastic programming as the sample average approximation (SAA) method, together with modern optimization techniques, we intend to show that it is possible to solve such type of problems in a considerably faster and more reliable way, and, furthermore, to validate quality of the obtained solutions by estimating errors resulting from the sampling approximations.

Although the proposed method is motivated and developed for latent variable models, similar estimation problems exist in many other areas, such as mixed-logit models in transportation (cf., Bhat, 2001) and generalized linear mixed models (cf., McCulloch and Searle, 2001), to which the proposed methodology can also be applied.

The remainder of the paper will unfold as follows. Section 2 discusses stochastic programming and SAA method. Section 3 introduces binary latent variable models. We discuss approaches to solving the ML problem by MC simulation in Section 4. Convergence analysis of the SAA method is provided in Section 5. In Section 6 we discuss methods for validation of SAA estimators. Section 7 illustrates the proposed method with two numerical examples involving binary latent trait models. A brief summary and some conclusions are outlined in Section 8.

Section snippets

Stochastic programming and sample average approximation method

Two-stage stochastic programming (with recourse), as an area in the field of optimization, can be traced back to Beale (1955) and Dantzig (1955). For an overview of recent theoretical and algorithmic developments in this field the interested reader can be referred to Ruszczyński and Shapiro (2003). In an abstract form a stochastic programming problem can be written in the formMinxX{f(x)EP[F(x,ξ)]},where xRm is a vector of decision variable, XRm is a given feasible set and ξ is a random

Binary latent variable models

Although the proposed method may work for general latent trait models, we specifically apply it to binary logit/normal models described as follows. Let x is a p-dimensional vector representing observed variables, which take binary values 0 or 1, and y be an m-dimensional vector of latent variables. The conditional density of x given y isg(x|y,λ)=i=1pπi(y;λ)xi1-πi(y;λ)1-xi,where the logit link is given asπi(y;λ)expλi0+j=1mλijyj1+expλi0+j=1mλijyj,λ=λij1ip,0jm are unknown parameters. Now

Solving the ML problem by Monte Carlo simulation

One of the main difficulties in solving the ML optimization problem (3.6) is that the functions φk(λ) are not given explicitly and involve calculations of multivariate integrals. In this section we discuss a numerical approach to solving (3.6) by using MC sampling techniques coupled with modern optimization algorithms. As it was discussed in Section 2, the basic idea of our approach is quite simple, in order to evaluate the expected value function φk(λ) we generate a random sample Y1,,YSk of Sk

Convergence analysis of SAA algorithm

In this section, we present a convergence analysis for the SAA program (4.3). Denote by v^S and λ^S the optimal value and an optimal solution of problem (4.3), respectively. We view v^S and λ^S as estimates (approximations) of their counterparts v* and λ* of the “true” ML problem (3.6). Note that we treat now the ML problem (3.6) as fixed (deterministic) and view (4.3) as its SAA. In that framework the estimates v^S and λ^S depend on the generated MC sample and therefore are treated as random

Validation of SAA estimators

In the Econometric (and Statistics) literature, statistical inference of MC simulation-based estimators is discussed in an asymptotic sense as the sample sizes tend to infinity (e.g., Gouriéroux and Monfort, 1996, Lee, 1997). One of the advantages of the proposed methodology is the ability to assess the accuracy of calculated solutions for a given (generated) MC sample. This is in contrast with existing methods, like the EM algorithm (Bartholomew and Knott, 1999, Moustaki and Knott, 2000),

Numerical examples

Our computation in the following sections are implemented to run in an IBM PC with Intel (R) Pentium (R) 4 CPU 2.66 GHz and Microsoft Windows 2000 operational system. We use the nonlinear optimization solver Systems Optimization Laboratory at Stanford University, MINOS (2005) in GAMS (2005) for numerical calculations and use R (2005) for MC and LH sampling. MINOS solves nonlinear optimization problems using a reduced-gradient algorithm (Murtagh and Saunders, 1978) combined with a quasi-Newton

Summary and conclusions

In this paper, we have developed a simulation-based method for calculating maximum likelihood estimators in latent variable models. The proposed method makes use of a sampling strategy recently developed in stochastic programming, namely the Sample Average Approximation (SAA) method, to efficiently compute the solutions for the estimation problem. Algorithmic and theoretical issues related to the convergence analysis of SAA method and validation of the SAA estimators have been discussed. The

Acknowledgement

We are grateful to Dr. Shabbir Ahmed for his suggestions and assistance in computing.

References (35)

  • GAMS Development Corporation, 2005. GAMS: A General Algebraic Modeling System, Version 2.5, available at...
  • C.J. Geyer et al.

    Constrained Monte Carlo maximum likelihood for dependent data (with discussion)

    J. Roy. Statist. Soc. Ser. B

    (1992)
  • C. Gouriéroux et al.

    Simulation-Based Econometric Methods

    (1996)
  • M.-gao. Gu et al.

    Maximum likelihood estimation for spatial models by Markov chain Monte Carlo stochastic approximation

    J. Roy. Statist. Soc. Ser. B

    (2001)
  • K. Joreskog et al.

    Factor analysis of ordinal variables: a comparsion of three approaches

    Multivariate Behavioral Res.

    (2001)
  • T.L. Lai

    Stochastic approximation: invited paper

    Ann. Statist.

    (2003)
  • A. Maydeu-Olivares et al.

    Limited and full-information estimation and goodness-of-fit testing in 2n contingency tables: a unified framework

    J. Amer. Statist. Assoc.

    (2005)
  • Cited by (6)

    • Indirect density estimation using the iterative Bayes algorithm

      2011, Computational Statistics and Data Analysis
    • Iterated filtering

      2011, Annals of Statistics
    • Planning and Integration of Refinery and Petrochemical Operations

      2010, Planning and Integration of Refinery and Petrochemical Operations
    • Integration and coordination of multirefinery networks: A robust optimization approach

      2008, Proceedings of the IASTED International Conference on Modelling and Simulation
    1

    Research of this author was partly supported by the NSF Grant DMS-0510324.

    View full text