Simulation-based approach to estimation of latent variable models
Introduction
In recent years a considerable attention was attracted to parametric statistical models in which the probability distributions are defined in terms of multivariate integrals. Since, typically, the involved integrals cannot be written in a closed form, a popular approach to solving the obtained estimation problems is by using Monte Carlo (MC) simulation techniques (e.g., Bhat, 2001, Geyer and Thompson, 1992, Gouriéroux and Monfort, 1996, Lee, 1997). At the same time it was shown theoretically, and verified in numerical experiments, that MC simulation techniques can be surprisingly efficient in solving large scale stochastic programming problems (see Shapiro, 2003, Shapiro and Nemirovski, 2005, and references therein).
In this paper we discuss applications of the methodology and some theoretical results borrowed from stochastic programming to a class of latent variable models. Recent years have witnessed an increasing interest in using latent variable models in a large spectrum of applications, ranging from business/social science to medical science to engineering (cf., Bartholomew and Knott, 1999, Joreskog and Moustaki, 2001, Medel and Kamakura, 2001, Moustaki and Knott, 2000). As a defining feature of such models, latent variables are incorporated for modeling variance, correlation, dependence or other unobserved quantities. Maximum likelihood (ML) method and variants are standard techniques for parameter estimation in these models. The likelihood of such latent variable models are in forms of multivariate integrals, which makes calculating the ML estimators a challenging task.
The EM algorithm has been widely used as a solution to mitigate these computational difficulties (cf., Bartholomew and Knott, 1999, Moustaki and Knott, 2000). It starts with approximating the involved integrals by some deterministic methods, like Gaussian quadratures, and then iterates between E and M steps to calculate the estimators based on the approximated integrals. There are two major difficulties associated with an implementation of the EM algorithm for the considered class of problems. First, it is known that numerical approximations for multi-dimensional integrals are quite inaccurate and deterministic methods do not work in high dimensions (see, e.g., Evans and Swartz, 2000). Actually this would be a problem with any numerical procedure for solving such type of problems. Second, the iterative E and M schemes in the EM method do not make much sense from the optimization point of view, and it is known that convergence rates of the EM algorithm are rather slow. When dealing with small optimization problems, the drawback of slow convergence of the EM algorithm could be counterbalanced by simplicity of its implementation. However, for larger problems this slow convergence could be prohibitively expensive.
In this paper we study the ML estimation problem from a stochastic programming perspective. Stochastic programming is an emerging and important area in modern optimization that studies mathematical programming problems involving uncertainty. We show that the estimation problem can be framed as a generalized stochastic program. By integrating a sampling methodology, which became known in the area of stochastic programming as the sample average approximation (SAA) method, together with modern optimization techniques, we intend to show that it is possible to solve such type of problems in a considerably faster and more reliable way, and, furthermore, to validate quality of the obtained solutions by estimating errors resulting from the sampling approximations.
Although the proposed method is motivated and developed for latent variable models, similar estimation problems exist in many other areas, such as mixed-logit models in transportation (cf., Bhat, 2001) and generalized linear mixed models (cf., McCulloch and Searle, 2001), to which the proposed methodology can also be applied.
The remainder of the paper will unfold as follows. Section 2 discusses stochastic programming and SAA method. Section 3 introduces binary latent variable models. We discuss approaches to solving the ML problem by MC simulation in Section 4. Convergence analysis of the SAA method is provided in Section 5. In Section 6 we discuss methods for validation of SAA estimators. Section 7 illustrates the proposed method with two numerical examples involving binary latent trait models. A brief summary and some conclusions are outlined in Section 8.
Section snippets
Stochastic programming and sample average approximation method
Two-stage stochastic programming (with recourse), as an area in the field of optimization, can be traced back to Beale (1955) and Dantzig (1955). For an overview of recent theoretical and algorithmic developments in this field the interested reader can be referred to Ruszczyński and Shapiro (2003). In an abstract form a stochastic programming problem can be written in the formwhere is a vector of decision variable, is a given feasible set and is a random
Binary latent variable models
Although the proposed method may work for general latent trait models, we specifically apply it to binary logit/normal models described as follows. Let is a p-dimensional vector representing observed variables, which take binary values 0 or 1, and be an m-dimensional vector of latent variables. The conditional density of given iswhere the logit link is given as are unknown parameters. Now
Solving the ML problem by Monte Carlo simulation
One of the main difficulties in solving the ML optimization problem (3.6) is that the functions are not given explicitly and involve calculations of multivariate integrals. In this section we discuss a numerical approach to solving (3.6) by using MC sampling techniques coupled with modern optimization algorithms. As it was discussed in Section 2, the basic idea of our approach is quite simple, in order to evaluate the expected value function we generate a random sample of
Convergence analysis of SAA algorithm
In this section, we present a convergence analysis for the SAA program (4.3). Denote by and the optimal value and an optimal solution of problem (4.3), respectively. We view and as estimates (approximations) of their counterparts and of the “true” ML problem (3.6). Note that we treat now the ML problem (3.6) as fixed (deterministic) and view (4.3) as its SAA. In that framework the estimates and depend on the generated MC sample and therefore are treated as random
Validation of SAA estimators
In the Econometric (and Statistics) literature, statistical inference of MC simulation-based estimators is discussed in an asymptotic sense as the sample sizes tend to infinity (e.g., Gouriéroux and Monfort, 1996, Lee, 1997). One of the advantages of the proposed methodology is the ability to assess the accuracy of calculated solutions for a given (generated) MC sample. This is in contrast with existing methods, like the EM algorithm (Bartholomew and Knott, 1999, Moustaki and Knott, 2000),
Numerical examples
Our computation in the following sections are implemented to run in an IBM PC with Intel (R) Pentium (R) 4 CPU 2.66 GHz and Microsoft Windows 2000 operational system. We use the nonlinear optimization solver Systems Optimization Laboratory at Stanford University, MINOS (2005) in GAMS (2005) for numerical calculations and use R (2005) for MC and LH sampling. MINOS solves nonlinear optimization problems using a reduced-gradient algorithm (Murtagh and Saunders, 1978) combined with a quasi-Newton
Summary and conclusions
In this paper, we have developed a simulation-based method for calculating maximum likelihood estimators in latent variable models. The proposed method makes use of a sampling strategy recently developed in stochastic programming, namely the Sample Average Approximation (SAA) method, to efficiently compute the solutions for the estimation problem. Algorithmic and theoretical issues related to the convergence analysis of SAA method and validation of the SAA estimators have been discussed. The
Acknowledgement
We are grateful to Dr. Shabbir Ahmed for his suggestions and assistance in computing.
References (35)
Quasi-random maximum simulated likelihood estimation of the mixed multinomial logit model
Transportation Res.
(2001)Simulated maximum likelihood estimation of dynamic discrete choice statistical models, Some Monte Carlo results
J. Econometrics
(1997)- et al.
Monte Carlo bounding techniques for determining solution quality in stochastic programs
Oper. Res. Lett.
(1999) - et al.
Integrated variance reduction strategies for simulation
Oper. Res.
(1996) Latent variable models and factor analysis
(1987)- et al.
Latent Variable Models and Factor Analysis
(1999) On minimizing a convex function subject to linear inequalities
J. Roy. Statist. Soc. Ser. B
(1955)Linear programming under uncertainty
Management Sci.
(1955)- et al.
Convergence of a stochastic approximation version of the EM algorithm
Ann. Statist.
(1999) - et al.
Approximating Integrals via Monte Carlo and Deterministic Methods
(2000)
Constrained Monte Carlo maximum likelihood for dependent data (with discussion)
J. Roy. Statist. Soc. Ser. B
Simulation-Based Econometric Methods
Maximum likelihood estimation for spatial models by Markov chain Monte Carlo stochastic approximation
J. Roy. Statist. Soc. Ser. B
Factor analysis of ordinal variables: a comparsion of three approaches
Multivariate Behavioral Res.
Stochastic approximation: invited paper
Ann. Statist.
Limited and full-information estimation and goodness-of-fit testing in contingency tables: a unified framework
J. Amer. Statist. Assoc.
Cited by (6)
Indirect density estimation using the iterative Bayes algorithm
2011, Computational Statistics and Data AnalysisRobust planning of multisite refinery networks: Optimization under uncertainty
2010, Computers and Chemical EngineeringIterated filtering
2011, Annals of StatisticsEnhancing the sample average approximation method with U designs
2010, BiometrikaPlanning and Integration of Refinery and Petrochemical Operations
2010, Planning and Integration of Refinery and Petrochemical OperationsIntegration and coordination of multirefinery networks: A robust optimization approach
2008, Proceedings of the IASTED International Conference on Modelling and Simulation
- 1
Research of this author was partly supported by the NSF Grant DMS-0510324.