Bayesian estimation based on trimmed samples from Pareto populations

doi:10.1016/j.csda.2005.11.010

Computational Statistics & Data Analysis

Volume 51, Issue 2, 15 November 2006, Pages 1119-1130

https://doi.org/10.1016/j.csda.2005.11.010 Get rights and content

Abstract

Trimmed samples are widely employed in several areas of statistical practice, especially when some sample values at either or both extremes might have been contaminated. The problem of estimating the inequality and precision parameters of a Pareto distribution based on a trimmed sample and prior information is considered. From an inferential viewpoint, the problem of finding the highest posterior density (HPD) estimates of the Pareto parameters is discussed. The existence and uniqueness of the HPD estimates are established under mild conditions; explicit and accurate lower and upper bounds are also provided. Adopting a decision-theoretic perspective, several Bayesian estimators for standard loss functions are presented. In addition, two-sided and HPD credibility intervals for each Pareto parameter and joint HPD credibility regions for both parameters are derived, which have the corresponding frequentist confidence level in the noninformative case. Finally, an illustrative example concerning annual wage data is included.

Introduction

Trimmed samples are commonly used for several applications including some robust estimation procedures (e.g., Prescott, 1978, Huber, 1981, Healy, 1982, Welsh, 1987, Wilcox, 1995). For instance, trimmed means make up a well-known class of robust location estimates that range from the sample mean to the sample median. The trimming removes a fixed proportion of the outlying sample values, i.e., certain proportions $q_{1}$ and $q_{2}$ of the smallest and largest observations are eliminated. On the other hand, in various situations some data may not be observed due to time limitations and other restrictions on data collection. In particular, a known number of observations in an ordered sample are missing at either end (single censoring) or at both ends (double censoring) in failure censored experiments. Specifically, double censoring has been widely treated in the literature (e.g., Ng, 1976, Healy, 1978, Prescott, 1979, Schneider, 1984, LaRiccia, 1986, Schneider and Weissfeld, 1986, Escobar and Meeker, 1994, Upadhyay and Shastri, 1997; Fernández, 2000a, Fernández, 2000b; Fernández, 2004, Fernández et al., 2002, Raqab and Madi, 2002, Ali Mousa, 2003).

The Pareto distribution provides a population model which has a wide variety of applications in many fields. Concretely, applications are seen in insurance risk studies, property values, income, stock price fluctuations, migration, size of cities and firms, word frequencies, availability of natural resources, service time in queuing systems, error clustering in communications circuits and business failures. The probability density function (pdf) and cumulative distribution function (cdf) of a random variable $X$ having a Pareto law $P (α, τ)$ with inequality (or shape) parameter $α > 0$ and precision parameter $τ > 0$ are given by $f (x; α, τ) = τ α (τ x)^{- α - 1} and F (x; α, τ) = 1 - (τ x)^{- α}, x ⩾ 1 / τ .$ From the Bayesian perspective, $(α, τ)$ is not merely an unknown fixed quantity but rather a random variable that is characterized by some prior pdf. This point of view has been considered by many authors (e.g., Lwin, 1972, Arnold and Press, 1983, Arnold and Press, 1986, Arnold and Press, 1989, Geisser, 1984, Geisser, 1985, Nigm and Hamdy, 1987, Tiwari and Zalkikar, 1990, Tiwari et al., 1996, Upadhyay and Shastri, 1997, Ali Mousa, 2003). The use of a Bayesian approach allows both sample and prior information to be incorporated into the statistical analysis, which will improve the quality of the inferences and permit a reduction in sample size. The decision-theoretic viewpoint takes into account additional information concerning the possible consequences of our decisions (quantified by a loss function).

There may be situations where it is convenient to choose a single value as an estimate of $(α, τ)$ . This paper discusses the problem of finding the Bayesian estimates of the Pareto parameters based on a trimmed sample, i.e., when some of the smallest and largest sample values are not available or have been discarded, applying both inferential and decision-theoretic viewpoints. The structure of the article is as follows. The next section presents the likelihood function and prior pdf for $(α, τ)$ . The joint posterior density of $(α, τ)$ and its marginals are derived in Section 3. The existence and uniqueness of the highest posterior density (HPD) estimates of $α$ and $τ$ are shown in Section 4 under mild conditions. Precise lower and upper bounds are also provided in closed forms. Section 5 presents several Bayesian estimators from a decisional perspective. Section 6 is devoted to Bayesian interval and region estimation. An illustrative example is given in Section 7. The paper concludes with several comments.

Section snippets

Sample and prior information

Consider a random sample of size $n$ from a $P (α, τ)$ distribution ( $α$ and $τ$ being unknown), and let $x_{r : n}, \dots, x_{s : n}$ be the ordered observations remaining when the $(r - 1) = {nq}_{1}$ smallest and $(n - s) = {nq}_{2}$ largest observations have been discarded or censored, where $1 ⩽ r ⩽ s ⩽ n$ . Then, given the trimmed sample $x = (x_{r : n}, \dots, x_{s : n})$ , the likelihood function of $(α, τ)$ can be written as $L (α, τ | x) = \frac{n! {\{F (x_{r : n}; α, τ)\}}^{r - 1} {\{1 - F (x_{s : n}; α, τ)\}}^{n - s} \prod_{i = r}^{s} f (x_{i : n}; α, τ)}{(r - 1)! (n - s)!} .$ Hence, in accord with (1) and (2), the likelihood becomes $L (α, τ | x) = n! α^{m} {\{1 - {(x_{r : n} τ)}^{- α}\}}^{r - 1}$

Posterior knowledge

The posterior distribution of $(α, τ)$ presents a complete description of what is known about the Pareto parameters from the sample information and prior knowledge, expressed, respectively, by (3) and (6).

The conditions $b ⩾ x_{r : n}$ , $m + c > 0$ and $W (x) = \log \{b^{a} U (x) / x_{r : n}^{n - r + 1 + a}\} + d > 0$ will be assumed to hold hereafter, unless otherwise specified. Then, utilizing Bayes’ theorem, the posterior pdf of $(α, τ)$ , given $x$ , is deduced to be $π (α, τ | x) = \frac{{α W (x)}^{m + c} {\{1 - {(τ x_{r : n})}^{- α}\}}^{r - 1} τ^{- (n - r + 1 + a) α - 1}}{Γ (m + c) B (r, n - r + 1 + a) {\{b^{a} \exp (d) U (x)\}}^{α}}$ for $α > 0$

Bayesian estimation: an inferential approach

Unless the researcher incorporates further information on the consequences of incorrect choices of $(α, τ)$ (i.e., a loss function), there would seem to be no other suitable criterion for choosing a single value to estimate $(α, τ)$ than to use the most likely value of posterior density (8), i.e., the posterior mode. The HPD estimate of $(α, τ)$ , $(\hat{α}, \hat{τ}) \equiv (\hat{α} (x), \hat{τ} (x))$ , is therefore an appropriate choice from a Bayesian inferential viewpoint, i.e., when there is no compelling reason to accept some specific

Bayesian estimation: a decision-theoretic perspective

From a decision-theoretic point of view, in order to select a single value as representing the “best” estimate of an unknown parameter, a convenient loss function must be specified. Assuming the commonly used squared-error loss function, $L (α, \tilde{α}) = {(α - \tilde{α})}^{2}$ , the Bayes estimate of $α$ (i.e., the value $\tilde{α}$ that minimizes the posterior expected loss) is the mean of the posterior density of $α$ , given $x$ . As $α | x \sim G (m + c, W (x))$ , this estimate is given by $\tilde{α} \equiv \tilde{α} (x) = E [α | x] = (m + c) / W (x) .$ The Bayes estimate of $τ$ under

Credibility intervals and regions

In practice, the researcher often has to find intervals that enclose unknown parameters with high credibility. For given $ε \in (0, 1)$ , it is clear that $(α_{ε / 2}^{*}, α_{1 - ε / 2}^{*})$ is the two-sided (equitailed) $100 (1 - ε) %$ credibility interval for $α$ , whereas the values $α_{ε}^{*}$ and $α_{1 - ε}^{*}$ are called the (one-sided) lower and upper $100 (1 - α) %$ credibility limits (or bounds) for $α$ , respectively. Nevertheless, in choosing a credibility set for an unknown parameter, it is usually desirable to minimize its size. In our case,

An illustrative example

Dyer (1981) reported annual wage data (in multiples of 100 US dollars) of a random sample of 30 production line workers in a large industrial firm, for which the Pareto distribution appeared to be adequate. For illustrative purposes, the six smallest and largest sample values will be discarded, i.e., the 40% symmetrically trimmed sample $(107, 107, 108, 108, 111, 112, 112, 112, 115, 115, 116, 119, 119, 119, 123, 125, 128, 132)$ will represent the available data x. Hence $n = 30$ , $q_{1} = q_{2} = 0.2$ , $r = 7$ , $s = 24$ , $m = 18$ , $x_{r : n} = 107$

Concluding remarks

Classical statistics usually considers current experimental data as its only source of relevant information. Bayesian statistics also utilizes the subjective and objective prior information accumulated from past or external experience, whereas Bayesian decision theory augments the inferential knowledge of the statistical study by also incorporating assessments of (potential) future consequences of alternative decisions, expressed through a loss function.

In many statistical studies, some extreme

Acknowledgement

I would like to thank the editor and three anonymous reviewers for valuable comments.

References (34)

B.C. Arnold et al.
Bayesian inference for Pareto populations
J. Econometrics
(1983)
A.J. Fernández
Bayesian inference from type II double censored Rayleigh data
Statist. Probab. Lett.
(2000)
S. Geisser
Interval prediction for Pareto and exponential observables
J. Econometrics
(1985)
R.C. Tiwari et al.
Empirical Bayes estimation of the scale parameter in a Pareto distribution
Comput. Statist. Data Anal.
(1990)
M.A.M. Ali Mousa
Bayesian prediction based on Pareto doubly censored data
Statistics
(2003)
B.C. Arnold et al.
Bayesian analysis of censored or grouped data from Pareto populations
B.C. Arnold et al.
Bayesian estimation and prediction for Pareto data
J. Amer. Statist. Assoc.
(1989)
J.O. Berger
Statistical Decision Theory and Bayesian Analysis
(1985)
S.R. Dalal et al.
Approximating priors by mixtures of natural conjugate priors
J. Roy. Statist. Soc. B
(1983)
D. Dyer
Structural probability for the strong Pareto law
Canad. J. Statist.
(1981)

L.A. Escobar et al.

Algorithm AS 292: Fisher information matrix for the extreme value, normal and logistic distributions and censored data

Appl. Statist.

(1994)

A.J. Fernández

On maximum likelihood prediction based on type II doubly censored exponential data

Metrika

(2000)

A.J. Fernández

One- and two-sample prediction based on doubly censored exponential data and prior information

Test

(2004)

A.J. Fernández et al.

Computing maximum likelihood estimates from type II doubly censored exponential data

Statist. Methods Appl.

(2002)

S. Geisser

Predicting Pareto and exponential observables

Canad. J. Statist.

(1984)

M.J.R. Healy

A mean difference estimator of standard deviation in symmetrically censored samples

Biometrika

(1978)

M.J.R. Healy

Algorithm AS 180: a linear estimator of standard deviation in symmetrically trimmed normal samples

Appl. Statist.

(1982)

Cited by (22)

Explicit quasi-optimal inspection schemes from nonconformity count data with controlled producer and consumer risks
2016, Applied Mathematical Modelling
Citation Excerpt :
The number of nonconforming units manufactured in a given length of time, the number of blemishes in sheets of glass of fixed size, the number of imperfections per unit of area in inspecting cloth, and the number of flaws in painted sheets of metal could be modeled by Poisson distributions. In practice it is generally advantageous to properly combine available sample data with existing technical information and experience [35–47]. Nonetheless, a fundamental supposition in conventional inspection from Poisson count data is that the process average λ is a constant number.
Assuming that maximum allowable risks are specified for both producer and consumer, an integer nonlinear programming problem is formulated in order to find the optimal inspection scheme for lot acceptance purpose based on defect count data and limited prior information. The number of nonconformities per unit is assumed to follow a Poisson distribution. Closed-form quasi-optimal inspection schemes for screening submitted lots of manufactured material are determined using normal transformations of chi-square distributions and second-order Taylor series expansions of the operating characteristic function. The full prior model on the process average is not needed. Instead only prior estimations of the means and variances for the quality levels of the acceptable and rejectable lots are necessary to derive explicit and accurate approximations of the smallest number of units to be tested per lot and the maximum tolerable number of nonconformities in the selected sample. The suggested methodology requires little prior information to judge the quality of the submitted material and greatly simplifies and quickens the determination of optimal acceptance sampling plans with controlled average risks. In general, the inclusion of prior knowledge yields substantial savings in sample size, as well as improved evaluations of the actual average producer and consumer risks. Moreover, our approach is quite robust against slight changes in the available prior knowledge and also provides a suitable way to properly combine multiple expert judgments. The developed procedure is applied to the manufacturing of paper and glass to show the accuracy of the proposed quasi-optimal plans and the corresponding average risks.
Optimum attributes component test plans for k-out-of-n:F Weibull systems using prior information
2015, European Journal of Operational Research
Citation Excerpt :
Our viewpoint is extremely advantageous when considering the expenses associated with testing procedures. It is well-known that the use of substantial prior information can significantly increase the quality of the inferential methods; see, e.g., Fernández (2006b, 2010c, 2012, 2014b), Aktekin and Caglar (2013), Aktekin (2014), Ausín, Galeano, and Ghosh (2014) and Ensthaler and Giebe (2014). The rest of this paper is organised as follows.
Assuming that the proportion defective in a production process p is variable and the component lifetimes are Weibull distributed, integer nonlinear programming problems are formulated and solved in order to determine the optimum component inspection scheme by attributes for k-out-of-n:F system reliability demonstration using available prior knowledge. A limited beta prior model is adopted to reflect the random fluctuations on p. The required quantity of components to test and the permissible number of component failures up to a specified censoring time are found by solving a minimisation problem with nonlinear constraints which are related to the tolerable average producer and consumer risks. First-order Taylor polynomials of the operating characteristic function are used to derive a quite accurate approximate solution. Lower and upper bounds are also deduced. Optimal solutions are usually robust to small variations in the Weibull parameters and prior information. Existing technical knowledge and experience are shown to be of great practical value in designing optimal sampling plans for lot acceptance purposes. The proposed approach generalises the classical viewpoint to those cases in which appreciable prior information on the fraction of defective systems exists, and also allows the analyst to determine the acceptability of a k-out-of-n:F system before assembly and to limit the range of p. Moreover, the practitioners may attain substantial savings in sample size and improved assessments of the true producer and consumer risks. A 4-out-of-5:F system of independent water pumps for cooling a reactor is considered to illustrate the suggested component test plans.
Computing tolerance limits for the lifetime of a k-out-of-n: F system based on prior information and censored data
2014, Applied Mathematical Modelling
Citation Excerpt :
General progressive censoring is a useful and more versatile scheme in which some early failure times may also be discarded or censored; e.g., Balakrishnan and Sandhu [32] and Fernández [33]. Clearly, this general scheme includes as special cases the right, left and double censoring situations; e.g., Khan et al. [34], Fernández [35–38] and Wu [39]. Nowadays, the Bayesian paradigm is riding a high tide of popularity in virtually all areas of statistical application; see, e.g., Li et al. [24], Lee et al. [28], Han [40,41], Jaheen and Okasha [42], Fernández [43,44] and Lin et al. [45], and references therein.
Exact guaranteed-coverage and expected-coverage Bayesian tolerance limits for the lifetime distribution of a k-out-of-n:F system are computed by solving nonlinear equations. The bounds are based on exponential component test data and available prior information concerning the expected component lifetime which is described by an inverted gamma distribution. The Bayesian tolerance limits are valid for single (right or left), double and progressive (standard or general) censoring, and even have frequentist validity in the noninformative case. The derived results allow the reliability engineer to judge the quality of a system prior to assembly, which offers obvious practical and economic benefits. Minimum and expected percentages of conforming systems are assessed by constructing suitable tolerance limits. Even though the viewpoints are different, the Bayesian tolerance limits that adopt the natural diffuse prior coincide numerically with recently published conditional tolerance limits in the double censoring case. The proposed Bayesian approach may be deemed as an extension of the existing frequentist methodology under double censoring that also takes into account the presence of prior information and general progressive censoring. The perspective developed simplifies and unifies the computation of tolerance limits with both frequentist and Bayesian interpretations, and also provides a probabilistic way of updating the tolerance limits in the light of new, relevant data, which is especially important in the dynamic analysis of a sequence of data. Moreover, the Bayesian approach is shown to outperform the frequentist viewpoint in terms of accuracy. In most situations, the use of substantial prior information significantly increases the accuracy level and considerably reduces the required number of failures to attain a specified degree of accuracy. Two illustrative numerical examples are studied, including the analysis of a system of water pumps for cooling a reactor. The results developed are extended to the Weibull case with unknown scale parameter and other probability models.
Computing optimal confidence sets for Pareto models under progressive censoring
2014, Journal of Computational and Applied Mathematics
Citation Excerpt :
Nowadays, the Pareto model is very common in many practical fields, including reliability and life-testing studies, property values, stock price fluctuations, service times in queuing systems, insurance risk studies, business failures, geological site sizes, availability of natural resources and species, errors clustering in communications circuits, city population sizes, areas burnt in forest fires, migrations, word frequencies and sizes of firms. Papers by Nigm and Hamdy [1], Hong et al. [7], Arnold and Press [19], Chen [20], Fernández [21,22], Wu et al. [23], Wu [24], Xia and Sun [25], Nadarajah [26] and Wang and Shi [27] make up a small list of references in the literature.
This paper presents two families of joint confidence sets for the Pareto shape and precision parameters based on progressively censored data, which include the existing regions in the literature as particular cases, and derives explicit expressions for the corresponding areas. Upon using fiducial arguments, an exact and quick computational method is then proposed in order to find the smallest-area Pareto region which depends on the minimal sufficient statistic with a specified confidence level. Shortest-length confidence intervals for the Pareto parameters are also provided. The reduction in area of the minimal joint confidence region with respect to the existing sets is substantial in most situations, and impressive in some cases. For illustrative and comparative purposes, two practical studies concerning device lifetimes and business failures are considered. Applications of the smallest confidence sets include uses in hypothesis testing and estimation. In particular, it is possible to construct confidence intervals for functions of the Pareto parameters, as well as pointwise and simultaneous confidence bands for the Pareto reliability function.
Smallest Pareto confidence regions and applications
2013, Computational Statistics and Data Analysis
Citation Excerpt :
Inferences based on right censored Pareto samples are widely considered in the scientific literature; see, e.g., Arnold and Press (1989), Tiwari and Zalkikar (1990), Chen (1996), Soliman (2000), Nigm et al. (2003), Hong et al. (2007, 2008) and Wu et al. (2007b). Among other authors, Upadhyay and Shastri (1997), Fernández (2006b) and Wu (2008) analyzed doubly censored Pareto data. Left censoring is also allowed in Wu et al. (2006, 2007a), Fernández (2008a) and Soliman (2008).
Available joint confidence sets for the parameters of the Pareto model are not the regions with minimum area. In order to determine the smallest joint confidence region among all those which are based on the minimal sufficient statistic, a computational procedure is proposed which is applicable even when some of the smallest and largest observations have been discarded or censored; i.e., both single (right or left) and double censoring are allowed. The smallest Pareto region is determined by using iterative linear interpolation, as well as numerical integration and optimization methods. A few iterations are often enough to achieve the optimal solution. The reduction in area of the smallest confidence regions with respect to the existing sets is substantial in most situations, and enormous in some cases. Applications of the present approach include uses in estimation and hypothesis testing. In particular, it permits to construct confidence intervals for functions of the Pareto parameters, as well as pointwise and simultaneous confidence bands for the Pareto distribution function. Data sets concerning component lifetimes, fire claims and business failures are studied for illustrative and comparative purposes.
Minimizing the area of a Pareto confidence region
2012, European Journal of Operational Research
Citation Excerpt :
This method, however, cannot be extended to the double censoring situation. Adopting a Bayesian viewpoint, Fernández (2006a, 2008a) derived joint highest posterior density credibility regions for α and τ using trimmed samples and multiply censored data, respectively. Recently, Wu (2008) has proposed a frequentist confidence region for (α, τ) under double censoring, which improves Chen’s method in terms of a smaller area in the complete and right censoring cases.
A constrained optimization problem is formulated and solved in order to determine the smallest confidence region for the parameters of the Pareto distribution in a proposed family of sets. The objective function is the area of the region, whereas the constraints are related to the required confidence level. Explicit expressions for the area and confidence level of a given region are first deduced. An efficient procedure based on minimizing the corresponding Lagrangian function is then presented to solve the nonlinear programming problem. The process is valid when some of the smallest and largest observations have been discarded or censored, i.e., both single (right or left) and double censoring are allowed. The optimal Pareto confidence region is derived by simultaneously solving three (four) nonlinear equations in the right (double) censoring case. In most practical situations, Newton’s method with the balanced set as the starting point only needs a few iterations to find the global solution. In general, the reduction in area of the optimal Pareto region with respect to the balanced set is considerable if the sample size, n, is small or moderately large, which is usual in practice. This reduction is sometimes impressive when n is quite small and the censoring degree is fairly high. Two numerical examples regarding component lifetimes and fire claims are included for illustrative and comparative purposes.

View all citing articles on Scopus

View full text

Bayesian estimation based on trimmed samples from Pareto populations

Abstract

Introduction

Section snippets

Sample and prior information

Posterior knowledge

Bayesian estimation: an inferential approach

Bayesian estimation: a decision-theoretic perspective

Credibility intervals and regions

An illustrative example

Concluding remarks

Acknowledgement

J. Econometrics

Statist. Probab. Lett.

J. Econometrics

Comput. Statist. Data Anal.

Bayesian prediction based on Pareto doubly censored data

Statistics

Bayesian analysis of censored or grouped data from Pareto populations

Bayesian estimation and prediction for Pareto data

J. Amer. Statist. Assoc.

Statistical Decision Theory and Bayesian Analysis

Approximating priors by mixtures of natural conjugate priors

J. Roy. Statist. Soc. B

Structural probability for the strong Pareto law

Canad. J. Statist.

Algorithm AS 292: Fisher information matrix for the extreme value, normal and logistic distributions and censored data

Appl. Statist.

On maximum likelihood prediction based on type II doubly censored exponential data

Metrika

One- and two-sample prediction based on doubly censored exponential data and prior information

Test

Computing maximum likelihood estimates from type II doubly censored exponential data

Statist. Methods Appl.

Predicting Pareto and exponential observables

Canad. J. Statist.

A mean difference estimator of standard deviation in symmetrically censored samples

Biometrika

Algorithm AS 180: a linear estimator of standard deviation in symmetrically trimmed normal samples

Appl. Statist.