BAYESIAN ESTIMATION OF THE
LINEAR REGRESSION MODEL
WITH AN UNCERTAIN
INTERVAL CONSTRAINT ON COEFFICIENTS
Alan T.K. Wan and William E. Griffiths
No. 81 - November 1995
ISSN
0 157 0188
ISBN
1 86389 286 9
BAYESIAN ESTIMATION OF THE LINEAR REGRESSION MODEL
WITH AN UNCERTAIN INTERVAL CONSTRAINT ON COEFFICIENTS
Alan T.K. Wan
Department oy Econometrics, University of New South Wales, Australia
and
Department of Applied Statistics and O.R., City University of Hong Kong
William E. Oriffiths
Department of Econometrics, University of New England, Australia
Keywords and Phrases : Bayesian; interval restriction; pre-test;
sampling theoretic risk.
ABSTRACT
This article considers Bayesian inference in the interval
constrained normal linear regression model. Whereas much of the
previous literature has concentrated on the case where the prior
constraint is correctly specified, our framework explicitly allows for
the possibility of an invalid constraint. We adopt a non-informative
prior and uncertainty concerning the
interval restriction is
represented by two prior odds ratios. The sampling theoretic risk of
the resulting Bayesian
interval pre-test
estimator is derived,
illustrated and explored.
I. INTRODUCTION
Non-sample prior information often specifies the unknown
coefficients in a regression model to lie within fixed intervals. Many
studies have considered the problem of estimating the linear model
subject to interval restrictions using the frequentist approach.
Examples are Escobar and Skarpness (1986, 1987), Ohtani (1987), Wan
(1994) and Srivastava and Ohtani (1995).
These papers are mainly
concerned with the sampling performance of the interval constrained
least squares estimator or its variants.
Although the alternative
Bay4~ian approach has also been discussed (see, for example, Davis
(1978), Geweke (1986), Griffiths (1988), Hasegawa (1989a), among
others), the vast majority of these studies focus on the special case
where the prior information exists as inequality constraints. Few have
explicitly considered the case of interval restrictions and the
sampling properties of the Bayesian estimator thereby generated. An
exception is Hasegawa (1989b), where the sampling theoretic risk of the
Bayesian estimator of the linear regression model with a single
interval constraint on the coefficients is derived and analyzed.
A problem which arises from much of these Bayesian analyses on the
inequality or interval constrained linear regression model, is that
they assume that our a pr£or~ belief of the constraint is unaltered,
even when faced with strong sample information to the contrary. Within
the frequentist framework, when faced with the possibility of an
invalid prior constraint, one would choose between imposing or
discarding the constraint according to the outcome of a preliminary
test. Hasegawa (1991) examines the MSE performance of the pre-test
estimator after a (two stage) preliminary test of an interval
restriction on the coefficients.
However, the properties of the
analogous Bayesian estimator has not been considered except for the
special case where the prior information exists as an inequality
constraint (see Griffiths and Wan (1994)).
In this article, we consider a Bayesian interval estimator which
explicitly allows for the possible violation of the prior constraint.
We adopt a non-informative prior and uncertainties concerning the
interval restriction are represented by two prior odds ratios defined
over the null and alternatives.
The sampling theoretic risk of the
Bayesian interval pre-test estimator is derived and compared with that
of its frequentist counterpart. The case of an inequality restriction
as analysed in Griffiths and Wan (1994) is nested as a special case in
this representation.
2. THE PROBLEM AND ESTIMATORS
We consider the standard linear regression model,
y = X/3 + ~ ;
e - N(0,~Zl)
2
C1)
where y is a n x 1 vector, X is a n × k matrix with rank k, ~ is a k x
1 vector of unknown coefficients, and ~ is n x 1 vector of
normally distributed error terms.
In addition to sample information, we assume that there exists
prior information in the form of an interval constraint
H0 : 1r -~ C’/~ -~ r
2
,
(2)
where C’ is a i x k known vector, and rI and r2 are known scalars such
that r < r. When r ÷ -~ or r ÷ =o, (2) reduces to an inequality
i
2
i
2
constraint.
Following Judge and Yancey (1981)., we repararneterize (i) and (2) as
y = H8 + ~
(3)
and
respectively, where H = XS-I/ZQ’; $ = X’X; O = QSI/2~; Pl = rl/hl; P2 =
rz/hl; hl is the first element of h = C’$-i/ZQ’ and is assumed to be
positive without loss of generality; O is the first element of 8 and Q
is an orthogonal matrix such as Q$-1/2C(C’ S-1C)-IC’S-~/ZQ" =
.
Now consider the non-informativeprior
(5)
f(8,~) a l[pi,pz](el)/~r,
where I[.](8I) is an indicator function which takes the value of i if
8 lies in the subscripted interval and 0 otherwise. Given the sample
information and the prior density function, it is readily shown that
the posterior pdf for @ can be written as
1
f(Ol[Y) =<~vsz+ (Ol-~)l)2)-(v+!)/2I[pl,P2](O1)!
where v = n-k; ~ is the maximum likelihood estimator of 01, and s =
(y-H~)’(y-H~)/v. The Bayesian interval constrained estimator is simply
the mean of the posterior pdf.
Hasegawa (1989b) shows that the
Bayesian interval constrained estimator can be expressed as
,, -(v-l)/2 (
3
where m = (F(Y-*~-)v)/(F(~)(r~.v)I/Z(v-1)), ~(I-Ioly)--~((0z-gl)/s)-
~/(pl-g~)/sl, and ~(.) is the cdf of the Student’s t distribution.
~(Holy) measures the sample support of HoWhen the validity of H is uncertain, the sampling theory approach
o
is to conduct a (two stage) preliminary test of H .
In the first
o
stage, Ho~ : el ~- p~ is tested against H~ : 01 < 01. If Hol is not
rejected, then Hoz : 01 -~ ~2 is tested against H~ : 01 ) .~ in the
second stage. Hasegawa (1991) examines the risk (under squared error
loss) of the resulting interval pre-test estimator.
prior
To consider the alternative Bayesian approach, we assisn
probabilities to the null and each of the alternative hypotheses.
Now,
P(Ho) - P(HI) be the prior
let P(Ho), P(HI) and P(H2) = i probabilities representing the investigator’s belief of Ho, H and H
1
2
respectively. The "Bayesian interval pre-test" estimator is simply the
weighted average of the posterior means corresponding to each
hypothesis, that is,
e^1 = P(HolY)e~,Ho + P(HIIy)@~,H~ + P(H21Y)eI,H2 ,
(8)
where
/13(H1 ~y)
I,H1
= ~ - ms 1 + (pz-~ll/(vsZ)
1 -(v-l)/2
(9)
and
gl,H2= g~
( CP2-~1)/(vs2)) -(v-D/2/~CH2
+ ms i +
[y)
(i0)
are the Bayesian inequality constrained estimators corresponding to H
and H2 respectively; ~(HllY) = ~ ((/Dl-~l)/s] and ~(Hz[y) = 1 kg((p2-~l)/S) measure the support for HI and H2 in the sample; and
P(HilY), i=0,i,2; are the posterior probabilities.
i
Now, using (8), and the definitions of ~
can write
we
and ~
I,HO’
I,H1
I,H2’
P(HI }y) }
2 ~-(v-I)/2(P(HolY)
= +
+ (p1-~1)/(vs)]
~
~I ~I ms(1
- ms (
1 + (pz-~1)/(vs2) l-(v-1’/z(P(H°Iy)
P(H
1
_P(HzlY) }
P(HzlY)
(11)
Along the lines of Oriffiths and Wan (1994), we can express the
posterior probabilities P(Hi[y) as
P(H~[y) =
~(HilY)P(H.)
~
,i=0,1,2 (12)
~(Ho]Y)P(Ho) + ~(Hlly)P(H~) + ~(Hzly)P(H2)
The proof of (12) is available upon request from the authors.
Now, using (12), we obtain,
P(Hily)
P(Ho[Y)P(Hi)
~(Hily)
~(HolY)P(Ho)
(13)
and
^
I
i
e = ~ + ms 1+
(
(p~_~1)/(vsZ)]-(v-1)/2
P(HolY)
P(H
1
{ P(Ho) }
I
P(Ho[Y)
)
{ P(H2)
i }
- ms (i + (pz-~1)/(vsZ))-(v-1)/2 P(H°[Y)
P(Ho)
P(HoIY)
.
(14)
Using (12) again, it is straightforward to show that
~(HoIY)
P(Ho ] y) =
,
(lS)
Ti+ (!-T)~(HI [y) + (T2-~:)~(H
0
I
2
where T 1 = P(HI)/P(Ho) and T 2= P(Hz)/P(Ho) are the prior odds ratios
measuring the (subjective) odds in favour of each of the alternative
hypothesis relative to the null. We assume that, a priori, H is at
o
least as likely to occur as its alternatives, hence ~ + T -< 1. This
1
2
assumption is in line with the sampling theory approach that
only
rejects H when there is convincing sample evidence against it.
0
Now, combining (14) and (15), we can write
01 = ~1 + ms (l-q:1) 1 +
- (i-’~z)
x (1+ (pZ-~1)Z/(vsZ))-(v-1)/2~/0,
(16)
where ..... = zl + (I-*I)~(Ho]Y) + (z2-zl)~(H2 ]y)"
Note that when P(H)I =
^
P(H2) = O, 0i reduces to ~
the Bayesian interval constrained
I ,H0,
estimator under Ho. Alternatively, when P(Ho) = P(HI) = P(Hz) = i/3,
81 collapses to ~.
When pl ÷ -~ or P2 -> m’ either 1 r or "[ vanish,
1
2
and (16) reduces to the expression for the Bayesian inequality pre-test
estimator given in Griffiths and Wan (1994).
^
3. THE SAMPLING THEORETIC RISK OF 8
1
In this section, we derive and evaluate the sampling theoretic risk
function of the Bayesian interval pre-test estimator according to a
squared error loss measure.
Making use of (16) and the usual
^
definition of the squared error loss, the relative risk of @ may be
expressed as
(17)
-(l-r2)(l +
Now, E((~I - el)2]/¢2 equals 1. Furthermore, making use of the n
change
z=
2, n~ = {pl-@~}/o- and
in variables w = (~-O }/¢, q =1vsZ/~
1
(Oz-O1)/~r, and after performing some manipulations, we can write
+(I-T2)2 [I
x
2(I-T1)(I-T2)
+ (n2-w)Z/qJ
~ -(v-l) -
[i + (Dl-w)Z/q)-(v-l)/Z(l
2
~-2e-{W
÷q)/ZqV/2
x
dq dw +
+ (Wz-w)Z/q)-(v-l)/z}
m
vl/Z2( v -~)/zFC!)F(V)
2 2
x
(i-’~1) 1 + (~1-W)2/q
- (1-’~2)
-~ 0
(18)
^
In terms of the /~ space, the risk of @ is equivalent to the risk of
1
the
Bayesian
interval pre-test estimator
of B1 assuming orthonormal
regressors.
^
In order to fully appreciate the risk behaviour of 01, we carried
out numerical evaluation of (18) for various parameter values.
The
subroutines BETAI and GAMMQ from Press et al. (1986) were used to
calculate the cumulative distribution function for the Student’s t
distribution and the Gamma function respectively.
Numerical
integrations were performed using the subroutine D01AMF from the NAG
(1991) library. These were incorporated into a FORTRAN program and
executed on the VAX6000 machine. For comparison purposes, the risk of
the sampling theory interval pre-test estimator derived by Hasegawa
(1991) was also evaluated.
Figures 1 and 2 illustrate some typical results. Comparing these
figures with those given in Griffiths and Wan (1994), we see that the
^
shape of the risk of @ is considerably different from that associated
with the Bayesian inequality pre-test estimator. The risk function of
^
81 depends on pl and P2 only through the difference between Pl and P2’
and is symmetric about (Pl + p2)/2 only if r = T. The latter feature
i
2
also contrasts with the corresponding property concerning the risk of
~.
When the interval of the constraint is relatively small, the
I’H0
^I
I,H
^
risks of 8 and ~ 0 are minimized at (Pl + Pz)/2" On the contrary,
as the interval becomes wider, the risks of 8 and ~
i
1,H0
take their
respective local maximums at (Pl + P2)/2" This property accords with
that of the
corresponding sampling theory estimators. The risk of 8
I,H0
moves towards that of ~
as T and r decrease, and vice versa. The
1
2
^I
I,H0
risk of ~
dominates that of 8 when the constraint is true.
However, over much of the
regions where the interval restriction is
7
invalid, the risk of @ is smaller than that of ~
1
I,H0
In terms of comparison ~vith the risk of the corresponding sampling
theory pre-test estimators (see figure 3), neither the frequentist nor
the Bayesian estimators dominate each other.
Contrary to the risk
behaviour of the sampling theory estimator, the risk of the Bayesian
estimator is always enveloped by the risks of the unrestricted and
interval constrained estimators. With the sampling theory estimators,
there is always a region where the sampling theory pre-test estimator
has risk greater than the risk of both the unrestricted and interval
constrained estimators. As 8 depends on the prior odds ratios, while
the corresponding sampling theory pre-test estimator depends on the
size of the pre-test, it is not yet clear how to place the two
estimators on an equal footing. For the special case of an inequality
constraint, Griffiths and Wan (1994) suggest choosing an "optimal odds
ratio" analogous to the procedure of selecting optimal pre-test size
using a minimax regret approach discussed in the literature (see Wan
(1995)).
For the problem considered in the current paper, the fact
that @ depends on two prior odds ratios complicates the situation.
The question of optimal pre-test size for the interval pre-test
estimator is also unexplored for similar reasons. These remain topics
for future research.
FIGURE I
RISK FUNCTIONS OF THE BAYESIAN INTERVAL PRE-TEST ESTIMATORS
H :-i-<8 -<I
o
1
2 i
1,H0
1.5
^
O (TI=I/2;TZ=I/6)
1
0.5
8
FIGURE 2
RISK FUNCTIONS OF THE BAYESIAN INTERVAL PILE-TEST ESTIMATORS
H:-3~e~3
o
2
I,H0
^
e (’r =1/8;’~z=1/8)
11
O 1(R:1=i/2;’~z=i/6)
1
0.8
0.6
-6
-5 -4-
~ -2 -i 0
1
2
5
a
FIGURE 3
RISK FUNCTIONS OF THE FREQUENTIST AND BAYESIAN
INTERVAL PILE-TEST ESTIMATORS
Ho :-3-<81 -<3
interval restricted
interva! pre-test (5~.’.
I,H0
^
8~ (-c~=i/8;-c2=i/8)
-6
-5 -<- -,5 -2
0
I0
5
REFERENCES
Davis, W.W. (1978), "Bayesian analysis of the linear model subject to
linear inequality constraints", Journal of the American Statistical
Association 73, 573-579.
Escobar, L.A. and B. Skarpness (1986), "The bias of the least squares
estimator over interval constraints", Economics Letters 20,
331-335.
Escobar and B. Skarpness (1987), "Mean square error and efficiency of
the
least
squares
estimator
over
interval
constraints",
Oommunications in Statistics : Theory and Methods 16, 397-406.
Geweke, .I. (1986), "Exact inference in the inequality constrained
normal linear regression model", Journal of Applied Econometrics I,
127-141.
Griffiths, W.E. (1989), "Bayesian Econometrics and how to get rid of
the wrong signs", Review of MarketinE and AEricultural Economics
56, 36-56.
Griffiths, W.E. and A.T.K. Wan (1994), "A Bayesian estimator of the
linear regression model with an uncertain inequality constraint",
under submission.
Hasegawa, H. (1989a), "On some comparisons between Bayesian and
sampling theoretic estimators of a normal mean subject to an
inequality constraint", Journal of
the Japan
Statisttca~
Society 19, 167-i77.
Hasegawa, H. (1989b), "Bayesian estimator
of the linear regression
model with an interval constraint on coefficients", Economics
Letters 31, 9-12.
Hasegawa, H. (1991), "The MSE of a pre-test estimator of the linear
regression model with an interval constraint on coefficients",
Journal of the Japan Statistical Society 21, 189-195.
Judge, G.G. and T.A. Yancey (1981), "Sampling properties of an
inequality restricted estimator", Economics Letters 7, 327-333.
Ohtani, K. {1987), "The MSE of least squares estimator over an interval
constraint", Economics Letters 25, 351-354.
Press, W.H., B.P. Flannery, S.A. Teukolsky and W.T. Vetterling (1986),
Numerical Recipes : The Art of Scientific Computing. New York:
Cambridge University Press.
Srivastava, V.K. and K. 0htani (1995), "A comparison of interval
constrained least squares and mixed regression estimator",
Communications in Statistics: Theory and Methods 24, 395-413.
11
Wan, A.T.K. (1994), "The non-optimality of the interval restricted and
Communications tn
pre-test estimators under squared error loss",
Statistics: Theory and Methods 23, 2231-2252.
Wan,
A.T.K. (1995), "The optimal critical value of a pre-test for an
regression model",
inequality restriction in a mis-specified
Australian Journa~ 03: Statistics 37, 73-82.
12
WORKING PAPERS IN ECONOMETRICS AND APPLIED STATISTICS
~o2~ic~ ~ ~o~. Lung-Fei Lee and William E. Griffiths,
No. I - March 1979.
Howard E. Doran and Rozany R. Deen, No. 2 - March 1979.
William Griffiths and Dan Dao, No. 3 - April 1979.
~oiaia:. G.E. Battese and W.E. Griffiths, No. 4 - April 1979.
D.S. Prasada Rao, No. 5 - April 1979.
~e!~ Re~~o~. George E. Battese and
Bruce P. Bonyhady, No. 7 - September 1979.
Howard E. Doran and David F. Williams, No. 8 - September 1979.
D.S. Prasada Rao, No. 9 - October 1980.
~ Do22~/~ - 1979. W.F. Shepherd and D.S. Prasada Rao,
No. 10 - October 1980.
~ ~-YenLeaand ~aaa~-~ecigon Do/ate g~EmaZea 9anduc_iiz~ ~nctian
~ ~o~o~ui Neq~~o~ ~k~. W.E. Griffiths and
J.R. Anderson, No. II - December 1980.
fo~-O~-~ ~n~he~a~02nceo/ R~. Howard E. Doran
and Jan Kmenta, No. 12 - April 1981.
~ Ooxie~ ~ D~. H.E. Doran and W.E. Griffiths,
No. 13 - June 1981.
Pauline Beesley, No. 14 - July 1981.
~o~ DoZe. George E. Battese and Wayne A. Fuller, No. 15 - February
1982.
13
~Dec~. H.I. Tort and P.A. Cassidy, No. 16 - February 1985.
H.E. Doran, No. 17 - February 1985.
J.W.B. Guise and P.A.A. Beesley, No. 18 - February 1985.
W.E. Griffiths and K. Surekha, No. 19 - August 1985.
~Zen~ ~a~. D.S. Prasada Rao, No. 20- October 1985.
H.E. Doran, No. 21- November 1985.
~ae-~eai gaiLrn~~aZAe ~~ade!. William E. Griffiths,
R. Carter Hill and Peter J. Pope, No. 22 - November 1985.
William E. Griffiths, No. 23 - February 1986.
~ed~ ~un~ ~aia~ ~ Doia. George E. Battese and
Sohail J. Malik, No. 25 - April 1986.
George E. Battese and Sohail 3. Malik, No. 26 - April 1986.
George E. Battese and Sohail J. Malik, No. 27 - May 1986.
George E. Battese, No. 28- June 1986.
NumAen~. D.S. Prasada Rao and J. Salazar-Carrillo, No. 29 - August
1986.
~uni~ Rea~on ~n~ gaii~~naa~R(1) ganea Made/. H.E. Doran,
W.E. Griffiths and P.A. Beesley, No. 30 - August 1987.
Za~ gco~ aadRa~ ta FeZ
William E. Griffiths, No. 31 - November 1987.
14
Chris M. Alaouze, No. 32 - September, 1988.
G.E. Battese, T.J. Coelli and T.C. Colby, No. 33- January, 1989.
Tim J. Coelli, No. 34- February, 1989.
~n~Ze ~ ~c~-~i/le~ed,/gt~. Colin P. Hargreaves,
No. 35 - February, 1989.
William Griffiths and George Judge, No. 36 - February, 1989.
~p2~ te ~o~ ~ ~n/~. Chris M. Alaouze, No. 38 July, 1989.
Chris M. Alaouze and Campbell R. Fitzpatrick, No. 39- August, 1989.
Doia. Guang H. Wan, William E. Griffiths and Jock R. Anderson, No. 40 September 1989.
o~ Fh/~ Rea~ 0~. Chris M. Alaouze, No. 41 - November,
1989.
~ Y/~ e~d ~~ ~. William Griffiths and
Helmut L~tkepohl, No. 42 - March 1990.
Howard E. Doran, No. 43 - March 1990.
~aZaq YAeXu2/nnn ~igteaga @~Yui-~o#~. Howard E. Doran,
No. 44 - March 1990.
Howard Doran, No. 45 - May, 1990.
Howard Doran and Jan Kmenta, No. 46 - May, 1990.
and ~~ 9alc2~. D.S. Prasada Rao and E.A. Selvanathan,
No. 47 - September, 1990.
15
~connmicY~aZ~e~~ o~ Nex~ ~ngZend. D.M. Dancer and
H.E. Doran, No. 48 - September, 1990.
D.S. Prasada Rao and E.A. Selvanathan, No. 49 - November, 1990.
~p/i~i~ ~ g~. George E. Battese,
No. 50 - May 1991.
S~ ganan~ o~ ~a~ ~onm. Howard E. Doran,
No. 51 - May 1991.
Howard E. Doran, No. 52 - May 1991.
~on~ ~nn~. C.J. O’Donnell and A.D. Woodland,
No. 53 - October 1991.
~ano2~nJ~ Se.zio~. C. Hargreaves, J. Harrington and A.M.
Siriwardarna, No. 54 - October, 1991.
Damna~ in ~ gcaanm@-1~ide 2~ad22a: Same
Colin Hargreaves, No. 55 - October 1991.
2.0. T.J. Coelli, No. 57- October 1991.
Barbara Cornelius and Colin Hargreaves, No. 58 -October 1991.
Barbara Cornelius and Colin Hargreaves, No. 59 - October 1991.
Duangkamon Chotikapanich, No. 60 - October 1991.
Colin Hargreaves and Melissa Hope, No. 61 - October 1991.
#aZeaex~ Ratea, YAe 5erun Stnuc2azve and
Colin Hargreaves, No. 62 - November 1991.
O&o~ :£o~ ~an~ge. Duangkamon Chotikapanich, No. 63 - May 1992.
16
l~ighmo~. G.E. Battese and G.A. Tessema, No. 64 - May 1992.
~ax%o~. Guang H. Wan and George E. Battese, No. 66 - June 1992.
9acarr~e ~~ ~r~ daze, 1960-1985: ~ De~
Ma. Rebecca J. Valenzuela, No. 67 - April, 1993.
~oien~ ~ g he ~.~. Alicia N. Rambaldi, R. Carter Hill and
Stephen Father, No. 68 - August, 1993.
#~ g~2.ci~. G.E. Battese and T.J. Coelli, No. 69 - October,
1993.
Tim Coelli, No. 70 - November, 1993.
~hnno~e~n ~eaZean ~h~Yumggen ~. Tim J. Coelli, No. 71 December, 1993.
G.E. Battese and M. Bernabe, No. 72 - December, 1993.
Getachew Asgedom Tessema, No. 73 - April, 1994.
~n~wn/d/~ ~o~. W.E. Griffiths and A.T.K. Wan, No. 74 May, 1994.
On~J~_xi ~i4oZe ~AoZce. C.J. O’Donnell and D.H. Connor, No. 75 September, 1994.
~~. T.J. Coelli and G.E. Battese, No. 76 - September, 1994.
Bayesian Predictors for an AR(1) Error Model. William E. Griffiths, No. 77 September, 1994.
Small Sample Performance of Non-Causality Tests in Cointegrated Systems.
Hector O. Zapata and Alicia N. Rambaldi, No. 78 - December, 1994.
~h%o2~. Ma. Rebecca J. Valenzuela, No. 79 - August, 1995.
~o~ @a%aend~Y~. William E. Griffiths and
Ma. Rebecca Valenzuela, No. 80 - November 1995.
17