RESTRICTED MAXIMUM LIKELIHOOD ESTIMATION

Chapter 12 RESTRICTED MAXIMUM LIKELIHOOD ESTIMATION

  Let us assume that the k × 1 vector of parametersθ 0 satisfy r constrains, i.e.

  ϕ (θ) = 0

  where ϕ (θ) and 0 are r × 1 vectors. Let us also assume that

  the r × k matrix of derivatives has rank r, i.e. there no redundant constrain.

  Under these conditions, MLE is still consistent but not is not asymptotically efficient, as it ignores the information in the constrains. To get an asymptotically efficient estimator we have to take into consideration the information of the constrains. Hence we form the Lagrangian:

  L (θ, λ) = (θ) + λ ϕ (θ) = (θ) + ϕ (θ) λ

  where λ is the r × 1 vector of Lagrange Multipliers. The first order conditions are:

  Let e θ and e λ the constrained ML estimators, i.e. the solution of the above first order conditions, i.e.

  ³ ´

  ³ ´

  s eθ +F eθ eλ = 0

  Restricted Maximum Likelihood Estimation

  ³ ´ ϕ eθ = 0.

  ³ ´ Applying the Mean Value Theorem around θ 0 we have to s eθ and ϕ eθ we get

  0 eθ − θ 0 = s (θ 0 ) + H (θ ∗ ) eθ − θ 0 (12.3)

  eθ − θ 0 = ϕ (θ 0 ) + F (θ ∗∗ ) eθ − θ 0 (12.4)

  ∂θ

  where θ ∗ and θ ∗∗ are defined as

  Notice that θ ∗ and θ ∗∗ are not necessarily the same.

  Substituting (12.3) into (12.1), (12.4) into (12.2) and taking into account that

  under the null ϕ (θ 0 )=0 we get

  Hence we get

  H (θ ∗ ) n eθ − θ 0 + √ F eθ eλ = 0 (12.5)

  Now e θ is consistent and so are θ ∗ and θ ∗∗ , i.e.

  eθ = θ 0 +o p (1) , θ ∗ =θ 0 +o p (1) , and θ ∗∗ =θ 0 +o p (1) .

  Furthermore, according to our assumptions we have that

  H (θ

  ∗ )= − J (θ 0 )+o p (1)

  n

  Restricted Maximum Likelihood Estimation

  where J (θ 0 ) the average information matrix. Hence, equations (12.5) and (12.6) become:

  Let us now define the matrix

  i −1 as J (θ 0 )>0 and F (θ 0 ) has rank r. Now multiplying (12.7) by F (θ 0 ) J (θ 0 )

  and

  add (12.8) we get:

  Hence, as P (θ 0 )>0 we have that

  Furthermore, substituting into (12.7) we get

  Now we can state the following Theorem: Theorem 42 Under the assumptions A4., A5., model identification and that the true

  parameter values satisfy the constrains, we have

  eλ d ¡

  −1 ¢

  √ →N

  0, [P (θ 0 )]

  n

  and

  √ ³

  ´

  µ

  d h _

  i −1

  ¶

  n eθ − θ 0 →N 0, J (θ 0 ) −A

  where

  h _

  i −1

  h _ i −1

  A= J (θ 0 ) F (θ 0 ) [P (θ 0 )] −1

  F (θ 0 ) J (θ 0 )

  Restricted Maximum Likelihood Estimation

  Proof: From assumption A4. we have that

  Hence from (12.9) we have

  Furthermore from (12.10) we have

  which is the result. ¥

  Hence we can reach the following Conclusion:

  Restricted Maximum Likelihood Estimation

  Corollary 43 The Restricted MLE is at least as efficient as the MLE, i.e.

  ³ ´

  µ ∧ ¶

  AsyV ar eθ ≤ AsyV ar θ .

  Restricted Maximum Likelihood Estimation

  Part V

  Neyman or Ratio of the

  Likelihoods Tests

  Let x = (x 1 ,x 2 , ..., x n ) be a random sample having a density function d (x, θ),

  where θ ∈ Θ ⊂ R k . We would like to test a simple hypothesis

  H 0 :θ=θ 0

  against a simple alternative

  H 1 :θ=θ 1 ∈ Θ − {θ 0 }.

  To simplify the notation we write

  d 0 (x) = d (x, θ 0 ), d 1 (x) = d (x, θ 1 ).

  The Neyman Ratio is the statistic

  If S is the sample space of x, then the Neyman Ratio Test is defined by the Rejection Region

  R= {x ∈ S : λ (x) ≥ c α }

  where c α >0 is a constant such that the Type I Error (Size) of the test is equal to α ∈ (0, 12), i.e.

  i.e. the probability of rejecting the null when it is correct. Notice that the Power of the above test, π R (α) is

  Z π R (α) = P (R |θ = θ 1 )= d 1 (x) dx,

  R

  i.e. the probability of rejecting the null when the alternative is correct.

  Lemma 44 (Neyman-Pearson) Let A be the Rejection Region of the test of H 0 with

  size less or equal to α, i.e.

  Z P (A |θ = θ 0 )= d 0 (x) dx ≤ α.

  A

  Then the Power of this test is less or equal to the Power of the Neyman Test, i.e.

  P (A |θ = θ 1 ) ≤ P (R|θ = θ 1 ).

  Proof: The difference in the Powers of the 2 tests is

  π R (α) −π A (α) = P (R |θ = θ 1 ) − P (A|θ = θ 1 )=

  Form the definition of the Rejection Region R we have that

  ∀x ∈ R d 1 (x) ≥c α d 0 (x) , and ∀x ∈ R d 1 (x) < c α d 0 (x) .

  Substituting to the above integrals we get

  Hence π R (α) ≥π A (α) .

  The above Lemma says that when we test a simple null versous a simple alternative the Neyman Test is the Most Powerful Test of all tests that have the same or smaller size.

  However, the usual tests are of simple H 0 versous composite alternative. In

  such cases the power of the test is, in general, a function of the parameters in the

  alternative, i.e. π A (α, θ) the power of the test is a function of the parameters as well.

  Definition 45 Let A be the rejection region of a test. Then if π A (α, θ) ≥ α for all

  θ ∈ Θ the test is unbiased.

  The comparison of unbiased tests is very difficult as the one is more powerful in one region of the parametric space and the other in another.

  Definition 46 If for any test, say B, we have that π A (α, θ) ≥π B (α, θ) for all θ ∈Θ

  the test A is called uniformly most powerful test.

  There is one way to find uniformly most powerful tests or alternatively to

  compare the power of any given test. Assume that we have to test the null H 0 :θ=0

  versous the alternative H 1 :θ 6= 0. We can calculate the power of the Neyman Ratio

  Test of the simple null H 0 :θ=0 versous the simple alternative H 1 :θ=1 . and

  graph the point in a π R (α, θ) − θ diagram. We repeat for the simple alternatives

  H 1 :θ= −1, H 1 :θ=2 ,H 1 :θ= −2, etc. The line we get by joining all these points

  together is called the Envelope Power Function. As an immediate consequence of the Neyman-Pearson Lemma we have:

  Theorem 47 Let π A (α, θ) be the power function of a test of size α of the null hy-

  pothesis H 0 :θ=θ 0 versous the alternative H 1 :θ ∈ Θ − {θ 0 }. If e α (θ) is the

  Envelope Power Function we have that

  ∀θ ∈ Θ π A (α, θ) ≤e α (θ) .

  For hypothesis testing the Envelope Power Function is the analogue of the Cramer-Rao bound in estimation. It is obvious that if the power function of a test is identical to the Envelope Power Function then this test is uniformly most powerful in the class of unbiased tests. Hence we have the following Corollary

  Corollary 48 If the Rejection Region of a Neyman Ratio test of a simple null H 0 : θ=θ 0 versous the alternative H 1 :θ ∈Θ 1 does not depend on θ 1 , a random point is

  Θ 1 , then the Neyman test is uniformly most powerful test.

  The proof is based on the fact that if the rejection region R does not depend on θ 1 , the power of the test is identical to the Envelope Power Function.

  Example 49 2 Let x = (x

  1 ,x 2 , ..., x n ) be a random sample from a N (μ, σ ) with un-

  known μ and known σ 2 . We want to test the hypothesis H 0 :μ=μ 0 versous the

  alternative H 1 :μ>μ 0 . For any μ 1 >μ the Likelihood Functions for μ j (j = 0, 1) is

  given by

  and the ratio of the likelihoods is

  Hence λ (x) ≥ c α is equivalent to

  ⇔x≥c α =

  where the constants c

  α and c α are determined by the size of the test, which should be

  α , i.e.

  Under H 0 , we have that x vN μ 0 , n and n x−μ 0 σ v N (0, 1). The above probabil-

  ity is equivalent to

  Let z α be the critical value that leaves α in the tail of a N (0, 1) distribution. Hence

  Hence we have shown that

  The Rejection Region of the Ratio of the Neyman Test is

  which is independent of μ 1 . Consequently, the test is Uniformly Most Powerful.

  The Neyman Ratio Test is generalised and for the testing of a composite null

  versous composite alternative. Let x = (x 1 ,x 2 , ..., x n ) be a sample with Likelihood

  Function

  d (x; θ) , k θ ∈Θ⊂R ,

  and Ω ⊂ R ν (ν < k) be a subset of the parametric space Θ. We would like to test the composite null

  H 0 :θ ∈Ω

  versous the alternative

  H 1 :θ ∈ Θ − Ω.

  The Neyman Ratio is the function

  and the Neyman Ratio Test is defined by the Rejection Region of H 0

  R= {x ∈ S : λ (x) ≥ c α },

  where c α >0 is a constant such that the probability of Type I Error is α, i.e.

  Z

  sup {P (R|θ ∈ Ω)} = sup

  Example 50 Let x = (x 1 ,x 2 , ..., x n ) be a random sample from a N (μ, σ 2 ) with un-

  known μ and σ 2 . We want to test the hypothesis H

  0 :μ=μ 0 versous the alternative

  H 1 :μ 6= μ 0 . The vector of parameters is

  ¡ 2 ¢ θ = μ, σ ∈Θ=R×R + .

  Under the null the parameter sample is Ω = μ 0 ×R + . Consequently we have that

  H 0 :θ ∈ Ω versous H 1 :θ ∈ Θ − Ω. For any θ the likelihood function is

  ¡

  2 ¢ −n2

  n

  1 X 2

  d (x; θ) = 2πσ

  exp − 2 (x i − μ)

  2σ i=1

  However, the maximum of sup θ∈Ω [d (x; θ)] = sup σ 2 [d (x; μ 2 0 ,σ )]. The maximum of

  With the same reasoning we find that

  Hence the Ratio of the Likelihoods is

  sup θ∈Θ−Ω [d (x; θ)] ¡ 2 2 ¢ −n2

  and the Rejection Region of the test is

  Notice that

  ∧ , where σ =

  σ 2 It is easy to show that under H 0 the ratio

  vF 1,n−1 .

  ∧

  σ 2

  Hence the null is rejected when