Explain properties of least-squares estimators: the Gauss–Markov theorem. (linear, Unbiased and efficient)
The correct answer and explanation is:
Answer:
The Gauss–Markov theorem states that in a linear regression model where the errors have expectation zero, are uncorrelated, and have constant variance (homoscedasticity), the Ordinary Least Squares (OLS) estimator is the Best Linear Unbiased Estimator (BLUE). This means the OLS estimator is:
- Linear: It is a linear function of the observed dependent variable.
- Unbiased: On average, the estimator equals the true parameter value.
- Efficient: Among all linear unbiased estimators, it has the smallest variance.
Explanation (about 300 words):
In the classical linear regression model, we assume the relationship between the dependent variable yy and independent variables XX is: y=Xβ+ϵy = X\beta + \epsilon
where β\beta is the vector of unknown parameters, and ϵ\epsilon is the error term. The key assumptions include:
- Linearity: The model is linear in parameters.
- Unbiased errors: E(ϵ)=0E(\epsilon) = 0.
- Homoscedasticity: The variance of errors is constant, Var(ϵ)=σ2I\text{Var}(\epsilon) = \sigma^2 I.
- No autocorrelation: Errors are uncorrelated, Cov(ϵi,ϵj)=0\text{Cov}(\epsilon_i, \epsilon_j) = 0 for i≠ji \neq j.
The Least Squares estimator is given by: β^=(X′X)−1X′y\hat{\beta} = (X’X)^{-1} X’y
Linearity:
The estimator β^\hat{\beta} is a linear function of yy because it is obtained by multiplying yy by a matrix (X′X)−1X′(X’X)^{-1} X’. This ensures that the estimator changes linearly with the observed data.
Unbiasedness:
Using E(y)=XβE(y) = X\beta, we get: E(β^)=E[(X′X)−1X′y]=(X′X)−1X′E(y)=(X′X)−1X′Xβ=βE(\hat{\beta}) = E[(X’X)^{-1} X’y] = (X’X)^{-1} X’ E(y) = (X’X)^{-1} X’ X \beta = \beta
So, on average, β^\hat{\beta} equals the true parameter β\beta.
Efficiency (Best):
The Gauss–Markov theorem proves that no other linear unbiased estimator of β\beta has a smaller variance than β^\hat{\beta}. This means β^\hat{\beta} has the minimum variance in the class of all linear unbiased estimators. Hence, it is the “best” (most efficient).
In summary, the Gauss–Markov theorem guarantees that under the standard linear model assumptions, the least squares estimator is the optimal linear unbiased estimator with the smallest possible variance, making it a foundational result in regression analysis.