{"id":22792,"date":"2025-06-16T22:06:07","date_gmt":"2025-06-16T22:06:07","guid":{"rendered":"https:\/\/gaviki.com\/blog\/?p=22792"},"modified":"2025-06-16T22:06:14","modified_gmt":"2025-06-16T22:06:14","slug":"explain-properties-of-least-squares-estimators","status":"publish","type":"post","link":"https:\/\/gaviki.com\/blog\/explain-properties-of-least-squares-estimators\/","title":{"rendered":"Explain properties of least-squares estimators"},"content":{"rendered":"\n<p>Explain properties of least-squares estimators: the Gauss\u2013Markov theorem. (linear, Unbiased and efficient)<\/p>\n\n\n\n<p><mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-vivid-cyan-blue-color\"><strong>The correct answer and explanation is:<\/strong><\/mark><\/p>\n\n\n\n<p><strong>Answer:<\/strong><br>The Gauss\u2013Markov theorem states that in a linear regression model where the errors have expectation zero, are uncorrelated, and have constant variance (homoscedasticity), the Ordinary Least Squares (OLS) estimator is the <strong>Best Linear Unbiased Estimator (BLUE)<\/strong>. This means the OLS estimator is:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Linear<\/strong>: It is a linear function of the observed dependent variable.<\/li>\n\n\n\n<li><strong>Unbiased<\/strong>: On average, the estimator equals the true parameter value.<\/li>\n\n\n\n<li><strong>Efficient<\/strong>: Among all linear unbiased estimators, it has the smallest variance.<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h3 class=\"wp-block-heading\">Explanation (about 300 words):<\/h3>\n\n\n\n<p>In the classical linear regression model, we assume the relationship between the dependent variable yy and independent variables XX is: y=X\u03b2+\u03f5y = X\\beta + \\epsilon<\/p>\n\n\n\n<p>where \u03b2\\beta is the vector of unknown parameters, and \u03f5\\epsilon is the error term. The key assumptions include:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li><strong>Linearity<\/strong>: The model is linear in parameters.<\/li>\n\n\n\n<li><strong>Unbiased errors<\/strong>: E(\u03f5)=0E(\\epsilon) = 0.<\/li>\n\n\n\n<li><strong>Homoscedasticity<\/strong>: The variance of errors is constant, Var(\u03f5)=\u03c32I\\text{Var}(\\epsilon) = \\sigma^2 I.<\/li>\n\n\n\n<li><strong>No autocorrelation<\/strong>: Errors are uncorrelated, Cov(\u03f5i,\u03f5j)=0\\text{Cov}(\\epsilon_i, \\epsilon_j) = 0 for i\u2260ji \\neq j.<\/li>\n<\/ol>\n\n\n\n<p>The <strong>Least Squares estimator<\/strong> is given by: \u03b2^=(X\u2032X)\u22121X\u2032y\\hat{\\beta} = (X&#8217;X)^{-1} X&#8217;y<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<p><strong>Linearity:<\/strong><br>The estimator \u03b2^\\hat{\\beta} is a linear function of yy because it is obtained by multiplying yy by a matrix (X\u2032X)\u22121X\u2032(X&#8217;X)^{-1} X&#8217;. This ensures that the estimator changes linearly with the observed data.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<p><strong>Unbiasedness:<\/strong><br>Using E(y)=X\u03b2E(y) = X\\beta, we get: E(\u03b2^)=E[(X\u2032X)\u22121X\u2032y]=(X\u2032X)\u22121X\u2032E(y)=(X\u2032X)\u22121X\u2032X\u03b2=\u03b2E(\\hat{\\beta}) = E[(X&#8217;X)^{-1} X&#8217;y] = (X&#8217;X)^{-1} X&#8217; E(y) = (X&#8217;X)^{-1} X&#8217; X \\beta = \\beta<\/p>\n\n\n\n<p>So, on average, \u03b2^\\hat{\\beta} equals the true parameter \u03b2\\beta.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<p><strong>Efficiency (Best):<\/strong><br>The Gauss\u2013Markov theorem proves that no other linear unbiased estimator of \u03b2\\beta has a smaller variance than \u03b2^\\hat{\\beta}. This means \u03b2^\\hat{\\beta} has the minimum variance in the class of all linear unbiased estimators. Hence, it is the &#8220;best&#8221; (most efficient).<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<p>In summary, the Gauss\u2013Markov theorem guarantees that under the standard linear model assumptions, the least squares estimator is the optimal linear unbiased estimator with the smallest possible variance, making it a foundational result in regression analysis.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Explain properties of least-squares estimators: the Gauss\u2013Markov theorem. (linear, Unbiased and efficient) The correct answer and explanation is: Answer:The Gauss\u2013Markov theorem states that in a linear regression model where the errors have expectation zero, are uncorrelated, and have constant variance (homoscedasticity), the Ordinary Least Squares (OLS) estimator is the Best Linear Unbiased Estimator (BLUE). This [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-22792","post","type-post","status-publish","format-standard","hentry","category-quiz-questions"],"_links":{"self":[{"href":"https:\/\/gaviki.com\/blog\/wp-json\/wp\/v2\/posts\/22792","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/gaviki.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/gaviki.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/gaviki.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/gaviki.com\/blog\/wp-json\/wp\/v2\/comments?post=22792"}],"version-history":[{"count":1,"href":"https:\/\/gaviki.com\/blog\/wp-json\/wp\/v2\/posts\/22792\/revisions"}],"predecessor-version":[{"id":22793,"href":"https:\/\/gaviki.com\/blog\/wp-json\/wp\/v2\/posts\/22792\/revisions\/22793"}],"wp:attachment":[{"href":"https:\/\/gaviki.com\/blog\/wp-json\/wp\/v2\/media?parent=22792"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/gaviki.com\/blog\/wp-json\/wp\/v2\/categories?post=22792"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/gaviki.com\/blog\/wp-json\/wp\/v2\/tags?post=22792"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}