Sum of Squares

Table of contents
  1. What is Sum of Squares?
  2. Total Sum of Squares
  3. Residual Sum of Squares
  4. Explained Sum of Squares
  5. Relationship Between Sum of Squares
  6. Ordinary Least Squares
    1. Properties of OLS
      1. Consistent
      2. Asymptotically Normal

What is Sum of Squares?

Sum of squares is a concept frequently used in regression analysis.

Depending on what we choose to square, we end up with many different sums of squares.


Total Sum of Squares

The total sum of squares (SStot) is the sum of the squared differences between the observed dependent variable yiY and its mean y¯:

SStot=i=1n(yiy¯)2

where n is the number of observations.

Graphically, in a simple linear regression with one independent variable, SStot is the sum of the areas of the purple squares in the figure below.

Total sum of squares

Residual Sum of Squares

Also known as sum of squared errors (SSE).

The residual sum of squares (SSres) is the sum of the squared differences between the observed dependent variable yiY and the predicted value y^i from the regression line:

SSres=i=1n(yiy^i)2

Graphically, in a simple linear regression with one independent variable, SSres is the sum of the areas of the red squares in the figure below.

Residual sum of squares

Explained Sum of Squares

Also known as model sum of squares.

The explained sum of squares (SSexp) is the sum of the squared differences between the predicted value y^i from the regression line and the mean y¯:

SSexp=i=1n(y^iy¯)2

Graphically, in a simple linear regression with one independent variable, SSexp is the sum of the areas of the blue squares in the figure below.

Explained sum of squares

Relationship Between Sum of Squares

For linear regression models using Ordinary Least Squares (OLS) estimation, the following relationship holds:

SStot=SSexp+SSres


Ordinary Least Squares

Least squares is a common estimation method for linear regression models.

The idea is to fit a model that mimimizes some sum of squares (i.e. creates the least squares).

Ordinary least squares (OLS) minimizes the residual sum of squares.

β^=argminβi=1n(yiy^i)2

Since we are trying to minimize the sum of squares with respect to the parameters, we solve for the partial derivatives. In simple linear regression, for example:

β0i=1n(yiβ0β1xi)2=0β1i=1n(yiβ0β1xi)2=0

Given that some conditions hold, there is a closed-form estimation for β:

β^=(XTX)1XTy

If εi|XiN(0,σ2), then OLS is the same as MLE.

Properties of OLS

Consistent

OLS estimators are consistent:

β^pβ

Asymptotically Normal

OLS estimators are asymptotically normal:

β^βse^(β^)N(0,1)

Hence you could find normal confidence intervals for β.