Sum of Squares

Table of contents

1. What is Sum of Squares?
2. Total Sum of Squares
3. Residual Sum of Squares
4. Explained Sum of Squares
5. Relationship Between Sum of Squares
6. Ordinary Least Squares
   1. Properties of OLS
      1. Consistent
      2. Asymptotically Normal

---

What is Sum of Squares?

Sum of squares is a concept frequently used in regression analysis.

Depending on what we choose to square, we end up with many different sums of squares.

---

Total Sum of Squares

The total sum of squares ($S{S}_{tot}$) is the sum of the squared differences between the observed dependent variable $y_i\in Y$ and its mean $\overline{y}$:

$$S{S}_{tot}=\sum _{i=1}^n(y_i-\overline{y}{)}^2$$

where $n$ is the number of observations.

Graphically, in a simple linear regression with one independent variable, $S{S}_{tot}$ is the sum of the areas of the purple squares in the figure below.

---

Residual Sum of Squares

Also known as sum of squared errors (SSE).

The residual sum of squares ($S{S}_{res}$) is the sum of the squared differences between the observed dependent variable $y_i\in Y$ and the predicted value ${\hat{y}}_i$ from the regression line:

$$S{S}_{res}=\sum _{i=1}^n(y_i-{\hat{y}}_i{)}^2$$

Graphically, in a simple linear regression with one independent variable, $S{S}_{res}$ is the sum of the areas of the red squares in the figure below.

---

Explained Sum of Squares

Also known as model sum of squares.

The explained sum of squares ($S{S}_{exp}$) is the sum of the squared differences between the predicted value ${\hat{y}}_i$ from the regression line and the mean $\overline{y}$:

$$S{S}_{exp}=\sum _{i=1}^n({\hat{y}}_i-\overline{y}{)}^2$$

Graphically, in a simple linear regression with one independent variable, $S{S}_{exp}$ is the sum of the areas of the blue squares in the figure below.

Relationship Between Sum of Squares

For linear regression models using Ordinary Least Squares (OLS) estimation, the following relationship holds:

$$S{S}_{tot}=S{S}_{exp}+S{S}_{res}$$

---

Ordinary Least Squares

Least squares is a common estimation method for linear regression models.

The idea is to fit a model that mimimizes some sum of squares (i.e. creates the least squares).

Ordinary least squares (OLS) minimizes the residual sum of squares.

$$\hat{\beta }=\underset{\beta }{\mathrm{argmin}}\sum _{i=1}^n(y_i-{\hat{y}}_i{)}^2$$

Since we are trying to minimize the sum of squares with respect to the parameters, we solve for the partial derivatives. In simple linear regression, for example:

$$\begin{array}{rl}& \frac{\partial }{\partial {\beta }_0}\sum _{i=1}^n(y_i-{\beta }_0-{\beta }_1{x}_i{)}^2=0\\ & \frac{\partial }{\partial {\beta }_1}\sum _{i=1}^n(y_i-{\beta }_0-{\beta }_1{x}_i{)}^2=0\end{array}$$

Given that some conditions hold, there is a closed-form estimation for $\beta$:

$$\hat{\beta }=(X^{T}X{)}^{-1}{X}^{T}y$$

If ${\epsilon }_i|X_i\sim N(0,{\sigma }^2)$, then OLS is the same as MLE.

Properties of OLS

Consistent

OLS estimators are consistent:

$$\hat{\beta }\stackrel{p}{\to }\beta$$

Asymptotically Normal

OLS estimators are asymptotically normal:

$$\frac{\hat{\beta }-\beta }{\widehat{\text{se}}(\hat{\beta })}⇝N(0,1)$$

Hence you could find normal confidence intervals for $\beta$.