One-Standard-Error Rule

Within a certain range of test error, select the simplest model.

During model selection, we plot the estimated test error against the model complexity, and we select a complexity at the minmum test error.

However, when the test error is sort of plateaued (flattened out over a range of complexity), we can introduce a heuristic to choose the simplest model instead of the strictly minimum.

This is because the test error is technically an estimate of the true test error.

This rule is often quite helpful with validation set and cross-validation.

Estimators that rely on randomness (e.g. validation set, cross-validation) have slightly varying results depending on the how the validation set is created.

So to account for this variability, we calculate the standard error of the test error estimate at the lowest point of the test error curve.

Then we choose the simplest model within one standard error of the minimum.

In the diagram above, it would be leftmost model with error in the yellow region.