[Theorem] Overfitting

1. Overfitting in Linear Regression

When degree of freedom is low, \(H(x)\) can only predict output in simple way and can't predict every case of x. This called 'underfitting' or 'high bias'. When degree of freedom is proper(not too low and not too high), predicting output is pretty well. When degree of freedom is high, model can predict output well, but can't generalize well to predict new data.

This case is called as 'overfitting' or 'high variance'.

 

2. Overfitting in Logsitic Regression

Same as linear regression, underfitting(high bias) is when the form of our hypothesis function maps pooly to the trend of data. It is usually caused by a function that is too simple or uses too few features. At the other extreme, overfitting(high variance) is caused by a hypothesis function that fits the available data but does not generalize well to predict new data. It is usually caused by complicated function that a lot of unnecessary curves and angles unrelated to the data.

 

3. Ways of solving overfitting

1. Reduce the number of features
   1. Manually select which features to keep
   2. Use a model selection algorithm
2. Regularization
   1. Keep all the features, but reduce the magnitude of parameter theta
   2. Regularization works well when have a lot of slightly useful features