[Theorem] Bias vs Variance

1. Intersection between Bias and Variance 

Let's review about overfitting problem and underfitting problem. Underfitting problem is the problem when we use too much low degree polynomial term. Overfitting problem is the problem when we use too much high degree polynomial term.

 

 

So, when we plot training error of \(J(\Theta)\) by degree of polynomial, we can see that in lower degree, error is high, and low in high degree. However test error of \(J(\Theta)\) are high in low degree and high degree but low in intermediate degree.(It has covex shape!) 

 

So we can divide case into Bias(underfitting) and Variance(overfitting). 

 

 2. Regularization 

To solve bias vs variance problem, we need to use regularization parameter lambda. Let's set list of \(\lambda\) for regularization. To find optimize parameter \(\theta\), we use \(J(\theta)\) with regularized model and use \(J(\theta)\) of cross validation and test set. 

 

3. Learning Curves 

Learning curves is the graph of the number of examples. In high bias model, low training set size cause \(J(\theta)\) of train and \(J(\theta)\) of test to be high. 

 

In high variance model, low training set size causes \(J(\theta)\) of train to be low and \(J(\theta)\) of test becomes decreases. So in high variance, getting more training data is likely to help. 

 

4. Summarization 

After making machine learning algorithm, we need to debug a learning algorithm with below option : 

1. Get more training examples(High variance) 
2. Try smaller sets of features(High variance) 
3. Try getting additional features(High bias) 
4. Try adding polynomial features(High bias) 
5. Try decreasing lambda(High bias) 
6. Try increasing labmda(High variance)