[Terms] Bias and Variance Tradeoff

Bias and Variance

Bias is the offset of the prediction while variance is the spread of the prediction.

So simply high bias means that the prediction is off the mark.

High variance means that the prediction is spread out, it doesn’t take the correctness into account.

 

Bias and Variance tradeoff

So, lowering bias means variance will increase and lowering variance will increase bias.

Noise is the irreversible error of the prediction. So, this part of the equation can’t be changed.

 

Overfitting and Underfitting

Overfitting is when the model is trained to fit the train data too well so when the valid data is used the loss is high. This is when the bias is low, and the variance is high.

Underfitting is when the model is not trained much so it produces high loss for both the train and valid data. This is when the bias is high, and the variance is low.

I think that because the bias is squared it is better to have a low bias than a low variance.

 

To get the best results we need to adjust the parameters and model to control the bias and variance.

 

Detection and Solution

Underfitting

• When the training error is higher than the valid error
  • Improve the models complexity
  • Add additional features
  • Set a higher epoch or learn rate

Overfitting

• When the training error is too low compared to the valid error
• When the training error continues to get lower while the valid error keeps getting higher
  • Add extra data to train
  • Reduce the models complexity
  • Set a lower epoch or learn rate