[Terms] Types of gradient descent

Momentum

• Use the moving average to move the data.
• Due to having the previous momentum as a value it reduces large changes.
• Think of a ball rolling down a hill into a flat surface.
• Creates cases where the ball moves in a different direction from the slope.

 

Nesterov accelerated gradient

• Uses the gradient from the expected direction.
• Fixes the problem of moving in a different direction from the minimum.
• Using the expected gradient will increase responsiveness.

 

Adagrad

• Sets the learn rate for each individual parameters.
• Frequently occurring features gets a lower learn rates, while un-frequent occurring features gets a higher learn rate.
• Works well on sparce data.