Developing any machine learning model involves a rigorous experimental process that follows the idea-experiment-evaluation cycle.
The above cycle is repeated multiple times until satisfactory performance levels are achieved. The “experiment” phase involves each the coding and the training steps of the machine learning model. As models turn into more complex and are trained over much larger datasets, training time inevitably expands. As a consequence, training a big deep neural network may be painfully slow.
Fortunately for data science practitioners, there exist several techniques to speed up the training process, including:
- Transfer Learning.
- Weight Initialization, as Glorot or He initialization.
- Batch Normalization for training data.
- Picking a reliable activation function.
- Use a faster optimizer.
While all of the techniques I identified are necessary, on this post I’ll focus deeply on the last point. I’ll describe multiple algorithm for neural network parameters optimization, highlighting each their benefits and limitations.
Within the last section of this post, I’ll present a visualization displaying the comparison between the discussed optimization algorithms.
For practical implementation, all of the code utilized in this text may be accessed on this GitHub repository:
Traditonally, Batch Gradient Descent is taken into account the default alternative for the optimizer method in neural networks.