Summary of hyperparameter tuning

The 3 parameters of hyperparameter tuning are: batch size, learning rate, and epoch length.

If the training loss graph is oscillating it means the learning rate is too high. 

If the training loss graph doesn't converge (slope approaches zero) towards the end, try adding more epochs.

If the training loss graph is converging too slowly, try to increase the learning rate.

Lowering the learning rate while increasing the number of epochs or the batch size is often a good combination.

The batch size has to fit into memory.

To tune batch size, start with a larger number then decrease it till you see degradation on results.

Because the hyperparameters tuning is data-dependent so we must experiment with the values.