Enable stepwise processing flag for schedulers
🚀 Feature
Asking if it makes sense adding a flag in the Trainer class for calling scheduler.step() after every update (per #640).
Motivation
This makes sense for training NLP models such as BERT/XLNet or any other that update the lr based on the current step (and training defined in terms of steps instead of epochs) instead of the current state that it is called after an epoch ends.
I'm aware that users can override optimizer_step of the model, however it's a quite common training pattern for training such NLP models.
I think this feature is worthwhile and I will contribute my changes. Let me know if not.
Pitch
Add scheduler.step() call after every step (modify the optimizer step accordingly)