Use scheduler.get_last_lr() instead of manually searching for optimizers.param_groups
🐛 Bug / Feature Request
I am not sure whether to classify this issue as a feature request or bug, but in your codebase, you get the learning rate from schedulers as scheduler.optimizer.param_groups[0]['lr'].
However, PyTorch provides the get_last_lr() method for this.
I created my own scheduler, which allows me to combine multiple schedulers together (eg. burn-in LR for the first few steps, and then step down LR later during training).
As this scheduler holds multiple different schedulers itself, there is no scheduler.optimizer argument and thus I cannot use this with PyTorch Lightning.
I did however implement the get_last_lr() method, which gets the LR from the underlying schedulers, so if you could modify the PyTorch Lightning code to use that method, it should all work!
Problems
Obviously, nothing is ever as straight forward with the PyTorch scheduler API, which means that there might be problems with this solution. One small issue I found is that ReduceLROnPlateau is behaving differently and does not implement the get_last_lr() method. (The PyTorch team should really look into unifying their API, but that should be discussed there...)
If you look at GH-5364, which I made on top of the release/1.2-dev branch, you will see that for the ReduceLROnPlateau scheduler, I still use the old method of finding the LR through the scheduler.optimizer object.
However, for any other scheduler, we use the get_last_lr() method.
I could adapt the code to be even more defensive and check whether there is a get_last_lr() method and only use it if it is available, but I don't know whether that is necessary, as all other official schedulers have the same base Scheduler class which implements this method.

I send in a Draft PR with the necessary changes.
Kind Regards,
0phoff