Learning Rate schedulers do not follow the optimizer frequencies.
🐛 Bug
I want to use two optimizers sequentially with different LR schedulers. For the first one, I want to use OneCycleLR and for the second optimizer I do not want to use a LR scheduler. The problem with OneCycleLR is that you need to specify the exact number of steps. I have the setup like this
def configure_optimizers(self):
 optimizer_one = torch.optim.SGD(self.model.parameters(), lr=self.hparams.lr, momentum=0.9, weight_decay=5e-4)
 optimizer_two = torch.optim.SGD(self.model.parameters(), lr=0.01, momentum=0.9, weight_decay=5e-4)
 return [
     {
       'optimizer': optimizer_one,
       'lr_scheduler': {
           'scheduler': torch.optim.lr_scheduler.OneCycleLR(optimizer, 0.1, epochs=self.hparams.first_opt_epochs, steps_per_epoch=math.ceil(45000/batch_size)),
           'interval': 'step',
       },
       'frequency': self.hparams.first_opt_epochs * math.ceil(45000/batch_size)
     }, 
     {
       'optimizer': optimizer_two,
       'frequency': (self.trainer.max_epochs - self.hparams.first_opt_epochs) * math.ceil(45000/batch_size)
   }]
The problem with this is that after the first_opt_epochs, I get a ValueError: Tried to step 28142 times. The specified number of total steps is 28140. This is because update_learning_rates is getting called from self.update_train_loop_lr_schedulers even after the frequency of the first optimizer has passed
Please reproduce using the BoringModel and post here
https://colab.research.google.com/drive/1A0p04A05ltiFkVEYPE-267LDopE3m8ju?usp=sharing
Expected behavior
The LR scheduler OneCycleLR of the first optimizer should only be updated when using the first optimizer.
Environment
Note: Bugs with code are solved faster ! Colab Notebook should be made public !


IDE: Please, use our python bug_report_model.py template.


Colab Notebook: Please copy and paste the output from our environment collection script (or fill out the checklist below manually).


You can get the script and run it with:
wget https://raw.githubusercontent.com/PyTorchLightning/pytorch-lightning/master/tests/collect_env_details.py
# For security purposes, please check the contents of collect_env_details.py before running it.
python collect_env_details.py


CUDA:

GPU:

Tesla P100-PCIE-16GB


available:         True
version:           10.1


Packages:

numpy:             1.18.5
pyTorch_debug:     True
pyTorch_version:   1.7.0+cu101
pytorch-lightning: 1.0.7
tqdm:              4.41.1


System:

OS:                Linux
architecture:

64bit



processor:         x86_64
python:            3.6.9
version:           #1 SMP Thu Jul 23 08:00:38 PDT 2020