Unable to use any scheduler ('scheduler' object has no attribute 'param_groups') [BUG]
I am trying to convert a number of Pytorch repos into Lightning, but I am unable to use any scheduler. I have tried both the origin custom made schedulers and official Pytorch schedulers, and always get the same 'scheduler' object has no attribute 'param_groups'
Traceback (most recent call last):
  File "tools/train_net_lightning.py", line 255, in <module>
    trainer.fit(trainer_sys)
  File "/home/anaconda3/envs/panoptic_cuda11/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 514, in fit
    self.dispatch()
  File "/home/anaconda3/envs/panoptic_cuda11/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 554, in dispatch
    self.accelerator.start_training(self)
  File "/home/anaconda3/envs/panoptic_cuda11/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 74, in start_training
    self.training_type_plugin.start_training(trainer)
  File "/home/anaconda3/envs/panoptic_cuda11/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 111, in start_training
    self._results = trainer.run_train()
  File "/home/anaconda3/envs/panoptic_cuda11/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 645, in run_train
    self.train_loop.run_training_epoch()
  File "/home/anaconda3/envs/panoptic_cuda11/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 493, in run_training_epoch
    batch_output = self.run_training_batch(batch, batch_idx, dataloader_idx)
  File "/home/anaconda3/envs/panoptic_cuda11/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 619, in run_training_batch
    self.run_train_split_start(split_idx, split_batch, opt_idx, optimizer)
  File "/home/anaconda3/envs/panoptic_cuda11/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 910, in run_train_split_start
    model.toggle_optimizer(optimizer, opt_idx)
  File "/home/anaconda3/envs/panoptic_cuda11/lib/python3.8/site-packages/pytorch_lightning/core/lightning.py", line 1267, in toggle_optimizer
    for group in opt.param_groups:
AttributeError: 'StepLR' object has no attribute 'param_groups'

My specs:
OS: Ubuntu 18.04
Conda environment:
cudatoolkit               11.0.3               h15472ef_8    conda-forge
python                    3.8.8           hffdb5ce_0_cpython    conda-forge
pytorch                   1.7.1           py3.8_cuda11.0.221_cudnn8.0.5_0    pytorch
pytorch-lightning         1.2.3              pyhd8ed1ab_0    conda-forge
torchvision               0.8.2                py38_cu110    pytorch
(also does't work in slighlty different environment like
cudatoolkit 11.0.221 h6bb024c_0
python 3.8.8 hdb3f193_4
pytorch 1.7.1 py3.8_cuda11.0.221_cudnn8.0.5_0 pytorch
pytorch-lightning 1.2.3 pyhd8ed1ab_0 conda-forge
torchvision 0.8.2 py38_cu110 pytorch
)
GPU: 2018 Ti