LR scheduler lags behind after resume_from_checkpoint
🐛 Bug
MultiStepLR lags behind by one epoch after resuming from checkpoint in Trainer.
See this collab to reproduce.
In the example below,
class BoringModel(LightningModule):
    ...
    def training_step(self, batch, batch_idx):
        print(f'Epoch {self.trainer.current_epoch} / Step {self.trainer.global_step}: lr {self.trainer.optimizers[0].param_groups[0]["lr"]}')
        ...
    def configure_optimizers(self):
        optimizer = torch.optim.SGD(self.layer.parameters(), lr=100)
        lr_scheduler = torch.optim.lr_scheduler.MultiStepLR(optimizer, milestones=[4], gamma=.1)
        return [optimizer], [lr_scheduler]

def test_x(resume_from_checkpoint=None):
    model = BoringModel()
    trainer = pl.Trainer(
        callbacks=[pl.callbacks.ModelCheckpoint(save_top_k=-1)],
        resume_from_checkpoint=resume_from_checkpoint
    )
    trainer.fit(model, train)

test_x()
test_x('.../checkpoints/epoch=2.ckpt')
in the run from scratch the learning rate is changed at epoch 4
Epoch 0 / Step 0: lr 100
Epoch 0 / Step 1: lr 100
Epoch 1 / Step 2: lr 100
Epoch 1 / Step 3: lr 100
Epoch 2 / Step 4: lr 100
Epoch 2 / Step 5: lr 100
Epoch 3 / Step 6: lr 100
Epoch 3 / Step 7: lr 100
Epoch 4 / Step 8: lr 10.0  # lr changed
Epoch 4 / Step 9: lr 10.0
Epoch 5 / Step 10: lr 10.0
Epoch 5 / Step 11: lr 10.0

and in the resumed run the learning rate is changed at epoch 5, i.e., the scheduler lags by one epoch.
Epoch 3 / Step 6: lr 100
Epoch 3 / Step 7: lr 100
Epoch 4 / Step 8: lr 100
Epoch 4 / Step 9: lr 100
Epoch 5 / Step 10: lr 10.0  # lr changed
Epoch 5 / Step 11: lr 10.0

Environment
The output from the environment collection script:

CUDA:

GPU:

Tesla T4


available:         True
version:           10.1


Packages:

numpy:             1.18.5
pyTorch_debug:     True
pyTorch_version:   1.7.0+cu101
pytorch-lightning: 1.0.6
tqdm:              4.41.1


System:

OS:                Linux
architecture:

64bit



processor:         x86_64
python:            3.6.9
version:           #1 SMP Thu Jul 23 08:00:38 PDT 2020