Custom callbacks are lost after resume_from_checkpoint
🐛 Bug

A checkpoint should state everything needed to restore a training session including the state of all callbacks. However, custom callbacks are lost after resuming the trainer from Trainer(resume_from_checkpoint="...ckpt").
To Reproduce
Steps to reproduce the behaviour:

Define a trainer with a custom callback.
trainer = pl.Trainer(gpus=1, max_epochs=3, progress_bar_refresh_rate=20, callbacks=[MyPrintingCallback()])
(Probably not necessary) run trainer.fit()
Save trainer trainer.save_checkpoint('checkpoint.ckpt')
Load trainer loaded_trainer = pl.Trainer(resume_from_checkpoint='checkpoint.ckpt')
Problem trainer.callbacks is not the same as loaded_trainer.callbacks


Code sample

import os

import torch
from torch.nn import functional as F
from torch.utils.data import DataLoader
from torchvision.datasets import MNIST
from torchvision import transforms
import pytorch_lightning as pl

class MyPrintingCallback(pl.Callback):

    def on_test_end(self, trainer, pl_module):
        print('Custom Callback!')

class MNISTModel(pl.LightningModule):

    def __init__(self):
        super(MNISTModel, self).__init__()
        self.l1 = torch.nn.Linear(28 * 28, 10)

    def forward(self, x):
        return torch.relu(self.l1(x.view(x.size(0), -1)))

    def training_step(self, batch, batch_nb):
        x, y = batch
        loss = F.cross_entropy(self(x), y)
        return {'loss': loss}

    def test_step(self, batch, batch_nb):
        x, y = batch
        loss = F.cross_entropy(self(x), y)
        loss = F.cross_entropy(self(x), y)
        return {'loss': loss}
        
    def configure_optimizers(self):
        return torch.optim.Adam(self.parameters(), lr=0.02)


data_loader = DataLoader(MNIST(os.getcwd(), train=True, download=True, transform=transforms.ToTensor()), batch_size=32)

model = MNISTModel()

trainer = pl.Trainer(gpus=1, callbacks=[MyPrintingCallback()])    
trainer.test(model, data_loader)  
trainer.save_checkpoint('checkpoint.ckpt')

loaded_model = MNISTModel.load_from_checkpoint(checkpoint_path="checkpoint.ckpt")
loaded_trainer = pl.Trainer(resume_from_checkpoint='checkpoint.ckpt')
loaded_trainer.test(loaded_model, data_loader) # using test because cpu

# MyPrintingCallback has disappeared from loaded_trainer
print('trainer callbacks', *trainer.callbacks)
print('loaded_trainer callbacks', *loaded_trainer.callbacks)
Colab with code and comments:
https://colab.research.google.com/drive/1ksECPaUWMPN76fuUWEa443ZVnvTwvbZB?usp=sharing
Expected behavior

The original trainer and loaded trainer should have the same callbacks.
trainer.callbacks == loaded_trainer.callbacks # Thought this would be True
Environment
Just google colab with gpu.

CUDA:

GPU:

Tesla K80


available:         True
version:           10.1


Packages:

numpy:             1.18.5
pyTorch_debug:     False
pyTorch_version:   1.5.1+cu101
pytorch-lightning: 0.8.3
tensorboard:       2.2.2
tqdm:              4.41.1


System:

OS:                Linux
architecture:

64bit



processor:         x86_64
python:            3.6.9
version:           #1 SMP Wed Feb 19 05:26:34 PST 2020



Additional context

This may be intentional and just haven't realized it. However, the docs do specify that the state of all callbacks are restored.
Also on a related note, a trainer resumed from a checkpoint is not sent to the gpu even if the original is using a gpu. This was also a little confusing since it felt unintuitive. Code sample below.
# Also I realized that the loaded_trainer is on cpu. Is this intentional?
print('model device', model.device)
print('loaded_model device', loaded_model.device)

# This means that using a loaded_trainer with the orignal model won't work (feels unintuitive)
try:
    loaded_trainer.test(model, test_loader)
except RuntimeError as e:
    print(e)