Pytorch lightning switched to cpu in the middle of training. How can I debug this?
🐛 Bug


So I am training a model and suddenly PyTorch lightning started using the CPU. I checked the GPU status and it is working normally.
I have never seen this before. Can Pytorch start using CPU in the middle of training (at the next epoch)?
The weird thing is the next epoch was using the gpu.
EDIT: Some asked how I noticed it is not using GPU.
I noticed the difference by looking at the it/s displayed in the verbose. It dropped to 13 it/s from 130 in one epoch and then rebounded to 150 it/s. I immediately took a look at the CUDA usage and It was very low.
To Reproduce
This is the training loop: Pretty standard
`    model = MatrixFactorizationAttention(att_mf_config)
train_loader = sample_generator.instance_a_train_loader_attention(att_mf_config['batch_size'])
val_loader = sample_generator.instance_val_loader_attention
# wandb_logger.log_hyperparams(att_mf_config)
early_stop_callback = EarlyStopping(
    monitor='HR',
    min_delta=0.01,
    patience=10,
    verbose=True,
    mode='max'
)

trainer = pl.Trainer(gpus=1, max_epochs=150, precision=16,checkpoint_callback=False,logger = False,early_stop_callback=early_stop_callback)
trainer.fit(model, train_dataloader=train_loader, val_dataloaders=val_loader)

`
EDIT: apart from this training loop, I am using a standard PyTorch model with a regular classification pipeline.
Environment
Win 10 with pytorch lightning running on pycharm
Cuda 10.2
GPU 2060 super
pytorch: nightly version