Precision 16 not transforming inputs to Float16 nor having LSTMs as halfs 
🐛 Bug
When using precision 16 to train a model, the LSTM layers are not transformed to accept FP16 and the inputs to the model are FP32 (as mention in the issue #1876). This can be seen with simple modifications of the MNIST model your provide in colab.
To Reproduce
Execute the following code:
import os

import pytorch_lightning as pl
import torch
from torch.nn import functional as F
from torch.utils.data import DataLoader
from torchvision import transforms
from torchvision.datasets import MNIST


class LitClassifier(pl.LightningModule):
    def __init__(self):
        super().__init__()
        self.l0 = torch.nn.Linear(28 * 28, 32)
        self.l1 = torch.nn.LSTM(32, 10, bidirectional=True)

    def forward(self, x):
        print('forward() input dtype: ', x.dtype)
        x1 = self.l0(x.view(x.size(0), -1))
        return torch.relu(self.l1(x1.view(x1.size(0), 1, -1))[0])

    def training_step(self, batch, batch_nb):
        x, y = batch
        loss = F.cross_entropy(self(x).squeeze(1), y)
        tensorboard_logs = {'train_loss': loss}
        return {'loss': loss, 'log': tensorboard_logs}

    def configure_optimizers(self):
        return torch.optim.Adam(self.parameters(), lr=0.02)


print('Pytorch version: ', torch.__version__)
print('PL version: ', pl.__version__)

train_loader = DataLoader(MNIST(os.getcwd(), train=True, download=True, transform=transforms.ToTensor()), batch_size=32)
model = LitClassifier()
trainer = pl.Trainer(gpus=1, precision=16)
trainer.fit(model, train_loader)


This gives the following error:
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "C:\Program Files\JetBrains\PyCharm 2018.3.5\plugins\python\helpers\pydev\_pydev_bundle\pydev_umd.py", line 197, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
  File "C:\Program Files\JetBrains\PyCharm 2018.3.5\plugins\python\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "C:/Users/rodri/PycharmProjects/TTMelGAN/src/data/__init__.py", line 38, in <module>
    trainer.fit(model, train_loader)
  File "C:\Users\rodri\Miniconda3\envs\TTMelGAN\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 921, in fit
    self.single_gpu_train(model)
  File "C:\Users\rodri\Miniconda3\envs\TTMelGAN\lib\site-packages\pytorch_lightning\trainer\distrib_parts.py", line 171, in single_gpu_train
    self.run_pretrain_routine(model)
  File "C:\Users\rodri\Miniconda3\envs\TTMelGAN\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 1091, in run_pretrain_routine
    self.train()
  File "C:\Users\rodri\Miniconda3\envs\TTMelGAN\lib\site-packages\pytorch_lightning\trainer\training_loop.py", line 374, in train
    self.run_training_epoch()
  File "C:\Users\rodri\Miniconda3\envs\TTMelGAN\lib\site-packages\pytorch_lightning\trainer\training_loop.py", line 457, in run_training_epoch
    _outputs = self.run_training_batch(batch, batch_idx)
  File "C:\Users\rodri\Miniconda3\envs\TTMelGAN\lib\site-packages\pytorch_lightning\trainer\training_loop.py", line 633, in run_training_batch
    loss, batch_output = optimizer_closure()
  File "C:\Users\rodri\Miniconda3\envs\TTMelGAN\lib\site-packages\pytorch_lightning\trainer\training_loop.py", line 594, in optimizer_closure
    output_dict = self.training_forward(split_batch, batch_idx,
  File "C:\Users\rodri\Miniconda3\envs\TTMelGAN\lib\site-packages\pytorch_lightning\trainer\training_loop.py", line 767, in training_forward
    output = self.model.training_step(*args)
  File "C:/Users/rodri/PycharmProjects/TTMelGAN/src/data/__init__.py", line 24, in training_step
    loss = F.cross_entropy(self(x).squeeze(1), y)
  File "C:\Users\rodri\Miniconda3\envs\TTMelGAN\lib\site-packages\torch\nn\modules\module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:/Users/rodri/PycharmProjects/TTMelGAN/src/data/__init__.py", line 20, in forward
    return torch.relu(self.l1(x1.view(x1.size(0), 1, -1))[0])
  File "C:\Users\rodri\Miniconda3\envs\TTMelGAN\lib\site-packages\torch\nn\modules\module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:\Users\rodri\Miniconda3\envs\TTMelGAN\lib\site-packages\torch\nn\modules\rnn.py", line 576, in forward
    result = _VF.lstm(input, hx, self._flat_weights, self.bias, self.num_layers,
RuntimeError: cuDNN error: CUDNN_STATUS_BAD_PARAM

If in the example we remove the l0 layer, the LSTM receives the input directly (in FP32) and therefore works without problem. When we keep it, l0 transforms the FP32 input into FP16 and breaks.
Environment
Pytorch version:  1.6.0.dev20200618
PL version:  0.8.0
(both downloaded before executing to check that I have the latest version in both cases. Error also occurs in Linux.)

CUDA:

GPU:

GeForce MX150


available:         True
version:           10.2


Packages:

numpy:             1.18.1
pyTorch_debug:     False
pyTorch_version:   1.6.0.dev20200618
pytorch-lightning: 0.8.0
tensorboard:       2.2.2
tqdm:              4.46.1


System:

OS:                Windows
architecture:

64bit
WindowsPE


processor:         Intel64 Family 6 Model 142 Stepping 10, GenuineIntel
python:            3.8.3
version:           10.0.18362