RAM not correctly released when training a pl module multiple times
🐛 Bug
When I use the pl.Trainer multiple times (for instance when doing cross-validation), it seems that the ram is not completely released, as the ram memory usage increases over runs, in a strange way.

To Reproduce
Steps to reproduce the behavior:

Define a pl module and a pl.trainer inside a function, let's call it train()
call train() multiple times and track ram usage with psutil

Code sample
import psutil
from loguru import logger
import numpy as np
import torch
import pytorch_lightning as pl
from torch.utils.data import DataLoader, random_split
from torch.nn import functional as F
from torchvision.datasets import MNIST
from torchvision import transforms
import os


class LightningMNISTClassifier(pl.LightningModule):
    def __init__(self):
        super(LightningMNISTClassifier, self).__init__()

        # mnist images are (1, 28, 28) (channels, width, height)
        self.layer_1 = torch.nn.Linear(28 * 28, 128)
        self.layer_2 = torch.nn.Linear(128, 256)
        self.layer_3 = torch.nn.Linear(256, 10)

    def forward(self, x):
        batch_size, channels, width, height = x.size()
        # (b, 1, 28, 28) -> (b, 1*28*28)
        x = x.view(batch_size, -1)
        # layer 1 (b, 1*28*28) -> (b, 128)
        x = self.layer_1(x)
        x = torch.relu(x)
        # layer 2 (b, 128) -> (b, 256)
        x = self.layer_2(x)
        x = torch.relu(x)
        # layer 3 (b, 256) -> (b, 10)
        x = self.layer_3(x)
        # probability distribution over labels
        x = torch.log_softmax(x, dim=1)
        return x

    def cross_entropy_loss(self, logits, labels):
        return F.nll_loss(logits, labels)

    def training_step(self, train_batch, batch_idx):
        x, y = train_batch
        logits = self.forward(x)
        loss = self.cross_entropy_loss(logits, y)

        logs = {'train_loss': loss}
        return {'loss': loss, 'log': logs}

    def prepare_data(self):
        # transforms for images
        transform = transforms.Compose([transforms.ToTensor(),
                                        transforms.Normalize((0.1307,), (0.3081,))])
        # prepare transforms standard to MNIST
        mnist_train = MNIST(os.getcwd(), train=True,
                            download=True, transform=transform)
        self.mnist_train, _ = random_split(mnist_train, [10_000, 50_000])

    def train_dataloader(self):
        return DataLoader(self.mnist_train, batch_size=1000, num_workers=3)

    def configure_optimizers(self):
        optimizer = torch.optim.Adam(self.parameters(), lr=1e-3)
        return optimizer


def train():
    model = LightningMNISTClassifier()
    trainer = pl.Trainer(
        checkpoint_callback=False,
        max_epochs=1
    )
    trainer.fit(model)


# train
rams_used = []
process = psutil.Process()
for _ in range(100):
    train()
    ram_used = process.memory_info()[0]/2.**30
    logger.warning(f"RAM USED : {ram_used}")
    rams_used.append(ram_used)

np.save("rams_used.npy", rams_used)
Expected behavior
I would expect to observe the almost exact same ram usage after the training of the model. Indeed I compared it with native pytorch and did not observe any ram usage increase over runs.
Environment
* CUDA:
        - GPU:
                - TITAN X (Pascal)
                - TITAN X (Pascal)
        - available:         True
        - version:           10.1
* Packages:
        - numpy:             1.18.4
        - pyTorch_debug:     False
        - pyTorch_version:   1.5.0+cu101
        - pytorch-lightning: 0.7.6
        - tensorboard:       2.2.1
        - tqdm:              4.46.0
* System:
        - OS:                Linux
        - architecture:
                - 64bit
                - ELF
        - processor:         x86_64
        - python:            3.7.4
        - version:           #192-Ubuntu SMP Fri Sep 13 12:02:50 UTC 2019

Additional context
I tried with older versions of lightning, and observed the same behaviour even in 0.7.1.
The example above is with a small dataset of Mnist, but I encountered more impressive increase with a bigger dataset, as it ram went from 20Go to 65Go after 20 runs.