auto_select_gpu does not free memory allocated on GPU for DDP/Horovod
🐛 Bug
If using both auto_select_gpu=True and the ddp or horovod accelerator, the memory allocated by pick_single_gpu is not freed by torch.
This can't be reproduced on Colab since DDP isn't supported.
To Reproduce
Paste this code into a file and run it:
import pytorch_lightning as pl
from torch import nn
from torchvision.models import resnet18
from torchvision.datasets import CIFAR10
from torchvision.transforms import ToTensor
from torch.utils.data import DataLoader
from torch.optim import SGD


class Model(pl.LightningModule):
    def __init__(self):
        super().__init__()
        self.model = nn.Linear(3072, 10)

    def train_dataloader(self):
        dataset = CIFAR10("__pycache__", download=True, transform=ToTensor())
        return DataLoader(dataset)

    def configure_optimizers(self):
        return SGD(self.model.parameters(), lr=0.01)

    def training_step(self, batch, _):
        x, label = batch
        y = self.model(x.view(x.size(0), -1))
        return nn.functional.cross_entropy(y, label)


trainer = pl.Trainer(accelerator="ddp", gpus=2, auto_select_gpus=True)

model = Model()
trainer.fit(model)

Expected behavior
The same amount of memory is allocated on both GPUs. However, nvidia-smi shows this:

Environment
* CUDA:
        - GPU:
                - GeForce RTX 2080 Ti
                - GeForce RTX 2080 Ti
                - GeForce RTX 2080 Ti
                - GeForce RTX 2080 Ti
        - available:         True
        - version:           10.2
* Packages:
        - numpy:             1.19.2
        - pyTorch_debug:     False
        - pyTorch_version:   1.6.0
        - pytorch-lightning: 1.0.3
        - tqdm:              4.50.1
* System:
        - OS:                Linux
        - architecture:
                - 64bit
                - ELF
        - processor:         x86_64
        - python:            3.8.5
        - version:           #110-Ubuntu SMP Tue Jun 23 02:39:32 UTC 2020