Memory allocated on gpu:0 when using torch.cuda.empty_cache()
🐛 Bug
Pytorch lightning calls torch.cuda.empty_cache() at times, e.g. at the end of the training loop. When the trainer is set to run on GPUs other than gpu:0, it still allocates memory on gpu:0 when running torch.cuda.empty_cache(). Apparently this is the initial device context, but it can be avoided. For example,
with torch.cuda.device('cuda:1'):
    torch.cuda.empty_cache()

If the cache is emptied in this way, it will not allocate memory on any other gpu other than the one specified
This seems to be the same issue as in #458, but was never resolved and is still an issue.
To Reproduce
Steps to reproduce the behavior:

Create a pl.Trainer with gpus=[1]
Fit a model on gpu:1
torch.cuda.empty_cache() runs in run_training_teardown at the end of the training loop
nvidia-smi shows memory usage on gpu:0
If gpu:0 already had high memory allocation because of another job, then it will throw a CUDA out of memory error

.../pytorch_lightning/trainer/training_loop.py in run_training_teardown(self)
   1153             model = self.get_model()
   1154             model.cpu()
-> 1155             torch.cuda.empty_cache()
   1156 
   1157     def training_forward(self, batch, batch_idx, opt_idx, hiddens):

.../torch/cuda/memory.py in empty_cache()
     84     """
     85     if is_initialized():
---> 86         torch._C._cuda_emptyCache()
     87 
     88 

RuntimeError: CUDA error: out of memory


Code sample
trainer = Trainer(gpus=[1])
trainer.fit(task, train_dataloader, val_dataloader)

Expected behavior
Only gpu:1 should be used when training this model.
Environment

CUDA:

GPU:

GeForce RTX 2080 Ti
GeForce GTX 1080 Ti


available:         True
version:           10.1


Packages:

numpy:             1.18.1
pyTorch_debug:     False
pyTorch_version:   1.5.0
pytorch-lightning: 0.9.0rc12
tensorboard:       2.2.1
tqdm:              4.46.0


System:

OS:                Linux
architecture:

64bit
ELF


processor:         x86_64
python:            3.8.3
version:           18.04.1-Ubuntu