Circulation training with different seed increases memory
🐛 Bug

I reproduce using [the BoringModel and post here]
https://colab.research.google.com/drive/1HvWVVTK8j2Nj52qU4Q4YCyzOm0_aLQF3?usp=sharing
Because of the needs of my project, I need to run the program over and over again to measure the performance of the model. So Each time I give the model a different SEED. This can also lead to memory leaks. This makes me wonder why giving different seedings would cause this problem.
based on BoringModel code ，I change seed. colab link :
https://colab.research.google.com/drive/1KUz-IFZ8RMK9O2Gd4XSaYbRe8iFFSD09?usp=sharing
Expected behavior
code：
for i in range(0,50): test_x(tmpdir) print(torch.cuda.max_memory_allocated()/ 1024**2,'\n','\n','\n','\n')
The torch.cuda.max_memory_allocated() is increasing.
I  ran out of memory in the second loop in real situation
Environment

CUDA:

GPU:

Tesla P4


available:         True
version:           10.1


Packages:

numpy:             1.18.5
pyTorch_debug:     False
pyTorch_version:   1.6.0+cu101
pytorch-lightning: 1.0.1
tqdm:              4.41.1


System:

OS:                Linux
architecture:

64bit



processor:         x86_64
python:            3.6.9
version:           #1 SMP Thu Jul 23 08:00:38 PDT 2020



Additional context
Wait online for a solution.  I've brought this up before, but it's off, I'm not familiar with Github, so I guess I should repost the issue to get attention again. #4161