Deepspeed + Auto Select GPUs = CUDA Out of Memory Error
🐛 Bug

Please reproduce using the BoringModel
https://colab.research.google.com/drive/17Bt2m570f4o16iwbEV1fpUhgO04cuCqg?usp=sharing
To Reproduce
You can see the code on the BoringModel above, but I don't think it'll run on Colab because it's a multigpu issue.
Basically, when I have a large-ish model (2M parameters), I find that deepspeed is incompatible with auto_select_gpus.
So,
trainer = pl.Trainer(
    gpus=8,
    accelerator='ddp',
    plugs='deepspeed',
    precision=16,
)

seems to work
But,
trainer = pl.Trainer(
    gpus=-1,
    auto_select_gpus=True,
    accelerator='ddp',
    plugs='deepspeed',
    precision=16,
)

causes a CUDA out of memory error.

Expected behavior
I'd expect it to select the GPUs and run it.
Environment

CUDA:
- GPU:
- Tesla V100-SXM2-16GB-N
- Tesla V100-SXM2-16GB-N
- Tesla V100-SXM2-16GB-N
- Tesla V100-SXM2-16GB-N
- Tesla V100-SXM2-16GB-N
- Tesla V100-SXM2-16GB-N
- Tesla V100-SXM2-16GB-N
- Tesla V100-SXM2-16GB-N
- available:         True
- version:           11.1
Packages:
- numpy:             1.20.2
- pyTorch_debug:     False
- pyTorch_version:   1.8.1+cu111
- pytorch-lightning: 1.2.6
- tqdm:              4.59.0
System:
- OS:                Linux
- architecture:
- 64bit
-
- processor:         x86_64
- python:            3.9.2
- version:           #16~16.04.1-Ubuntu SMP Thu Apr 5 12:19:23 UTC 2018

Additional context
Semi-related, am I supposed to specify an accelerator when using deepspeed? In the docs, none is specified, but when i run without an accelerator, it complains and says I should be setting one.