ddp opens processes on GPUs (and consumes memory) not assigned to the Trainer
🐛 Bug
When training on multiple GPUs with ddp, it is still starting processes on the unused GPUs (with a lot less memory, but still). It's not running any work there (GPU-Util is 0% in nvidia-smi).
How to fix this manually: I can run CUDA_VISIBLE_DEVICES="2,3" python train.py --gpus 0,1 and then it works, it only runs processes on GPUs 2,3 and nothing on 0,1.
It also crashes when you use GPU 0 or GPU 1 and then some others (for example 0, 2, 3 crashes).
To me it looks like the ddp rank is used to index into the list of visible GPUs and there is some error in how that is handled.
To Reproduce
Run training on GPU index 2,3 and it will also create a process on 0 and 1 (can be verified in nvidia-smi):
trainer = pl.Trainer(gpus=[2,3], accelerator='ddp')
trainer.fit(lightning_model)

Expected behavior
I should be able to run any GPU combination and the other GPUs should not be touched.
Environment

CUDA:

GPU:

GeForce RTX 2080 Ti
GeForce RTX 2080 Ti
GeForce RTX 2080 Ti
GeForce RTX 2080 Ti


available:         True
version:           10.2


Packages:

numpy:             1.18.5
pyTorch_debug:     True
pyTorch_version:   1.7.0
pytorch-lightning: 1.0.8
tqdm:              4.54.1


System:

OS:                Linux
architecture:

64bit
ELF


processor:         x86_64
python:            3.8.5
version:           #124-Ubuntu SMP Thu Oct 15 13:03:05 UTC 2020