[distributed] set_nvidia_flags doesn't affect dp, does affect ddp
🐛 Bug
When default CUDA GPU order differs from PCI_BUS_ID order the user won't use the same GPUs in DP and DDP modes.
Trainer(gpus='0,1,2', distributed_backend='dp') uses gpus with PCI_BUS_ID's: '0,1,3' (on my machine) whereas Trainer(gpus='0,1,2', distributed_backend='ddp') uses gpus with PCI_BUS_ID's: '0,1,2'
It seems that setting environment variables programmatically doesn't actually affect that process, but it does affect spawned processes (ex. torch.multiprocessing.spawn)
os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
os.environ["CUDA_VISIBLE_DEVICES"] = $GPU_STR
To Reproduce
Steps to reproduce the behavior (I don't know how to reproduce GPU ordering, let me know if I should provide my own outputs)

Have machine with multiple GPUs ordered differently by CUDA and PCI_BUS_ID
pip install pytorch-lightning (or just pip install torch, mine is 1.4.0)
Use loop to check device order (and environment variables) before setting os.environ:

for x in range($NUM_GPUS):
        print(f'Data parallel device id: {x}: name: {torch.cuda.get_device_name(x)}


Set os.environ programmatically as in pytorch-lightning distrib set_nvidia_flags
Use loop in 2. and observe no change

Multiprocessing success example gist available:
2a. Run gist with cuda_gpu_test.py . Observe Device order is changed only in spawned processes
2b. Run gist with CUDA_DEVICE_ORDER=PCI_BUS_ID cuda_gpu_test.py. Observe Device order is same in spawned processes.
2c. [optional] run gist with CUDA_VISIBLE_DEVICES=0,1 cuda_gpu_test.py. Observe pytorch error because certain gpus are not available
Expected behavior
Using the same gpus parameter / flag for trainer should use same local gpus no matter dp or ddp.
Setting CUDA_DEVICE_ORDER to PCI_BUS_ID should change the order no matter dp or ddp.
Environment
PyTorch version: 1.4.0
Is debug build: No
CUDA used to build PyTorch: 10.1
OS: Ubuntu 19.10
GCC version: (Ubuntu 9.2.1-9ubuntu2) 9.2.1 20191008
CMake version: Could not collect
Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: Could not collect
GPU models and configuration:
GPU 0: GeForce RTX 2080 Ti
GPU 1: GeForce RTX 2080 Ti
GPU 2: GeForce GT 710
GPU 3: GeForce RTX 2080 Ti
Nvidia driver version: 435.21
cuDNN version: Could not collect
Versions of relevant libraries:
[pip] numpy==1.18.1
[pip] pytorch-lightning==0.7.1
[pip] torch==1.4.0
[conda] pytorch-lightning         0.7.1                    pypi_0    pypi
[conda] torch                     1.4.0                    pypi_0    pypi
(Installed pytorch-lightning from master in this environment, but pytorch is the relevant package here, as pytorch-lightning code I'm talking about hasn't changed in 5 months)
Additional context
Obviously a niche case, mostly wanted to bring to attention that the programmatic "PCI_BUS_ID" order doesn't actually do anything in DP mode. I think for most users they won't notice this, but my build can't seem to figure out which slots go to which GPUs.
This does have implications about CUDA_VISIBLE_DEVICES in ddp vs dp, as the environment variable goes into effect on ddp spawned processes but doesn't affect dp.
If the user doesn't set CUDA_VISIBLE_DEVICES from the command line I believe all devices are available to torch, but if the user explicitly sets that env variable they will be limited to the env variable (tested on minimal gist, torch throws "invalid device id" if cuda makes the device not visible).
An obnoxious workaround is importing torch after setting CUDA_DEVICE_ORDER, as noted on stack overflow. I tried this in an edit to the minimal gist  but I'm pretty sure that's not going to be very useful to pytorch-lightning.
It seems to me that DP might need to launch a new process to put the lightning defined environment variables in effect. This would make it functionally more similar to ddp and ddp2 (haven't tested ddp2 personally), but does not make much sense from the pytorch / python side of things.
I don't have a minimal example with a pytorch lightning module as I ran into this today on a research project I've been working on. Would be glad to set one up with one of the lightning examples, but this might be specific to my machine.
Been loving the repo so far, cheers!