Questions about DDP and slurm
Hi,
When running on slurm with say 8 GPUs on a single node with ddp as backend, each process occupies all GPUs. It only uses small portion (less than 1gb) of non-root GPUs. However, multiplying it by 7 is still a large chuck of memory unused for training. I followed the multi-GPUs and slurm document. Maybe I am missing something but I could not empty that memory. Unfortunately, I cannot find a similar issue. Could you send me some pointers? Thanks!
version
pytorch                   1.5.1           py3.7_cuda10.1.243_cudnn7.6.3_0    pytorch
pytorch-lightning         0.8.1                    pypi_0    pypi

nvidia-smi (similar output for all GPUs)
|    0     163      C   python 24409MiB |
|    0     164      C   python   933MiB |
|    0     165      C   python   933MiB |
|    0     166      C   python   933MiB |
|    0     167      C   python   933MiB |
|    0     168      C   python   933MiB |
|    0     169      C   python   933MiB |
|    0     170      C   python   933MiB |
|    1     163      C   python   933MiB |
|    1     164      C   python 24411MiB |
|    1     165      C   python   933MiB |
|    1     166      C   python   933MiB |
|    1     167      C   python   933MiB |
|    1     168      C   python   933MiB |
|    1     169      C   python   933MiB |
|    1     170      C   python   933MiB |
...

log
1: GPU available: True, used: True
1: TPU available: False, using: 0 TPU cores
1: Multi-processing is handled by Slurm.
1: CUDA_VISIBLE_DEVICES: [0,1,2,3,4,5,6,7]
1: Using 16bit precision.
1: initializing ddp: GLOBAL_RANK: 1, MEMBER: 2/8
6: GPU available: True, used: True
6: TPU available: False, using: 0 TPU cores
6: Multi-processing is handled by Slurm.
6: CUDA_VISIBLE_DEVICES: [0,1,2,3,4,5,6,7]
6: Using 16bit precision.
6: initializing ddp: GLOBAL_RANK: 6, MEMBER: 7/8
4: GPU available: True, used: True
4: TPU available: False, using: 0 TPU cores
4: Multi-processing is handled by Slurm.
4: CUDA_VISIBLE_DEVICES: [0,1,2,3,4,5,6,7]
4: Using 16bit precision.
4: initializing ddp: GLOBAL_RANK: 4, MEMBER: 5/8
7: GPU available: True, used: True
7: TPU available: False, using: 0 TPU cores
7: Multi-processing is handled by Slurm.
7: CUDA_VISIBLE_DEVICES: [0,1,2,3,4,5,6,7]
7: Using 16bit precision.
7: initializing ddp: GLOBAL_RANK: 7, MEMBER: 8/8
5: GPU available: True, used: True
5: TPU available: False, using: 0 TPU cores
5: Multi-processing is handled by Slurm.
5: CUDA_VISIBLE_DEVICES: [0,1,2,3,4,5,6,7]
5: Using 16bit precision.
5: initializing ddp: GLOBAL_RANK: 5, MEMBER: 6/8
3: GPU available: True, used: True
3: TPU available: False, using: 0 TPU cores
3: Multi-processing is handled by Slurm.
3: CUDA_VISIBLE_DEVICES: [0,1,2,3,4,5,6,7]
3: Using 16bit precision.
3: initializing ddp: GLOBAL_RANK: 3, MEMBER: 4/8
2: GPU available: True, used: True
2: TPU available: False, using: 0 TPU cores
2: Multi-processing is handled by Slurm.
2: CUDA_VISIBLE_DEVICES: [0,1,2,3,4,5,6,7]
2: Using 16bit precision.
2: initializing ddp: GLOBAL_RANK: 2, MEMBER: 3/8
0: GPU available: True, used: True
0: TPU available: False, using: 0 TPU cores
0: Multi-processing is handled by Slurm.
0: CUDA_VISIBLE_DEVICES: [0,1,2,3,4,5,6,7]
0: Using 16bit precision.
0: initializing ddp: GLOBAL_RANK: 0, MEMBER: 1/8
0: ----------------------------------------------------------------------------------------------------
0: distributed_backend=ddp
0: All DDP processes registered. Starting ddp with 8 processes
0: ----------------------------------------------------------------------------------------------------
1: Set SLURM handle signals.
2: Set SLURM handle signals.
5: Set SLURM handle signals.
6: Set SLURM handle signals.
7: Set SLURM handle signals.
0: Set SLURM handle signals.
3: Set SLURM handle signals.
4: Set SLURM handle signals.