Segmentation Fault when training 3D CNN using 4 GPUs with batch_size=8
🐛 Bug
Trying to train 3D CNN using 4 GPUs, batch_size = 8, and num_workers >= 4 (ddp backend). I'm using a GCP VM with 16 cores and 60GB memory. The data is stored on a mounted disk and is roughly 3 TB.
I can successfully train using 2 GPUs, batch_size=4, and num_workers=4, but whenever I try increasing number of GPUs, batch size, and num_workers, a segmentation fault get's thrown. It's not the same segfault stack trace everytime, but the image below is most common. I tried increasing cores and memory to 32/120 and it still errors out. The memory used doesn't stay stable either. I see the buff/cache increasing to about 85% of available mem, then segfault gets thrown shortly after.
I followed the PyTorch Lightning website carefully and made sure the code didn't have anything that could cause slow down or memory leak. I followed all the tips for fast performance, multi-gpu training, etc. that's listed on docs website.
One more observation:
To test my code, I run training for 3 epochs using about 60 training examples and 60 validation. The test run works perfectly fine with batch_size=8, 4 GPUs, and num_workers=4. When I switch to normal training mode with full dataset, it errors out. I've noticed that anything more than 64 examples causes segfault.

Expected behavior
Environment

PyTorch Version (e.g., 1.0): 1.6.0+cu101
OS (e.g., Linux): Linux (Debian)
How you installed PyTorch (conda, pip, source): pip
Build command you used (if compiling from source):
Python version: 3.7.6
CUDA/cuDNN version: 10.1
GPU models and configuration: 4 x Nvidia Tesla V100
Any other relevant information:

Additional context

Data is stored as .npy files