Random CUDA OOM error when starting SLURM jobs
🐛 Bug
When submitting jobs to SLURM, some jobs (around 1-2% of them) will randomly encounter a CUDA OOM error during the setup prior to training. I can confirm it's not an issue with the configuration of the job vs hardware itself, since I can resubmit the exact same job script and it will work. I also know that my resource consumption is not even close to the limit, since the jobs that do work use ~10Gb on 32Gb GPUs.
I already checked with nvidia-smi that the GPUs I get allocated are empty at the start of the training, confirming that it's not a problem with the cluster:
echo $(nvidia-smi --query-gpu=memory.used --format=csv,noheader) # At the beginning of the script
# Output: 0 MiB
Here is the output I get when my jobs crash:
CometLogger will be initialized in online mode
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
Using native 16bit precision.
Traceback (most recent call last):
  File "my_code.py", line XX, in XX
    trainer.fit(model)
  File "my_virtualenv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 424, in fit
    self.accelerator_backend.setup(model)
  File "my_virtualenv/lib/python3.8/site-packages/pytorch_lightning/accelerators/gpu_accelerator.py", line 36, in setup
    model.cuda(self.trainer.root_gpu)
  File "my_virtualenv/lib/python3.8/site-packages/pytorch_lightning/utilities/device_dtype_mixin.py", line 124, in cuda
    return super().cuda(device=device)
  File "my_virtualenv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 463, in cuda
    return self._apply(lambda t: t.cuda(device))
  File "my_virtualenv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 359, in _apply
    module._apply(fn)
  File "my_virtualenv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 359, in _apply
    module._apply(fn)
  File "my_virtualenv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 359, in _apply
    module._apply(fn)
  [Previous line repeated 2 more times]
  File "my_virtualenv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 381, in _apply
    param_applied = fn(param)
  File "my_virtualenv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 463, in <lambda>
    return self._apply(lambda t: t.cuda(device))
RuntimeError: CUDA error: out of memory
Please reproduce using [the BoringModel and post here]
Unfortunately, given the nature of the bug, it's not something I can reproduce with the BoringModel.
To Reproduce

Here are any of the SLURM parameters that could be relevant (only excludes logging and job time):
#SBATCH --gres=gpu:v100l:1
#SBATCH --cpus-per-task=8
#SBATCH --mem=48000M
The related non-default parameter to the trainer and dataloaders are the following:
Trainer(..., benchmark=true, precision=16, ...)

# Inside the lightning module
DataLoader(..., shuffle=True, num_workers=7, pin_memory=self.on_gpu, ...)
From experience, I set num_workers to 1 lower than the number of CPUs I request. On my local machine, that's never caused me any issues.
Expected behavior
SLURM jobs should not encounter random CUDA OOM error when configured with the necessary ressources.
Environment
PyTorch and CUDA are provided by the cluster's manager, so that they are optimized for the hardware. PyTorch Lightning I installed myself using pip.

CUDA:

GPU:

Tesla V100-SXM2-32GB


available:         True
version:           10.2


Packages:

numpy:             1.19.1
pyTorch_debug:     True
pyTorch_version:   1.7.0
pytorch-lightning: 1.0.4
tqdm:              4.50.2


System:

OS:                Linux
architecture:

64bit
ELF


processor:
python:            3.8.0
version:           #1 SMP Tue Feb 4 23:02:59 UTC 2020



Additional context
The PyTorch version reported here is 1.7, but I updated it just yesterday. I encountered the reported bug using both PyTorch 1.6 and 1.7.
EDIT: Fixed mistakes when detailing trainer & dataloader arguments.