Very slow training on SLURM cluster
🐛 Bug
When switching from my local machine (old and slow 960m laptop GPU) to a SLURM cluster with Titan X GPU's, I see a significant drop in performance from 4.71it/s to 7s/it (!) - so basically training becomes unusably slow.
To Reproduce

Take a model and submit it as a SLURM job. Happy to provide further details if requested, not sure what is relevant here.

2020-09-10 22:11:58,979 : lightning    : INFO       GPU available: True, used: True
2020-09-10 22:11:58,979 : lightning    : INFO       TPU available: False, using: 0 TPU cores
2020-09-10 22:11:58,979 : lightning    : INFO       CUDA_VISIBLE_DEVICES: [0,1]
2020-09-10 22:12:01,698 : lightning    : INFO       Set SLURM handle signals.
Epoch 1:  40%|███▉      | 150/377 [19:11<29:02,  7.68s/it, loss=2.401, v_num=19]

Code sample
The is setup as follows
logger = TensorBoardLogger(str(DATA_DIR_PATH / 'lightning_logs'), name=Path(__file__).stem)
trainer_kwargs = {'logger': logger,
                  'default_root_dir': str(DATA_DIR_PATH / 'lightning_logs'),
                  'val_check_interval': 0.2,  # check (1 / value) * times per train epoch
                  'gpus': 1,
                  # 'distributed_backend': 'ddp',  # checked with ddp enabled, 2 GPUs, multiple workers etc., same slow behaviour
                  'fast_dev_run': False}
trainer = pl.Trainer(**trainer_kwargs)
Environment

Packages:

numpy:                 1.18.1
pyTorch_debug:   False
pyTorch_version: 1.4.0
pytorch-lightning: 0.8.5
tensorboard:        1.15.0
tqdm:                   4.43.0




PyTorch Version (e.g., 1.0):
OS (e.g., Linux): Linux
How you installed PyTorch (conda, pip, source): conda
Python version: 3.7.9
CUDA/cuDNN version: cudatoolkit 10.0
GPU models and configuration: TitanX (on SLURM cluster)