Multi-GPU on AWS p2.8xlarge instance (ddp2 and ddp)
As information, AWS p2.8xlarge has 8 K80s all on the same node.
I have tried my model gpus=1 and distributed_backend=None on an AWS p2.xlarge instance (1 K80) and it works.
When I try gpus=8 and distributed_backend='ddp2' on an AWS p2.8xlarge, I get the following error:
  File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/trainer.py", line 335, in fit
task = int(os.environ['SLURM_LOCALID'])
File "/usr/lib/python3.6/os.py", line 669, in __getitem__
raise KeyError(key) from None
KeyError: 'SLURM_LOCALID'