Slurm GPUs Not Properly Detected
🐛 Bug
I'm using slurm and the GPUs don't seem to be detected properly, the doc at https://pytorch-lightning.readthedocs.io/en/stable/clouds/slurm.html implies it's as simple as passing the number of GPUs and nodes to the trainer with a suitable sbatch script. But when I do this for a single node, 2 GPUs, I get the following error:
pytorch_lightning.utilities.exceptions.MisconfigurationException: 
                You requested GPUs: [0, 1]
                But your machine only has: [0]
            

There's no output printed about slurm, I have created a minimal reproduction of the error
To Reproduce
slurm_test.py:
import pytorch_lightning as pl

pl.Trainer(gpus=2, num_nodes=1, accelerator="ddp")

slurm_test.sh
#!/bin/bash

# Parameters
#SBATCH --cpus-per-task=4
#SBATCH --gres=gpu:p6000:1
#SBATCH --mem=32GB
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=2
#SBATCH --open-mode=append
#SBATCH --qos=high
#SBATCH --signal=USR1@90
#SBATCH --time=2160

# command
srun --output /cfarhomes/maxehr/mm/%j_%t_log.out --error /cfarhomes/maxehr/mm/%j_%t_log.err --unbuffered /cfarhomes/maxehr/.cache/pypoetry/virtualenvs/mm-XIIWUkOg-py3.8/bin/python  /cfarhomes/maxehr/mm/slurm_test.py /cfarhomes/maxehr/mm

Parameters (gres) may need to be adjusted, this is how my cluster calls them. Run this with sbatch slurm_test.sh
As you can see there are 2 tasks per node and 1 node and the syntax matches with that used in the docs. This fails with the following exception:
Traceback (most recent call last):
  File "/cfarhomes/maxehr/mm/slurm_test.py", line 3, in <module>
    pl.Trainer(gpus=2, num_nodes=1, accelerator="ddp")
  File "/cfarhomes/maxehr/.cache/pypoetry/virtualenvs/mm-XIIWUkOg-py3.8/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/env_vars_connector.py", line 41, in overwrite_by_env_vars
    return fn(self, **kwargs)
  File "/cfarhomes/maxehr/.cache/pypoetry/virtualenvs/mm-XIIWUkOg-py3.8/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 348, in __init__
    self.accelerator_connector.on_trainer_init(
  File "/cfarhomes/maxehr/.cache/pypoetry/virtualenvs/mm-XIIWUkOg-py3.8/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator_connector.py", line 104, in on_trainer_init
    self.trainer.data_parallel_device_ids = device_parser.parse_gpu_ids(self.trainer.gpus)
  File "/cfarhomes/maxehr/.cache/pypoetry/virtualenvs/mm-XIIWUkOg-py3.8/lib/python3.8/site-packages/pytorch_lightning/utilities/device_parser.py", line 78, in parse_gpu_ids
    gpus = _sanitize_gpu_ids(gpus)
  File "/cfarhomes/maxehr/.cache/pypoetry/virtualenvs/mm-XIIWUkOg-py3.8/lib/python3.8/site-packages/pytorch_lightning/utilities/device_parser.py", line 139, in _sanitize_gpu_ids
    raise MisconfigurationException(f"""
pytorch_lightning.utilities.exceptions.MisconfigurationException: 
                You requested GPUs: [0, 1]
                But your machine only has: [0]

Both jobs produce the same exception and neither job has anything written to stdout.
Expected behavior
It works.
Environment

CUDA:
- GPU:
- available:         False
- version:           10.2
Packages:
- numpy:             1.20.1
- pyTorch_debug:     False
- pyTorch_version:   1.7.1
- pytorch-lightning: 1.1.6
- tqdm:              4.57.0
System:
- OS:                Linux
- architecture:
- 64bit
- ELF
- processor:         x86_64
- python:            3.8.1
- version:           #1 SMP Thu Jan 21 16:15:07 EST 2021

Additional context
I would also like to know if there's a way to disable automatic requeueing that I've read about, supposing this eventually works.