ddp: Consider warning / failing fast if single proc relies on `len(gpus) >= 8` for CUDA, due to peer-to-peer resource limitations?
🚀 Feature
If using accelerator="ddp" (or any other backend that may trigger this error), ideally there's a warning and/or error about possibly encountering this issue, e.g.:
WARNING: You are requesting > 8 GPU devices be shared via CUDA's peer-to-peer protocol, but it generally
supports <= 8 devices. Please consider distributing this across multiple nodes (e.g. instead of `--gpus 8`,
use `--num_nodes 2 --gpus 4`).

Motivation
I just ran into this failure mode on the same instance type (p2.16xlarge) today.
https://forums.developer.nvidia.com/t/cuda-peer-resources-error-when-running-on-more-than-8-k80s-aws-p2-16xlarge/45351#4994583
(FWIW, I uh, locked the EC2 instance, tried to reboot which didn't work, then tried to stop-then-start and was unable to relaunch due to zone availability 😅)
We are using pytorch-lightning == 1.2.0.
Scrubbed stacktrace:
Traceback (most recent call last):
  File "{source}/train_pl.py", line 248, in <module>
    main()
  File "{source}/train_pl.py", line 235, in main
    trainer.fit(model, dataloaders["train"], dataloaders["val"])
  File "{venv}/pytorch_lightning/trainer/trainer.py", line 510, in fit
    self.pre_dispatch()
  File "{venv}/pytorch_lightning/trainer/trainer.py", line 539, in pre_dispatch
    self.accelerator.pre_dispatch()
  File "{venv}/pytorch_lightning/accelerators/accelerator.py", line 84, in pre_dispatch
    self.training_type_plugin.pre_dispatch()
  File "{venv}/pytorch_lightning/plugins/training_type/ddp.py", line 254, in pre_dispatch
    self.model_to_device()
  File "{venv}/pytorch_lightning/plugins/training_type/ddp.py", line 279, in model_to_device
    self.model.to(self.root_device)
  File "{venv}/pytorch_lightning/core/decorators.py", line 89, in inner_fn
    module = fn(self, *args, **kwargs)
  File "{venv}/pytorch_lightning/utilities/device_dtype_mixin.py", line 120, in to
    return super().to(*args, **kwargs)
  File "{venv}/pytorch/torch/nn/modules/module.py", line 612, in to
    return self._apply(convert)
  File "{venv}/pytorch/torch/nn/modules/module.py", line 359, in _apply
    module._apply(fn)
  File "{venv}/pytorch/torch/nn/modules/module.py", line 359, in _apply
    module._apply(fn)
  File "{venv}/pytorch/torch/nn/modules/module.py", line 359, in _apply
    module._apply(fn)
  File "{venv}/pytorch/torch/nn/modules/module.py", line 381, in _apply
    param_applied = fn(param)
  File "{venv}/pytorch/torch/nn/modules/module.py", line 610, in convert
    return t.to(device, dtype if t.is_floating_point() else None, non_blocking)
RuntimeError: cuda runtime error (711) : peer mapping resources exhausted at /pytorch/aten/src/THC/THCGeneral.cpp:139 

Pitch
See above.
Alternatives
Can't think of a quick one.
Additional context
None necessary.