Sequential model parallel with multi nodes regarding --num_nodes
Sequential model parallel with multi nodes regarding --num_nodes
Please reproduce using the BoringModel
8 MP with 2 DP as follow.
# node 0 
MASTER_ADDR=XXX.XXX.XXX.1 MASTER_PORT=7874 NODE_RANK=0 python train.py --gpus 8 --accelerator ddp  ....  --use_ddp_sequential --num_nodes 2

# node 1
MASTER_ADDR=XXX.XXX.XXX.1 MASTER_PORT=7874 NODE_RANK=1 python train.py --gpus 8 --accelerator ddp  ....  --use_ddp_sequential --num_nodes 2
It produces error as follow.
File "/home/hey/.local/lib/python3.7/site-packages/pytorch_lightning/plugins/ddp_sequential_plugin.py", line 233, in _infer_check_num_gpus
    "Pipe currently only supports splitting the module onto all available GPUs"

So I edited _infer_check_num_gpus as follow.
haven-jeon@c96f144
After that, learning proceeds normally.
Environment

PyTorch Version (e.g., 1.0): 1.6.0
OS (e.g., Linux): CentOS 7
How you installed PyTorch (conda, pip, source):  pip
Build command you used (if compiling from source): pip install -U .
Python version: 3.7
CUDA/cuDNN version: 10.1
GPU models and configuration: P40
Any other relevant information: