ValueError: host not found: Name or service not known in _env_rendezvous_handler
🐛 Bug

To Reproduce
Steps to reproduce the behavior:

Go to pl_examples/basic_examples/
modify the script to fit environment

# SLURM SUBMIT SCRIPT
#SBATCH --account=gpu-s2-intelperf-0
#SBATCH --partition=gpu-s2-core-0
#SBATCH --nodes=2
#SBATCH --gres=gpu:2
#SBATCH --time=0-02:00:00
#SBATCH --ntasks-per-node=2
# activate conda env
source  activate $1
export NCCL_DEBUG=INFO
export PYTHONFAULTHANDLER=1
export NCCL_SOCKET_IFNAME=^ib0,lo
# ib0 is looked up from ifconfig
module load cuda10.1
# run script from above
srun python multi_node_ddp_demo.py


See error

Traceback (most recent call last):
  File "multi_node_ddp_demo.py", line 51, in <module>
    main(hyperparams)
  File "multi_node_ddp_demo.py", line 37, in main
    trainer.fit(model)
  File "/data/gpfs/home/hsinpaic/anaconda3/envs/pose/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 684, in fit
    self.ddp_train(task, model)
  File "/data/gpfs/home/hsinpaic/anaconda3/envs/pose/lib/python3.6/site-packages/pytorch_lightning/trainer/distrib_data_parallel.py", line 308, in ddp_train
    model.init_ddp_connection(self.proc_rank, self.world_size)
  File "/data/gpfs/home/hsinpaic/anaconda3/envs/pose/lib/python3.6/site-packages/pytorch_lightning/core/lightning.py", line 951, in init_ddp_connection
    torch_distrib.init_process_group('nccl', rank=proc_rank, world_size=world_size)
  File "/data/gpfs/home/hsinpaic/anaconda3/envs/pose/lib/python3.6/site-packages/torch/distributed/distributed_c10d.py", line 397, in init_process_group
    store, rank, world_size = next(rendezvous_iterator)
  File "/data/gpfs/home/hsinpaic/anaconda3/envs/pose/lib/python3.6/site-packages/torch/distributed/rendezvous.py", line 168, in _env_rendezvous_handler
    store = TCPStore(master_addr, master_port, world_size, start_daemon)
ValueError: host not found: Name or service not known
srun: error: gpu-1: task 2: Exited with exit code 1
srun: error: gpu-1: task 3: Exited with exit code 1
srun: error: gpu-0: task 0: Exited with exit code 1
srun: error: gpu-0: task 1: Exited with exit code 1


Code sample

Expected behavior
I was following the README in basic_examples folder, I can pass through single node example. But it shows this error in multi-nodes.

Environment
 CUDA:
	- GPU:
		- Tesla P100-SXM2-16GB
		- Tesla P100-SXM2-16GB
	- available:         True
	- version:           10.1
* Packages:
	- numpy:             1.18.1
	- pyTorch_debug:     False
	- pyTorch_version:   1.4.0
	- pytorch-lightning: 0.7.3
	- tensorboard:       1.14.0
	- tqdm:              4.45.0
* System:
	- OS:                Linux
	- architecture:
		- 64bit
		-
	- processor:         x86_64
	- python:            3.6.10
	- version:           #1 SMP Mon Jul 29 17:46:05 UTC 2019
* CUDA:
	- GPU:
		- Tesla P100-SXM2-16GB
		- Tesla P100-SXM2-16GB
	- available:         True
	- version:           10.1
* Packages:
	- numpy:             1.18.1
	- pyTorch_debug:     False
	- pyTorch_version:   1.4.0
	- pytorch-lightning: 0.7.3
	- tensorboard:       1.14.0
	- tqdm:              4.45.0
* System:
	- OS:                Linux
	- architecture:
		- 64bit
		-
	- processor:         x86_64
	- python:            3.6.10
	- version:           #1 SMP Mon Jul 29 17:46:05 UTC 2019```