0.8.0-dev hydra changed working directory for other GPUs with DDP
🐛 Bug
When I moved from 0.7.6 to 0.8.0-dev for DicConfig support for saving model hparams, I found that working directory changed for GPUs in DDP setting. I modify from huggingface.
Code sample
python: can't open file '/home/joe/summarization/models/bart/outputs/2020-06-05/11-57-25/finetune.py': [Errno 2] No such file or directory
python: can't open file '/home/joe/summarization/models/bart/outputs/2020-06-05/11-57-25/finetune.py': [Errno 2] No such file or directory
python: can't open file '/home/joe/summarization/models/bart/outputs/2020-06-05/11-57-25/finetune.py': [Errno 2] No such file or directory
python: can't open file '/home/joe/summarization/models/bart/outputs/2020-06-05/11-57-25/finetune.py': [Errno 2] No such file or directory
python: can't open file '/home/joe/summarization/models/bart/outputs/2020-06-05/11-57-25/finetune.py': [Errno 2] No such file or directory
python: can't open file '/home/joe/summarization/models/bart/outputs/2020-06-05/11-57-25/finetune.py': [Errno 2] No such file or directory
python: can't open file '/home/joe/summarization/models/bart/outputs/2020-06-05/11-57-25/finetune.py': [Errno 2] No such file or directory
initializing ddp: LOCAL_RANK: 0/7 WORLD_SIZE:8
[2020-06-05 11:57:57,549][lightning][INFO] - initializing ddp: LOCAL_RANK: 0/7 WORLD_SIZE:8

where finetune.py is in /home/joe/summarization/models/bart/
and the program just stuck here like forever.
My hydra config.yaml is
defaults:
  - trainer: train
  - data: example
model_name_or_path: bart-large
output_dir: /data/models/
cache_dir: ""
config_name: bart-large
tokenizer_name: bart-large
downgrade: True
gradient_accumulation_steps: 1
max_grad_norm: 1
fp16: False
n_tpu_cores: 0
weight_decay: 0
adam_epsilon: 1e-8

Expected behavior
Programs on other GPUs should load finetune.py from '/home/joe/summarization/models/bart/
Environment

version: 0.8.0-dev
PyTorch Version (e.g., 1.0): 1.4
OS (e.g., Linux): ubuntu 18.04
How you installed PyTorch (conda, pip, source): conda
Python version: 3.6