DDP Bug with Model Checkpoint parsing
🐛 Bug
My script works with CPU, single-GPU and dp.
I need ddp to do 16 bit training. Also even on a single machine ddp is faster.
Here is my ModelCheckpoint code:
def setup_model_checkpoint(config):
    kwargs = config["model_checkpoint_kwargs"]
    metrics = kwargs.pop("metrics", ["val_loss"])
    if isinstance(metrics, str):
        metrics = [metrics]

    fp = "checkpoints/{epoch}"
    for metric in metrics:
        fp += "-{"
        fp += str(metric)
        fp += ":.2f}"

    return ModelCheckpoint(filepath=fp, **kwargs)

In my case it would generate the checkpoint: checkpoints/epoch=4_val_loss=0.6_auc=0.85 for example.
Although I even tried it with just checkpoints and it's the same issue.
The issue is the following:
2020-06-20T14:50:19.704+01:00
File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 891, in fit
2020-06-20T14:50:19.704+01:00
File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 891, in fit
2020-06-20T14:50:19.704+01:00
File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 891, in fit
2020-06-20T14:50:19.705+01:00
self.ddp_train(task, model)
2020-06-20T14:50:19.705+01:00
self.ddp_train(task, model)
2020-06-20T14:50:19.705+01:00
self.ddp_train(task, model)
2020-06-20T14:50:19.705+01:00
File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/pytorch_lightning/trainer/distrib_data_parallel.py", line 530, in ddp_train
2020-06-20T14:50:19.705+01:00
File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/pytorch_lightning/trainer/distrib_data_parallel.py", line 530, in ddp_train
2020-06-20T14:50:19.705+01:00
File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/pytorch_lightning/trainer/distrib_data_parallel.py", line 530, in ddp_train
2020-06-20T14:50:19.705+01:00
self.run_pretrain_routine(model)
2020-06-20T14:50:19.705+01:00
self.run_pretrain_routine(model)
2020-06-20T14:50:19.705+01:00
File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 1046, in run_pretrain_routine
2020-06-20T14:50:19.705+01:00
File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 1046, in run_pretrain_routine
2020-06-20T14:50:19.705+01:00
self.run_pretrain_routine(model)
2020-06-20T14:50:19.705+01:00
File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 1046, in run_pretrain_routine
2020-06-20T14:50:19.705+01:00
self.configure_checkpoint_callback()
2020-06-20T14:50:19.705+01:00
self.configure_checkpoint_callback()
2020-06-20T14:50:19.705+01:00
File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/pytorch_lightning/trainer/callback_config.py", line 60, in configure_checkpoint_callback
2020-06-20T14:50:19.705+01:00
File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/pytorch_lightning/trainer/callback_config.py", line 60, in configure_checkpoint_callback
2020-06-20T14:50:19.705+01:00
self.configure_checkpoint_callback()
2020-06-20T14:50:19.705+01:00
File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/pytorch_lightning/trainer/callback_config.py", line 60, in configure_checkpoint_callback
2020-06-20T14:50:19.705+01:00
"checkpoints"
2020-06-20T14:50:19.705+01:00
"checkpoints"
2020-06-20T14:50:19.705+01:00
File "/home/user/miniconda/envs/py36/lib/python3.6/posixpath.py", line 94, in join
2020-06-20T14:50:19.705+01:00
File "/home/user/miniconda/envs/py36/lib/python3.6/posixpath.py", line 94, in join
2020-06-20T14:50:19.705+01:00
"checkpoints"
2020-06-20T14:50:19.705+01:00
genericpath._check_arg_types('join', a, *p)
2020-06-20T14:50:19.705+01:00
File "/home/user/miniconda/envs/py36/lib/python3.6/genericpath.py", line 149, in _check_arg_types
2020-06-20T14:50:19.705+01:00
File "/home/user/miniconda/envs/py36/lib/python3.6/posixpath.py", line 94, in join
2020-06-20T14:50:19.705+01:00
genericpath._check_arg_types('join', a, *p)
2020-06-20T14:50:19.705+01:00
File "/home/user/miniconda/envs/py36/lib/python3.6/genericpath.py", line 149, in _check_arg_types
2020-06-20T14:50:19.705+01:00
(funcname, s.__class__.__name__)) from None
2020-06-20T14:50:19.705+01:00
genericpath._check_arg_types('join', a, *p)
2020-06-20T14:50:19.705+01:00
TypeError: join() argument must be str or bytes, not 'NoneType'
2020-06-20T14:50:19.706+01:00
File "/home/user/miniconda/envs/py36/lib/python3.6/genericpath.py", line 149, in _check_arg_types
2020-06-20T14:50:19.706+01:00
(funcname, s.__class__.__name__)) from None
2020-06-20T14:50:19.706+01:00
TypeError: join() argument must be str or bytes, not 'NoneType'
2020-06-20T14:50:19.706+01:00
(funcname, s.__class__.__name__)) from None
2020-06-20T14:50:19.706+01:00
TypeError: join() argument must be str or bytes, not 'NoneType'

Environment

PyTorch Version (e.g., 1.0): 1.4
OS (e.g., Linux): Linux
How you installed PyTorch (conda, pip, source): Conda
Build command you used (if compiling from source):
Python version: 3.6.5
CUDA/cuDNN version: 10.1
GPU models and configuration: 4 x V100
Any other relevant information: Pytorch lightning 0.8.0

Additional context