In multi-gpus, trainer.test Error when ModelCheckpoint specifies the filename.
🐛 Bug

In ddp mode (trainer(accelerator='ddp') with multi-gpus, the trainer could not find the saved custom-named checkpoints when .test(), which would lead to the following error:

In addition, the program could hang, not be closed (even with CTRL-C)   and the main GPU's Volatile is always 100%:

The following modelcheckpoint would lead to the problem:
ckpt = ModelCheckpoint(dirpath='./checkpoints/', monitor='valid_loss', mode='min', filename="{epoch}-{valid_loss:.4f}")

If removing the filename, the program can work well:
ckpt = ModelCheckpoint(dirpath='./checkpoints/', monitor='valid_loss', mode='min')

Please reproduce using the BoringModel
The code is based on the pl_example  in https://github.com/PyTorchLightning/pytorch-lightning/blob/master/pl_examples/basic_examples/autoencoder.py.
The whole code could be found in https://paste.ubuntu.com/p/2Qp4GRns7f/

Environment

CUDA:
- GPU:
- GeForce RTX 3090
- GeForce RTX 3090
- available:         True
- version:           11.1
Packages:
- numpy:             1.19.4
- pyTorch_debug:     False
- pyTorch_version:   1.8.0+cu111
- pytorch-lightning: 1.2.5
- tqdm:              4.53.0
System:
- OS:                Linux
- architecture:
- 64bit
- ELF
- processor:         x86_64
- python:            3.6.9
- version:           #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021

Additional context