External MLFlow logging failures cause training job to fail
🐛 Bug
I am using a pytorch_lightning.loggers.mlflow.MLFlowLogger during training, with the MLFlow tracking URI hosted in Databricks. When Databricks updates, we sometimes lose access to MLFlow for a brief period. When this happens, logging to MLFlow fails with the following error:
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host=XXX.cloud.databricks.com, port=443): Max retries exceeded with url: /api/2.0/mlflow/runs/get?XXX (Caused by NewConnectionError(<urllib3.connection.HTTPSConnection object at 0x7fbbd6096f50>: Failed to establish a new connection: [Errno 111] Connection refused))
Not only does logging fail, but with PyTorch Lightning, an error logging means the entire training pipeline will also fail, losing progress on a potentially long-running job with limited error handling options currently available.
Ideally, there would be flexibility in PyTorch Lightning to allow users to handle logging errors such that it will not always kill the training job.
Please reproduce using the BoringModel
https://colab.research.google.com/drive/17TqdKZ8SjcdpiCWc76N5uQc5IKIgNp7g?usp=sharing
To Reproduce
Attempt to use a logger that fails to log. The training job will fail, losing all progress.
Expected behavior
There is an option to handle exceptions from the logger such that the job does not automatically die if logging a parameter fails.
Environment

CUDA:

GPU:

Tesla T4


available:         True
version:           10.1


Packages:

numpy:             1.19.5
pyTorch_debug:     False
pyTorch_version:   1.8.0+cu101
pytorch-lightning: 1.2.4
tqdm:              4.41.1


System:

OS:                Linux
architecture:

64bit



processor:         x86_64
python:            3.7.10
version:           #1 SMP Thu Jul 23 08:00:38 PDT 2020



Additional context