Saving meta tags hparams file in Tensorboard Logger can occur multiple times
🐛 Bug
The bug exists exactly here https://github.com/PyTorchLightning/pytorch-lightning/blob/master/pytorch_lightning/loggers/tensorboard.py#L220. The if statement is checking using os.path rather than self._fs.
As written, when using a non-local filesystem (such as S3, etc.), the os.path.isfile will always return false, meaning the content in the if-statement is always executed.
This is counter to expectations where normally the file is not overwritten if it already exists.
Please reproduce using the BoringModel and post here
No reproduction.
To Reproduce
The bug is not immediately obvious, but we found it when using Lightning with FB's internal distributed storage. In a workflow where multiple trainers are launched with the same hypers, the distributed storage system will throw when the same file is being written to at the same time. This is how we discovered this bug.
Expected behavior
From the code and from behavior on the local filesystem, it seems clear that the expected behavior is to check if the file exists and write it only if it does not.
Environment
N/A
Additional context
I'll be sending a pull-request to fix this minor bug shortly.