All TPU cores create tensorboard logs
🐛 Bug
With TPUs, TestTubeLogger writes many empty tensorboard logs, one log per TPU core except one. This confuses tensorboard and prevents it from updating. This is happening because the logger is created before spawning processes then the logger is replicated in each process.
To Reproduce
Train any model with ptl.Trainer(logger=TestTubeLogger(), num_tpu_cores=8) then check the tf directory, you will find 1 file with real log and 7 empty files.
Expected behavior
Only the main process creates a tensorboard log.
Environment
pytorch-lightning==0.8.5