Race condition and repeated os.remove in load_spawn_weights
In multi-gpu, multi-node training, after training is finished, the following error occurs:
0:   File ".../estevens/pytorch/lightning_resnet50.py", line 126, in <module>
0:     ResnetLightningExample.init_from_cli(sys.argv[1:]).main()
0:   File "...//lightning_utils/lightning_utils.py", line 215, in main
0:     trainer.fit(self)
0:   File ".../pytorch_lightning/trainer/trainer.py", line 593, in fit
0:     self.load_spawn_weights(model)
0:   File ".../pytorch_lightning/trainer/distrib_data_parallel.py", line 372, in load_spawn_weights
0:     os.remove(path)
0: FileNotFoundError: [Errno 2] No such file or directory: '/mnt/sun-pcs01/estevens/resnet50/__temp_weight_ddp_end.ckpt'

I think this is a similar issue to other os.remove bugs, but I'm worried that merely adding an if rank == 0 will move the issue to a race condition (where rank 0 removes the file before other processes can load it).