prepare_data called multiple times per node for slurm and elastic training
🐛 Bug
Slurm and elastic training create the training processes per node outside of the lightning context. This means that when the fit function calls prepare_data, the assumption that it's only being called on proc 0 is broken and it gets called for each process.
This is an issue computational reasons (e.g. downloading a whole dataset) and for training stability if the data preparation process isn't deterministic.
See calling code here:

  
    
      pytorch-lightning/pytorch_lightning/trainer/trainer.py
    
    
         Line 825
      in
      7c7e50c
    
  
  
    

        
          
           model.prepare_data() 
        
    
  


To Reproduce
Steps to reproduce the behavior:

Add print statements to prepare_data
Train a lightning model with either slurm or elastic training
See that it's being called multiple times.

Expected behavior
Expected prepare_data to only be called once per node.