DDP: Multiple processes try to create the logger directory tree
🐛 Bug
An user from our supercomputing center run into an issue which I think turned out to be a bug in PyTorch-Lightning.
When using the DDP accelerator together with a logger, multiple processes will try creating the logger directory tree, causing some errors about already existing directories or files.
Troubleshooting
PyTorch-Lightning uses extensively the rank_zero_only function to ensure that some actions are only performed by the process with rank 0:

  
    
      pytorch-lightning/pytorch_lightning/utilities/distributed.py
    
    
        Lines 35 to 42
      in
      b3b8f95
    
  
  
    

        
          
           def rank_zero_only(fn): 
        

        
          
            
        

        
          
               @wraps(fn) 
        

        
          
               def wrapped_fn(*args, **kwargs): 
        

        
          
                   if rank_zero_only.rank == 0: 
        

        
          
                       return fn(*args, **kwargs) 
        

        
          
            
        

        
          
               return wrapped_fn 
        
    
  


rank_zero_only.rank default value is set there:

  
    
      pytorch-lightning/pytorch_lightning/utilities/distributed.py
    
    
        Lines 45 to 46
      in
      b3b8f95
    
  
  
    

        
          
           # add the attribute to the function but don't overwrite in case Trainer has already set it 
        

        
          
           rank_zero_only.rank = getattr(rank_zero_only, 'rank', int(os.environ.get('LOCAL_RANK', 0))) 
        
    
  


but can be set in other modules, for example in our case DDP:

  
    
      pytorch-lightning/pytorch_lightning/plugins/training_type/ddp.py
    
    
        Lines 227 to 228
      in
      b3b8f95
    
  
  
    

        
          
           # set warning rank 
        

        
          
           rank_zero_only.rank = self.global_rank 
        
    
  


Unfortunately it seems that the initialization by the DDP module happens too late, I think because of commit da6dbc8:

self.setup_trainer(model) gets called on line 467 effectively initializing the logger and creating the logger directory tree
DDP initialization and thus rank_zero_only.rank getting the correct value only happens at line 477 when calling self.training_type_plugin.pre_training().

To Reproduce
I have attached the code the user provided together the Slurm script: only_rank_zero.tar.gz.
I understand that you would prefer a BoringModel and Collab based reproducer but I am from the HPC world and I am not used to those. Let me know if I can help in any other way. I hope that my own digging into the code will hep.
Environment (probably not relevant in this case)

PyTorch Version: 1.7.1
OS: Linux (Red Hat 8.1)
How you installed PyTorch: conda, I tried the latest version of  PyTorch-Lightning available on conda but also tested installing the current master branch from source and the behavior is still the same.
Python version: 3.7.10
CUDA/cuDNN version: 11.0.221/8.0.5
GPU models and configuration: NVIDIA V100