seed for DistributedSampler
🐛 Bug
torch.utils.data.distributed.DistributedSampler defaults to random seed 0 to shuffle the sampler if shuffle=True.
As lightning does not specify a seed, multi-gpu training with DistributedSampler in lightning will use a deterministic order independently from the user-specified seed (seed_everything).

  
    
      pytorch-lightning/pytorch_lightning/trainer/data_loading.py
    
    
        Lines 198 to 202
      in
      b85cfbe
    
  
  
    

        
          
           def _get_distributed_sampler(self, dataloader, shuffle): 
        

        
          
               kwargs = self.distributed_sampler_kwargs 
        

        
          
               kwargs['shuffle'] = shuffle and not self.overfit_batches 
        

        
          
               sampler = DistributedSampler(dataloader.dataset, **kwargs) 
        

        
          
               return sampler 
        
    
  



  
    
      pytorch-lightning/pytorch_lightning/plugins/training_type/ddp.py
    
    
        Lines 91 to 94
      in
      33cc9fe
    
  
  
    

        
          
           @property 
        

        
          
           def distributed_sampler_kwargs(self): 
        

        
          
               distributed_sampler_kwargs = dict(num_replicas=(self.num_nodes * self.num_processes), rank=self.global_rank) 
        

        
          
               return distributed_sampler_kwargs 
        
    
  


As a workaround, users can specify replace_sampler_ddp when creating the Trainer, but I think this is something that lightning should handle well out of the box.