[BUG] `auto_move_data` does not work with `DataParallel`
🐛 Bug
In case your forward function is wrapped with auto_move_data it will not work with DataParallel because it will try to send the data to self.device which in dataparallel is always the main device.
i.e. the following won't work with accelerator="dp" (and probably also with "ddp"):
class Module(pl.LightningModule):
     ...
     
     @auto_move_data
     def forward(x):
         ...

     def training_step(self, batch, batch_idx):
         x = self.forward(batch[0])
         ...
The error comes from this line: 
  
    
      pytorch-lightning/pytorch_lightning/core/hooks.py
    
    
         Line 646
      in
      b190403
    
  
  
    

        
          
           device = device or self.device 
        
    
  


self.device should probably be replaced by torch.distributed.get_rank() when torch.distributed.is_available() and torch.distributed.is_initialized()