TypeError with multiple validation loaders and overfit_batches
🐛 Bug
A TypeError when using multiple validation datasets and  overfit_batches != 0
To Reproduce
Steps to reproduce the behavior:

Use multiple val_dataloaders
Use overfit_batches != 0, e.g. overfit_batches=0.5

Code sample
https://colab.research.google.com/drive/1BtQBCoP5fK-aZm_2uLMOUbf2c9cu-yFb?usp=sharing
Traceback
TypeError                                 Traceback (most recent call last)

<ipython-input-5-c33b987ae54f> in <module>()
      1 trainer = pl.Trainer(overfit_batches=0.5)
----> 2 trainer.fit(model)

3 frames

/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/trainer.py in fit(self, model, train_dataloader, val_dataloaders)
   1018             self.optimizers, self.lr_schedulers, self.optimizer_frequencies = self.init_optimizers(model)
   1019 
-> 1020             self.run_pretrain_routine(model)
   1021 
   1022         # callbacks

/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/trainer.py in run_pretrain_routine(self, model)
   1137                                           self.val_dataloaders,
   1138                                           max_batches,
-> 1139                                           False)
   1140 
   1141             # allow no returns from eval

/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/evaluation_loop.py in _evaluate(self, model, dataloaders, max_batches, test_mode)
    291                         output = self.evaluation_forward(model, batch, batch_idx, dataloader_idx, test_mode)
    292                 else:
--> 293                     output = self.evaluation_forward(model, batch, batch_idx, dataloader_idx, test_mode)
    294 
    295                 # on dp / ddp2 might still want to do something with the batch parts

/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/evaluation_loop.py in evaluation_forward(self, model, batch, batch_idx, dataloader_idx, test_mode)
    483             output = model.test_step(*args)
    484         else:
--> 485             output = model.validation_step(*args)
    486 
    487         return output

TypeError: validation_step() missing 1 required positional argument: 'dataloader_idx'

Expected behavior
If the codebase is working with multiple validation loaders, it should continue to work even when using overfit_batches != 0
Possible solution

Check if there were multiple val dataloaders, in case call validation_step with dataloader_idx=0
Repeat the train loader to match the number of val dataloaders
Add the possibility to overfit on train but validate and test normally. It is already possible with limit_train_batches, so it would be only a doc change "If there are multiple val_dataloaders, use limit_train_batches instead of overfit_batches"

Reason
When using multiple validation loaders, validation_step takes a dataloader_idx.
However if later on we set the overfit_batches to something that is not 0, line 268 is executed to use the train loader instead than the validation loaders:

  
    
      pytorch-lightning/pytorch_lightning/trainer/data_loading.py
    
    
        Lines 266 to 270
      in
      a91b06e
    
  
  
    

        
          
           # use the training loader as val and test when overfitting 
        

        
          
           if self.overfit_batches > 0: 
        

        
          
               dataloaders = self.request_dataloader(getattr(model, 'train_dataloader')) 
        

        
          
           else: 
        

        
          
               dataloaders = self.request_dataloader(getattr(model, f'{mode}_dataloader')) 
        
    
  


Now there is only one validation loader, thus the validation_step function that had a dataloader_idx parameter breaks.
Environment

CUDA:

GPU:
available:         False
version:           10.1


Packages:

numpy:             1.18.5
pyTorch_debug:     False
pyTorch_version:   1.5.1+cu101
pytorch-lightning: 0.8.4
tensorboard:       2.2.2
tqdm:              4.41.1


System:

OS:                Linux
architecture:

64bit



processor:         x86_64
python:            3.6.9
version:           1 SMP Wed Feb 19 05:26:34 PST 2020